May 6, 2013 - Geeklist at Runway SF

A Realtime Synchronization Network

Build Apps fast without managing servers

Data persistence, but in realtime

With authentication & security rules, serves as the complete backend

 

Why?

 

Users don't like waiting

Users want their data wherever they are, between all their devices
(and sometimes their friends)

The save and refresh buttons are quickly becoming obsolete

Ergo, Sync should be made available to all players

Evolution of the industry

"All problems in computer science can be solved by adding another layer of abstraction"

Real servers

Virtual machines
 

PaaS
   

Next?
 

10 Lessons We Learned

1. Data Synchronization > Message Passing

In an always connected world, request-response just isn't good enough

Data synchronization provides a far easier programming model

    At the "cost" of increased provider complexity

Most apps needs some form of persistence anyway

 

With message passing, what happens to missed messages? History?

A data synchronization approach can easily subsume message passing

2. Event-driven APIs are not Ubiquitous (Yet)

It's a big leap from the request-response world

Developers "get" AJAX, but don't always grok events

Very important to make APIs intuitive, driving developers into the "pit of success"

One of our FAQs:

    "How do I get the current value of foo?"

var ref = new Firebase("https://myfirebase.firebaseio.com/foo");
ref.on("child_added", function(snapshot) {
  console.log(snapshot.val());
});

3. De-normalization isn't Intuitive (Yet)

Most developers think in terms of SQL

Even with "NoSQL" DBs, SQL-like queries are possible

But not with Firebase - duplication neccessary for performance

  users: {
    user1: {
      name: "Alice",
      comments: { comment1: true }
    }
  },
  comments: {
    comment1: {
      body: "Geeklist is awesome!",
      author: user1
    }
  }

4. Realtime isn't Easy

The nature of data sync requires a reliable, in-order transport

WebSocket support can be spotty

  • Mobile networks
  • Firewalls / Proxies

Always need to have a "works-everywhere" fallback: long-polling

  • Invisible iframes with JSONP callbacks
  • Always one outstanding request
  • Request and response order doesn't always match
  • Server hangs on to a request for about 20-25 seconds if there's no data

4. Realtime isn't Easy

5. The Browser World is Ugly

There's plenty of browser quirks for everyone!

  • IE messes with document.domain when creating iframes
  • Creating an iframe in the callback for another iframe's completion fails in Firefox
  • navigator.onLine is just a big lie

Always, always test on all browsers before a code push

6. Immutability is King

Whenever concurrency is involved, immutability shines

Most functional languages, Channels in Go

JS is single threaded, but support for other language drove the need for "snapshots"

Snapshot system developed in JavaScript first, now core to the API

var ref = new Firebase("https://myfirebase.firebaseio.com/foo");
ref.once("value", function(snapshot) {
  console.log(snapshot.val());
});

7. Latency Compensation is Important

We always fire local events acknowledging changes

This prevents jitter in some cases

Also makes offline mode easier to reason about

var ref = new Firebase("https://myfirebase.firebaseio.com/foo");
ref.on("child_added", function(snapshot) {
  console.log(snapshot.val());
});
ref.push("I'm a new child!");

8. Virtualization Doesn't Always Work

Hypervisors could not keep up with our traffic patterns

THe architecture we have generates large amounts of small packets

VMs can push about 15,000-75,000 packets per second

 

The "thundering herd" is a problem, but even steady-state for large number of concurrent connections isn't easy

9. Client Complexity Helps Scalability

Our client is a lot more complex than our server

Diffing, caching and event triggers are all client-side

Server doesn't keep track of what each client knows

    This would never work with >100k concurrent users anyway

10. Declarative Security Works Pretty Well

Security in a client-side app was a big open question

Tie it to authentication and express logic as lightweight expressions

{
  "rules": {
    "$userid": {
      ".read": true,
      ".write": "auth.id == $userid"
    }
  }
}

Thank You!

anant@firebase.com