Event Sourcing for Local-first Sync

With crsqlite going into maintenance mode, I’m on the hunt for a new way of managing sync in vfdir. The most complicated part is “syncing” data from are.na’s API. Contrary to what most frameworks and libraries assume, node syncing doesn’t work for 3rd-party services and I also don’t control the server on the other end. Thanks to a random comment from Yonz I decided to look into event sourcing. As usual leaning towards the most concrete info I could find.

Toumas Artman’s talk on Linear’s sync engine bookmarked for months and it finally made sense in the middle of working on something similar. The transaction queue is good reference for building in an event log without necesarily structuring the entire application around Event Sourcing(TM). A problem I’d tried to solve on the first version of vfdir was duplicate data (and memory use) in javascript after querying a sqlite database. The object pool Toumas mentions solves that in a different way. It reminds me a bit of TinyBase, creating regular js objects from persisted data.

He also mentions Even Wallace’s blog post on how Figma’s multiplayer was built. Here again no actual OT or CRDTs, although the system is influenced by both.

Intuitively, we want to show the user our best prediction of what the eventually-consistent value will be. Since our change we just sent hasn’t yet been acknowledged by the server but all changes coming from the server have been, our change is our best prediction because it’s the most recent change we know about in last-to-the-server order. So we want to discard incoming changes from the server that conflict with unacknowledged property changes.

Other similarities here:

  • Figma object IDs are generated client side with data from each clients unique client ID, like the originEventId in Andrey Salomatin’s blog post.
  • Figma’s tree structure is basically the same as the object pool in Linear. Both store objects separately and represent a tree by referencing the ID of related objects. Looking at bothe of these along with notes from del.ici.ous’s founder calling relational databases ‘a poor fit’ for social boookmarking, maybe I should consider other options.

The final bow to tie this all together comes from Matt Wonlaw, right on the crsqlite page, comparing Last write Wins to a P2P Event Log.

The event based approach gives you a set of facts about the system that never change.

The “list of facts” approach gives us:

  • An audit trail
  • The ability to change our interpretation of facts/events as requirements change
  • Freedom to pick how to resolve conflicts