Module 12 · Lesson 10
Design a file sync service
Metadata consistency is the hard part; the bytes are just blobs in a bucket.
What you will be able to do
- Design a content-addressed chunk store
- Version folder metadata without a global lock
- Notify other devices of changes efficiently
- Resolve offline conflicts with a stated policy
Requirements and estimates
- Functional: upload and download files, sync across devices, version history, shared folders, offline edits.
- Non-functional: no data loss, sync within seconds when online, resume large uploads.
- Scale: 500M users, 1 TB average quota, 1% daily change.
average file 1 MB, 4 MB chunks -> most files are 1 chunk change rate 1%/day of 500 PB = 5 PB/day ingress metadata ops: much higher than byte ops -> metadata service is read/write heavy, bytes are write-once
Two planes
Separate the metadata plane from the block plane. Metadata — names, folders, versions, permissions, chunk lists — lives in a transactional database and is queried constantly. Blocks live in object storage, written once, addressed by hash, and served through a CDN or signed URLs.
Chunk and hash
Split the file into fixed or content-defined chunks; hash each chunk and the whole file.
Check which chunks exist
Send hashes to the server; upload only missing chunks. This is how deduplication becomes a bandwidth win.
Upload missing chunks
Direct to object storage, resumable, retried per chunk.
Commit metadata
In one transaction: create the new version, record the chunk list, advance the folder version.
Notify devices
Publish a change event; connected devices fetch the delta since their cursor.
Content-defined chunking matters
Fixed 4 MB chunks break deduplication when one byte is inserted at the start, shifting every boundary. Rolling-hash chunking (like the approach used by restic and Borg) limits the damage to a local region.
Versioning and conflicts
| Event | Metadata action | Outcome |
|---|---|---|
| Same file, one device offline | New version with parent pointer | Both versions kept |
| Two devices edit offline | Two children of the same version | Conflict copy created |
| Rename and content change | One new version, path updated | Ordered by folder version |
| Delete while offline | Tombstone with timestamp | Delete wins unless a newer edit exists |
The folder version is a monotonically increasing integer per user or per shared folder. A device stores its last-seen version and asks for changes since then, which beats walking the tree and comparing timestamps. The service, not the client clock, assigns versions — that is the only way two devices agree on order.
The two-plane answer
"Metadata is a versioned transactional store with a per-folder change cursor; blocks are content-addressed in object storage and deduplicated by hash. Devices sync deltas since their cursor, and conflicting offline edits produce conflict copies rather than silent overwrites."
Worth remembering
- Deduplicate at the chunk level with content hashes; identical chunks are stored once across all users.
- A file is metadata plus an ordered list of chunk hashes; the metadata service is the authority.
- Syncing by polling the whole tree does not scale — send deltas with a version cursor.
- Conflicts are a product decision: keep both copies, or let the last writer win with history preserved.
Checkpoint
Answer all 3 questions to finish the lesson.
1. Why use content-defined chunk boundaries instead of fixed-size chunks?
Fixed boundaries shift after an insertion, changing all downstream chunk hashes and destroying deduplication.
2. How should devices find changes without rescanning the whole tree?
A server-assigned version cursor gives an ordered delta and avoids trusting client clocks or full tree comparisons.
3. Two devices edit the same file offline. What is the safest default?
Silent overwrite loses data. Conflict copies preserve both edits and let the user decide.