03. The reader

1. Finding index packs

A pull or clone already lists bare/packs/commits/ and bare/packs/snapshots/ (Vault__Packs.sync_down). When the branch index has the feature pack-indexes, it also lists bare/packs/indexes/commits/ and bare/packs/indexes/snapshots/. That is two more list requests, made only on vaults that have index packs.

Pair each index pack with its data pack by (folder, epoch, commit id). An index pack whose data pack is in neither its folder nor archive/ is ignored.

Download:

• the snapshot index of the snapshot the clone takes;
• the commit index of every commit pack not yet absorbed;
• nothing from indexes/archive/, except when a clone takes an archived history.

Index packs are small (01, section 5), so they go through the batch read with the data packs.

2. Verifying an index pack

In order, before anything uses it:

1. The header. Magic and format byte; envelope decrypts under the read key; schema known.
2. The signature. Verify with the key named by author_key_id (bare/keys/<id>), by the vault's commit policy:

• under signatures-required, a missing or bad signature means ignore the index pack;
• otherwise an unsigned index is accepted, subject to step 3.

1. The anchor. commit_id, parents, tree_id and timestamp_ms must equal the commit object's, which the data pack carries and is verified by its id. For a snapshot, the root of flat must equal tree_id, and the root folder member's entries must equal the root tree's.
2. Each member, when it is first read. Decrypt, inflate, and check its SHA-256 against the header's.

A failure at any step drops that index pack: the reader walks instead, for that pack only, and reports it once (sgit pack index verify gives the detail). Nothing an index says is ever written to the working copy or the object store without the object it names verifying by its content address. That is today's rule for pack members, unchanged.

3. The local index

.sg_vault/local/index/
  packs/<folder>/<name>.zip     the index packs, as downloaded: encrypted
  state.json                    {schema, snapshot: name, deltas: [names, oldest first], gaps: [commit ids]}

• Index packs stay encrypted on disk. A member is decrypted when it is read, at about a microsecond per KB. This adds nothing in clear beyond what 0.25.0's flat and scan caches already hold, and it stays compatible with sealed vault keys.
• The assembly is in memory, per process. It is cheap enough to redo per process (section 4), so nothing assembled has to be written. A later optimisation may cache it under the same rules as the flat cache.
• state.json is reset by a new snapshot. When compaction makes a new snapshot index, the deltas start over.

4. Assembling

The local index answers four kinds of question. Each has its own structure, built on first use:

A gap is a commit pack without a usable index. Compute its delta locally, which is what today's code does anyway: diff its tree against its first parent's, and take parents and generation from its commit object. Then carry on. A clone with gaps is as correct as one without, and only as slow as today for those commits.

5. Command by command

pull

The dirty-tree check (the stat pass) and writing the changed files do not change.

clone (full)

• Take the snapshot index with the snapshot data pack. Seed the flat cache from flat plus the deltas; that is no tree walk, where today the clone flattens after extraction.
• Seed the commit graph from commits:*.
• Extract the working copy as today. Phase 3 changes the object writes (section 6).

clone --path <folder> and a scoped pull (Phase 2)

1. List; download the snapshot index and the commit indexes.
2. Find folder:<folder>: the newest version of it and of every folder below it, walking dirs. Measured: 1,161 members for a subtree of 1,520 files, in 36.5 ms. totals gives the size before anything is downloaded.
3. Collect the at of every file and tree_at of every folder in the scope, plus the trees on the spine from the root, which Vault__Scope needs to commit later. Group by data pack and merge ranges less than 64 KB apart.
4. Take each range with fetch_presigned_range when the host has presigned reads. Otherwise, and for a pack under PACK_BATCH_MAX, take the whole pack as today.
5. Verify every object by its id, write the scope, and record the scope as today.

Measured on the benchmark vault, with blobs in path order: one folder of 1,520 files was 1 range of 0.22 MB. In today's id order the same blobs were 265 ranges over 30.4 MB, and today's clone takes the whole pack.

history log, log --stat, diff <a> <b>

• history log reads the changes of each commit in the range; its counts come from the lists. No tree is opened. log --stat is the same.
• diff <a> <b> for commits on one line of history is the union of their changes. Otherwise it is two flat maps, as today.
• A path's history (history log -- <path>) is file-commits plus the newer deltas.

status, commit, push

Unchanged for the reader. push does the writer's work (02).

A read without a clone (Phase 4)

sgit cat <vault> <path>:

• list;
• download the snapshot index and the commit indexes, a few MB;
• the newest folder:<parent of path> gives the blob's at;
• one ranged read, verify, decrypt.

That is three or four requests, whatever the size of the vault.

6. The pack as the store (Phase 3)

This is the CLI team's architecture, phase 1, unchanged in substance and listed here for completeness. A pack clone keeps the data packs it downloads under .sg_vault/bare/packs/, and the object store reads a member by offset. Offsets come from each pack's central directory, read once per pack: 0.56 s for 134,092 members. With Phase 2, the folder members' at give them too.

• Verification moves to the first read of each object, and the verdict is remembered.
• Loose objects win over packed ones; a clone's own new objects stay loose until pushed.
• Measured: 5 µs per read in place, against 11.6 s to write every object of the benchmark vault loose, and many minutes on a slow disk.

7. Fallbacks, all of them

This page as markdown · its source, admin/content/dev/pack-indexes/03-reader.md