Design 06: sgit worktree (roadmap 2.13, priority 2)

Read with design__00__index-and-reconciliation.md: where the two disagree (shared foundations F1–F9, the in-progress rules), the index wins.

Status: design, not built. Date: 28 Oct 2026. Roadmap: roadmap__git-parity.md item 2.13, approach A (agreed with Dinis: shared objects only, Git's alternates; no shared refs). Code read at: 3a1d72a on claude/charming-babbage-5hzcxq. Every file:line below is at that commit.

In one paragraph. sgit worktree add <path> makes a second working copy of the vault in seconds, with no network. The new folder is a real sgit clone: its own .sg_vault/local/ (clone branch, signing key, config, caches, merge state, reflog) and its own small .sg_vault/bare/ (branch index, refs, public keys). Its object store has no objects of its own at first. It reads objects from the main vault's store, then from the network, in that order. New objects are written to the worktree's own store and never to the main one. Merging works as today: push, then branch merge. Phase 2 adds branch merge from another local worktree without a push.

1. Summary and the exact Git behaviour matched

1.1 What Git does (checked on git 2.43.0, installed in this container)

1.2 What sgit matches, and how

2. Scope and non-goals

In scope, phase 1 (2–3 days):

1. sgit worktree add <path> [--name N] [--branch <named> | -b <new named>] [--path F ...] [--no-checkout] [--vault-key K].
2. sgit worktree list [--json], remove <name|path> [--force], prune [--dry-run], lock <name|path> [--reason R], unlock <name|path>, dissociate <name|path>.
3. The read path: Vault__Object_Store reads through one alternate store, verifying each borrowed object by its content address.
4. Guards in the main vault: vault uninit, vault rekey, vault move, migrate apply refuse while worktrees are registered (names listed, --force or worktree remove first). Guards in a worktree: vault uninit points to worktree remove; vault backup materialises borrowed objects into the zip; a first push of a never-pushed vault is refused.
5. The working-copy scan never descends into a nested vault (a sub-folder that holds its own .sg_vault). This matters when a worktree is created inside the main working copy, and it fixes a latent bug for any nested clone.

Phase 2 (1–1.5 days): sgit branch merge --worktree <name|path>: merge another local worktree's clone head without pushing. It imports the commits and objects it needs, verified by content address, plus the other worktree's public key, then runs the existing pull merge step.

Non-goals:

3. CLI surface

3.1 Placement (Rule 9) — decision D1

Git has git worktree at the top level, and agents type it from habit. A worktree is about working copies, which are daily-use, not about "the vault as a whole". Recommendation: top-level sgit worktree, with TOP_LEVEL_COMMANDS in tests/unit/cli/test_CLI__B07__Namespace_Moves.py:104 gaining 'worktree' in the same commit (a deliberate edit, as Rule 9 asks). The alternative is sgit vault worktree …. The rest of this section assumes top level. sgit help all (sgit_ai/cli/CLI__Main.py:1491) must list it (it already omits several commands; that list is fixed separately).

3.2 Commands

<name|path> resolves first as a registry name, then as a path (absolute or relative to the current directory). Every subcommand except add and phase 2 can run from the main folder or from any of its worktrees. Inside a worktree, the registry used is the main vault's, found through alternates.json.

3.3 Output

$ sgit worktree add ../agent-7 -b agent-7
Worktree agent-7 at /work/agent-7
  vault       y8pwtjlw
  branch      agent-7 (new, from current at 3f9a1c2)
  clone       branch-clone-5e1b… (registered on first push)
  objects     borrowed from /work/vault (0 downloaded)
  files       9,404 written in 2.1 s

$ sgit worktree list
/work/vault       3f9a1c2  current   (main)
/work/agent-7     3f9a1c2  agent-7
/work/agent-8     b07e2d4  current   locked: long run
/mnt/usb/agent-9  -        -         missing (sgit worktree prune)

--porcelain prints Git's shape: blocks of worktree <path>, HEAD <commit id>, branch <named>, locked <reason>, prunable <reason>, separated by a blank line.

3.4 --json shapes

worktree add --json:

{ "name": "agent-7", "path": "/work/agent-7", "main": "/work/vault", "vault_id": "y8pwtjlw",
  "named_branch": "agent-7", "named_branch_new": true, "clone_branch_id": "branch-clone-5e1b…",
  "start_commit": "obj-cas-imm-3f9a1c2…", "files_written": 9404, "objects_downloaded": 0,
  "scope_paths": [], "selective": false, "seconds": 2.1 }

worktree list --json: {"main": {...}, "worktrees": [{"name", "path", "head", "named_branch", "clone_branch_id", "created_at", "locked", "lock_reason", "state"}]}, where state is ok | missing | foreign. foreign means the folder exists but its alternates.json points to another main vault.

3.5 Exit codes

As the rest of the CLI (sys.exit(1) is used 116 times in sgit_ai/cli/*.py, sys.exit(2) for argparse misuse, 130 for Ctrl-C): 0 success, 1 refused or failed (the message says why and names the remedy), 2 bad arguments. remove and prune --dry-run exit 0 when there is nothing to do.

4. How the code works today

4.1 Why not Git-style shared refs: the counts (verified)

grep -rn "'\.sg_vault'" sgit_ai --include=*.py, excluding Vault__Storage.py: 52 occurrences in 22 files.

grep -rn "os.path.join([^)]*'bare'", same exclusion: 29 occurrences in 18 files.

On top of these, string paths such as 'bare/data/' appear in about 60 more places. Most are wire file ids (the server's path), but Step__Move__Build_Temp_Vault.py:103 and Step__Move__Validate_Local.py:61 use them on disk.

Shared refs would mean: every one of those 81 paths goes through one resolver, bare/refs, bare/indexes and bare/keys become shared between processes, and the ref, index and baseline writes need cross-process locks. Those writes are compare-and-swap on the server today and unlocked on disk (save_branch_index Vault__Branch_Manager.py:76-86 writes in place). That is a 1–2 week change touching most commands, with a concurrency bug class sgit does not have today. Approach A changes the object read path in one class, plus about 10 small guards.

5. Design

5.1 On-disk layout

/work/vault/                        main clone (unchanged layout)
  .sg_vault/bare/{data,refs,keys,indexes,packs,…}
  .sg_vault/local/
      worktrees.json                NEW: registry of this vault's worktrees (Schema__Worktree__Registry)
      …everything as today

/work/agent-7/                      worktree: a real clone
  .sg_vault/bare/
      data/                         own objects only (new commits, trees, blobs; fetched objects the main lacks)
      refs/                         copied at add time, then its own
      keys/                         copied at add time (public keys, encrypted), then its own
      indexes/<branch index id>     copied at add time, then its own
  .sg_vault/local/
      alternates.json               NEW: the pointer to the main store (Schema__Worktree__Alternates)
      config.json  vault_key  <key>.pem  remotes.json  token  base_url  remote_heads.json  …
  <working copy files>

There are no symlinks. The worktree's .sg_vault/ is a real directory, so find_vault_root and Vault__Context.detect work unchanged. A worktree is an ordinary clone plus one file.

5.2 Schemas and safe types (new)

Every schema gets the round-trip test (cls.from_json(obj.json()).json() == obj.json()). Both JSON files are written through Vault__Storage.write_private (0600, temp file, fsync, rename): alternates.json names a path to a folder holding a key, and worktrees.json names paths.

5.3 The read path (one class changes)

Vault__Object_Store (sgit_ai/storage/Vault__Object_Store.py) gains:

Vault__Alternates (new, sgit_ai/storage/Vault__Alternates.py, Type_Safe) does all pointer handling:

Why verify borrowed reads when own loose reads are not verified:

• the main store is written by another process, non-atomically (store :68, Verified_Write.save :72), so a half-written file is a real possibility;
• the main folder may be on another disk that changes under the worktree;
• the cost is one SHA-256 per borrowed object, about 1 GB/s, the same rule kept packs already follow.

A mismatch is reported once per run as a warning (object <id> in the main store does not match its id; fetched from the server instead).

5.4 The write path: new objects go to the worktree's own store

Decision: own store only. Git never writes into an alternate, and the same reasons hold:

1. Ownership. The main folder's store belongs to the main clone. Two processes writing into one bare/data/ meet the non-atomic writes above and the CAS guard's read-compare (store :70-75), which would raise a false collision on a half-written file.
2. Removal is clean. worktree remove deletes the worktree folder and everything it created. Nothing the worktree wrote stays behind in the main store.
3. Kept packs. A --pack-store main has a SQLite table per clone (local/pack-store.sqlite). A worktree writing loose objects into the main bare/data/ would bypass that table's intake rules.
4. Cost. The duplicate is only what the worktree itself creates or fetches: its own commits, plus what it pulls that the main has not pulled yet.

A pull in the worktree downloads only objects missing from both stores, because exists covers the alternate. When the main clone has already pulled, the worktree's pull downloads only refs.

5.5 The registry in the main vault

Vault__Worktree_Registry (new, sgit_ai/storage/Vault__Worktree_Registry.py):

Concurrency: worktree add and remove read-modify-write the registry. Two concurrent add calls could lose an entry. The registry write uses a lock file worktrees.json.lock (os.open(O_CREAT|O_EXCL), removed in finally, stale after 60 s). This is the only cross-process lock in the design.

5.6 worktree add, step by step

New class Vault__Worktree (sgit_ai/core/actions/worktree/Vault__Worktree.py, a Vault__Sync__Base subclass like the other actions), method add(main_directory, path, name=None, branch=None, new_branch=None, from_rev=None, scope_paths=None, no_checkout=False, vault_key=None, on_progress=None) -> dict. It runs Workflow__Worktree__Add (sgit_ai/workflow/worktree/Workflow__Worktree__Add.py, registered with @register_workflow as the clone workflows are):

Failure handling. Any failure after step 4 removes <path>/.sg_vault and the files written (the step runner's keep_work=False cleanup plus an explicit rollback in Vault__Worktree.add). Nothing in the main vault is written before step 11.

The vault key: copied, not referenced (decision D2). The worktree is a full clone and must keep working if the main folder moves or is removed. A reference would also make every worktree command open the main's key file. Git shares config the same way, in effect. The copy is written through the single secret writer, as every clone's is.

5.7 remove, prune, lock, unlock, dissociate, list

5.8 Phase 2: sgit branch merge --worktree <name|path>

Goal: merge another worktree's work into this clone without a push. This is what makes the main folder (or a coordinating agent) able to integrate agents' worktrees the way git merge feature does.

Flow (Workflow__Merge_Local, sgit_ai/workflow/pull/Workflow__Merge_Local.py; reuses the pull workspace):

What it does not touch: this clone's named-branch ref and baselines (remote_heads.json). A local merge is not a fetch of the named branch: the rewind guard and the baselines only ever follow the server. The merge commit's message is Merge worktree <name> (<clone branch>) into <this clone branch>.

Where the switch goes: CLI__Branch merge parser gains --worktree NAME (mutually exclusive with fetching from the server). Vault__Sync.merge_branch gains worktree: str = None and calls Vault__Sync__Pull.merge_local(…). Pull__Workspace.merge_from stays for named branches.

5.9 Compatibility

5.10 Nested vaults in the scan (fix included in phase 1)

_scan_local_directory (Vault__Sync__Base.py:555-562) and the other working-copy walks (:659, :752, Vault__Merge.remove_conflict_files / has_conflicts, Vault__Sync__Lifecycle.uninit's count) must not descend into a sub-folder that holds its own .sg_vault/. The check costs nothing extra: os.walk already lists each folder's sub-directories, so when root != directory and '.sg_vault' in dirs, clear dirs and skip files. Same rule as Git for a nested repository. One shared helper, Vault__Ignore.is_nested_vault_root(root_listing_dirs, root, directory) -> bool, so all walks use the same rule. status reports Nested vault skipped: wt/ once, so the user knows why those files are not tracked.

5.11 File-by-file changes

6. Security and threat model

6.1 What the worktree trusts

6.2 Cases

6.3 Threat-model rows to add

These use the next free TM numbers at implementation time (TM-R44 and TM-F38 at the time of writing; other designs may take numbers first).

6.4 Path-guard test to add

tests/security/test_Security__Fixed__Hostile_Paths.py:

• test_worktree_pointer_cannot_be_planted_by_vault_content. A hostile commit carries sub/.sg_vault/local/alternates.json and a file sub/.sg_vault. Clone, pull, branch switch, checkout, write, stash pop and revert must write neither. is_protected already refuses both; the test pins it for the new file name.
• Extend the existing parametrised test_spellings_of_structural_folders_are_protected with .sg_vault/local/alternates.json and .SG_VAULT/local/worktrees.json.

7. Interactions

7.1 Commands run in a worktree

7.2 Commands run in the main folder while worktrees exist

7.3 Clone shapes

7.4 Other designs

8. Performance

Reference numbers, 76,000 files in 58,553 folders, in-memory host, 4 cores (10/26/release-readiness__pack-indexes-branch.md §3): full clone 61 s; clone --no-checkout 28 s, so the network and store part is about 28 s and the checkout about 33 s; clone --path raw/c09 (1,520 files) 2.8 s.

Disk:

A full clone of the 76,000-file vault holds about 174 MB in .sg_vault; a worktree holds a few MB.

Read-path cost for clones without worktrees: one os.path.isfile on local/alternates.json per object-store instance (probed once), so nothing measurable.

9. Test plan

All tests are real: in-memory API (Vault__API__In_Memory) or Vault__Test_Env vaults, no mocks, no __init__.py under tests/. File names follow test_<Class>.py.

9.1 Unit tests

9.2 Security tests

9.3 Integration

tests/integration/test_Worktree__Live.py (Python 3.12 venv, the in-memory sgraph server): add, commit, push, main merge, pull, remove: the full loop against the real server code. One run with a pack vault.

9.4 Acceptance checklist (implementation and deployment)

10. Implementation steps (ordered commits)

Phase 1: about 3.25 days (commits 0–6). Phase 2: 1–1.5 days. Commits 0 and 2 are useful on their own and can land first.

11. Open decisions for Dinis

This page as markdown · its source, admin/content/research/git-parity-design-worktree.md