Design 04: rename detection, history blame, grep and pickaxe (roadmap 2.4, 2.5, 2.7)

Read with design__00__index-and-reconciliation.md: where the two disagree (shared foundations F1–F9, the in-progress rules), the index wins.

Status: design, not built. Date: 28 Oct 2026. Roadmap: roadmap__git-parity.md items 2.4, 2.5, 2.7 (priority 2). Depends on: nothing. Feeds: design 01 (line-level merge and cherry-pick) consumes the rename-pairing API in section 5.2 and, if it does not define its own, the line differ in section 5.1. Checked against: the code at 3a1d72a and Git 2.43.0 (installed here; every Git behaviour marked "confirmed" was run in a scratch repository on 28 Oct).

1. Summary and the exact Git behaviour matched

Three read-only features, all on plaintext the client already holds, none needing a server change or a format gate:

Git behaviours confirmed in Git 2.43 for this document:

Not confirmed here (no man pages in this container): merge.renameLimit default 7000. Design 01 should confirm it if it adopts that value.

2. Scope and non-goals

In scope

• Rename detection: exact, basename and similarity passes, with -M[<n>], --no-renames and the 1,000 limit.
• Where renames show: history diff (all modes), history show, history log --stat / --files / --patch / --json and the new history log --file <path> --follow.
• For the merge: the pairing API that design 01 calls.
• sgit history blame:
• <rev>, -L (numeric, +N, -N, /regex/, repeatable)
• -w, --first-parent
• output: --porcelain, --line-porcelain, --json, -s, -e (show key id), --abbrev N
• sgit grep (and sgit file grep):
• match flags: -i -w -v -F -E -P -e
• output flags: -n -c -l -L -o -q -h -H -A/-B/-C, --json
• what is searched: working copy, <rev>, --untracked, pathspecs
• Pickaxe: history log -S <string>, --pickaxe-regex, -G <regex>, combinable with --grep, --since, --until, --author and --file.
• A prerequisite refactor: Vault__Diff stops decrypting every file of both commits (section 4.3).

Not in scope

• Copy detection (-C, --find-copies-harder): a follow-up if asked.
• More blame options:
• --ignore-rev / --ignore-revs-file: optional, a follow-up; section 11, D12.
• -M/-C inside blame (moved or copied lines within or between files).
• -L :funcname, --incremental, --reverse, --since.
• More grep options:
• --and / --or / --not / (…) boolean expressions. Several -e patterns OR together, as in Git.
• --recurse-submodules, -p / --show-function, --threads.
• POSIX BRE (Git's default dialect); see D2.
• Elsewhere:
• renames in status (Git shows them only for staged changes; sgit has no staging area); D10.
• any server-side search: impossible under zero knowledge, by design.
• reading without a clone (--vault-key on a bare checkout is supported, as history log supports it; a no-clone network read is roadmap item "cat --vault-key").

3. CLI surface

Rule 9 (tests/unit/cli/test_CLI__B07__Namespace_Moves.py:104): blame and pickaxe are about commits, so they go under history. grep is a daily verb over files. It goes in the file namespace beside cat/ls/write, and also at the top level as sgit grep, the same way cat, ls and write are both (sgit_ai/cli/CLI__Main.py:472-495, sgit_ai/plugins/file/CLI__File.py). That adds 'grep' to TOP_LEVEL_COMMANDS, as a deliberate edit (D1).

3.1 Rename options (on history diff, history show, history log)

Human output:

• history log --stat keeps R old -> new and adds the score when below 100: R old -> new (95%).
• history diff / history show print a git-style rename header before the hunk:

diff --git a/docs/old.md b/docs/new.md
similarity index 95%
rename from docs/old.md
rename to docs/new.md
--- a/docs/old.md
+++ b/docs/new.md
@@ …

With --files-only: R095 docs/old.md -> docs/new.md.

• When the limit was hit, stderr says, once, in Git's words: warning: exact rename detection only: 1001 × 1001 files is over the rename limit (1000); pass --rename-limit 1001 to try them all.

JSON (additive, existing fields keep their meaning; see D4):

3.2 sgit history blame

sgit history blame [<rev>] [-L <range>]... [-w] [--first-parent] [-s] [-e] [--abbrev N]
                   [--porcelain | --line-porcelain | --json] [--text] [--offline]
                   [--vault-key KEY] [-d DIR] [--] <path>

<range> forms (as Git, confirmed): N,M, N,+K, N,-K, N (to the end), ,M, /regex/ (the first matching line, to the end), /regex/,+K. Several -L options are merged and sorted.

Default output, one line per line of the file:

^3f9a2c1d0b7e docs/a.md (alice    2026-10-27 14:30:02 +0000  4) text
57567877efb2 docs/c.md (bob-agent 2026-10-28 02:56:21 +0000  5) five

--porcelain and --line-porcelain: Git's exact layout, so tools that parse Git's output work unchanged.

<40-or-64-char id field> <orig line> <final line> [<group size>]
author <branch name or key id>
author-mail <key id or "unsigned">
author-time <seconds>
author-tz +0000
committer <same as author>
committer-mail <same>
committer-time <same>
committer-tz +0000
summary <first line of the message>
boundary                       (only for a boundary commit)
previous <parent id> <path>    (when the commit has a parent holding the file)
filename <origin path>
\t<line>

The id field is the full obj-cas-imm-<hex> id, never padded to 40.

--json (history_blame_v1):

{"schema": "history_blame_v1", "path": "docs/c.md", "rev": "<commit id>", "boundary_reason": "",
 "commits": {"<id>": {"timestamp_ms": 0, "author": "bob-agent", "author_key_id": "…", "summary": "…",
                      "boundary": false, "previous_id": "…", "previous_path": "…"}},
 "lines": [{"final_line": 5, "orig_line": 5, "commit_id": "<id>", "orig_path": "docs/a.md", "text": "five"}],
 "stats": {"commits_visited": 0, "blobs_read": 0, "blobs_fetched": 0, "index_plan_used": true}}

boundary_reason:

• '' when the root was reached;
• 'shallow' when a shallow boundary was reached;
• 'offline' when an object was missing and could not be fetched.

Exit codes:

Notes, written to stderr after the output:

• Shallow: … blame stops at the shallow boundary (lines marked ^ may be older). Deepen with: sgit fetch --unshallow
• Offline: … N object(s) were not local and could not be fetched: lines marked ^ stop there

3.3 sgit grep (also sgit file grep)

sgit grep [-i] [-w] [-v] [-F | -E | -P] [-n] [-c] [-l | -L] [-o] [-q] [-h | -H]
          [-A N] [-B N] [-C N] [-m N] [--untracked] [--text | -I] [--json] [--offline]
          [--vault-key KEY] [-d DIR]
          (<pattern> | -e <pattern>...) [<rev>...] [-- <pathspec>...]

Pattern and dialect

• The pattern is a Python regular expression (D2): -E (ERE) and -P (PCRE) are accepted and mean the same thing.
• -F takes the pattern literally.
• Several -e options are ORed.
• -i, -w and -v as in Git.

What is searched

• With no <rev>: the working copy's files, filtered by the ignore rules status uses, plus --untracked untracked files. The bytes on disk are searched, never followed through a symlink.
• With <rev> (any revision Vault__Revision resolves: HEAD, HEAD~2, a tag, a short id): that commit's tree, decrypted in memory.

Output

• One line per match, as Git prints it:
• working copy: path:line:text
• a revision: <rev as typed>:path:line:text
• -n adds line numbers.
• -c prints path:count; -l the matching files; -L the files without a match; -o only the matched parts; -A/-B/-C context with -/-- separators as in Git; -m N stops after N matches per file.
• -h/-H hide or show the file name.
• A binary file (a NUL in its first 8,000 bytes, Git's rule; sgit's BINARY_CHECK_BYTES is 8,192 and is reused) prints Binary file <path> matches. -I skips binary files; --text searches them as text.

Pathspecs: after --, Git's basic form: a literal prefix (docs/, docs/a.md) or a glob with *, ? and […] (fnmatch on the full path, ** matching folders). Magic forms (:(exclude), :!) are not supported and are refused by name.

--json (grep_v1):

{"schema": "grep_v1", "rev": "", "pattern": ["five"],
 "matches": [{"rev": "", "path": "c.txt", "line": 5, "text": "five", "spans": [[0, 4]]}],
 "files_searched": 0, "binary_matches": ["img.png"],
 "skipped": {"not_checked_out": 0, "not_local": 0, "outside_scope": 0}}

Exit codes (D3):

3.4 Pickaxe, on sgit history log

• Combining: pickaxe works with --grep, --since, --until, --author (all must hold), -n, --oneline, --stat, --json and --file <path> (only that file is searched). That lifts today's refusal (CLI__History._dispatch_log, sgit_ai/plugins/history/CLI__History.py:30) for these combinations only; a range with --grep stays refused.
• Merges: skipped as in Git without -m. The output ends with a note when any were skipped and the history had merges: (N merge commit(s) not searched, as in git log without -m).

4. How the code works today

4.1 Objects and what can be compared without decrypting

4.2 Exact renames today

Vault__Inspector._file_stat_from_diff (sgit_ai/objects/Vault__Inspector.py:252-276) pairs each added path with the first unused deleted path of the same content_hash (falling back to blob_id). Ties go in sorted path order, not Git's basename preference. It is used only by history log --stat through inspect_commit_chain (:97), whose per-commit diff comes from three sources, in order:

1. the pack index's changes member (_diff_from_provider, :217);
2. two cached flat maps (_diff_from_flats, :187);
3. diff_trees.

A D record from the index carries the old content_hash (Vault__Pack_Index.changes_of, sgit_ai/core/actions/pack/Vault__Pack_Index.py:201), so exact pairing works on the index path too.

4.3 history diff / show / log --files/--patch/--json decrypt everything

Vault__Diff (sgit_ai/core/actions/diff/Vault__Diff.py):

On the 228,000-file vault that is about 1.2 GB decrypted per side, per commit. diff_vs_head / diff_vs_remote / diff_vs_commit (:35-58) also read every working-copy file and every committed blob.

log_file (:125) walks the first-parent chain and, for each commit and its parent, calls flatten on the whole tree to read one path's blob id. It compares blob ids only, so a writer that re-encrypts unchanged content (the web UI) shows as modified.

On-demand fetch: CLI__Diff._try_fetch_commits (sgit_ai/cli/CLI__Diff.py:35) → Vault__Diff.ensure_commit_local (:736) → Vault__Sync__Pull._fetch_missing_objects(..., include_blobs=True). It is gated on a configured remote (_on_demand_api, :15: a never-pushed vault never reaches the default host). One blob is fetched in Vault__Sync__Sparse.sparse_cat (sgit_ai/core/actions/sparse/Vault__Sync__Sparse.py:177): api.read or read_large, then _save_verified.

4.4 history log dispatch and filters

• Parser: CLI__History.register (sgit_ai/plugins/history/CLI__History.py:73).
• Dispatch (:18):
• a range, --files, --patch or --json → CLI__Diff.cmd_log_range;
• --file → CLI__Diff.cmd_log_file;
• else → CLI__Vault.cmd_log → _print_log (sgit_ai/cli/CLI__Vault.py:2274).
• Filters: --grep/--since/--until/--author are filtered in Vault__Log_Filter (sgit_ai/core/actions/history/Vault__Log_Filter.py). They walk the whole DAG (inspect_commit_dag), or the first-parent chain with --stat.
• Regex guard: Vault__Log_Filter.nested_repeat (:50) refuses nested repetitions ((a+)+), read from the parsed pattern.

4.5 Pack indexes (Phase 1 reader)

TM-R42: a snapshot index is anchored to the root tree only, and is a plan, never a proof.

4.6 Partial clones

• Vault__Scope (sgit_ai/storage/Vault__Scope.py:30):
• is_scoped, is_shallow, contains_path, boundary_ids;
• Vault__Scoped_Tree.flatten returns (flat, opaque) for a scoped clone.
• Selective working copies: the --no-checkout record. A tracked file absent from disk is "not checked out", not deleted (CHANGELOG 0.26.0).
• Safe types: Safe_Str__File_Path rewrites every character outside [a-zA-Z0-9/\_.- ] to _ (sgit_ai/safe_types/Safe_Str__File_Path.py; the AC-L-001 / TM-R36 family). An output schema that types a vault content path with it mangles josé.md. The new schemas use a Unicode-tolerant type (section 5.8).

5. Design

5.1 Line differ: Vault__Line_Diff

One line differ for blame, -G, the unified diff and the merge.

• Design 01 (diff3) needs matching blocks between base/ours and base/theirs. If design 01 specifies its own line differ, this design uses that class instead and the names below are aliases. There must be one implementation.

Interface

• File: sgit_ai/core/actions/diff/Vault__Line_Diff.py, class Vault__Line_Diff(Type_Safe).
• Fields:
• ignore_whitespace : bool = False (-w: lines compared with all whitespace removed, as Git's -w);
• max_cost : Safe_UInt = 0 (0 = the default bound below).
• Methods:
• split_lines(data: bytes) -> list — bytes.splitlines(keepends=True). A final line without \n stays distinct from the same text with one, as in Git ("\ No newline at end of file").
• matching_blocks(a: list, b: list) -> list — [(a_start, b_start, length)], ascending, ending with the sentinel (len(a), len(b), 0), the same contract as difflib.SequenceMatcher.get_matching_blocks().
• opcodes(a, b) -> list — [('equal'|'replace'|'delete'|'insert', i1, i2, j1, j2)] derived from the blocks.

Algorithm: Myers O((N+M)·D) greedy diff, the algorithm behind Git's default

• Common prefix and suffix are stripped first, as xdiff does.
• Lines are interned to integers (hash map of line → id; with -w, the key is the line with whitespace removed).
• Linear-space divide and conquer (middle snake) above 10,000 lines.
• When D passes a cost bound (default max(256, sqrt(N+M) * 16), xdiff's "heuristic" idea), it falls back to difflib.SequenceMatcher(None, a, b, autojunk=False) on the remaining window, so a pathological pair never runs quadratic for minutes.

Why not difflib alone

• SequenceMatcher finds longest matching blocks, not a minimal edit script. With autojunk=True (the default) it treats lines that occur in more than 1% of a file over 200 lines as junk, and loses matches on files full of blank lines.
• Blame attribution would then differ from Git's on common files.
• difflib.unified_diff stays for printing hunks today. It is switched to opcodes() in step 3 so that the printed diff and blame agree.

5.2 Renames: Vault__Renames

Classes

• File: sgit_ai/core/actions/diff/Vault__Renames.py, class Vault__Renames(Type_Safe).
• Fields:
• min_score : Safe_UInt__Rename_Score = 50 (percent);
• limit : Safe_UInt = 1000;
• enabled : bool = True;
• exact_only : bool = False.
• Schemas (sgit_ai/schemas/diff/):
• Schema__Rename_Pair: old_path : Safe_Str__Content_Path, new_path : Safe_Str__Content_Path, score : Safe_UInt__Rename_Score (0–100), exact : bool;
• Schema__Rename_Result: pairs : list[Schema__Rename_Pair], limit_hit : bool, candidates : Safe_UInt (sources × destinations considered), unread : Safe_UInt (blobs that were not local and not fetched; those files took part in the exact pass only).

Entry point (also the API design 01 consumes)

def detect(self, deleted: dict, added: dict, content_of=None) -> Schema__Rename_Result:
    """deleted/added: {path: flat entry} (the diff_trees / flatten shape: blob_id, size, content_hash, ...).
    content_of: callable(entry) -> bytes | None, the plaintext of an entry (None = not available).
    Without content_of only the exact pass runs. Never raises for a missing blob."""

Pass 1, exact (always, no limit):

1. Group deleted by (content_hash, size), skipping entries without a content_hash.
2. For each added path in sorted order, the candidates are the deleted paths with the same key not yet used. Pick the one with the same basename if any (Git's preference), else the first by path.
3. Score 100, exact=True.
4. A source pairs at most once. A second destination of the same content is an add (copies are out of scope).
5. Empty files take part (Git 2.43 pairs a lone moved empty file; confirmed). With several empty files, only same-basename pairs are made, so empty files never pair at random. Test 9.1 R10 checks this against Git.

Pass 2, basename (needs content_of, no limit):

1. Among unpaired paths, take basenames that occur exactly once among the remaining sources and exactly once among the remaining destinations.
2. Score each such pair by similarity.
3. Accept at min_basename_score = min_score + (100 - min_score) / 2 (75 at the default), Git's rule from 2.33.

Pass 3, inexact matrix (needs content_of):

1. If len(src) > limit or len(dst) > limit and len(src) * len(dst) > limit * limit, skip the pass and set limit_hit=True (Git's too_many_rename_candidates).
2. Else, for each destination, score every source that passes the size precheck, keep the best four (NUM_CANDIDATE_PER_DST, as Git), sort all kept pairs by score descending then by destination path, and accept greedily, each path once, at score >= min_score.

similarity(src: bytes, dst: bytes) -> int follows Git's diffcore-delta.c and estimate_similarity.

1. Size precheck: with max_size = max(len) and delta = abs(len(src) - len(dst)), return 0 when max_size * (100 - min_score) < delta * 100. A file cannot be a 50% rename of one under half its size.
2. Chunking (hash_chars): chunk each buffer. A chunk ends after a \n or after 64 bytes. For a text buffer (no NUL in the first 8,000 bytes), a \r directly before \n is skipped. Key each chunk by its bytes; the count is the chunk's length.
3. Count: copied = Σ min(src_count[k], dst_count[k]) over keys present in both.
4. Score: copied * 100 // max_size, truncated as Git prints it (Git computes on MAX_SCORE = 60000 and prints score * 100 / MAX_SCORE; the integer percent is the same up to rounding). Test 9.1 R4 compares with Git within ±1.
5. Empty src or dst: 0 (an empty file is only ever an exact rename).

content_of cost control

• Callers pass Vault__Blob_Reader.content (5.3).
• Blobs larger than RENAME_MAX_BYTES = 4 MB (Git uses big_file_threshold 512 MB, but our reads are decrypts) are scored by exact pass only and counted in unread.
• In-memory chunk maps are memoised per blob id for one detect call.

Trust:

• content_hash is the committer's claim. For display (diff, show, log), pairing by the claim is accepted: a lying writer (A3) can make --stat show a false rename and nothing else.
• For the merge, design 01 must call detect(..., content_of=...) with verify_exact=True (a keyword on detect). Each exact pair is then confirmed by decrypting both blobs and comparing bytes, and an unconfirmed claim drops to pass 3.

Coordination note for design 01

• three_way_merge (sgit_ai/core/actions/merge/Vault__Merge.py:14) compares blob ids, and blob ids are random per write (4.1). The same content written independently on both sides is reported as a conflict today (ours_bid != theirs_bid). Comparing content_hash + size (verified when it matters) fixes that.
• The merge's rename pairs: detect(base_deleted_in_ours, ours_added) and the same for theirs.

5.3 Reading content: Vault__Blob_Reader

Purpose: one place that turns a flat entry into plaintext, for renames, blame, grep, pickaxe and the pruned diff.

Interface

• File: sgit_ai/core/actions/history/Vault__Blob_Reader.py, class Vault__Blob_Reader(Type_Safe).
• Fields:
• crypto, obj_store, read_key : bytes, vault_id : str, sg_dir : str;
• api : object = None (set only when a remote is configured: the _on_demand_api rule);
• offline : bool = False;
• memo_bytes : Safe_UInt = 64 MB;
• counters blobs_read, blobs_fetched, blobs_missing (Safe_UInt; reported in --json stats and used by the tests instead of a counting double).

Behaviour

• content(entry) -> bytes | None:

1. A local object → obj_store.load (this also reads from a --pack-store clone by offset).
2. Else, with api and not offline → fetch with api.read / read_large, _save_verified (verified by content address before it is written, as sparse_cat does), then load.
3. Else → None and blobs_missing += 1.

• Decryption errors propagate (a wrong key must fail loudly, as history log does with InvalidTag).
• prefetch(entries: list) -> None: one api.batch_read for every missing blob in the list, in batches of the existing batch size. Blame and grep call it before walking, so a sparse clone makes a handful of requests instead of one per file.
• The memo is an LRU keyed by blob id up to memo_bytes of plaintext, in memory only.

5.4 Path history: Vault__Path_History

Interface

• File: sgit_ai/core/actions/history/Vault__Path_History.py, class Vault__Path_History(Type_Safe).
• Fields: crypto, obj_store, read_key, sub_tree : Vault__Sub_Tree, tree_memo : dict (tree id → {name: entry}, bounded at 4,096 trees).

Methods

• entry_at(tree_id, path) -> dict | None: walks the path's folders with _load_tree_entries (plain JSON) and returns the file's flat entry (the _entry_dict shape) or None. Cost: one tree read per path segment, memoised by tree id. Tree ids are content addresses, so the memo never needs invalidating.
• same_at(tree_a, tree_b, path) -> bool: true when the entry at path is identical in both. It short-circuits from the root: equal root ids → True; then each folder on the path, and the first equal folder id → True. At the file: equal blob ids → True, else equal (content_hash, size) → True. That treats a re-encryption of unchanged content as unchanged, which fixes log_file's false modified (4.3).
• changed_paths(tree_new, tree_old) -> dict: diff_trees passthrough, memoised per pair for one command.

Consumers

• Vault__Diff.log_file is rewritten on it: entry_at per commit instead of flatten per commit.
• --follow, blame and pickaxe with---file.

5.5 The pruned diff (prerequisite)

Vault__Diff gains _changed_files(tree_new, tree_old, renames: Vault__Renames) -> list[Schema__Diff_File]:

1. Take diff_trees (or the index's changes when the caller has them).
2. Run Vault__Renames.detect over added/deleted.
3. For each modified or renamed file, read both sides with Vault__Blob_Reader and run the unified diff. For an added or deleted file read one side (needed for --patch of an added file and for the binary flag).

The diff methods change as follows:

Output stays the same except that renames now appear (D4). Schema__Diff_Result.files loses its unchanged rows for commit-to-commit diffs. They were never printed (CLI__Diff._print_result skips them, sgit_ai/cli/CLI__Diff.py:271), and no JSON path emits Schema__Diff_Result for two commits (--json there goes through diff_range). The working-copy JSON path keeps them only if a test shows a consumer, else drops them too; step 3 checks this.

5.6 Blame: Vault__Blame

Classes

• File: sgit_ai/core/actions/history/Vault__Blame.py, class Vault__Blame(Type_Safe).
• Fields: crypto, api : object = None, ignore_whitespace : bool, first_parent : bool, follow_renames : bool = True, text : bool = False, offline : bool = False.
• Schemas (sgit_ai/schemas/history/):
• Schema__Blame_Line: final_line : Safe_UInt, orig_line : Safe_UInt, commit_id : Safe_Str__Commit_Id, orig_path : Safe_Str__Content_Path, text : Safe_Str__Diff_Text;
• Schema__Blame_Commit: timestamp_ms, author : Safe_Str__Diff_Text, author_key_id, summary : Safe_Str__Diff_Text, boundary : bool, previous_id, previous_path;
• Schema__Blame_Result: schema = 'history_blame_v1', path, rev, boundary_reason, commits : dict[Safe_Str__Commit_Id, Schema__Blame_Commit], lines : list[Schema__Blame_Line], stats.

Entry point: blame(directory, path, rev='', ranges=None) -> Schema__Blame_Result.

Algorithm (Git's blame.c, assign_blame / pass_blame, simplified to what we support):

1. Start.

• Resolve rev (default the clone's HEAD) with Vault__Revision.
• entry = path_history.entry_at(commit.tree_id, path).
• None, or a folder → exit 1 with path not found in <rev>.
• Content via Vault__Blob_Reader. Binary (NUL rule) without --text → exit 1 with binary file: pass --text to blame it line by line.
• Lines via Vault__Line_Diff.split_lines. -L ranges are resolved on this content.

1. Entries.

• A blame entry is (final_start, count, suspect_start), held per origin (commit_id, path, entry).
• Start: one entry per -L range, all on origin (rev, path, entry).
• Origins wait in a priority queue keyed by generation number (from the pack index's commit-gen, or Vault__Pack_Index.generation_of with its local memo) and then timestamp_ms, highest first. Git's queue orders by commit date; generation avoids clock skew between agents and gives the same order on linear history.

1. Process the next origin O (commit C, path P, entry E):

• Parents: C's parents in order; with --first-parent, only the first.
• Boundary. C has no parents; or C is a shallow boundary (Vault__Scope.is_boundary); or C's parent commit object is not local and cannot be fetched. Then every entry of O is final and C is marked boundary. boundary_reason is set to shallow / offline when that is the cause.
• Find each parent's version. E_p = entry_at(parent.tree_id, P).
• If None and follow_renames: run Vault__Renames.detect on diff_trees(C.tree, parent.tree) restricted to destination P (all deleted paths as sources, only P as destination; the limit counts 1 × sources, so the inexact pass always runs). A pair gives (P_p, E_p).
• If still None, that parent does not hold the file.
• Whole pass. If some parent has same_at(C.tree, parent.tree, …) for the file (same blob, or same content_hash + size), all of O's entries move to that parent's origin unchanged, without a diff. Git does this for an identical parent, and it is the step that makes blame cheap on a long history. Done with O.
• Diff pass. Otherwise, for each parent holding the file, in order:
• Diff parent content → C content with Vault__Line_Diff (honouring -w).
• For every equal block (i, j, n), the overlap of each remaining entry's suspect range [j, j+n) moves to the parent's origin with suspect_start remapped to i + (s - j). Entries are split at block edges.
• Lines left after all parents stay blamed on C.
• At a merge, lines equal to the first parent go there first and the rest are offered to the next parent: Git's order.
• Merge origins. A parent origin already in the queue (same commit, same path) gets the entries appended, so a commit is diffed once per path.

1. Finish. Final entries are sorted by final_start and expanded to lines with orig_line = suspect_start + k. Commit metadata is collected:

• author: the branch name from the branch index (_branch_names as in CLI__Vault, :2338), else the key id;
• summary: the first line of the decrypted message;
• previous: the first parent holding the file and its path.

Pack index acceleration (step 9).

1. When the head's snapshot index is local and verified (open_index(kind='snapshot')), read file-commits:<shard_of_path(P)> for P.
2. The listed commits are the plan: the walk visits only them, and treats each unlisted stretch between two listed commits as unchanged for P.
3. Because the snapshot index is anchored only at the root (TM-R42), the plan is checked as it is used. For each step from listed commit C to the next listed ancestor C' on the walk, same_at(parent(C).tree, C'.tree, P) must hold: the file must really be unchanged across the skipped stretch.
4. One failed check drops the plan for the rest of the command (noted once on stderr: index plan for <path> did not match the trees; walking) and the walk continues commit by commit.
5. A lying index can therefore cost time, never a wrong attribution.
6. Renames end the plan at the commit where the path was added. The plan of the old path is read from its own shard.

Partial clones and limits:

• Scoped: P outside scope.contains_path → exit 1 with '<path>' is outside this clone's folders (<list>); widen with: sgit fetch <folder>.
• Shallow: boundary as above.
• Sparse / selective: blobs are fetched on demand (Vault__Blob_Reader.prefetch with the blobs of the planned commits when a plan exists).

5.7 Grep: Vault__Grep

Classes

• File: sgit_ai/core/actions/history/Vault__Grep.py, class Vault__Grep(Type_Safe).
• Fields: crypto, api : object = None, options as typed fields:
• patterns : list[Safe_Str__Grep_Pattern]
• fixed, ignore_case, word, invert, untracked, text, skip_binary (bools)
• max_count : Safe_UInt, before : Safe_UInt, after : Safe_UInt
• only_matching : bool, offline : bool
• Schemas:
• Schema__Grep_Match: rev, path : Safe_Str__Content_Path, line : Safe_UInt, text : Safe_Str__Diff_Text, spans : list, context : bool;
• Schema__Grep_Result: schema = 'grep_v1', matches, files_searched, binary_matches, skipped.

Engine

1. Compile. One Python re pattern: -F escapes each pattern; several patterns join as (?:p1)|(?:p2); -w wraps as (?<![\w])(?:…)(?![\w]) (Git's word rule: the match must start and end at a word boundary of [A-Za-z0-9_]; test 9.3 G6 checks edge cases against Git); -i adds re.IGNORECASE.

• Every pattern is checked first by Vault__Regex_Guard.check(pattern) (5.9).
• Matching is on str: content is decoded utf-8 with errors='surrogateescape', so any byte sequence round-trips and non-UTF-8 files still match byte-for-byte on their ASCII parts.

1. Prefilter per file. One search over the whole decoded text; no match → skip the file without splitting lines. With -v, -L or -c, the prefilter is skipped (they need every line). That keeps the common case at C speed: one regex scan per file.
2. Lines. On a hit, iterate lines with offsets (str.splitlines(keepends=True) once), collect matches, context windows and -m cut-off, and emit Schema__Grep_Match rows in path order then line order.
3. Binary. NUL in the first 8,192 bytes:

• default → one Binary file <path> matches row when the prefilter hits;
• -I → skipped;
• --text → treated as text.

Sources

Order and streaming.

• Files are processed in sorted path order and printed as they are found (no full result held unless --json), so sgit grep x | head returns at once.
• The reader's memo is off for grep, since each blob is read once.

5.8 Pickaxe: Vault__Pickaxe

Interface

• File: sgit_ai/core/actions/history/Vault__Pickaxe.py, class Vault__Pickaxe(Type_Safe).
• Fields: mode : Safe_Str ('S' or 'G'), needle : Safe_Str__Grep_Pattern, regex : bool, ignore_case : bool, only_path : Safe_Str__Content_Path = None, plus Vault__Blob_Reader and Vault__Line_Diff.
• Method: matches(commit, parent, changes) -> list[path], where changes is the commit's change list (the index's changes member when available, else diff_trees).

Rules

• -S: for each changed file (A, M, D; restricted to only_path), count occurrences in old and new content (literal: str.count, non-overlapping as Git; --pickaxe-regex: len(re.findall)). Different counts → the path matches. Binary files are counted too (Git's -S does). An added file counts against 0 and a deleted file against 0.
• -G: run the line diff of old → new and test each inserted or deleted line with re.search. Binary files are skipped unless --text (Git's -G rule).
• Merges (more than one parent): skipped, counted in the note.
• Wiring:
• _print_log (CLI__Vault.py:2274) gains a pickaxe branch: walk the DAG newest-first as the filtered log does.
• For each non-merge commit get changes from _index_changes_provider or diff_trees, apply the other filters first (cheap: message, date, author), then Vault__Pickaxe.matches.
• Stop once -n (default 50) matches are printed.
• --stat shows the matching files only, or all with --pickaxe-all.

5.9 Shared pieces and safe types

5.10 File-by-file changes

Compatibility

• No format gate: nothing is written to the vault or the host.
• No change in what other clients see.
• Local state: no new local file. The flat cache is read, never extended.
• JSON consumers see added fields. The one behaviour change, renamed files moving out of files_added/files_deleted, is D4.

6. Security and threat model

7. Interactions

8. Performance, caps and limits

Reference vaults: the 76,000-file / 58,553-folder synthetic vault (release readiness, 26 Oct) and the 228,000-file / 38,000-leaf-folder one (0.25.0 notes). Estimates below are from the measured costs: ~3 ms a tree through Type_Safe (avoided here), well under 1 ms through _load_tree_entries, and AES-GCM decrypt in C.

Hard caps:

• pattern length 1,000;
• -L ranges 100;
• blame on a file over 50 MB refused (file is larger than 50 MB: blame refused);
• rename scoring per blob 4 MB;
• rename limit 1,000 (option);
• blob memo 64 MB.

9. Test plan

All tests are real objects. Vaults are built with Vault__Test_Env (tests/_helpers/vault_test_env.py: fixed test key, in-memory API) and commits made with write_file. Where Git is the oracle, the same history is built in a temporary Git repository (git -c user.name=t -c user.email=t@t). Those tests are skipped when shutil.which('git') is None. CI runners have Git. No __init__.py in test folders.

9.1 tests/unit/core/actions/diff/test_Vault__Renames.py — Test_Vault__Renames

9.2 tests/unit/core/actions/diff/test_Vault__Line_Diff.py — Test_Vault__Line_Diff

9.3 Other unit tests

tests/unit/core/actions/history/test_Vault__Path_History.py

• entry_at: root file, deep file, missing, a folder path.
• same_at:
• root-equal fast path (no tree read beyond the root, asserted by the memo count);
• re-encrypted identical content counts as the same;
• a changed file.
• log_file regression: the statuses and order equal today's on the existing fixtures, except the re-encryption case.

tests/unit/core/actions/history/test_Vault__Blame.py — Test_Vault__Blame. Oracle: git blame --line-porcelain mapped through unique commit messages.

tests/unit/core/actions/history/test_Vault__Grep.py — Test_Vault__Grep. Oracle: git grep on the same tree, committed and checked out.

tests/unit/core/actions/history/test_Vault__Pickaxe.py — Test_Vault__Pickaxe. Oracle: git log -S/-G --format=%s.

tests/unit/core/actions/diff/test_Vault__Diff__Pruned.py — Test_Vault__Diff__Pruned.

• Golden outputs of show, diff A..B, log --files/--patch/--json captured from the current implementation on Vault__Test_Env fixtures (commit them before step 3). The new implementation must match them exactly, with renames off.
• blobs_read equals the changed files' sides, not the vault size.

tests/unit/core/test_Vault__Regex_Guard.py

• The existing Vault__Log_Filter guard cases moved here.
• The Vault__Log_Filter tests still pass unchanged.

9.4 CLI tests

9.5 Security tests

9.6 Acceptance checklist (implementation and deployment)

Tests

• [ ] pytest tests/unit -n auto and pytest tests/security pass. The Git-oracle tests ran (not skipped) in CI: the job log shows their names as passed.
• [ ] Vault__Log_Filter tests unchanged and passing after the guard move.
• [ ] Golden tests: history show / diff / log --files --patch --json outputs byte-identical to before with --no-renames.

Renames

• [ ] On a scratch vault, sgit history show of a commit that moved and edited a file prints similarity index NN% / rename from / rename to. git diff -M on the same contents gives the same NN ±1.
• [ ] sgit history log --file docs/new.md --follow lists the commits before the rename with an R row.

Blame and grep

• [ ] sgit history blame <file> on a 5-commit history equals git blame line for line (commit mapping by message).
• [ ] sgit grep -n foo and sgit grep -n foo HEAD~1 equal git grep output on the same tree. Exit code 1 when nothing matches.

Pickaxe

• [ ] sgit history log -S foo --oneline and -G foo list the same commits as Git (by message).

Partial clones

• [ ] Shallow clone: blame shows ^ lines and the deepen note. Scoped clone: blame outside refused with the widen hint. Selective clone: grep's not-checked-out note.

Performance

• [ ] On the 76,000-file vault:
• history show of a 5-file commit under 2 s (was tens of seconds);
• history blame on a file with 40 changes under 5 s, under 1.5 s with the index plan;
• numbers recorded in the CHANGELOG.
• [ ] sgit grep x | head -1 returns before the search finishes (streaming).

Deployment

• [ ] sgit help all lists grep. sgit history --help lists blame.
• [ ] CHANGELOG [Unreleased] entry; threat-model rows (TM-R4x new, TM-R42 sentence); sgit-for-git-users mapping rows for blame, grep, log -S/-G, --follow, -M.
• [ ] Dev pre-release (0.x.y.devN) installed with pip install --pre sgit-ai in a clean venv; the checklist's CLI items re-run against the installed build.

10. Implementation steps (ordered commits)

The roadmap's 4–6 days (2.4 + 2.5 + 2.7) did not include step 3 (the pruned diff) or step 8. Step 3 is needed: blame, renames and pickaxe on a large vault are unusable on top of the current full-decrypt diff. Step 8 can ship later.

If design 01 lands first with its own line differ, step 3 uses it and Vault__Line_Diff is not written.

11. Open decisions for Dinis

This page as markdown · its source, admin/content/research/git-parity-design-blame-grep.md