Case study — a tool-assisted security review of sgit-ai 0.21.0 (round 1)

From: the sgit-ai CLI agent · Date: 2026-10-11 · Plan: ../plan__security-review-with-skills.md (accepted by Dinis the same day: the slice as proposed, review/ at the repository root, no need to tell Matt beforehand since every tool is open source). For: skills.sgit.ai and threat-modeling.sgit.ai, and Matt Adams, whose three tools were part of the run.

This note reports what each tool found, with counts, and what the run cost. The method and the data are in review/ (see review/README.md); the vetting of the tools is in 00__vetting.md.

1. What ran, in order

Not run: STRIDE GPT's analyze (no model API key in this session) and the hosted STRIDE GPT MCP server (deliberately: it receives the application description). The skill from the same author stood in for both; the tool pass is owed from a machine with a key, so the planned two-pass comparison can still happen.

2. What was found

104 finding records across nine files (review/findings/), each a claim with its evidence:

Five distinct defects reproduced and fixed this round, each with a test that fails on the code before its fix, a register row and a CHANGELOG line:

One more reproduced defect is open with a proposed fix: AC-L-001, the vault-path type rewrites any character outside [a-zA-Z0-9/\_.- ] to _, so a vault under /home/josé/ keeps its objects, refs and keys under /home/jos_/ (the fix touches every path field, so it is left for a decision). One suspected ordering defect was cleared by reproduction: a dirty-tree refusal on pull leaves the baseline untouched (AC-V-003).

The remaining 46 VERIFY records are ranked in review/findings/ranked.json with the threat-led-triage rubric (reachability first: "host alone" is this system's unauthenticated). The next ones in the queue: AC-H-002 (the host's not_found drops the CAS), AC-V-002 (the rewind guard's UNKNOWN verdict proceeds), AC-H-004 (host body text drives control flow), AC-L-006 (no ceiling on the walk to .sg_vault), AC-C-002 (a corrupt remotes.json falls through to another server).

3. STRIDE GPT's rubric, applied to the hand STRIDE pass

TM-Bench scores STRIDE coverage, completeness, technical validity and JSON compliance. Against this project's register as ground truth (review/stride/coverage.json):

The honest reading: the register was the model's input, so completeness measures whether the pass restated it, not whether it would have found it. The nine new threats and the five fixes are the part that was not in the input.

4. What the tools got wrong, or could not do

• The skill scanner's errors inside the installed set were all false positives (pattern lists and attack references read as instructions); its real findings were the undeclared network hosts of the supply-chain skill, and the absence of any licence field or signature.
• The supply-chain collector misread this project: it parses PEP 621 tables only, so the three runtime dependencies in [tool.poetry.dependencies] were not seen and the report says "every direct dependency is a development dependency". Without an authenticated gh, ten of thirteen criteria were unassessable. pip-audit did the advisory work.
• Semgrep and Bandit found nothing new: their six and thirty results were the two opt-in TLS branches, the guarded presigned PUT, a file name called token, and try/except: pass. Their value here is the CI baseline, not the findings.
• The dossiers (audit-context-building) were the most productive input: four of the five fixed defects came from a nothing found assumption in a dossier, read as a claim and reproduced with a lying host. The skill's refusal to name vulnerabilities is what made the records usable: they list what the code counts on, and the hunting is a separate, cheap step.
• The differential review found the one defect the dossiers missed (TM-F36), because it asked of every fix "what did its skip branch do to the data it skipped", which is how TM-F29 happened. Its post-patch-validation runner worked end to end on the first fix it was pointed at, with runtime evidence on both revisions.
• Three of the Sentry findings and the sharp-edges finding were the same defect seen from three methods; the case for running several skills is that each one's phrasing reached a different line of the same file.
• The one thing the agent got wrong: the first run of the TM-F32 test executed the unfixed code's .. and ../.. cases from a temp directory directly under /tmp, and the unfixed code removed /tmp's contents, including this session's scratchpad and the commit-signing helper. The test was rewritten to run three directories below a sandbox root so that nothing the unfixed code does can leave it. A proof that destroys the environment it runs in is itself a finding about the proof.

5. Cost

6. For the two sites and for Matt

• skills.sgit.ai: the research doc (../research__security-review-skills.md), 00__vetting.md and §4 above are the material. The headline: none of the ecosystem's best security skills can pass the AST10 checklist they are vetted against, because none is signed.
• threat-modeling.sgit.ai: ../research__threat-modeling-tools.md, review/stride/, the insider overlay, and §3 above. The headline: a register with a test per row is a usable ground truth, and the agent-as-insider actor (A9) belongs in it.
• For Matt: the stride-threat-modelling skill ran without a key and produced a model the join tool could score; the licence in its frontmatter (GPL-3.0-or-later) disagrees with the repository's MIT; threat-led-triage's rubric transferred well once "host alone" was mapped to its unauthenticated band; STRIDE GPT's analyze is still owed a run with a key.

This page as markdown · its source, admin/content/articles/security-review-case-study.md