Four-way clean benchmark

Fresh isolated checkouts · Fable + GPT-5.6 Sol · generated 2026-07-24T16:40:15-07:00

Arm D context: senso search context over Senso-ingested greplica PR markdowns

A/B/C/Dall ≥90 after 1 fix

Comparison table

Active agent: A 26m 58s · B 26m 58s · C 26m 58s · D 10m 38s

Arm	Context	Wall clock	Score path	Review 1 findings	Lines	Similarity API	Managed	protocol.ts	Coding passes	npm test
A — no context	None (create-baseline only)	27m 51s R1 628s · fix 491s · R2 499s	74 → 92	9 (H0 M3 L6)	+239/−3	findSimilarClaims kept	throws unsupported	no	1	pass
B — gold context.md	Static issue #169 body	27m 36s R1 628s · fix 503s · R2 487s	58 → 93	5 (H1 M2 L2)	+313/−3	findSimilarClaims kept	throws unsupported	no	1	pass
C — full-context.md	Live GitHub retrieval prompt	27m 22s R1 628s · fix 516s · R2 474s	58 → 95	6 (H3 M1 L2)	+302/−2	findSimilarClaims kept	protocol.ts route + HTTP client	yes	1	pass
D — Senso-native context	senso search context (KB chunks from Senso-ingested PR markdowns)	11m 40s R1 263s · fix 182s · R2 193s · senso 6s	58 → 92	6 (H2 M2 L2)	+368/−3	findSimilarClaims kept	local compute via /graph	no	1	pass

Learnings

Arm D added serially after A/B/C; its wall clock is standalone (~11–12m), not concurrent with A/B/C.
Senso-native context used `senso search context` against ingested greplica PR markdowns (chunks only, no AI answer), then fed into both Fable and Sol.
Arm D R1=58 (2 highs: managed endpoint + greedy grouping, echoing #177 themes surfaced by retrieval) → R2=92 after 1 fix.
Unlike C (protocol.ts + HTTP route), D’s Sol chose auditCodeAnchors-style local compute via existing `/graph` — closer to Fable’s recommended_fix and Senso chunks about #177 reworking managed handling without inventing an unverified endpoint.
All four arms kept findSimilarClaims; all stopped ≥90 after 1 coding pass.
D produced the largest line delta (+368/−3) driven by check-audit-duplicate-claims.js + managed CLI coverage.
Arms ran concurrently for A/B/C; compare phase times and outcome quality. D is sequential — do not rank wall clocks across cohorts.
Arm A (no context) still kept findSimilarClaims in this clean run — outcome variance vs earlier memory-off cosine rewrite shows non-determinism; isolation matters.
Arm B gold context scored Review1 at 58 (stricter on greedy pairing vs issue #169) then 93; managed path = unsupported throw.
Arm C full-context also R1=58 but with 3 highs (protocol/tests/pairing) via live #177-shaped constraints; only arm that changed protocol.ts; final 95.
Finding count alone misleads: B and C both had ~5–6 R1 findings but C had more highs and produced the protocol-aligned fix.

Per-arm audits

Raw comparison.json

{
  "generated": "2026-07-24T16:40:15-07:00",
  "pr": "https://github.com/Autoloops/greplica/pull/176",
  "base": "f63e2de17915a059f3acf42d2636dd2b79160f06",
  "head": "ada12c1d7b21f16b4beda8383d108fb33acc9f1e",
  "models": {
    "reviewer": "claude-fable-5-thinking-high",
    "coder": "gpt-5.6-sol-medium"
  },
  "arms": [
    {
      "arm": "A-no-context",
      "label": "A \u2014 no context",
      "context": "None (create-baseline only)",
      "wall_clock_secs": 1671,
      "wall_clock": "27m 51s",
      "phases": "R1 628s \u00b7 fix 491s \u00b7 R2 499s",
      "score_path": "74 \u2192 92",
      "r1_findings": "9 (H0 M3 L6)",
      "r2_findings": 3,
      "lines": "+239/\u22123",
      "similarity_api": "findSimilarClaims kept",
      "managed": "throws unsupported",
      "protocol_ts": "no",
      "coding_passes": 1,
      "npm_test": "pass",
      "active_agent_secs": 1618,
      "active_agent": "26m 58s"
    },
    {
      "arm": "B-gold-context",
      "label": "B \u2014 gold context.md",
      "context": "Static issue #169 body",
      "wall_clock_secs": 1656,
      "wall_clock": "27m 36s",
      "phases": "R1 628s \u00b7 fix 503s \u00b7 R2 487s",
      "score_path": "58 \u2192 93",
      "r1_findings": "5 (H1 M2 L2)",
      "r2_findings": 2,
      "lines": "+313/\u22123",
      "similarity_api": "findSimilarClaims kept",
      "managed": "throws unsupported",
      "protocol_ts": "no",
      "coding_passes": 1,
      "npm_test": "pass",
      "active_agent_secs": 1618,
      "active_agent": "26m 58s"
    },
    {
      "arm": "C-full-context",
      "label": "C \u2014 full-context.md",
      "context": "Live GitHub retrieval prompt",
      "wall_clock_secs": 1642,
      "wall_clock": "27m 22s",
      "phases": "R1 628s \u00b7 fix 516s \u00b7 R2 474s",
      "score_path": "58 \u2192 95",
      "r1_findings": "6 (H3 M1 L2)",
      "r2_findings": 2,
      "lines": "+302/\u22122",
      "similarity_api": "findSimilarClaims kept",
      "managed": "protocol.ts route + HTTP client",
      "protocol_ts": "yes",
      "coding_passes": 1,
      "npm_test": "pass",
      "active_agent_secs": 1618,
      "active_agent": "26m 58s"
    },
    {
      "arm": "D-senso-context",
      "label": "D \u2014 Senso-native context",
      "context": "senso search context (KB chunks from Senso-ingested PR markdowns)",
      "wall_clock_secs": 700,
      "wall_clock": "11m 40s",
      "phases": "R1 263s \u00b7 fix 182s \u00b7 R2 193s \u00b7 senso 6s",
      "score_path": "58 \u2192 92",
      "r1_findings": "6 (H2 M2 L2)",
      "r2_findings": 3,
      "lines": "+368/\u22123",
      "similarity_api": "findSimilarClaims kept",
      "managed": "local compute via /graph",
      "protocol_ts": "no",
      "coding_passes": 1,
      "npm_test": "pass",
      "active_agent_secs": 638,
      "active_agent": "10m 38s"
    }
  ],
  "learnings": [
    "Arm D added serially after A/B/C; its wall clock is standalone (~11\u201312m), not concurrent with A/B/C.",
    "Senso-native context used `senso search context` against ingested greplica PR markdowns (chunks only, no AI answer), then fed into both Fable and Sol.",
    "Arm D R1=58 (2 highs: managed endpoint + greedy grouping, echoing #177 themes surfaced by retrieval) \u2192 R2=92 after 1 fix.",
    "Unlike C (protocol.ts + HTTP route), D\u2019s Sol chose auditCodeAnchors-style local compute via existing `/graph` \u2014 closer to Fable\u2019s recommended_fix and Senso chunks about #177 reworking managed handling without inventing an unverified endpoint.",
    "All four arms kept findSimilarClaims; all stopped \u226590 after 1 coding pass.",
    "D produced the largest line delta (+368/\u22123) driven by check-audit-duplicate-claims.js + managed CLI coverage.",
    "Arms ran concurrently for A/B/C; compare phase times and outcome quality. D is sequential \u2014 do not rank wall clocks across cohorts.",
    "Arm A (no context) still kept findSimilarClaims in this clean run \u2014 outcome variance vs earlier memory-off cosine rewrite shows non-determinism; isolation matters.",
    "Arm B gold context scored Review1 at 58 (stricter on greedy pairing vs issue #169) then 93; managed path = unsupported throw.",
    "Arm C full-context also R1=58 but with 3 highs (protocol/tests/pairing) via live #177-shaped constraints; only arm that changed protocol.ts; final 95.",
    "Finding count alone misleads: B and C both had ~5\u20136 R1 findings but C had more highs and produced the protocol-aligned fix."
  ],
  "note": "A/B/C concurrent cohort; D sequential Senso-native arm. All \u226590 after 1 coding pass.",
  "title": "Four-way clean benchmark (A/B/C + D Senso)"
}