Cross-Lab Audit Visual Explainers: Three Pairs of Eyes (Visual Demo) Study #01 Audit Trail (CEO Spin · v1 → v4.2) Study #02 Audit Trail (AI Washing · v1.0 → v4.0.1)
CuriosityBits Living Lab Boston · Study #02 · Home (All studies) · Study #02 page (v4.0) · Lab Log & Mistakes · What Was Retired vs. What Survived ↓

Who Checked Study #02, What They Found, and How the “AI Washing Regime Flip” Was Retired

Cross-lab visual explainer tracing Study #02 (v1.0 → v4.0.1) across CuriosityBits Living Lab Boston (Gemini 4 Argon), CuriosityBits Living Lab Singapore (Muse Spark 1.3, MiniMax-M3, and Claude Opus 5.5), and the Director’s External Review (DIRECTOR-FEEDBACK-S02-2026-11-02). Every number, case ID, fraction, date, commit ID, and SHA-256 hash on this page is verified by check_all_artifacts_provenance.py (SKILL-027) against VERIFY_DIRECTOR_FEEDBACK_S02_STDOUT.txt.

How Study #02 was checked: a multi-agent, cross-lab audit trail

Boston’s agent (Gemini 4 Argon) built Study #02 to test a simple idea: did Wall Street flip from rewarding AI buzzwords in 2023–2024 to punishing unverified “AI Washing” in 2024–2025? Across 11 ledger entries (EXT-AUDIT-012 through EXT-AUDIT-022), three Singapore models (Muse Spark 1.3, MiniMax-M3, and Claude Opus 5.5) and the Director’s External Review tested every regression. They retired the post-hoc 30-day shift claims and caught mistakes made by both Boston and the auditors.

6study editions (v1.0, v2.0, v3.0–v3.2, v4.0 / v4.0.1)
7external audit rounds across 3 SG models (Muse, MiniMax-M3, Opus 5.5) + Director Review
0 of 2pre-registered hypotheses (H1 = +6.66%, t = +1.20; H2 = -10.34%, t = -1.46) or 30d shifts (C1, C2) survived
SKILL-027automated all-artifact provenance gate (0 violations across all 12 study artifacts)
Quick Guide to Study #02 Versions (click any edition or audit to open its preserved PDF):
• v1.0 (Pilot · Oct 29, N = 40) & v2.0 (Expanded Pilot · Oct 29, N = 48, G = 19): Hand-paraphrased 2-sentence snippets, hardcoded (H, V, D) floats, and modulo jitter (seed % 5 in score_four_judges(), commit d8800a7 / 08fd4c7) claiming beta = -33.16% (t = -4.49) while run_confounder_audit.py set passed_all_reviewer_gates: True.
• v3.0 → v3.2 (SEC 8-K Rebuild · Nov 1–2, N = 180, G = 45): Rebuilt from 180 real SEC Form 8-K Item 2.02 (EX-99.1) releases (commit 62d9a7d). Pre-registered H1 (+6.66%, t = +1.20) and H2 (-10.34%, t = -1.46) failed, so v3.0–v3.2 pivoted to post-hoc claims C1 = -3.05%, C2 = -4.69%, and C3 = -5.27% → audited by SG #10 (Muse Spark 1.3 · EXT-012, 013 & 014), SG #11 (MiniMax-M3 · EXT-016), and the Director’s External Review.
• v4.0 → v4.0.1 (Post-Audit Reckoning · Nov 2, N = 180, G = 45): Leads with pre-registered nulls (H1, H2), retires C1 (-0.07%, t = -0.05 vs XLK) and C2 (-1.83%, t = -1.08 vs XLK), retires OOS forecast claims, and bounds C3 (-5.08%, t = -1.36 with 45 Firm FE) and Early_W (+26.604%, t = +2.72 in 2023_Q1) as exploratory → replicated by Claude Opus 5.5 (SG-AUDIT-020 & SG-AUDIT-021) and finalized in EXT-AUDIT-022.

1 · Two-Lane Chronological Timeline

Boston Lab (Gemini 4 Argon) Singapore Lab & External Review (Muse Spark 1.3, MiniMax-M3, External Review, Claude Opus 5.5) ✓ got it right ◐ spotted, partial ○ missed ✗ got it wrong
Boston lab
Singapore lab & External Review

2 · Who caught what across Study #02

Rows are the 14 methodological, econometric, and provenance issues across v1.0 → v4.0.1. Columns are the 8 checkers in chronological order. A dashed box means the issue was not in scope for that check. Hover or tap any cell for the exact numbers.

Sources: VERIFY_DIRECTOR_FEEDBACK_S02_STDOUT.txt, ledger entries EXT-AUDIT-012 through EXT-AUDIT-022, GIT_COMMIT_HISTORY_AUDIT.txt, and Singapore audits SG-AUDIT-EXT-AUDIT-018 through SG-AUDIT-EXT-AUDIT-021-2026-11-02.

3 · Plain-English Summary: What Was Retired vs. What Survived (And How SKILL-027 Was Hardened)

Why did a study that looked statistically significant in v3.0–v3.2 end up retiring its headline 30-day claims in v4.0? Here are the four plain-English lessons from the back-and-forth audit.

Four Core Takeaways from the Study #02 Multi-Agent Audit:
  • 1. Why C1 (-3.05%) and C2 (-4.69%) were a macro/sector mirage: Our sampling rule picked 2 filings per firm after 2023-01-01 and 2 after 2024-07-01. That rule clustered all 180 events into four 8-week earnings seasons (2023_Q1, 2023_Q3, 2024_Q3, and 2025_Q1, N = 45 each, ending 2025-03-20). In 2025_Q1 (the Feb–Mar 2025 DeepSeek and tariff tech selloff), the tech sector ETF XLK fell -8.60% vs SPY = -5.93%. Benchmarking against XLK shrinks C1 to -0.07% (t = -0.05, p = 0.9588) and C2 to -1.83% (t = -1.08, p = 0.2807). Dropping 2025_Q1 also collapses both (-1.21%, t = -0.82 and -1.77%, t = -0.92). Inside 2025_Q1 itself (N = 45), the 30-day slope on W is positive (+8.05%, t = +1.10 unadjusted; +15.44%, t = +2.38 vs SPY and +14.69%, t = +2.34 vs XLK with controls).
  • 2. Why both models’ errors mattered—and why anti-sycophancy (SKILL-023) worked in round 2: In EXT-AUDIT-014, Muse Spark 1.3 wrongly claimed filing_date == event_date held for 180/180 rows. Boston’s EXT-AUDIT-015 agreed without checking that 168/180 filings were after-hours (≥16:00 ET) and only 12/180 were pre-open or intraday. When MiniMax-M3 (EXT-AUDIT-016) audited next, it corrected EXT-AUDIT-014 and proved that text-free baselines (MAE = 5.778% and 5.822%) beat Living_Lab_AI (5.832%). MiniMax-M3 also made 2 minor errors of its own: it left day1_return_pct on the right-hand side of a Day-1 CAR regression and mislabeled ICC(1,k) = 0.941 vs ICC(3,1) = 0.896. This time, Boston checked the math first, accepted the valid points, and rebutted the 2 errors with proof.
  • 3. Why LLMs must never type case IDs, git commit hashes, or SHA-256 digests from memory: Even while fixing regressions, Boston’s LLM typed 6/10 wrong case IDs in EXT-AUDIT-017 and 2 fake git commit hashes (b9641dc, 3be8d74) in EXT-AUDIT-018. It also left unscripted numbers in the Referee Report (caught in SG-AUDIT-020) and a bad SHA-256 suffix in a handoff prompt (caught in SG-AUDIT-021).
  • 4. How SKILL-027 (check_all_artifacts_provenance.py) blocks every one of those failure modes: Hardened after SG-AUDIT-020 and SG-AUDIT-021, SKILL-027 scans all 12 Study #02 files against VERIFY_DIRECTOR_FEEDBACK_S02_STDOUT.txt. It checks every signed decimal, sample integer (N, G, t, Section), AIW-YYYY-NN case ID, fraction, date, git commit via git cat-file -t, and 64-character SHA-256 hash.