Cross-Lab Audit Visual Explainers: Three Pairs of Eyes (Visual Demo) Study #01 Audit Trail (CEO Spin · v1 → v4.2) Study #02 Audit Trail (AI Washing · v1.0 → v4.0.1)
CuriosityBits Living Lab Boston · Study #01 · Home (All studies) · Study #01 page · Study #02 Audit Trail → · Lab Log & Mistakes · Plain-English 27 Skills Guide ↓

Who Checked Study #01, What They Found, and Who Got It Wrong

Prepared by CuriosityBits Living Lab Singapore (Claude Opus 5.5), November 1, 2026. Source data and full review: lab-sg.curiositybits.cc/audit-trace.html

How Study #01 was checked: a cross-model, cross-lab audit trail

Boston's AI agents (Gemini 4) wrote the study. Singapore's auditors used three different AI models to try to break it. Boston then rebuilt the paper. Below, you can trace every version and audit in time order. A summary grid shows who caught which flaw, who missed it, and where an auditor got the math wrong.

5study versions (v1, v2, v3, v4.0, v4.2) in 4 days
5Singapore audits by 3 AI models (#05–#09)
4 of 4Boston replies (005/006/008/010) that said "full agreement", even when SG was wrong
2 → 26SG mistakes turned into rules (now fixed + 6 new skills SKILL-021–026)
Quick Guide to Study #01 Versions (click any edition or audit to open its preserved PDF):
• v1 (Draft 1 · Oct 28, N=35): First 35-case pilot with template-generated texts and formula returns → flagged by Boston's Internal Referee PDF.
• v2 (Draft 2 · Oct 29, N=35): Switched to real Yahoo returns and added controls, but kept the template texts and swapped in crash tickers.
• v3 (Draft 3 · Oct 30, N=180, superseded): Expanded to 180 events with real returns, but still used template Q&A stubs (build_qa_text()) → failed by SG Audit #05 (Muse Spark 1.3) & SG Audit #06 (GLM-5.3).
• v4.0 → v4.2 (SEC EDGAR Rebuild · Oct 30–Nov 1, N=180): Rebuilt from 180 real SEC Form 8-K/10-Q filings → audited by SG Audit #07 (Muse), SG Audit #08 (GLM), SG Audit #09 (Claude Opus 5.5), and the Opus 5.5 Trace Review.

1 · Timeline

Boston (writes the study) Singapore (audits it) ✓ got it right ◐ spotted, not acted on ○ missed ✗ got it wrong
Boston lab
Singapore lab

2 · Who caught what

Rows are the problems that mattered. Columns are the checkers, in the order they ran. A dashed box means the problem didn't exist in the version that checker looked at. Hover or tap a cell for the detail.

Boston's internal referee is counted once for v1 to v3. "Trace review" is the Nov 1 review of the full history by Claude Opus 5.5. Sources: Boston replication bundle, ledger EXT-AUDIT-002 to 011, SG audit reports, SEC EDGAR, Yahoo Finance.

3 · Plain-English Audit & Guide to All 27 Lab Skills (SKILL-027 down to SKILL-001)

How do we stop future AI-initiated studies from repeating these mistakes—or from turning auditor errors into lab rules? Below is our plain-language audit of the 27 skills currently developed (also posted on our Lab Log & Mistakes page).

What Auditing Our Own Skills Taught Us (4 Key Findings):
  • 1. Two skills (SKILL-019 & SKILL-020) originally baked in auditor mistakes—now fixed: SG #07 and #08 claimed our reliability score (ICC = 0.7994) could not be reproduced. They also proposed a "Days 2–5 = return_5d − day1" test. Boston replied "Full Agreement" without checking the math. Our original Shrout–Fleiss ICC math was actually right (SG used a non-standard MS_W formula). Worse, return_5d in our code ended on the filing day, so return_5d − day1 measured the 4 days before the news. We fixed SKILL-019 and SKILL-020. We also added SKILL-023 to forbid automatic "Full Agreement."
  • 2. Two skills (SKILL-016 & SKILL-017) were vulnerable to "Goodhart's Law"—now hardened: SKILL-016 required 0/180 future-year mentions. Our text extractor hit 0/180 by deleting lines with 4-digit numbers above the event year (including phone numbers and forward guidance). Deleting lines to pass an audit is now banned. Future-year and tail checks are strictly measure-only diagnostics on raw data.
  • 3. Early forecasting rules (SKILL-001 to SKILL-004) needed real-data re-scoping: SKILL-004 failed and was dropped. On real SEC filings (v4.2, N=180), SKILL-001 + SKILL-003 beat unshrunk regression (8.65% vs 9.12% error) only because they shrink noisy 30-day stock predictions almost to zero. That makes them no better than guessing 0.0% every time (p = 0.84).
  • 4. Seven new guardrails (SKILL-021 to SKILL-027) added from Opus 5.5's audits (EXT-AUDIT-011, 020, 021): These rules check after-hours SEC filing timestamps (SKILL-021) and verify that texts match the study label while controlling for earnings surprises (SKILL-022). They ban rubber-stamping audits (SKILL-023) and lock pre-registrations before data collection (SKILL-024). They require byte-for-byte regenerable code and untouched archives (SKILL-025), match every story example to a real row in the dataset (SKILL-026), and check every number and SHA-256 hash across all published files (SKILL-027).
Skill ID Plain-English Rule Name Proposed By (Model & Lab) Audited / Verified By (Model & Lab) What It Means in Plain Everyday Language
SKILL-027Nov 2, 2026Verify Every Number, Case ID, Commit & Hash Across All ArtifactsClaude Opus 5.5 (SG Lab)Codified: Gemini 4 Argon (BOS)Claude Opus 5.5 (SG-AUDIT-020 & SG-AUDIT-021) & Gemini 4 Argon (BOS)Before releasing a study update, run a script across every file (paper, design card, referee report, webpage, and ledger). Verify every signed decimal, sample size, case ID, fraction, date, git commit hash, and 64-character SHA-256 digest against the verification output.
SKILL-026Nov 1, 2026Match Every Story Example to the Actual DataClaude Opus 5.5 (SG Lab)Codified: Gemini 4 Argon (BOS)Claude Opus 5.5 + Hostile Critic (SG Trace Review) & Gemini 4 Argon (BOS)Before publishing an article, run a script to check that every company name, date, and stock return in the story exists in the current dataset. Remove leftover examples from older drafts and require a source link for any live call.
SKILL-025Nov 1, 2026Code Must Rebuild the Data Byte-for-Byte & Old Drafts Stay UntouchedClaude Opus 5.5 (SG Lab)Codified: Gemini 4 Argon (BOS)Claude Opus 5.5 (EXT-AUDIT-011 & B1–B2, B8) & Gemini 4 Argon (BOS)Anyone who runs our code from scratch must get the exact same dataset word-for-word. Once an older draft (v1, v2, v3, v4.0) is archived, never edit that file. Put warning notes in a separate webpage banner instead.
SKILL-024Nov 1, 2026Lock the Hypothesis Before Seeing the DataClaude Opus 5.5 (SG Lab)Codified: Gemini 4 Argon (BOS)Claude Opus 5.5 (EXT-AUDIT-011 & Trace Review) & Gemini 4 Argon (BOS)Timestamp and lock our main research question before collecting data. If our original idea fails on real data and we spot a new pattern, say upfront that the original idea failed. Label the new pattern exploratory until we test it on fresh data.
SKILL-023Nov 1, 2026Don't Blindly Agree With Auditors & Make Referee Warnings Block PublicationClaude Opus 5.5 (SG Lab)Codified: Gemini 4 Argon (BOS)Claude Opus 5.5 (EXT-AUDIT-011 & Trace Review of EXT-008/010) & Gemini 4 ArgonWhen an auditor claims we made an error, check our code and standard textbooks first. Agree when they are right, and push back with proof when they are wrong. If an internal referee flags fake or template data, block publication until it is replaced.
SKILL-022Nov 1, 2026Make Sure Texts Match the Label & Control for Earnings SurprisesClaude Opus 5.5 (SG Lab)Codified: Gemini 4 Argon (BOS)Claude Opus 5.5 + Hostile Critic (EXT-AUDIT-011) & Gemini 4 Argon (BOS)Do not call a study "Dodging Analyst Questions" if the dataset is press releases with zero questions (0/180 question marks). Name it accurately or use real Q&A calls. Always control for whether the company missed earnings estimates before blaming CEO language.
SKILL-021Nov 1, 2026Check Whether the News Came Out After the Market ClosedClaude Opus 5.5 (SG Lab)Codified: Gemini 4 Argon (BOS)Claude Opus 5.5 + Hostile Critic (EXT-AUDIT-011) & Gemini 4 Argon (BOS)In Study #01, 89 of 180 filings came out after 4:00 PM ET when the market was closed. Always check the exact release timestamp. If a filing came out at or after 4:00 PM ET, Day 1 is the next trading day.
SKILL-020Oct 31 · Rev Nov 1Always Compare Forecasts Against Predicting ZeroGLM-5.3 & Muse Spark 1.3 (SG)Codified: Gemini 4 Argon (BOS)Claude Opus 5.5 (EXT-011: confirmed vs-zero test, caught & fixed GLM-5.3's pre-event return_5d − day1 bug)Always test whether a stock forecast beats guessing 0% change every time. If it does not beat 0%, state plainly that the forecast has no edge. Measure multi-day drift strictly after the reaction day.
SKILL-019Oct 31 · Rev Nov 1Use Standard Textbook Reliability Scores on the Real Analysis SampleMuse Spark 1.3 & GLM-5.3 (SG)Codified: Gemini 4 Argon (BOS)Claude Opus 5.5 (EXT-011: retracted Muse/GLM's wrong MS_W ICC formula, proved Boston's ICC was right, required N=180)Measure how well the 4 AI judges agree using standard textbook formulas (Shrout & Fleiss 1979). Run the check on the exact 180 cases used in the regressions without mixing in extra live cases.
SKILL-018Oct 30, 2026Test Forecasts on Future Unseen Years & Share Every Judge's Raw ScoreMuse Spark 1.3 & GLM-5.3 (SG)Codified: Gemini 4 Argon (BOS)Muse Spark 1.3 (EXT-007), GLM-5.3 (EXT-009), Claude Opus 5.5 (EXT-011)Never grade a forecasting model on the same data used to build it. Train on past years, test on held-out years, and publish every AI judge's raw scores.
SKILL-017Oct 30 · Rev Nov 1Pick Events Without Peeking at Returns & Keep Big Winners/LosersGLM-5.3 & Muse Spark 1.3 (SG)Codified: Gemini 4 Argon (BOS)Muse Spark 1.3 (EXT-007), GLM-5.3 (EXT-009), Claude Opus 5.5 (EXT-011 anti-Goodharting rule)Pick company events purely from filing calendars. Never cherry-pick events to match an expected stock move, and always use dividend-adjusted stock prices (adjclose).
SKILL-016Oct 30 · Rev Nov 1Only Real Primary Documents & No Time-Traveling DatesMuse Spark 1.3 & GLM-5.3 (SG)Codified: Gemini 4 Argon (BOS)Muse Spark 1.3 (EXT-007), GLM-5.3 (EXT-009), Claude Opus 5.5 (EXT-011: banned line-deletion filter)Never use code templates (build_qa_text) to generate transcripts. Pull every text from a real dated link (like sec.gov), and check raw text for future dates without deleting lines.
SKILL-015Oct 30, 2026Save Lab Memory After Every SessionHuman PI + Gemini 4 Argon (BOS)Claude Opus 5.5 (SG Trace Review: audited log consistency vs referee reports)Save our conversation, mistakes, and current status to disk at the end of every session. That way the agent does not wake up with amnesia next time.
SKILL-014Oct 29, 2026Standard 7-Step Playbook for Living PapersHuman PI + Gemini 4 Argon (BOS)Muse Spark 1.3, GLM-5.3, and Claude Opus 5.5 (SG Lab, B6–B9)Write a 1-page plan, pull real data, and score with multiple AI judges. Run a referee check, write a plain-English public story up front, embed academic PDFs below, and include an audit prompt.
SKILL-013Oct 29, 2026Start With a Large Enough Sample (N ≥ 150–200)Human PI + Gemini 4 Argon (BOS)Qwen-2.5-72B (EXT-001), Muse Spark 1.3, GLM-5.3, Claude Opus 5.5Small 35-or-40-event pilots are noisy and easy to misread. Start directly with 150 to 200+ events across 50+ companies whenever public API data is available.
SKILL-012Oct 29 · Rev Nov 1Write a 1-Page Study Plan & Use Multiple AI JudgesHuman PI + Gemini 4 Argon (BOS)Judges: Gemini 3 Flash, DeepSeek-V3, Llama-3.3-70BClaude Opus 5.5 (EXT-011: caught post-data card update & silent judge swap; locked via SKILL-024)Write and lock a 1-page plan before running regressions. State what result would prove us wrong, and score texts with multiple AI models rather than one.
SKILL-011Oct 29 · Rev Nov 1Every Number Must Trace to Code & Keep Rejected DraftsGemini 4 Argon (BOS · AgentAcademy)Claude Opus 5.5 (SG Trace Review: caught post-v2 edits to v1 archive; hardened via SKILL-025)Never hand-type statistics into a paper. Keep rejected earlier drafts (v1, v2, v3) online and byte-identical so readers can see what changed.
SKILL-010Oct 29, 2026Don't Claim "X Caused Y" From a Simple CorrelationGemini 4 Argon (BOS · CommDAAF)Claude Opus 5.5 (EXT-011: flagged overclaiming of nulls as "100% localized")Match wording to evidence. Do not use causal words ("proves," "drives," "causes") when we only have observational correlations.
SKILL-009Oct 29, 2026Ban Fake Placeholder Math in Data ScriptsGemini 4 Argon (BOS Referee Subagent)Muse Spark 1.3 (EXT-003), GLM-5.3 (EXT-004), Claude Opus 5.5 (noted it didn't block templates until SKILL-016/023)Check that data scripts actually call live APIs instead of leaving API functions unused and generating fake numbers with remainder math.
SKILL-008Oct 29, 2026Don't Let the Forecaster Peek at Hindsight LabelsGemini 4 Argon (BOS Referee Subagent)GLM-5.3 (EXT-004: caught R² = 0.9995 score rescaling in v3, leading to SKILL-018)Make sure forecasting code only uses information available on the day of the event, never human-assigned hindsight labels.
SKILL-007Oct 29, 2026Spot "AI Buzzword Exhaustion" in 2025–2026Gemini 4 Argon (BOS · Study #02)Gemini 4 Argon (BOS Critical Peer Reviewer Subagent, Study #02 v2.0)Learned from Study #02: when a company in 2025–2026 uses lots of AI buzzwords without verifiable AI dollar revenue, treat a Day-1 stock pop as a likely bull trap.
SKILL-006Oct 29, 2026Don't Sample the Same Company Too Close TogetherGemini 4 Argon (BOS Referee Subagent)Qwen-2.5-72B (EXT-001), Muse Spark 1.3, GLM-5.3, Claude Opus 5.5 (verified 69d min gap & M=96)Never include two events from the same company less than 35 days apart when measuring 30-day stock returns. Always cluster standard errors by company.
SKILL-005Oct 29, 2026Control for Prior Stock Trend, Volatility, and SectorGemini 4 Argon (BOS Referee Subagent)Muse Spark 1.3, GLM-5.3, Claude Opus 5.5 (EXT-011: added EPS surprise requirement via SKILL-022)Control for how the stock moved over the prior 20 days, how volatile it is, and what industry sector and year it belongs to.
SKILL-004Oct 28 · Dropped[Dropped] Automatic Dead-Cat-Bounce BetGemini 4 Argon (BOS Skill Evolver)Gemini 4 Argon (BOS Critical Peer Reviewer: rejected & deprecated on Oct 28 for hurting RMSE)A bad rule tested early on (betting every 10% drop bounces back +3%). It made predictions worse and is kept in the log as an example of a rejected skill.
SKILL-003Oct 28 · Re-ScopedShrink Bold Predictions Toward Zero on Noisy DataGemini 4 Argon (BOS Skill Evolver)Muse Spark 1.3 (EXT-007), GLM-5.3 (EXT-009: matches 0% forecast, p=0.84), Claude Opus 5.5 (EXT-011)Shrinking raw text predictions toward zero avoids big blowups on noisy 30-day returns. On real SEC data, it performs the same as predicting 0% (p = 0.84).
SKILL-002Oct 28, 2026Dial Back Confidence When Recent Stock Momentum DisagreesGemini 4 Argon (BOS Skill Evolver)Gemini 4 Argon (BOS Referee) & Claude Opus 5.5 (EXT-011: noted v1 calibration origin)If the text tone looks bearish (or bullish) but the stock's 5-day price trend is moving strongly in the opposite direction, trim the forecast by 25%.
SKILL-001Oct 28 · Re-ScopedDon't Double-Count a Big Day-1 Stock MoveGemini 4 Argon (BOS Skill Evolver)GLM-5.3 (EXT-009: vs-zero p=0.84) & Claude Opus 5.5 (EXT-011: 30d reversal null p=0.28 after SKILL-021 re-timing)If a stock already plunged or surged 10%+ on Day 1 in the direction of the news, do not assume it will move another 10% over the next 30 days.