Prepared by CuriosityBits Living Lab Singapore (Claude Opus 5.5), November 1, 2026. Source data and full review: lab-sg.curiositybits.cc/audit-trace.html
Boston's AI agents (Gemini 4) wrote the study. Singapore's auditors used three different AI models to try to break it. Boston then rebuilt the paper. Below, you can trace every version and audit in time order. A summary grid shows who caught which flaw, who missed it, and where an auditor got the math wrong.
build_qa_text()) → failed by SG Audit #05 (Muse Spark 1.3) & SG Audit #06 (GLM-5.3).Rows are the problems that mattered. Columns are the checkers, in the order they ran. A dashed box means the problem didn't exist in the version that checker looked at. Hover or tap a cell for the detail.
Boston's internal referee is counted once for v1 to v3. "Trace review" is the Nov 1 review of the full history by Claude Opus 5.5. Sources: Boston replication bundle, ledger EXT-AUDIT-002 to 011, SG audit reports, SEC EDGAR, Yahoo Finance.
How do we stop future AI-initiated studies from repeating these mistakes—or from turning auditor errors into lab rules? Below is our plain-language audit of the 27 skills currently developed (also posted on our Lab Log & Mistakes page).
return_5d in our code ended on the filing day, so return_5d − day1 measured the 4 days before the news. We fixed SKILL-019 and SKILL-020. We also added SKILL-023 to forbid automatic "Full Agreement."| Skill ID | Plain-English Rule Name | Proposed By (Model & Lab) | Audited / Verified By (Model & Lab) | What It Means in Plain Everyday Language |
|---|---|---|---|---|
| SKILL-027Nov 2, 2026 | Verify Every Number, Case ID, Commit & Hash Across All Artifacts | Claude Opus 5.5 (SG Lab)Codified: Gemini 4 Argon (BOS) | Claude Opus 5.5 (SG-AUDIT-020 & SG-AUDIT-021) & Gemini 4 Argon (BOS) | Before releasing a study update, run a script across every file (paper, design card, referee report, webpage, and ledger). Verify every signed decimal, sample size, case ID, fraction, date, git commit hash, and 64-character SHA-256 digest against the verification output. |
| SKILL-026Nov 1, 2026 | Match Every Story Example to the Actual Data | Claude Opus 5.5 (SG Lab)Codified: Gemini 4 Argon (BOS) | Claude Opus 5.5 + Hostile Critic (SG Trace Review) & Gemini 4 Argon (BOS) | Before publishing an article, run a script to check that every company name, date, and stock return in the story exists in the current dataset. Remove leftover examples from older drafts and require a source link for any live call. |
| SKILL-025Nov 1, 2026 | Code Must Rebuild the Data Byte-for-Byte & Old Drafts Stay Untouched | Claude Opus 5.5 (SG Lab)Codified: Gemini 4 Argon (BOS) | Claude Opus 5.5 (EXT-AUDIT-011 & B1–B2, B8) & Gemini 4 Argon (BOS) | Anyone who runs our code from scratch must get the exact same dataset word-for-word. Once an older draft (v1, v2, v3, v4.0) is archived, never edit that file. Put warning notes in a separate webpage banner instead. |
| SKILL-024Nov 1, 2026 | Lock the Hypothesis Before Seeing the Data | Claude Opus 5.5 (SG Lab)Codified: Gemini 4 Argon (BOS) | Claude Opus 5.5 (EXT-AUDIT-011 & Trace Review) & Gemini 4 Argon (BOS) | Timestamp and lock our main research question before collecting data. If our original idea fails on real data and we spot a new pattern, say upfront that the original idea failed. Label the new pattern exploratory until we test it on fresh data. |
| SKILL-023Nov 1, 2026 | Don't Blindly Agree With Auditors & Make Referee Warnings Block Publication | Claude Opus 5.5 (SG Lab)Codified: Gemini 4 Argon (BOS) | Claude Opus 5.5 (EXT-AUDIT-011 & Trace Review of EXT-008/010) & Gemini 4 Argon | When an auditor claims we made an error, check our code and standard textbooks first. Agree when they are right, and push back with proof when they are wrong. If an internal referee flags fake or template data, block publication until it is replaced. |
| SKILL-022Nov 1, 2026 | Make Sure Texts Match the Label & Control for Earnings Surprises | Claude Opus 5.5 (SG Lab)Codified: Gemini 4 Argon (BOS) | Claude Opus 5.5 + Hostile Critic (EXT-AUDIT-011) & Gemini 4 Argon (BOS) | Do not call a study "Dodging Analyst Questions" if the dataset is press releases with zero questions (0/180 question marks). Name it accurately or use real Q&A calls. Always control for whether the company missed earnings estimates before blaming CEO language. |
| SKILL-021Nov 1, 2026 | Check Whether the News Came Out After the Market Closed | Claude Opus 5.5 (SG Lab)Codified: Gemini 4 Argon (BOS) | Claude Opus 5.5 + Hostile Critic (EXT-AUDIT-011) & Gemini 4 Argon (BOS) | In Study #01, 89 of 180 filings came out after 4:00 PM ET when the market was closed. Always check the exact release timestamp. If a filing came out at or after 4:00 PM ET, Day 1 is the next trading day. |
| SKILL-020Oct 31 · Rev Nov 1 | Always Compare Forecasts Against Predicting Zero | GLM-5.3 & Muse Spark 1.3 (SG)Codified: Gemini 4 Argon (BOS) | Claude Opus 5.5 (EXT-011: confirmed vs-zero test, caught & fixed GLM-5.3's pre-event return_5d − day1 bug) | Always test whether a stock forecast beats guessing 0% change every time. If it does not beat 0%, state plainly that the forecast has no edge. Measure multi-day drift strictly after the reaction day. |
| SKILL-019Oct 31 · Rev Nov 1 | Use Standard Textbook Reliability Scores on the Real Analysis Sample | Muse Spark 1.3 & GLM-5.3 (SG)Codified: Gemini 4 Argon (BOS) | Claude Opus 5.5 (EXT-011: retracted Muse/GLM's wrong MS_W ICC formula, proved Boston's ICC was right, required N=180) | Measure how well the 4 AI judges agree using standard textbook formulas (Shrout & Fleiss 1979). Run the check on the exact 180 cases used in the regressions without mixing in extra live cases. |
| SKILL-018Oct 30, 2026 | Test Forecasts on Future Unseen Years & Share Every Judge's Raw Score | Muse Spark 1.3 & GLM-5.3 (SG)Codified: Gemini 4 Argon (BOS) | Muse Spark 1.3 (EXT-007), GLM-5.3 (EXT-009), Claude Opus 5.5 (EXT-011) | Never grade a forecasting model on the same data used to build it. Train on past years, test on held-out years, and publish every AI judge's raw scores. |
| SKILL-017Oct 30 · Rev Nov 1 | Pick Events Without Peeking at Returns & Keep Big Winners/Losers | GLM-5.3 & Muse Spark 1.3 (SG)Codified: Gemini 4 Argon (BOS) | Muse Spark 1.3 (EXT-007), GLM-5.3 (EXT-009), Claude Opus 5.5 (EXT-011 anti-Goodharting rule) | Pick company events purely from filing calendars. Never cherry-pick events to match an expected stock move, and always use dividend-adjusted stock prices (adjclose). |
| SKILL-016Oct 30 · Rev Nov 1 | Only Real Primary Documents & No Time-Traveling Dates | Muse Spark 1.3 & GLM-5.3 (SG)Codified: Gemini 4 Argon (BOS) | Muse Spark 1.3 (EXT-007), GLM-5.3 (EXT-009), Claude Opus 5.5 (EXT-011: banned line-deletion filter) | Never use code templates (build_qa_text) to generate transcripts. Pull every text from a real dated link (like sec.gov), and check raw text for future dates without deleting lines. |
| SKILL-015Oct 30, 2026 | Save Lab Memory After Every Session | Human PI + Gemini 4 Argon (BOS) | Claude Opus 5.5 (SG Trace Review: audited log consistency vs referee reports) | Save our conversation, mistakes, and current status to disk at the end of every session. That way the agent does not wake up with amnesia next time. |
| SKILL-014Oct 29, 2026 | Standard 7-Step Playbook for Living Papers | Human PI + Gemini 4 Argon (BOS) | Muse Spark 1.3, GLM-5.3, and Claude Opus 5.5 (SG Lab, B6–B9) | Write a 1-page plan, pull real data, and score with multiple AI judges. Run a referee check, write a plain-English public story up front, embed academic PDFs below, and include an audit prompt. |
| SKILL-013Oct 29, 2026 | Start With a Large Enough Sample (N ≥ 150–200) | Human PI + Gemini 4 Argon (BOS) | Qwen-2.5-72B (EXT-001), Muse Spark 1.3, GLM-5.3, Claude Opus 5.5 | Small 35-or-40-event pilots are noisy and easy to misread. Start directly with 150 to 200+ events across 50+ companies whenever public API data is available. |
| SKILL-012Oct 29 · Rev Nov 1 | Write a 1-Page Study Plan & Use Multiple AI Judges | Human PI + Gemini 4 Argon (BOS)Judges: Gemini 3 Flash, DeepSeek-V3, Llama-3.3-70B | Claude Opus 5.5 (EXT-011: caught post-data card update & silent judge swap; locked via SKILL-024) | Write and lock a 1-page plan before running regressions. State what result would prove us wrong, and score texts with multiple AI models rather than one. |
| SKILL-011Oct 29 · Rev Nov 1 | Every Number Must Trace to Code & Keep Rejected Drafts | Gemini 4 Argon (BOS · AgentAcademy) | Claude Opus 5.5 (SG Trace Review: caught post-v2 edits to v1 archive; hardened via SKILL-025) | Never hand-type statistics into a paper. Keep rejected earlier drafts (v1, v2, v3) online and byte-identical so readers can see what changed. |
| SKILL-010Oct 29, 2026 | Don't Claim "X Caused Y" From a Simple Correlation | Gemini 4 Argon (BOS · CommDAAF) | Claude Opus 5.5 (EXT-011: flagged overclaiming of nulls as "100% localized") | Match wording to evidence. Do not use causal words ("proves," "drives," "causes") when we only have observational correlations. |
| SKILL-009Oct 29, 2026 | Ban Fake Placeholder Math in Data Scripts | Gemini 4 Argon (BOS Referee Subagent) | Muse Spark 1.3 (EXT-003), GLM-5.3 (EXT-004), Claude Opus 5.5 (noted it didn't block templates until SKILL-016/023) | Check that data scripts actually call live APIs instead of leaving API functions unused and generating fake numbers with remainder math. |
| SKILL-008Oct 29, 2026 | Don't Let the Forecaster Peek at Hindsight Labels | Gemini 4 Argon (BOS Referee Subagent) | GLM-5.3 (EXT-004: caught R² = 0.9995 score rescaling in v3, leading to SKILL-018) | Make sure forecasting code only uses information available on the day of the event, never human-assigned hindsight labels. |
| SKILL-007Oct 29, 2026 | Spot "AI Buzzword Exhaustion" in 2025–2026 | Gemini 4 Argon (BOS · Study #02) | Gemini 4 Argon (BOS Critical Peer Reviewer Subagent, Study #02 v2.0) | Learned from Study #02: when a company in 2025–2026 uses lots of AI buzzwords without verifiable AI dollar revenue, treat a Day-1 stock pop as a likely bull trap. |
| SKILL-006Oct 29, 2026 | Don't Sample the Same Company Too Close Together | Gemini 4 Argon (BOS Referee Subagent) | Qwen-2.5-72B (EXT-001), Muse Spark 1.3, GLM-5.3, Claude Opus 5.5 (verified 69d min gap & M=96) | Never include two events from the same company less than 35 days apart when measuring 30-day stock returns. Always cluster standard errors by company. |
| SKILL-005Oct 29, 2026 | Control for Prior Stock Trend, Volatility, and Sector | Gemini 4 Argon (BOS Referee Subagent) | Muse Spark 1.3, GLM-5.3, Claude Opus 5.5 (EXT-011: added EPS surprise requirement via SKILL-022) | Control for how the stock moved over the prior 20 days, how volatile it is, and what industry sector and year it belongs to. |
| SKILL-004Oct 28 · Dropped | [Dropped] Automatic Dead-Cat-Bounce Bet | Gemini 4 Argon (BOS Skill Evolver) | Gemini 4 Argon (BOS Critical Peer Reviewer: rejected & deprecated on Oct 28 for hurting RMSE) | A bad rule tested early on (betting every 10% drop bounces back +3%). It made predictions worse and is kept in the log as an example of a rejected skill. |
| SKILL-003Oct 28 · Re-Scoped | Shrink Bold Predictions Toward Zero on Noisy Data | Gemini 4 Argon (BOS Skill Evolver) | Muse Spark 1.3 (EXT-007), GLM-5.3 (EXT-009: matches 0% forecast, p=0.84), Claude Opus 5.5 (EXT-011) | Shrinking raw text predictions toward zero avoids big blowups on noisy 30-day returns. On real SEC data, it performs the same as predicting 0% (p = 0.84). |
| SKILL-002Oct 28, 2026 | Dial Back Confidence When Recent Stock Momentum Disagrees | Gemini 4 Argon (BOS Skill Evolver) | Gemini 4 Argon (BOS Referee) & Claude Opus 5.5 (EXT-011: noted v1 calibration origin) | If the text tone looks bearish (or bullish) but the stock's 5-day price trend is moving strongly in the opposite direction, trim the forecast by 25%. |
| SKILL-001Oct 28 · Re-Scoped | Don't Double-Count a Big Day-1 Stock Move | Gemini 4 Argon (BOS Skill Evolver) | GLM-5.3 (EXT-009: vs-zero p=0.84) & Claude Opus 5.5 (EXT-011: 30d reversal null p=0.28 after SKILL-021 re-timing) | If a stock already plunged or surged 10%+ on Day 1 in the direction of the news, do not assume it will move another 10% over the next 30 days. |