My Claude Code Setup

Actively maintained. A summary of how I use Claude Code for academic work — slides, papers, data analysis, and more — packaged so you can fork it for your own research. See CHANGELOG.md for the latest changes.

Live site: psantanna.com/claude-code-my-workflow

A ready-to-fork foundation for AI-assisted academic work. You describe what you want — lecture slides, a research paper, a data analysis, a replication package — and Claude plans the approach, runs specialized agents, fixes issues, verifies quality, and presents results. Like a contractor who handles the entire job. Extracted from a production PhD course and extended by a growing community.

Quick Start (5–10 minutes, plus ~30 min for first-time installs)

Before you start: Claude Code + git are the minimum. To run the included HelloWorld demos end-to-end you also need XeLaTeX (Beamer sample) and Quarto (Quarto sample). R and the GitHub CLI are recommended. Python 3 runs the gate suite (./scripts/backtest.sh — 10 checkers) and the quality scorer, and is pre-installed on macOS/Linux. Full list in Prerequisites below. Fastest path: clone first, then run ./scripts/validate-setup.sh — it reports exactly what's missing with install links.

Only need Python/R/markdown? You don't need XeLaTeX or Quarto. The agents, rules, skills, and orchestration patterns work for any text/code artifact. Skip the HelloWorld demos and head straight to /data-analysis, /review-paper, /lit-review, or /review-r.

Session 2 onwards: MEMORY.md (committed) collects generic [LEARN] entries that help all forkers; machine-specific notes accumulate in Claude Code's native auto memory (~/.claude/projects/<project>/memory/, machine-local, never committed). See .claude/rules/meta-governance.md for the distinction.

1. Fork & Clone

# Fork this repo on GitHub (click "Fork" on the repo page), then:
git clone https://github.com/YOUR_USERNAME/claude-code-my-workflow.git my-project
cd my-project
./scripts/validate-setup.sh        # reports missing tools with install links

Replace YOUR_USERNAME with your GitHub username.

2. Start Claude Code and Paste This Prompt

claude

Using VS Code? Open the Claude Code panel instead. Everything works the same — see the full guide for details.

Avoid prompt fatigue. New interactive sessions on Pro/Max/Team start in auto mode (classifier-gated — most actions run, risky ones prompt); on plans and providers without auto, Normal mode prompts per risky tool call. If you still see too many prompts, toggle Auto-accept edits mode (a keybinding; see the permission modes section of the guide) or run claude --permission-mode acceptEdits. For fully-autonomous runs on a trusted repo, Bypass mode skips prompts entirely. The template's .claude/settings.json ships defaultMode: bypassPermissions with broad catch-all allows (Bash(*), Edit(**), Write(**) — 7 wildcard rules, not a curated list), so out of the box almost nothing prompts. That is a deliberate power-user default: to tighten it, set defaultMode: "default" in .claude/settings.json and approve tools as you go, or remove the override to fall back to the platform's auto mode.

Then paste the starter prompt from the guide, filling in your project details:

I am starting to work on [PROJECT NAME] in this repo. [Describe your project in 2–3 sentences.] I've set up the Claude Code academic workflow... Please read the configuration files and adapt them for my project. Enter plan mode and start.

The full guide has the complete starter prompt with all the details.

What this does: Claude reads all the configuration files, fills in your project name, institution, and preferences, then enters contractor mode — planning, implementing, and (within the skill you invoke) running the review + verify loop. You approve the plan, invoke a skill, and the skill handles the rest within its scope.

Heavily adapting CLAUDE.md for a non-academic project? Anthropic's built-in /init command will re-derive a CLAUDE.md from your codebase as a starting point. The pre-shipped CLAUDE.md in this template already covers the academic setup — you only need /init if your fork diverges substantially (e.g., a Python/ML project that doesn't use LaTeX or Quarto).

3. Verify Your Setup

Before building real lectures, confirm your environment works:

./scripts/validate-setup.sh        # Checks XeLaTeX, Quarto, Python, git, etc.

Then inside Claude:

/compile-latex HelloWorld          # Compiles Slides/HelloWorld.tex to PDF
/deploy HelloWorld                 # Renders Quarto/HelloWorld.qmd to HTML

If both succeed, delete Slides/HelloWorld.tex and Quarto/HelloWorld.qmd and start on your real work.

How It Works

Goal-first, gate-enforced (the v2.0 shift)

You don't craft a perfect prompt — you state a goal and let the work loop toward it under gates. Specialist agents do the labor; enforcing gates decide when it's good enough; you adjudicate the disagreements they surface. Three things make that trustworthy:

Real gates, not reminders. One command — ./scripts/backtest.sh — runs ten gates: surface-sync, skill integrity, model currency against the SSoT, link and anchor resolution, Agent Skills spec conformance, staleness (including source-vs-published divergence), repo hygiene, derived counts (enumerable claims re-counted from disk), ledger coverage (the qualification ledger and the checks that actually run must agree in both directions, and every hook declared in settings must exist and be invocable — a one-character path typo no longer disables a hook in silence), and a seeded hook battery (every active guard hook is re-fired against the failure it targets, alongside clean controls, on every run). A version-controlled pre-commit hook (run ./scripts/install-hooks.sh once) runs it plus the quality check (≥80) on every commit — bypassing the skill no longer bypasses the review. A git-guardrails hook blocks destructive git (reset --hard, clean -f, push --force, add -A) and refuses a merge, rebase, or pull while the tree is dirty — reading the tree as it is rather than predicting what a chained command might do to it, so git stash && git merge is denied too and you run the two steps separately (ALLOW_DIRTY_MERGE=1 if you mean it); like its sibling it is a textual check over the command line, so an op carried inside an interpreter, an alias, or a script is outside what it can see, and its docstring says which forms those are. Its sibling root-of-trust-guard denies the common shell write paths into the files that define the gates themselves (.claude/settings*.json, .claude/hooks/, .githooks/) — redirection, tee, cp/mv/rm, in-place sed — and, because it already unwraps bash -c and env -S payloads for those rules, it hands the unwrapped payload to the same destructive-git deny list, so bash -c 'git reset --hard' and bash -c 'git clean -fdx' no longer fall between the two hooks. Read that one as a tripwire, not a lock, and read the name as a filename rather than a claim: it is a best-effort textual scan that fails open on its own errors, and the files it watches stay replaceable through channels the template deliberately allows — an Edit/Write/MultiEdit, a branch switch, a clean merge, or a bug in the hook itself. What it buys is a change of channel — a change to a gate arrives as a reviewable diff instead of an invisible overwrite — not a guarantee that the gates cannot be disabled. Nothing here recovers anything either: the transcript and git reflog are an audit trail and a commit-history aid, and neither holds the bytes of an uncommitted edit or an untracked file. The review runtime re-checks any reviewer-introduced "fatal" finding before it counts.
Every gate is qualified, and the ledger is itself a gate. Each one has been shown a planted defect and confirmed to go red, with recall and false-alarm rate recorded in quality_reports/qualification/LEDGER.md — and that ledger is now load-bearing rather than aspirational: a registered check with no row there fails the build, and a row naming a checker that no longer exists fails too. Checks that have not been qualified are listed there by name as visible debt — because an unqualified check is not weak evidence, it is none. Run /vaccinate to qualify one.
A real orchestration runtime. Reviews fan out to forked specialist agents, reduce over a shared finding schema, judge with a hallucination gate, and loop until dry — see orchestrator-protocol.md.
Ground truth as a process. A mismatch isn't always a failure: a defensible, named alternative is recorded as EXPLAINED and carried into your response-to-referees, while genuine errors stay fail-closed.

This is not an autonomous daemon — the loop is always you- or skill-initiated, and you stay the auditor. Scheduled automation handles recurring chores and notifies only on findings — cloud Routines for committed-repo checks (weekly lit-delta, inbox triage), Desktop scheduled tasks for anything touching local data (the nightly reproducibility check's home).

Contractor Mode

You describe a task. For complex or ambiguous requests, Claude first creates a requirements specification with MUST/SHOULD/MAY priorities and clarity status (CLEAR/ASSUMED/BLOCKED). You approve the spec, then Claude plans the approach and invokes the right skill (e.g. /create-lecture, /qa-quarto, /review-paper --adversarial). That skill implements the orchestrator runtime internally — implement, verify, review, fix, re-verify, score — and returns a summary when the work meets quality standards. Say "just do it" and it runs the full loop; commits still require an explicit /commit (which the pre-commit hook then gates).

Specialized Agents

Instead of one general-purpose reviewer, 18 focused agents each check one dimension. A representative sample:

proofreader — grammar/typos
slide-auditor — visual layout
pedagogy-reviewer — teaching quality
r-reviewer — R code quality
domain-reviewer — field-specific correctness, slides (template — customize for your field)
domain-referee / methods-referee / editor — manuscript peer-review pipeline (/review-paper --peer)

Each is better at its narrow task than a generalist would be. The /slide-excellence skill runs the slide-review agents in parallel; /review-paper --peer runs the paper-review pipeline. The same pattern extends to any academic artifact — manuscripts, data pipelines, proposals.

Adversarial QA

Two agents work in opposition: the critic reads both Beamer and Quarto and produces harsh findings. The fixer implements exactly what the critic found. They loop until dry — converging when a round surfaces no new issue (a 5-round cap is the fallback, not the primary stop). This catches errors that single-pass review misses.

Quality Review

Every artifact gets a score (0–100). Scores below threshold halt the workflow and surface the findings — the user decides whether to fix or explicitly override:

80 — commit threshold
90 — PR threshold
95 — excellence (aspirational)

Framing honesty: Thresholds are advisory at the harness level — the /commit skill runs quality checks and halts on failure. And as of v2.0, running ./scripts/install-hooks.sh once installs a real pre-commit hook (.githooks/pre-commit) that runs the full backtest gate suite plus the quality (≥80) gate on every commit, so bypassing the skill no longer bypasses the review. Opt out per-commit with SKIP_QUALITY_GATE=1 or git commit --no-verify.

Context Survival

Plans, specifications, and session logs survive auto-compression and session boundaries. The PreCompact hook saves a context snapshot before Claude's auto-compression triggers, ensuring critical decisions are never lost. MEMORY.md accumulates learning across sessions, so patterns discovered in one session inform future work.

For forced compression (long pipelines, mid-plan handoffs), /compress-session (v1.9.0) distils the conversation into a structured note — decisions, next actions, and discarded-as-noise — instead of letting auto-compaction truncate. /promote-memory (v1.9.0) periodically harvests generic learnings from native auto memory to committed MEMORY.md via a five-critic council.

Verification Discipline (v1.7.0+)

Multiple complementary verification layers run before submission:

/verify-claims (v1.7.0) — Chain-of-Verification with a forked verifier that cannot self-confirm because it has never seen the draft. v1.9.0 adds HIGH/MED/LOW-WARN severity tiers; HIGH-WARN findings (fabricated citation, numerical contradiction) are must-fix — resolve them before committing.
/audit-reproducibility (v1.7.0; Stata coverage v1.9.0) — every numeric claim in the manuscript is cross-checked against the script output that produced it. v1.9.0 adds passport.yaml — a per-paper YAML state file with PASS/FAIL/STALE/UNVERIFIED status per claim.
/humanize (v1.9.0) — detect AI-voice tells (boilerplate transitions, hedging stacking, sycophancy) before submission. Read-only by design; auto-rewriting degrades quality.
/review-paper --variance N (v1.9.0) — runs N referees with sampled dispositions and reports a decision distribution, not a point estimate. Motivated by AgentReview (EMNLP 2024) finding 37% of decisions vary purely from disposition sampling.

The Guide

For a comprehensive walkthrough, read the full guide (or see the source).

It covers:

Why This Workflow Exists — the problem and the vision
Getting Started — fork, paste one prompt, and Claude sets up the rest
The System in Action — specialized agents, adversarial QA, quality scoring
The Building Blocks — CLAUDE.md, rules, skills, agents, hooks, memory
Workflow Patterns — slides, research, reproducibility, presentation rhetoric, sequential adversarial audits, and more
The Ecosystem — extensions by clo-author, claudeblattman, MixtapeTools, autoresearch, ClaudeCodeTools, and a growing community
Customizing for Your Domain — creating your own reviewers and knowledge bases

2026 Features

The guide covers Claude Code's latest capabilities:

Model lineup — Fable 5 (claude-fable-5, opt-in via /model fable or the best alias) is the top tier for long-horizon work. Current Opus/Sonnet point versions and the provider-dependent alias table live in the single source of truth, model-versions.md — surfaces here stay tier-abstract so they cannot go stale, and the staleness gate fails the build when the SSoT's own expiry passes.
Effort levels — /effort sets cost vs. thoroughness (low / medium / high / xhigh / max). Fable 5 defaults to high (per the model SSoT); set effort explicitly on other tiers — reserve xhigh for extended exploration and ultracode (xhigh + dynamic workflows) for the largest autonomous runs.
/goal <verifiable condition> (v1.9.0; Anthropic May 2026) — keep working across turns until a fast model confirms the condition holds. Pairs with /commit quality gates for verified-end-state runs.
claude agents dashboard (v1.9.0; Anthropic May 2026) — single screen for parallel review work (/review-paper --peer, /slide-excellence).
Cost-Conscious Composition — prompt-cache TTL (5-min default on API keys; 1-hour automatic on Claude subscriptions), 70/20/10 model routing (Haiku/Sonnet/Opus), /cost + /usage monitoring, Agent SDK credit-pool split (2026-06-15).
Skill frontmatter — effort, context: fork, agent, hooks, disable-model-invocation (v1.8.0+), disallowed-tools (the actual tool restriction — allowed-tools only pre-approves), paths (glob-scoped auto-activation), and dynamic content ($ARGUMENTS, !command syntax)
Permission modes — Normal, Auto-accept, Plan, Auto (classifier-gated; since 2026-08-14 the default starting mode for new interactive sessions on Pro, Max, and Team, and available on Bedrock / Google Cloud / Foundry without an opt-in flag), Bypass
Hook handler types — command, prompt, and HTTP handlers with 20+ hook events; hooks see effort.level and $CLAUDE_EFFORT (Apr 2026 Week 19)
Advanced agent configuration — model, maxTurns, isolation, tool restrictions; model-routing.md rule codifies per-agent tier (v1.9.0)
Worktree base ref (v1.9.0; Anthropic Apr 2026) — worktree.baseRef setting controls fresh (default; remote default-branch) vs head (local HEAD) for new worktrees
Built-in skills — /fewer-permission-prompts, /team-onboarding, /autofix-pr, /powerup, Ultraplan, /loop (self-pacing)
Plugins — /discover-plugins for third-party extensions

Use Cases

Academic Task	How This Workflow Helps
Lecture slides (Beamer/Quarto)	Full creation, translation, multi-agent review, deployment
Research papers	Literature review, manuscript review, simulated peer review (`/review-paper --peer [journal]`), reviewer-disposition variance reporting (`--variance N`)
Data analysis	End-to-end R pipelines (`/data-analysis`) or Stata pipelines via `stata-mcp` (`/stata-replication`, v1.9.0), replication verification, publication-ready output
Monte Carlo simulations	Reproducible simulation studies (`/simulation-study`, v1.10.0) — parameterized DGP, estimator grid, bias/RMSE/coverage/size/power with Monte Carlo SEs, dedicated `sim-reviewer` review pass
Package development	R package release gate (`/r-package-check`, v1.10.0) — `devtools::document()` + tests + `R CMD check --as-cran` + CRAN-policy triage + `r-package-reviewer` (Stata / Python checks on the roadmap)
Replication packages	AEA-compliant packaging, reproducibility audit trails, `passport.yaml` claims provenance (v1.9.0)
Presentations	Rhetoric of decks principles, visual audit, cognitive load review
Research proposals	Structured drafting with adversarial critique
Preregistration	OSF / AsPredicted / AEA RCT Registry-ready document (`/preregister --style`) — full workflow in Pattern 16
Manuscript submission discipline	`/humanize` (detect AI voice), `/verify-claims` HIGH-WARN gate (block fabricated citations), reviewer-disposition variance

Disciplines preloaded: Economics (top-5 journal profiles, R conventions) and Political Science (APSR / AJPS / JOP profiles, formal-theory + survey-experiment paper types, conjoint/cjoint conventions). Forkers extend for psych / sociology / public-health via journal profiles + paper types + discipline cards.

One repo, many project types

This workflow is designed as a single hub for an entire research program — not one paper at a time. The same CLAUDE.md, rules, agents, and quality gates serve courses and lectures, papers and referee reports, data analysis and replication packages, Monte Carlo simulation studies (/simulation-study + sim-reviewer), and the R package release gate (/r-package-check + r-package-reviewer) — all new in v1.10.0. On the roadmap: Stata / Python package checks (SSC / PyPI) and personal-productivity workflows. See .claude/references/v2.0-backlog.md for what's next.

What's Included

<details> <summary><strong>18 agents, 60 skills, 37 rules, 8 hooks</strong> (click to expand)</summary>

Agents (`.claude/agents/`)

Agent	What It Does
`proofreader`	Grammar, typos, overflow, consistency review
`slide-auditor`	Visual layout audit (overflow, font consistency, spacing)
`pedagogy-reviewer`	13-pattern pedagogical review (narrative arc, notation density, pacing)
`r-reviewer`	R code quality, reproducibility, and domain correctness
`tikz-reviewer`	Merciless TikZ diagram visual critique
`beamer-translator`	Beamer-to-Quarto translation specialist
`quarto-critic`	Adversarial QA comparing Quarto against Beamer benchmark
`quarto-fixer`	Implements fixes from the critic agent
`verifier`	End-to-end task completion verification
`domain-reviewer`	Template for your field-specific substance reviewer
`claim-verifier` (v1.7.0)	Chain-of-Verification fact-checker in a forked context
`editor` (v1.5.0)	Journal editor for `/review-paper --peer` (desk review + referee selection + synthesis)
`domain-referee` (v1.5.0)	Disposition-primed substance referee for `--peer` mode
`methods-referee` (v1.5.0+)	Paper-type-aware methodology referee (6 paper types)
`humanize-auditor` (v1.9.0)	Read-only AI-voice auditor invoked by `/humanize`
`promote-memory-council` (v1.9.0)	Five-critic council for `[LEARN]` promotion to MEMORY.md
`sim-reviewer` (v1.10.0)	Monte Carlo simulation reviewer — DGP/estimand match, Monte Carlo SE, coverage-vs-truth, claims↔tables parity
`r-package-reviewer` (v1.10.0)	R package-source reviewer — DESCRIPTION/NAMESPACE hygiene, roxygen completeness, testthat coverage, CRAN-policy red flags

Skills (`.claude/skills/`)

Skill	What It Does
`/compile-latex`	3-pass XeLaTeX compilation with bibtex
`/deploy`	Render Quarto + sync to GitHub Pages
`/extract-tikz`	TikZ diagrams to PDF to SVG pipeline
`/proofread`	Launch proofreader on a file
`/visual-audit`	Launch slide-auditor on a file
`/pedagogy-review`	Launch pedagogy-reviewer on a file
`/review-r`	Launch R code reviewer
`/qa-quarto`	Adversarial critic-fixer loop (loops until dry; 5-round cap is a fallback)
`/slide-excellence`	Combined multi-agent review
`/translate-to-quarto`	Full Beamer-to-Quarto translation — Phase 0 pre-flight plus 11 translation phases
`/vaccinate`	Measure whether a check, gate, or AI reviewer actually detects the failure it targets — seeds defects + a clean control, reports recall and false-positive rate into a qualification ledger
`/adjudicate-review`	Turn incoming findings — AI review, referee report, linter, second model — into verified fixes. Every finding is a CANDIDATE until checked against the source
`/blast-radius`	Before and after changing anything shared (return value, schema, default, units), enumerate every consumer and actually run them
`/credible-claims`	Research brief before delegating, claim record after. Keeps faster execution from being mistaken for credible evidence
`/differential-audit`	Compare two implementati

…

vaccinate

name: vaccinate description: Qualify a check before it is allowed to clear anything — prove it can detect the failure it is meant to catch. Seeds known defects into a copy of a real artifact plus a clean control, runs the checker, and reports recall and false-positive rate into a qualification ledger. Use when the user says "does this check work", "qualify the gate", "test my reviewer", "seed defects", "vaccinate", "qualify the checks", "can I trust this review", "does it pass for the right reason", or before relying on any automated check or referee simulation for a decision that matters. NOT a code fixer and NOT a reviewer itself — it grades the grader. argument-hint: "[checker or skill to qualify] [artifact to seed]" allowed-tools: ["Read", "Write", "Bash", "Glob", "Grep", "Agent"] disable-model-invocation: true metadata: protocol: check-qualification

Vaccinate — grade the grader

Twenty bugs were once planted in a working codebase and the review agents were asked to check it again. They reported everything was fine. Recall: 0/20.

A vaccine is a small, controlled dose of error that strengthens the whole system. This skill administers one.

The rule it enforces: an unqualified check is not weak evidence — it is none.

When to run it

Before a referee simulation, reproducibility gate, or review agent is used to make a decision that matters (a submission, a release, a deposit).
After changing a checker — a modified gate is unqualified until re-measured.
On a schedule for gates that guard load-bearing claims. Detection decays as artifacts drift.

Protocol

1. Name the failure

State the defect class the check is supposed to catch. "Catches problems" is not a class. "Detects a coefficient in the text that no longer matches its table" is.

2. Build the seeded set + a clean control

Work on a copy, never the live artifact. Produce:

N seeded variants, one defect each, drawn from references/defect-library.md.
At least one clean control — an unmodified copy.

The control is not optional. Without it you measure recall and call it accuracy.

Verify each seed actually violates something. A seed that the artifact already permits creates no defect, and the checker correctly reporting "pass" will look like a broken gate. This is the most common way a qualification run produces a false alarm about itself.

3. Run the checker blind

Run the check or agent against each variant in a fresh context, one variant per run. It must not know which variant it has, how many defects exist, or that a qualification is underway. For an AI reviewer, spawn via the Agent tool with context: fork.

4. Score

Metric	Definition
Recall	seeded defects correctly identified / seeded defects planted
False-positive rate	findings on the clean control that are factually false / total findings on the control
Localization	did it name the right location, or just report unease?
Baseline delta	recall of a simpler alternative (a grep, a diff, a one-line assertion)

A finding on the clean control counts as a false positive only when it is factually wrong — not merely unwelcome. A reviewer prompted to find gaps will report some in sound work; that is expected behaviour, not a failure.

The baseline is load-bearing. A five-agent panel that scores no better than grep -n has not earned its cost.

5. Write the ledger row

Append to quality_reports/qualification/LEDGER.md:

| date | target | artifact | defect classes | N | recall | FPR | baseline | verdict |

Verdicts: PASS (detects its named class at an agreed threshold) · FAIL (misses it) · BLOCKED (could not be run — say why; do not record as PASS).

6. Act on the result

FAIL → the check does not license its claim. Fix the check or stop citing it. Do not weaken the seed until it passes.
PASS → record the threshold. A PASS at one difficulty is not a PASS at another.
Either way, a checker with no ledger row is unqualified, and its green light means nothing.

Worked example

/vaccinate check-model-versions.sh

Failure class: "a superseded model presented as current".
Seed: append The newest model is Opus 4.8 and it is the default. to README.md. Control: unmodified README.md.
Run: bash scripts/check-model-versions.sh; echo $?
Score: seeded → exit 1 (detected). Control → exit 0 (no false alarm). Recall 1/1, FPR 0/0. Baseline: grep -c "Opus 4.8" README.md also detects — so the gate's value is its allow-marker logic, not raw detection.
Ledger: PASS.
Restore the artifact and re-run to confirm you are back to green.

Anti-patterns

Seeding into an artifact that already permits the seed — measures nothing, looks like a broken gate.
Telling the reviewer it is a test — it will look harder than it does in production.
Counting any finding as a hit — a finding at the wrong location is not detection.
One seed, one run — a single trial does not distinguish detection from luck. Use ≥2 replicates per class where cost allows.
Weakening the seed until it passes — that is fitting the test to the checker.
Skipping the clean control — the most common omission, and it hides the cost.

Reference files

File	Read when
`references/defect-library.md`	choosing what to seed — defect classes by artifact type
`evals/README.md`	the complementary question: does the skill produce better output than not having it?

Doctrine: what qualification means

Do not assume more machinery is better

A second model, more agents, or a longer debate is not presumed to verify better. Before an elaborate procedure earns extra weight, show it outperforms a simpler check on the same prespecified seeded failures and valid cases, reporting both detection and false alarms. Complexity that has not beaten a baseline is cost, not assurance.

Treat AI verdicts as predictions, not facts

When a model grades, triages, or reviews at scale:

keep a sampled set for qualified human review, and record how it was sampled (retain coverage of hard subgroups — do not sample only the easy middle);
keep fitting/prompt-tuning cases separate from evaluation cases;
report where AI and expert judgments diverge;
remember a well-calibrated average score certifies no individual verdict;
agreement between models is not independent evidence — they share failure modes and converge on the same wrong answer at a meaningful rate.

Any material change to the model, prompt, rubric, or target population requires fresh human labels and recalibration.

Requalify after material change

A check qualified against an old interface, schema, or scale may silently stop testing anything. Re-run the seeded-defect proof after material changes to the object under test or to the check itself.

Distinguish qualified checks from scientific judgments

Qualified checks have a defensible reference answer: unique keys, units convert, a table regenerates, an estimator recovers an analytic special case, a seeded fault triggers a failure. These can be automated and rerun forever.
Scientific judgments — whether a field measures the intended construct, whether an identifying assumption is plausible, whether a result deserves causal language — cannot be automated, and no volume of qualified checks substitutes for one.

Confirm the check actually ran

A missing, substituted, or degraded check is missing evidence, not a pass. Verify the run happened (log, exit status, artifact timestamp — not an assumption); that it ran on the current object, not a cached one; that nothing was skipped, filtered, or swallowed into a default; and that the tolerance was fixed before the comparison. A tolerance loosened after a failed comparison converts evidence into decoration. If it must be loosened, record it as an approved divergence with a reason.

Cross-references

verification-ladder.md — rung 0; why this comes before everything
Merged with the former /qualify-checks (2026-08-21): same goal — one skill, not two
external-oracle-process.md — qualifying an external referee

Claude Code My Workflow

Skill 详情