AhaLab

AhaLab is an OpenAI Build Week Education project that turns a student's plausible wrong answer into a diagnostic question and a prediction-first interactive counterexample. It does not treat one wrong answer as proof of a misconception. Instead, it gathers another piece of evidence, makes the learner commit to a prediction, and records an auditable Understanding Trace.

Production: https://ahalab.vercel.app

The public deployment includes an explicitly labelled Demo Mode that runs without an API key. It uses synthetic cached AI artifacts, but still executes the real schemas, guarded state machine, deterministic calculations, renderers, transfer scoring, and session recovery.

The problem

Many AI tutoring flows explain the correct procedure immediately. That can fix an answer while leaving the learner's underlying model unchanged. For example, a learner who writes 1/3 + 1/4 = 2/7 may be applying a coherent but incorrect rule: add every visible number. AhaLab delays explanation until it has discriminated among competing hypotheses and made the contradiction observable.

Learning workflow

Every supported journey follows the same forward-only sequence:

1. Infer: interpret the submitted question, answer, visible work, and uncertainty. Propose two or three tentative hypotheses, including a possible non-conceptual slip when plausible.
2. Discriminate: ask exactly one question selected to separate the leading hypotheses. A hypothesis remains tentative rather than being presented as confirmed.
3. Predict: compile a constrained lab specification and require the learner to commit to an outcome before the result can be revealed.
4. Contradict: render an exact visual counterexample. Fraction quantities or motion states are computed locally rather than accepted from model prose.
5. Explain: ask the learner to revise their explanation after observing the result.
6. Transfer: score a structurally related problem with deterministic code and assemble the Understanding Trace.

The resulting trace separates the initial work, corrected interpretation, model hypotheses, diagnostic evidence, prediction, deterministic observation, revised explanation, transfer result, and remaining uncertainty.

Supported scope

V1 intentionally supports four curated misconception families:

Domain	Supported family	Interactive evidence
Fractions	Adding unlike denominators directly	Exact quantity bars compare the student's rule with the common-denominator sum
Fractions	Assuming a larger denominator means a larger fraction	Equal-whole bars and exact comparisons expose piece size
Physics	Believing force is required to maintain constant motion	A one-dimensional, zero-net-force simulation preserves velocity
Physics	Believing heavier objects fall faster	Two idealized masses share the same gravitational acceleration without air resistance

The build does not claim arbitrary-subject tutoring. It has no generated JavaScript, classroom management, authentication, database, vector store, or student grading model.

Architecture

Student submission
  -> server-only AI interpretation
  -> student confirmation/correction
  -> server-only diagnostic hypotheses + one discriminator
  -> constrained LabSpec proposal
  -> Zod schema validation
  -> fraction or motion domain validation
  -> deterministic calculation
  -> fixed React renderer
  -> guarded learning-state transition
  -> Understanding Trace

Area	Responsibility
src/contracts	Strict, versioned TypeScript and Zod contracts for submissions, AI artifacts, LabSpecs, results, and traces
src/flow	Pure forward-only reducer; rejects skipped, mismatched, stale, or invalid transitions
src/server/ai	OpenAI Responses API integration, Fireworks-compatible test path, structured-output parsing, retry, provenance, and safe fallback behavior
src/domain/fraction	Exact rational arithmetic and validation of every supported fraction claim
src/domain/motion	SI-unit, one-dimensional motion calculations and deterministic force-arrow data
src/labs	Fixed accessible React/SVG renderers; no model-generated layout or executable code
src/features/journey	Student workflow, fixture mode, recovery, and Understanding Trace UI
evals/ai	Synthetic, labelled fraction evaluation cases and an auditable validation runner
tests/e2e	Desktop and mobile browser journeys, recovery, accessibility, and renderer checks

The frozen contract and transition details are documented in docs/contracts.md.

GPT-5.6 responsibilities

The OpenAI path uses the official JavaScript SDK and Responses API:

Stage	Model and effort	Allowed responsibility
Submission interpretation	gpt-5.6, medium	Transcribe typed or handwritten work, recognize the supported problem, preserve visible reasoning, and report uncertainty
Diagnosis	gpt-5.6-sol, high	Return two or three competing working hypotheses, evidence for and against each, calibrated confidence, and exactly one discriminating question
Lab proposal	gpt-5.6-sol, medium; one high-effort retry after schema or domain failure	Select one supported LabSpec plus bounded reflection and transfer materials

OpenAI response continuity is used only where the validated diagnostic response ID is available. The diagnostic call stores a text-only response for that continuation; submission and lab calls use store: false. Raw reasoning and reasoning summaries are not requested or shown. Fireworks completion IDs are provenance rather than continuity handles, so that path resends the minimum validated text context.

Deterministic safety boundaries

GPT-5.6 may interpret evidence and propose bounded pedagogical content. It is not trusted to calculate, grade, simulate, choose renderer layout, or execute code.

• Every external and model-produced object passes a strict Zod schema.
• A discriminated LabSpec union allows only the four renderer families above.
• Fraction results use exact rational arithmetic, not floating-point approximations.
• Motion results use explicit SI units and tested equations for idealized one-dimensional motion.
• Domain validators reject contradictory operands, claims, units, assumptions, ranges, and transfer items.
• Prediction, observation, reflection, and transfer must traverse the guarded state machine in order.
• Observations and transfer scores are created with origin: "deterministic" and are recomputed when the Understanding Trace is validated.
• An invalid model draft is discarded. Lab generation retries once at high effort, then uses a separately labelled curated specification when that fallback is supported.

Local setup

Requirements:

• Node.js 24.x
• npm 11.x (package.json records npm 11.9.0)

After cloning the repository, run these commands from its root:

npm ci
cp .env.example .env.local
npm run dev

PowerShell equivalent for the environment file:

Copy-Item .env.example .env.local
npm run dev

Open http://localhost:3000. No API key is required for Demo Mode.

Environment variables

Keep secrets only in .env.local or encrypted deployment settings. Never use a NEXT_PUBLIC_ prefix for an AI key.

Variable	Required	Purpose
OPENAI_API_KEY	Optional	Enables the judged GPT-5.6 live path
FIREWORKS_API_KEY	Optional	Enables the Fireworks-compatible live testing path
AI_PROVIDER	Optional	openai or fireworks; when omitted, OpenAI is preferred if both keys exist
FIREWORKS_MODEL	Optional	Account-scoped Fireworks model or router ID; defaults to accounts/fireworks/models/kimi-k2p6

Restart the development server after changing provider variables. Without a configured provider, Live AI returns a recoverable configuration error and preserves the learner's draft; Demo Mode remains fully usable.

Sample data

Demo Mode provides four synthetic journeys directly in the interface. For a manual handwriting upload, use tests/fixtures/handwritten-fraction.png. Its expected transcription and synthetic provenance are documented in tests/fixtures/README.md. The image contains no real student work or embedded personal metadata.

Tests

Run the baseline verification suite:

npm run lint
npm run typecheck
npm test
npm run build

Or run the same baseline as one command:

npm run verify

Install the browser once, then run the desktop Chromium and Pixel 7 projects:

npx playwright install chromium
npm run test:e2e

Run the synthetic fraction evaluation:

npm run eval:fraction

Live golden paths are opt-in because they use provider quota. Configure one provider, start from a fresh server, and run:

LIVE_AI=1 npx playwright test tests/e2e/live-golden.spec.ts

PowerShell:

$env:LIVE_AI="1"
npx playwright test tests/e2e/live-golden.spec.ts

Set PLAYWRIGHT_BASE_URL=https://ahalab.vercel.app to run browser tests against production without starting a local server.

Evaluation

The repository contains 16 synthetic, non-personal fraction cases: eight unlike-denominator additions and eight denominator-magnitude comparisons. They cover explicit misconceptions, arithmetic slips, ambiguous evidence, correct answers with flawed explanations, correct reasoning, and four unsupported inputs.

The checked-in runner reports schema validity, deterministic lab validation, supported-case completion, unsupported-case rejection, and whether the teacher-tagged family appears in the leading two hypotheses. It deliberately reports no learning-effectiveness or belief-change metric.

The current 100% run uses teacher-derived reference predictions. Those predictions are intentionally derived from the same fixtures, so the result verifies wiring and regression behavior only; it is not GPT-5.6 performance and is not evidence that AhaLab improves learning. Labels are single-author synthetic review labels rather than independently adjudicated ground truth.

During implementation, the evaluation cases were used to make ambiguity and arithmetic slips explicit in the diagnostic contract, require one discriminator covering the leading hypotheses, reject unsupported inputs before rendering, and add a regression test that fails any generated numerical claim bypassing deterministic validation. No prompt-quality improvement or medium-versus-high model advantage is claimed from the leaked reference run. The implemented effort policy is tested as a reliability rule: high for diagnosis, medium for routine LabSpec generation, and one high-effort retry after validation failure.

More detail is available in evals/ai/README.md.

How Codex was used

Codex accelerated the project by turning the fixed product scope into typed contracts, domain engines, renderers, API routes, fixtures, browser journeys, and deployment checks within the build-week schedule. It was also used to generate adversarial schema and domain cases, run repeated math and physics audits, inspect responsive states with Playwright, trace failures across the state machine, and keep setup and release checks executable rather than narrative-only.

The important decisions were human-reviewed before or during implementation:

• limit V1 to four misconception families;
• ask a discriminator before presenting an intervention;
• require prediction before reveal;
• compile declarative LabSpecs instead of executing generated code;
• keep all numerical observations and transfer scoring deterministic;
• keep hypotheses tentative and show uncertainty in the Understanding Trace;
• provide an offline deterministic Demo Mode for judges;
• avoid authentication, databases, classrooms, and arbitrary-subject claims;
• process handwriting in memory and exclude raw images from recovery state.

Codex proposed and implemented within those boundaries; human review retained responsibility for product scope, pedagogical wording, trust boundaries, privacy decisions, and release acceptance.

Privacy behavior

• JPEG and PNG uploads are limited to 4 MiB and checked by declared type and file signature.
• Handwriting bytes are processed in memory and are not written to application storage.
• Raw images, filenames, model payloads, student content, and provider errors are not logged.
• The diagnosis stage receives validated text context, not the original image.
• Same-tab recovery uses versioned sessionStorage and excludes raw images, filenames, keys, and model payloads.
• Demo and evaluation data are synthetic and explicitly labelled.
• API errors are recoverable structured responses and do not expose upstream details.

This prototype is not a compliance certification. A deployment serving children under 13, or the applicable age of digital consent, requires organization-level retention controls, consent and safeguarding review, and compliance with the provider's under-18 requirements. Application-level store: false is not a substitute for those deployment controls.

Limitations

• Only the four listed misconception families are supported.
• Physics labs are idealized, one-dimensional, use SI units, and exclude air resistance.
• The synthetic evaluation set is small and single-author; it does not establish educational effectiveness or general model accuracy.
• Live AI depends on provider availability, credentials, quota, and latency.
• Demo Mode uses curated cached AI artifacts and therefore does not measure live model behavior.
• Recovery is same-tab sessionStorage; there is no account, cross-device history, or durable student record.
• The project has not been tested as a production service for minors or sensitive educational records.

License

Released under the MIT License.