How accurate is the AI?

If you are a teacher about to put a paper in front of a class, read it through first. We would rather say so here than have you find out during the lesson.

Once the questions exist, they go through the following passes in order. These rewrite items, and delete some outright:

• Visual-stem consistency. Where a question carries a chart or figure, the two are checked against each other. A mismatch means the visual is removed and the stem rewritten so the item still works without it.
• Renderability. Charts that cannot be drawn correctly — a reversed axis, a bar chart containing a negative value, a function plotted in the wrong variable — are dropped rather than drawn wrongly.
• Distractor audit. Each wrong option is checked for whether anyone would actually pick it. Options that are obviously wrong, duplicated, or quietly defensible get rewritten.
• Answer-length balance. Models habitually write the correct option long and complete and the distractors short and blunt, so “pick the longest” beats chance. The bias is measured, and the distractors are rewritten to match the answer’s length and shape — never the other way round.
• The correctness gate. Last, and the one that matters most: the whole paper is solved again independently and compared against the marked answers.

Each item comes back with one of five verdicts. Five rather than a yes/no, because they call for different actions:

• ok — the marked answer is the only right one. Kept.
• fix — there is a single right answer but it was not the one marked. Corrected automatically.
• none — no option is right. Not salvageable, so the item is deleted.
• ambiguous — more than one option is defensible. Also deleted, because such an item measures nothing.
• unsure — the audit could not decide. This is not the same as “no correct answer”, so the item is kept.

“unsure” and “none” are separate because they once were not. An earlier version returned the same value for both, so “this item has no correct answer” was read as “unsure” and kept, and broken items shipped. It is the most instructive bug this project has had; the five explicit verdicts exist so the two can never collapse again.

Deleted items are regenerated, so a request for ten questions returns ten — deletions do not shrink your paper.

Generation runs against a time budget. In an earlier version, when the shaping passes ran long the correctness gate was simply skipped — dropping the one pass that should never be dropped. The budget is now split in two, and the gate owns the second half outright; no earlier pass can eat into it however long it takes.

The limits are more useful than the capabilities. These are what the audits miss:

• Out-of-syllabus items: correct in themselves, but testing something the curriculum does not cover.
• Curriculum lag: after a syllabus change, the model may still reflect the old one.
• Faulty reasoning in an explanation: the answer is right but the working skips or misstates a step.
• Both passes wrong together: the re-solve uses the same model family, so it can repeat the same misconception. This is the deepest limit — independent re-solving is not independent of the model itself.
• Errors in the source you pasted: the AI faithfully writes questions from what you gave it, mistakes included.

Send the question — a screenshot or the URL — to willson114478@gmail.com. Reported items feed back into the prompts and the audit rules, and they are the most useful improvements available, precisely because they point at what the automated checks did not see.

See also: Terms §6, AI-generated content