Forensic Psychometrics & Peer Benchmarking

1. Overview & Purpose

Individual telemetry (a single blur event, a single fast answer) is weak evidence on its own. Forensic Psychometrics strengthens that evidence by comparing every candidate against their own cohort on the same quiz, same question slot — turning raw timing and response data into statistically grounded anomalies.

This runs automatically as part of the existing quizaccess_simpleproctor\task\evaluate_attempt ad-hoc task (the same task that produces the 🟢/🟡/🔴 verdict on the review screen). No separate setup is required — it activates as soon as a quiz has more than a handful of finished attempts to benchmark against.

2. Cohort-Relative Timing Z-Scores

For every question slot in a quiz, the evaluator queries all finished attempts and computes the dwell time (first step timestamp to last step timestamp) per student per slot.

mean    = average dwell time across all finished attempts for that slot
stddev  = sample standard deviation across those attempts
z_score = (this candidate's dwell time − mean) ÷ stddev

Requires more than 2 finished attempts for a slot before a z-score is computed (small cohorts are too noisy to benchmark).
A z_score is attached to every response sent to the LLM evaluator, alongside the raw time_sec.
The AI prompt explicitly instructs the model: “Z-Score < -2.0 indicates suspiciously fast compared to peers.”

Why relative, not absolute? A fixed “under 8 seconds is suspicious” rule breaks down across question types and difficulty. A z-score adapts per-question automatically — a trivial recall MCQ and a dense multi-step calculation each get their own baseline.

3. Distractor Collusion Fingerprinting

For multiple-choice questions, the evaluator also tallies how often each exact response summary was chosen across the cohort for that slot.

Only triggers when more than 3 total responses exist for the slot (avoids false positives on small cohorts).
If a candidate’s incorrect answer was chosen by less than 15% of their peers, it is tagged as a collusion_marker, e.g.:

“Obscure distractor chosen (Selected by only 8.3% of cohort)”
This does not by itself mean the student cheated — some genuinely tricky distractors are rare simply because they’re well-designed. It is a signal fed to the LLM, not an automatic penalty, and is intended to help markers notice patterns (e.g. the same rare distractor shared by several papers submitted close together in time).

4. Where This Surfaces for Markers

These two signals (z_score, collusion_marker) are injected directly into the evaluation prompt sent to the local Ollama model (llama3.2:1b / llama3.2:3b) alongside blur counts, snapshot counts, and response text. The model’s resulting JSON verdict (status, confidence score, flagged anomalies) is what actually appears as the 🟢/🟡/🔴 AI Integrity & Telemetry Evaluation card on:

https://examnode.com.au/mod/quiz/accessrule/simpleproctor/review.php?attempt=<id>

The raw z-scores and collusion percentages are not currently rendered as separate numbers on the review page — they are consumed by the LLM and summarised in plain language in the verdict card and flagged-anomalies list. If you need the raw numbers for a formal academic integrity case, they are visible in the evaluate_attempt task’s log output (mtrace) and can be queried directly from mdl_quizaccess_simpleproctor_evals.flags.

5. Current Limitations & Roadmap

Cold-start problem: The very first attempts on a freshly provisioned quiz have no cohort to compare against — z-scores will be null until enough finished attempts accumulate. Consider this when auditing the first few submissions of a new exam.
No raw numeric display yet: Markers currently only see the LLM’s narrative summary, not the underlying z-score/percentage. A future iteration could add a compact stats table to review.php for formal integrity hearings that need the raw figures.
Per-slot only: Z-scores compare timing on the same question across students, not a student’s overall exam pace. A candidate who is consistently fast across all questions (rather than one outlier question) is not yet specifically flagged as a distinct pattern.