Pre-Flight Exam Rigor & Security Audit CLI

1. Overview & Purpose

audit_quiz.php is a read-only diagnostic CLI tool in local_examprovision that inspects a quiz’s settings, review options, SimpleProctor configuration, and question content for common rigor and security gaps — before the exam opens to students. It is designed to run in under a second against a fully provisioned exam and give a pass/fail-style summary.

It does not modify anything. It only reads and reports.

2. Usage

docker exec moodle-exam-php-fpm-1 php /var/www/html/public/local/examprovision/cli/audit_quiz.php --quiz=<quizid>

Option	Description
`-q`, `--quiz`	Required. The numeric quiz ID (not course ID) to audit.
`-h`, `--help`	Print usage help and exit.

Find the quiz ID from the Moodle quiz settings URL (mod/quiz/view.php?id=<cmid> → Settings → the id query parameter on the resulting edit page is the quiz ID), or from the provision_exam.php output (Quiz Instance ID: X).

3. What It Checks

[1] Security Settings

Answer shuffling enabled (shuffleanswers).
Overdue handling set to auto-submit (prevents lost work / open-ended attempts).
Notes if browser security (Safe Exam Browser lockdown) is unset — expected when relying on SimpleProctor instead of SEB.

[2] Review Options (Leakage Risk)

Fails if correct answers are visible during the attempt.
Warns if correct answers are shown immediately after submission or while the quiz is still open — both create a window for answers to leak to later sitters.

[3] SimpleProctor Monitoring

Confirms quizaccess_simpleproctor is enabled for the quiz.
Confirms focus-loss / window-blur tracking specifically is turned on (it can be disabled independently of the base plugin).

[4] Question Content & AI Resilience Analysis

Tallies question types (MCQ / Essay / RecordRTC oral-video).
Flags MCQ answer-length giveaways: if the correct answer’s text is more than 2× longer than the longest distractor, it’s a well-known tell that experienced test-takers (and LLMs) exploit.
Flags quizzes with zero RecordRTC oral/video questions — an entirely written, click-based exam is far more exposed to generative-AI misuse than one with an impromptu oral bookend.

4. Example Output

==================================================
PRE-FLIGHT EXAM AUDIT
Course: ICT302 Pickup Verification Exam (ICT302)
Quiz:   ICT302 Pickup Verification Exam
==================================================

[1] Security Settings
  [WARN] Answer shuffling is DISABLED. Increases risk of collusion.
  [INFO] No browser security set (expected if using SEB/SimpleProctor).
  [OK] Overdue handling is set to auto-submit.

[2] Review Options (Leakage Risk)
  [OK] Correct answers are hidden during the attempt.
  [OK] Correct answers are hidden while the quiz is open.

[3] SimpleProctor Monitoring
  [OK] Webcam monitoring is ENABLED.
  [OK] Focus-loss / Window blur tracking is ENABLED.

[4] Question Content & AI Resilience Analysis
  Analyzing 3 questions...
  Summary: 0 MCQs, 1 Essays, 2 Oral/Video.
  [OK] Impromptu oral assessment detected. Good resilience against AI generation.

==================================================
⚠️ AUDIT COMPLETE: Found 1 potential issues to review.
==================================================

Known finding: As of this guide, quizzes created by provision_exam.php do not explicitly set shuffleanswers, so they inherit Moodle’s unshuffled default and are flagged by this exact check on every freshly provisioned exam. This is a genuine, easy fix tracked for the provisioner — set shuffleanswers => 1 in the $quizdata array in classes/service/provisioner.php — but has not been applied yet pending confirmation it won’t affect any already-authored question banks that assume fixed answer order.

5. Recommended Workflow

Provision the exam via provision_exam.php (with --oralbookends and --proctor as appropriate).
Run audit_quiz.php --quiz=<id> immediately afterwards as a standard part of exam setup.
Resolve any [FAIL] items before publishing the LTI launch URL to the parent LMS. [WARN] items are judgement calls — document the decision if you choose to leave one as-is (e.g. deliberately disabling shuffle for a paper-based equivalence requirement).
Re-run the audit after manually editing quiz settings or adding further questions, since it is a point-in-time snapshot, not a continuously enforced policy.