Generative tools compress course production from weeks to hours. But that relocates risk rather than removing it: an AI-generated course is fluent and visually complete by default, and that polish masks the errors it may contain. Surface polish is no longer evidence of substantive correctness.
I am not part of my company’s instructional design process. This audit began as a debugging task: learners were getting stuck at a point in some of our courses, and I opened the compiled SCORM package to find out why. Decoding Articulate Rise’s base64 payload turned an opaque export into readable data — and once the course was legible, and an AI agent could both reason over the corpus and drive a browser against the rendered application, the narrow question (“why do users get stuck?”) widened into a full audit of an AI-generated ISO 45001 internal-auditor course along two axes that are usually treated separately: content (what the course says) and implementation (how it behaves at runtime). This case study documents the resulting method: (1) a static extraction technique that converts the compiled package into machine-reviewable artifacts; (2) an agent-driven browser-automation procedure that inspects completion logic, the embedded AI tutor, and network and runtime behavior; (3) an evaluation framework combining an ISO 45001:2018 / ISO 19011:2018 compliance module with an adult-learning lens (andragogy, Bloom-based method-to-objective analysis, and the cognitive mechanisms of durable learning); and (4) a persistent, AI-service-agnostic architecture on the Model Context Protocol (MCP) that lets the same audit run identically across heterogeneous AI clients. The audit surfaced eleven content findings (three high-severity clause-level errors), a systemic mismatch between the course’s higher-order objectives and its recognition-level assessment, and runtime defects invisible to static review — including that the course’s embedded AI tutor, because it answers from the course’s own content, faithfully reproduces the content errors. I close by reflecting on the epistemic tension of using AI to audit AI-generated artifacts, and on why the human subject-matter expert, not the apparatus, remained decisive.
This project did not start as an audit. Learners were stalling at a point in a live course, and — as an engineer with no role in our instructional design process — I approached it the way I would approach any misbehaving web application: open the artifact, read the source, reproduce the behavior. The compiled SCORM package turned out to be decodable into a complete, readable representation of the course, and an AI agent could both reason over that corpus and drive a browser against the rendered application. At that point, the scope of what could be inspected changed. The narrow debugging question widened into a full review of everything the package contained: its claims, its configuration, and its runtime behavior.
That widening matters because of where e-learning production is heading. Generative authoring tools now compress course production from weeks to hours. This is a genuine capability gain, but it relocates rather than removes risk: the bottleneck moves from creation to verification. An AI-generated course is fluent, well-structured, and visually complete by default; these very qualities mask the errors it may contain, because surface polish is no longer evidence of substantive correctness. In regulated domains — occupational safety, maritime operations, compliance auditing — an undetected conceptual error is not a cosmetic defect but a transmission of incorrect professional practice to every learner who completes the course.
The problem is compounded by two blind spots common to e-learning reviews. First, most reviews examine content (the narrative, the questions) while ignoring implementation (how completion is scored, what the runtime does, whether embedded services behave as claimed). Second, evaluations that do address learning quality frequently invoke “pedagogy” generically, without grounding the critique in the learning sciences appropriate to the audience — here, experienced adult professionals.
This case study reports a concrete method that addresses both blind spots and packages the capability for reuse. The subject is a live artifact: an ISO 45001 internal-auditor course generated with AI tooling by subject-matter experts and instructional designers and delivered via a SCORM-compliant LMS. The contribution is fourfold — an extraction technique, a browser-automation procedure for implementation audit, an evaluation framework, and a service-agnostic tool architecture — each documented in enough detail to be reproduced. A deliberate feature of the case: the auditor is an outsider to the design process, and the audit’s most important methodological correction came from the subject-matter expert, not from the AI.
Using AI tools for content authoring and the evaluation gap: Tools such as Articulate Rise’s Concept-to-Course generate a full course from a prompt. The output is structurally sound but statistically “plausible,” and plausibility is not accuracy. Evaluation frameworks for training (e.g., Kirkpatrick’s four levels) address outcomes post-delivery; they do not provide a pre-delivery inspection of an artifact’s correctness and design integrity. This case targets the pre-delivery gap.
Standards for the domain: ISO 45001:2018 specifies requirements for occupational health-and-safety management systems, organized as clauses 4–10 (Context, Leadership & worker participation, Planning, Support, Operation, Performance evaluation, Improvement). Internal auditing methods derive from ISO 19011:2018 (Guidelines for auditing management systems). A course that trains auditors must represent both correctly.
Adult learning: The learners are working professionals, so the governing frame is andragogy (Knowles, 1980), not pedagogy: adults are self-directed, bring experience as a resource, are problem-centered, and are internally motivated. Method selection should further be conditioned on the cognitive level of each objective (Anderson & Krathwohl, 2001, the revised Bloom taxonomy): recall-level objectives tolerate direct instruction, whereas the judgment an auditor exercises (Analyze/Evaluate) requires scenario-based and experiential methods (Kolb, 1984; Merrill, 2002).
Cognitive mechanisms for permanent learning: Independent of frame and method, retention depends on encoding mechanisms: retrieval practice (the testing effect; Roediger & Karpicke, 2006), distributed practice (the spacing effect; Cepeda et al., 2006; cf. Ebbinghaus, 1885), and elaborated feedback, while extraneous cognitive load should be minimized (Sweller, 1988; Mayer, 2009). Passive progression produces a fluency illusion — the subjective feeling of having learned without the corresponding retention.
The Model Context Protocol (MCP): MCP is an open protocol that standardizes how AI clients discover and call external tools. It is the technical basis for the service-agnostic layer described in section 4.6: one tool server, many clients. (Anthropic, 2024)
The artifact is ISO 45001 Internal Auditor: Practical Skills for Safer Workplaces, a seven-section course (~230 content blocks) authored with Articulate Rise, exported as SCORM 2004 4th Edition, and delivered on our LMS. The course additionally embedded an AI tutor (an Articulate feature scoped to course content) and a graded final assessment delivered in the LMS rather than inside the package.
The audit itself began without a mandate. I opened the package to investigate a learner-progression problem, found the course could be decoded and inspected end to end, and continued until the review covered content, design, and runtime. The findings were then written up and shared with the course’s owners as a structured quality review.
I separated the audit into a content layer (static analysis of what the course asserts and how it is configured) and an implementation layer (dynamic analysis of how the compiled course behaves in a browser). The two layers see different defects: static analysis finds conceptual and structural errors; dynamic analysis finds runtime, completion-logic, and integration errors that no amount of reading the source can reveal.
Rise compiles the entire course into a single base64-encoded JSON payload passed to a deserialize() call inside index.html. Rather than review the rendered course lesson by lesson — slow and subject to the course’s own sequential locking — I decoded this payload directly. A small, dependency-free Python extractor locates the payload, base64-decodes it, and emits three artifacts:
course.json (the full object graph);content.txt (every lesson and block, with headings, prose, media references, interaction items, and quiz options flattened for review);summary.md inventory: completion/SCORM configuration, block-type census, media counts, and a knowledge-check table recording, per item, the correct-answer count and whether feedback text is present.This converts an opaque package into a corpus that can be reasoned over systematically and re-audited deterministically. Crucially, the summary exposes configuration facts — for example that completion was set to report on 90% of content viewed with no quiz gating (completeWith: “reporting”) — that are invisible when reading the narrative.
Static extraction cannot answer questions such as Does the completion logic actually behave as configured? Does the embedded AI tutor function, and how is it framed to the learner? What does the runtime request, and does anything fail? These require executing the course. I therefore drove a headless Chromium instance under AI-agent control (via the Chrome DevTools Protocol; equivalently deliverable as a browser extension or a CLI browser tool) and performed a scripted inspection:
file:// (the export is configured to run outside an LMS for local inspection) and confirm initialization.This layer produced findings unavailable to static review: an unused lazy-loaded ai_scenario module logged as a missing manifest entry (showing a branching-scenario interaction was scaffolded but never authored); subtitle assets being fetched from a CDN rather than the bundled local file (a risk for offline/air-gapped LMS hosting); and confirmation that the AI tutor is responsibly scoped but network-dependent. It also lets me verify the completion model against the rendered behavior rather than trusting the configuration alone.
Findings from both layers were assessed against a six-dimension rubric applied with a fixed severity scale (High / Medium / Low):
The three-layer structure of Dimension 3 was not in the rubric’s first version; it is a correction that came from human subject-matter feedback during the audit, and its origin is discussed as a finding in its own right.
Domain knowledge was encoded as a discrete, swappable checklist rather than baked into the reviewer. The ISO 45001 module enumerates, per clause (4–10), what the course must convey and the failure modes typical of generated content, plus key-concept checks (e.g., the hierarchy of controls under clause 8.1.2), the independence requirement for auditors (9.2.2), and citation expectations (ISO 45001:2018; ISO 19011:2018). Because the module is separable from the engine, the same apparatus reviews an ISO 14001, ISO 9001, STCW, or ISM Code course by substituting the module.
A one-off prompt is not an audit capability; it is not reproducible, versionable, or shareable across a team using different AI clients. I therefore externalized the capability into persistent configuration — version-controlled files that define the audit — and exposed it through an AI-service-agnostic layer.
The single source of truth is four artifacts: the extractor, the rubric, the standards modules, and the output template. On top of these sits an MCP server that presents the capability as callable tools (extract_rise_course, get_rubric, list_standards, get_standard, get_review_template). Because MCP is client-neutral, the identical server is consumed by any MCP-capable client. Around this core, thin per-tool adapters provide native ergonomics:
A universal "AGENTS.md" at the project root allows any file-aware agent to discover the audit procedure automatically. The design consequence is that audit policy (what “good” means) lives in one place and evolves once, while audit access (which AI tool a given team member uses) is free to vary. Updating the rubric — for instance, adding the andragogy layering of section 4.4- propagates to every client without editing any adapter. The local-model path (LM Studio) matters for confidentiality: sensitive management-system documents can be audited entirely on-device.
The apparatus is therefore designed to survive the audit that produced it. Reproducibility is concrete: re-running the extractor on the same package yields the same inventory; the rubric and standards are diffable text under version control; and the same MCP tools answer identically regardless of which AI client a reviewer uses. Extending coverage to a new standard is an additive change (a new module), not a modification of the engine. The intended operating model is triage-plus-verification: the toolkit produces a consistent, evidence-quoted first pass that a human subject-matter expert confirms, particularly for high-severity findings.
The static content audit produced eleven findings; the three high-severity ones were clause-level conceptual errors that would misdirect a trainee auditor:
Medium and low findings included misfiling training under Operation, the absence of any reference to ISO 19011 (the actual source of the audit method taught), the auditor-independence rule left unstated, conflation of internal audit with compliance evaluation, an unreconciled double use of the PDCA model, an imprecise definition of nonconformity, and the absence of versioned/clause-level citation. Notably, the correct material in the course was also acknowledged; the method is designed to be fair, not merely fault-finding.
Applying Dimension 3’s three layers:
With the graded examination residing in the LMS, the summative gap identified at first pass is mitigated at the delivery layer. However, the package itself reports completion on 90% of content viewed, independent of assessment. I raised this as an integration caveat: certification must be gated on the examination pass, not on the package’s “viewed” status, or a learner may be recorded complete without demonstrating competence.
The browser-automation layer confirmed a responsibly scoped AI tutor with an appropriate accuracy disclaimer, but also that (a) the tutor and captions are network-dependent and would degrade in offline hosting, (b) a scenario module was scaffolded but never authored, and (c) the completion behavior matched the permissive configuration. A corollary implementation finding: because the AI tutor answers from the course’s own content, it will faithfully reproduce the content errors of 5.1 — so the content and implementation layers are not independent, and fixing the source remediates both.
Using AI to audit: The design contains an obvious circularity: a human-guided generative system produced the course, and AI systems assisted its audit. I mitigate this in three ways. First, the audit is grounded — against an external standards module and an explicit rubric, not against the auditor model’s unconstrained judgment. Second, findings are evidence-bound: every finding quotes an exact block, or a captured runtime observation, which is human-checkable. Third, and decisively, the loop is closed by a human expert. The most consequential methodological improvement in this study — restructuring Dimension 3 so that andragogy is the governing frame, methods are matched to each objective’s cognitive level, and cognitive mechanisms are treated as machinery inside a method rather than as competing “approaches” — originated from human subject-matter feedback, not from the model. I am not an instructional designer; the apparatus let me carry the audit to where the expert’s correction had something precise to act on. The apparatus makes the expert more efficient; it does not replace the expert’s authority.
Content and implementation are complementary, not redundant. Neither layer subsumes the other. The clause-level errors were only visible in the decoded content; the missing scenario module, the CDN dependency, and the completion behavior were only visible at runtime; and the AI-tutor error-propagation risk is intelligible only when both layers are held together.
Generalizability. Nothing in the method is specific to this course beyond the ISO 45001 module. The extraction technique applies to Rise exports generally; the browser-automation procedure applies to any web-delivered course; the rubric is not Rise-specific and should transfer to other e-learning artifacts, though it has so far been applied to one; and the standards mechanism is designed for substitution. The service-agnostic layer means an organization can adopt the capability without standardizing on a single AI vendor — a practical property for teams whose members legitimately prefer different tools.
The extractor is specific to the Articulate Rise export format; other authoring tools (Storyline, Captivate) would require their own extractors. The browser-automation audit inspects behavior reachable from a local launch; LMS-mediated behaviors (real SCORM sequencing, gradebook integration) require in-LMS testing. AI-assisted findings, though grounded and evidence-bound, remain subject to model error and must be verified for high-stakes claims; the toolkit is explicitly positioned as first-pass triage. The rubric’s transferability beyond this single course is a design intention, not yet a demonstrated result. Finally, the learning-experience critique is analytic (design-level), not empirical: it predicts weak retention from the absence of retrieval and spacing but does not measure learner outcomes, which would require a separate evaluation study.
Generative AI has made course production cheap and course verification scarce. This case study shows that verification can be systematized by someone outside the design process: decode the compiled artifact for content analysis, drive a browser to audit implementation and runtime behavior, evaluate against an explicit rubric that unites domain compliance with adult-learning science, and package the whole as persistent configuration behind a service-agnostic MCP layer so the capability outlives the single review and travels across a team’s toolchain. The audit found errors that matter — an auditor course that misstated the standard it teaches — and, in the same pass, produced a reusable instrument for finding them again. The lesson is not that AI can grade AI. It is that a disciplined, grounded, human-verified apparatus makes generated content auditable at something closer to the rate at which it is now produced — and that the remaining open question is what happens in organizations where neither the engineer nor the subject-matter expert is in that loop.