Module eval 

The eval-case model: a typed model and its parse boundary.

Holds EvalSuite/EvalCase, the ExpectedVerdict polarity enum, the typed EvalParseError, and parse_eval_suite — the pure parse that turns an instructor’s eval_<lesson>.yaml into a typed suite. Each case pairs a synthetic learner submission with the polarity the grader should return; ExpectedVerdict is a dedicated sum type rather than the message-carrying runtime llm::Verdict (ADR-0007), so an illegal verdict is rejected here instead of travelling downstream as data.

The public types are serde-free domain values; deserialization runs through the private Raw* boundary DTOs, mirroring the llm layer’s Feedback → Verdict split (ADR-0006). That two-phase parse is what lets a bad verdict be reported against its case index rather than a YAML line.

The module owns two responsibilities at different altitudes. The data shape and its parse are pure: reading files is the caller’s concern (§2.1). Scoring (score_case, aggregate, CaseResult, EvalReport) is the pure core (§2.3) — exact polarity equality with no model in sight. run_eval is the thin effectful shell (§2.4) that drives the suite through the same pipeline run uses (so the feedback evaluated is the feedback shipped, §3.2) and hands each verdict to the pure scorer.

Structs

CaseResult

One scored eval case: the expected polarity, the polarity the grader actually returned, whether they matched, and the grader’s verbatim feedback message.

EvalCase

A single eval case: a synthetic submission and the polarity it should grade to.

EvalReport

The result of scoring an eval suite: every CaseResult in suite order plus the aggregate accuracy.

EvalSuite

A lesson’s eval suite: its cases in authored order.

Enums

EvalParseError

Why a YAML document is not a valid eval suite.

EvalRunError

Why scoring an eval suite failed.

ExpectedVerdict

The verdict an eval case expects the grader to return: a polarity, with no learner-facing message.

Functions

aggregate

The aggregate accuracy of scored cases: the fraction whose polarity matched.

parse_eval_suite

Parse an eval suite from a YAML document.

run_eval

Run every case in suite through the same pipeline run uses and score each verdict against its expected polarity — or, when case is Some(n), only the one 1-based case n.

score_case

Score one case: whether the grader’s actual polarity matched the expected one.