Expand description
smevals eval-dir generation: a pure translator from a lesson + eval suite to
the file tree smevals run/build consume.
An eval dir is any directory with an eval.yaml, tasks/, configs/, and
graders/ — the smevals v0.2.0 layout. generate_eval_dir produces that
tree as (relative_path, contents) pairs: eval.yaml, configs/default.yaml,
graders/default.yaml, and tasks/case-N.yaml (one per suite case, 1-based).
It is pure (§2.1): no filesystem, no clock, no iteration-order randomness —
identical inputs yield byte-identical output, and every value is emitted
through an injection-proof YAML scalar discipline proven by round-trip tests
(§1.3.1). The model id in configs/default.yaml is single-sourced from
ProviderChoice::default_model(), never a divergent literal.
This module owns only the lesson+suite → smevals-dir translation. It is
NOT the runner (AC-3), the LLM judge (AC-4), or the report command (AC-5);
it merely emits the templates those later slices wire into. Script paths in
the templates (scripts/smevals/run.sh, scripts/smevals/check_polarity.sh,
scripts/smevals/judge_feedback.py) are emitted relative to the generated
configs//graders/ file, as smevals resolves them relative to the file
that names them.
YAML emission: serde-saphyr 0.0.27 is parse-only, so the emitter is
hand-rolled. Hostile content (a submission containing : , &anchor,
---, or a leading - ) is emitted as a double-quoted scalar with every
control character escaped — byte-exact under re-parse (serde_saphyr’s |+
chomping is broken and |- drops trailing newlines, so block scalars cannot
round-trip arbitrary strings; the round-trip test in
crates/core/tests/generate_eval_dir.rs pins this). Safe-by-construction
values (slugs, verdict tokens, the pinned script paths) are plain scalars.
Enums§
- GenError
- Why an eval dir could not be generated.
Functions§
- course_
root_ for - The course root for
lesson_path: the nearest ancestor directory containing ablendtutor.tomlmanifest (the course boundary, per scaffold.rs). - generate_
eval_ dir - Generate the smevals eval-dir file tree for
suite, tagged withlesson_id. - lesson_
id_ from_ path - The lesson id for a lesson file: its stem, so
lessons/foo.yamlisfoo. - write_
eval_ dir - Write the generated eval dir into
dir/.smevals/, wherediris the course root.