run_eval

Function run_eval 

Source
pub async fn run_eval(
    lesson: &Lesson,
    suite: &EvalSuite,
    provider: ProviderChoice,
    base_url_override: Option<&str>,
    case: Option<usize>,
) -> Result<EvalReport, EvalRunError>
Expand description

Run every case in suite through the same pipeline run uses and score each verdict against its expected polarity — or, when case is Some(n), only the one 1-based case n.

The thin effectful shell over the pure scorer (§2.4): for each case it runs the submission through run_lesson — execute, grade, ask provider for a verdict — then pairs the verdict’s polarity with the expected one as a CaseResult. Driving the same run_lesson is what keeps the evaluated feedback identical to the shipped feedback (§3.2). An out-of-range case selection is rejected up front with EvalRunError::CaseOutOfRange — never clamped — before any case runs; a run failure stops scoring and names the offending case (EvalRunError::Run). base_url_override points the provider at a stub in tests, exactly as run does, and is None in production.