Repository navigation
fix(evaluation): don't pass AgentEvaluator metrics on runs the evaluator left unscored - #8277
Open
EdwardGlockner wants to merge 1 commit into
Open
EdwardGlockner wants to merge 1 commit into
EdwardGlockner wants to merge 1 commit into
Conversation
…tor left unscored AgentEvaluator averaged a metric's per-invocation scores and ignored the run's overall result. When an evaluator scores some invocations but leaves the run as a whole NOT_EVALUATED with no score (hallucinations_v1 does this when the judge cannot label a response), the scored invocations alone decided the metric, so the run could pass. The invocations of such a run are now left out of the metric's score. A metric left unscored in every run is reported as not evaluated, and an unscored run no longer changes the mean of the runs that were scored.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Link to Issue or Description of Change
Problem:
AgentEvaluatorscores a metric by averaging its per-invocation scores and skippingNone. It never looks at the run's overall result. So whenhallucinations_v1leaves a runNOT_EVALUATEDbut has scored some of its responses, those responses alone decide the metric, and it can pass.Solution:
I added a
run_unscoredfield to_EvalMetricResultWithInvocation._get_eval_metric_results_with_invocationsets it on every invocation of a run whose overall result for the metric isNOT_EVALUATEDwith no score, and_process_metrics_and_get_failuresleaves those out of the average. They're still printed and written to the CSV. If every run is unscored, the metric is reported as not evaluated, as it already is when no response has a score. If only some runs are, they're just left out. That's the same rule #8238 uses in the aggregator, soadk eval --num_runsandAgentEvaluatorwould agree.Testing Plan
Unit Tests:
The two new tests in
test_agent_evaluator.pycheck that unscored runs with one scored turn are reported as not evaluated, and that an unscored run's scored turn doesn't change the mean of a scored run. Both fail on main.test_process_metrics_and_get_failures_printingbuilds its input from aMagicMock, whoserun_unscoredwould be truthy, so I set it toFalsethere.pytest tests/unittests/evaluationpasses (1343 tests).Manual End-to-End (E2E) Tests:
The script in #8276: on main
evaluate_eval_setpasses withhallucinations_v1at1.0. With this change it fails withhallucinations_v1 for geo was not evaluated.Checklist