Skip to content

fix(evaluation): don't pass AgentEvaluator metrics on runs the evaluator left unscored - #8277

Open
EdwardGlockner wants to merge 1 commit into
google:mainfrom
EdwardGlockner:fix/agent-evaluator-unscored-runs
Open

EdwardGlockner wants to merge 1 commit into
google:mainfrom
EdwardGlockner:fix/agent-evaluator-unscored-runs

Conversation

@EdwardGlockner

Copy link
Copy Markdown

Link to Issue or Description of Change

Problem:
AgentEvaluator scores a metric by averaging its per-invocation scores and skipping None. It never looks at the run's overall result. So when hallucinations_v1 leaves a run NOT_EVALUATED but has scored some of its responses, those responses alone decide the metric, and it can pass.

Solution:
I added a run_unscored field to _EvalMetricResultWithInvocation. _get_eval_metric_results_with_invocation sets it on every invocation of a run whose overall result for the metric is NOT_EVALUATED with no score, and _process_metrics_and_get_failures leaves those out of the average. They're still printed and written to the CSV. If every run is unscored, the metric is reported as not evaluated, as it already is when no response has a score. If only some runs are, they're just left out. That's the same rule #8238 uses in the aggregator, so adk eval --num_runs and AgentEvaluator would agree.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

The two new tests in test_agent_evaluator.py check that unscored runs with one scored turn are reported as not evaluated, and that an unscored run's scored turn doesn't change the mean of a scored run. Both fail on main. test_process_metrics_and_get_failures_printing builds its input from a MagicMock, whose run_unscored would be truthy, so I set it to False there.

pytest tests/unittests/evaluation passes (1343 tests).

Manual End-to-End (E2E) Tests:

The script in #8276: on main evaluate_eval_set passes with hallucinations_v1 at 1.0. With this change it fails with hallucinations_v1 for geo was not evaluated.

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.

…tor left unscored

AgentEvaluator averaged a metric's per-invocation scores and ignored the
run's overall result. When an evaluator scores some invocations but leaves
the run as a whole NOT_EVALUATED with no score (hallucinations_v1 does this
when the judge cannot label a response), the scored invocations alone
decided the metric, so the run could pass.

The invocations of such a run are now left out of the metric's score. A
metric left unscored in every run is reported as not evaluated, and an
unscored run no longer changes the mean of the runs that were scored.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AgentEvaluator can pass hallucinations_v1 even when the run is unscored

2 participants