Metrics reference

Harnest delegates metric computation to the Google ADK 2.x evaluator for both ADK and LangGraph agents. It reflects the installed ADK registry rather than freezing a second allowlist. The registry shipped at Harnest’s Google ADK 2.8 dependency floor contains the 13 metrics below. A compatible later ADK 2.x release can add registry entries without a Harnest code change. Use the playground’s Evals catalog to inspect the exact installed list.

Built-in metrics

Reference and single-turn metrics

Metric	What it measures	Score	Needs golden result	Backend
`tool_trajectory_avg_score`	Tool names and arguments across each invocation	0–1	Tool calls	Local deterministic comparison
`response_match_score`	ROUGE-1 similarity between actual and golden final text	0–1	Final response	Local deterministic comparison
`response_evaluation_score`	Coherence of the actual response	1–5	No	Vertex Gen AI evaluation service
`safety_v1`	Harmlessness of the actual response	0–1	No	Vertex Gen AI evaluation service
`final_response_match_v2`	Meaning-level validity against the golden final response	0–1	Final response	Configured ADK judge model
`rubric_based_final_response_quality_v1`	Final answer behavior against authored rubrics	0–1	Rubrics	Configured ADK judge model
`hallucinations_v1`	Fraction of claims supported by available context and evidence	0–1	No	Configured ADK judge model
`rubric_based_tool_use_quality_v1`	Actual tool use against authored rubrics	0–1	Rubrics	Configured ADK judge model

tool_trajectory_avg_score gives each invocation either 0 or 1, then averages the invocation scores. Choose business or strict to control matching. final_response_match_v2 produces a binary verdict for each invocation. It uses majority vote across judge samples, then reports the fraction of valid invocations.

Multi-turn and simulation metrics

Metric	What it measures	Score	Backend
`per_turn_user_simulator_quality_v1`	Fraction of simulated user messages that follow the conversation plan	0–1	Configured ADK judge model
`multi_turn_task_success_v1`	Whether the agent achieves the conversation’s goals	0–1	Vertex Gen AI evaluation service
`multi_turn_trajectory_quality_v1`	Quality of the full path used to achieve the goals	0–1	Vertex Gen AI evaluation service
`multi_turn_tool_use_quality_v1`	Function-call quality across the conversation	0–1	Vertex Gen AI evaluation service
`rubric_based_multi_turn_trajectory_quality_v1`	Cumulative responses and tool behavior against authored rubrics	0–1	Configured ADK judge model

The three Vertex multi-turn metrics are reference-free and evaluate the complete actual conversation. Their per-invocation result is not_evaluated on earlier turns; the last turn carries the conversation-level score. The rubric-based multi-turn metric uses the same final-turn convention.

Configure thresholds

Use a number for a metric that needs only a threshold:

evals/test_config.json

{
  "criteria": {
    "response_match_score": 0.8,
    "response_evaluation_score": 3.0,
    "safety_v1": 0.9,
    "multi_turn_task_success_v1": 0.8,
    "multi_turn_trajectory_quality_v1": 0.8,
    "multi_turn_tool_use_quality_v1": 0.8
  }
}

Scores equal to the threshold pass. Choose a threshold inside the metric’s documented score interval.

Configure model-judged metrics

Judge-backed criteria use the properties below. The final-output option applies only where the evaluator consumes final answer text.

Property	Default	Applies to	Purpose
`threshold`	Required	Every metric	Pass boundary
`judgeModelOptions.judgeModel`	`OPENAI_MODEL`, or `gpt-4.1-mini`	Model-judged metrics	Omit to use the shared OpenAI-compatible model; explicit IDs select an ADK model provider
`judgeModelOptions.judgeModelConfig`	None	Model-judged metrics	Google GenAI generation options for judge calls
`judgeModelOptions.numSamples`	`5`	Final-response match, rubric metrics, and simulator-quality metrics	Judge calls per invocation before aggregation
`judgeModelOptions.parallelismLimit`	`1`	Final-response match and rubric metrics	Maximum concurrent judge calls
`includeIntermediateResponsesInFinal`	`false`	`final_response_match_v2` and rubric-based final response quality	Include visible text emitted before tool calls with the final output

evals/test_config.json

{
  "criteria": {
    "final_response_match_v2": {
      "threshold": 0.8,
      "judgeModelOptions": {
        "numSamples": 3,
        "parallelismLimit": 1
      }
    },
    "hallucinations_v1": {
      "threshold": 0.9,
      "evaluateIntermediateNlResponses": false
    }
  }
}

hallucinations_v1 uses evaluateIntermediateNlResponses, not includeIntermediateResponsesInFinal, to opt visible intermediate natural-language responses into its analysis.

Use a custom judge model

Model precedence is an explicit judgeModel, then OPENAI_MODEL, then gpt-4.1-mini. Use an explicit OpenAI-compatible model ID when the judge should differ from the agent:

evals/test_config.json

{
  "criteria": {
    "final_response_match_v2": {
      "threshold": 0.8,
      "judgeModelOptions": {
        "judgeModel": "openai/my-judge-model",
        "numSamples": 3
      }
    }
  }
}

Replace my-judge-model with a model served by your configured endpoint. This explicit ID overrides OPENAI_MODEL. The judge reuses a compatible agent model transport when one is available, including a lifecycle-owned client, custom gateway, and headers. Otherwise it uses the normal OPENAI_BASE_URL and OPENAI_API_KEY configuration. To select a native non-OpenAI provider instead, use its ADK model ID and supply that provider’s credentials through the same process environment.

Harnest runs each eval set once. That num_runs=1 safeguard does not replace judgeModelOptions.numSamples: a judge-backed metric can still call its judge several times for one invocation. Lower the sample count deliberately when cost matters, and raise it when judge stability matters more.

Configure rubric metrics

Each rubric criterion requires a non-empty rubrics list. Every rubric has a stable ID and one testable text property.

evals/test_config.json

{
  "criteria": {
    "rubric_based_final_response_quality_v1": {
      "threshold": 1.0,
      "judgeModelOptions": {
        "numSamples": 3
      },
      "rubrics": [
        {
          "rubricId": "grounded_answer",
          "rubricContent": {
            "textProperty": "The answer states only claims supported by tool evidence."
          }
        }
      ]
    },
    "rubric_based_tool_use_quality_v1": {
      "threshold": 1.0,
      "judgeModelOptions": {
        "numSamples": 3
      },
      "rubrics": [
        {
          "rubricId": "verify_first",
          "rubricContent": {
            "textProperty": "The agent calls get_city_fact before making a factual claim."
          }
        }
      ]
    }
  }
}

Use rubric_based_multi_turn_trajectory_quality_v1 with the same criterion shape when a property spans the complete conversation. Rubric verdicts, scores, and rationales are preserved in the complete CLI result.

Configure evaluation backends

The agent under test and every metric backend inherit the same process environment. Follow Configure model credentials for local, CI, and deployment setup. Metrics may add a second provider to the agent’s normal model and tool stack:

Local deterministic
ADK judge model
Vertex evaluation service

tool_trajectory_avg_score and response_match_score need no metric service. The agent itself can still make live model or tool calls.

Omit judgeModel to use OPENAI_MODEL and the shared OPENAI_API_KEY and OPENAI_BASE_URL configuration. An explicit judgeModel, such as gemini-2.5-flash, opts into that ADK model provider and requires its own credentials. Harnest does not replace explicit model IDs.

response_evaluation_score, safety_v1, and the three multi_turn_*_v1 metrics call the Vertex Gen AI evaluation client. If GOOGLE_API_KEY is set, ADK constructs that client with the key. Otherwise set both GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION and make Application Default Credentials available. When all are set, the API key path takes precedence.

test_config.json selects metric backends and judge model IDs. Never put credentials in that file.

Overview

Get started

Guides

Metrics reference

Built-in metrics

Reference and single-turn metrics

Multi-turn and simulation metrics

Configure thresholds

Configure model-judged metrics

Use a custom judge model

Configure rubric metrics

Configure evaluation backends

​Built-in metrics

​Reference and single-turn metrics

​Multi-turn and simulation metrics

​Configure thresholds

​Configure model-judged metrics

​Use a custom judge model

​Configure rubric metrics

​Configure evaluation backends

Built-in metrics

Reference and single-turn metrics

Multi-turn and simulation metrics

Configure thresholds

Configure model-judged metrics

Use a custom judge model

Configure rubric metrics

Configure evaluation backends