Your agent framework and your evaluation engine are separate choices. A LangGraph team keeps its LangGraph agent and runs evaluations through Harnest. You do not need to rewrite the agent in ADK or create a separate ADK application.
Start here
- To check known answers or tool calls, author an eval set and select a built-in metric.
- To score your own business rule, follow Create custom evals.
- To test follow-up behavior, put multiple turns in one case or use a user simulation.
- To make evaluation a CI gate, run and save results.
Choose an evaluation
Start with deterministic reference metrics where they fit. Add model-judged metrics for meaning and qualitative behavior. Judge and simulation metrics add cost and can vary between runs.
How evaluation works
1
Author an eval set
Put one or more
*.evalset.json files directly under the root agent’s evals/ directory. Each case contains either a static golden conversation or a conversation scenario.2
Select metrics
Add
evals/test_config.json. Set a threshold for every metric that should contribute to the case result.3
Run the quality gate
Run
harnest test AGENT_DIR --evals. Unit tests run first, followed by every validated eval set in filename order.4
Inspect or retain the result
Read the structured
EvalRunResult printed to the terminal, or select a JSON file with --eval-output FILE.Framework support
For a LangGraph run, Harnest starts the compiled LangGraph runtime, sends the evaluation’s user turns to it, and converts public responses and tool events into ADK invocations for scoring. Custom scorers therefore receive ADK
Invocation objects, not LangChain messages or raw graph state. The result’s framework: "langgraph" identifies the agent runtime; it does not select a different metric engine.
Keep the same text eval sets, criteria, and custom scoring functions when you switch frameworks. Actual tool trajectories can differ between runtimes, so review expectations after a switch. LangGraph non-text user content fails instead of being silently discarded. Use a smoke test for LangGraph media or bidirectional behavior.
ADK evaluation does not require a Google model for every metric. Deterministic metrics need no judge model; compatible LLM judges and text simulators can use your configured model and agent transport. Some service-backed metrics have their own credentials. See model configuration and the metric prerequisites.
Learn each part
Eval sets and configuration
Define static conversations, expected responses, tools, sessions, and shared criteria.
Metrics reference
Compare every built-in metric, score range, input requirement, and evaluation backend.
Simulations and live evaluation
Generate user turns, configure text or audio simulation, and select ADK live inference.
Create custom evals
Build and test a business-specific scorer, register it, and run it on either framework.
Custom metric API
Look up scorer arguments, result types, score intervals, and registration rules.
Run and inspect results
Run locally or in CI and capture the complete versioned JSON result.