智能体评估：衡量任务完成度与推理质量

学习如何用严格的评估来衡量你的智能体是否真的在正常工作。

🎯 你将学到

• 理解并应用本课涵盖的核心概念

「它能用吗？」这个问题的问题

你构建了一个网页研究智能体。你在几个测试查询上跑它，它产出看起来合理的答案，然后你就发布了。三周后，用户抱怨它答错问题、原地打转，有时还会返回自信的胡说八道。

哪里出错了？你从未真正衡量过它是否 work。

通过盯着几个输出「测试」一个智能体，不是评估——那是祈祷。真正的评估意味着定义成功长什么样、构建覆盖你用例的测试集、并运行能告诉你一个数字指标的自动化测量。这个数字应在你改进智能体时上升，在下降时提醒你。

智能体评估比模型评估更难。你不是只把输出和一个标签比对——你是在衡量一个过程：智能体是否以正确的顺序、用正确的工具、采取了正确的步骤？本教程给你一个严格做这件事的框架。

智能体质量的四个维度

1. 任务完成率（Task Completion Rate）

智能体究竟有没有完成这个任务？

这是最基本的指标。一个遇到错误、在达到最大迭代次数后无结果、或对有效任务产出「我无法帮你」的智能体，是完成失败。

def measure_task_completion(agent_response: dict) -> bool:
    """Check if the agent completed the task."""
    # Check for explicit failure signals
    failure_phrases = [
        "i'm unable to", "i cannot", "i don't have access",
        "task incomplete", "max iterations", "error occurred"
    ]

    output = agent_response.get("output", "").lower()

    if any(phrase in output for phrase in failure_phrases):
        return False

    # Check if the agent actually produced meaningful content
    if len(output.strip()) < 50:  # Suspiciously short response
        return False

    return True

一个值得瞄准的好基线：在你定义的测试用例上 95%+ 的任务完成率。低于 90% 意味着你的智能体有会惹恼用户的可靠性问题。

2. 工具准确性（Tool Accuracy）

智能体是否用正确的参数调用了正确的工具？

这更难衡量，因为你需要定义「正确」的工具调用长什么样。对一个研究智能体，一个关于近期事件的问题应当触发 web_search，而非 wikipedia。一个关于成熟概念的问题应当优先用 wikipedia 以取深度。

def evaluate_tool_calls(
    actual_steps: list,
    expected_tool_sequence: list[str]
) -> dict:
    """
    Compare actual tool calls against expected sequence.

    actual_steps: list of (AgentAction, observation) tuples from the agent
    expected_tool_sequence: list of expected tool names in order
    """
    actual_tools = [step[0].tool for step in actual_steps]

    # Check if all expected tools were called (order-independent)
    expected_set = set(expected_tool_sequence)
    actual_set = set(actual_tools)

    coverage = len(expected_set & actual_set) / len(expected_set) if expected_set else 1.0

    # Check for unnecessary tool calls (hallucinated steps)
    unnecessary = [t for t in actual_tools if t not in expected_set]

    return {
        "tool_coverage": coverage,           # Did it use all expected tools?
        "unnecessary_calls": unnecessary,    # What extra calls did it make?
        "total_calls": len(actual_tools),    # How many total calls?
        "expected_calls": len(expected_tool_sequence)
    }

3. 答案质量（Answer Quality）

最终答案是否真的正确且有用了？

这是哲学上变难的地方。对研究摘要而言，「正确」不是二元的对/错——它是一个光谱。你需要二者之一：

• 参考答案： 人工撰写的黄金标准，用于比对
• LLM-as-judge： 用另一个 LLM 给答案打分（见下文）
• 事实校验： 把答案里的具体声明与已知事实比对

4. 轨迹效率（Trajectory Efficiency）

智能体是否走了一条高效的路径？

一个该用 2 次却调用了 7 次搜索的智能体，是低效的——它更慢且成本更高。衡量它：

def measure_trajectory_efficiency(
    actual_steps: int,
    optimal_steps: int
) -> float:
    """
    Returns a score between 0 and 1.
    1.0 = perfectly efficient, took exactly optimal steps.
    0.5 = took twice as many steps as needed.
    """
    if actual_steps == 0:
        return 0.0
    return min(1.0, optimal_steps / actual_steps)

构建测试套件

一个好的评估需要一个系统化的测试集。下面是一个研究智能体评估的模板：

# test_cases.py

TEST_CASES = [
    {
        "id": "TC001",
        "query": "What is Flash Attention and why was it invented?",
        "expected_tools": ["wikipedia"],  # Background concept, not breaking news
        "expected_answer_contains": ["memory", "attention", "GPU", "efficient"],
        "optimal_steps": 2,
        "category": "technical_background"
    },
    {
        "id": "TC002",
        "query": "What LLM models were released in the last 3 months?",
        "expected_tools": ["web_search"],  # Requires current info
        "expected_answer_contains": ["model", "release", "2025", "2026"],
        "optimal_steps": 2,
        "category": "current_events"
    },
    {
        "id": "TC003",
        "query": "Compare BERT and GPT architectures. What are their key differences?",
        "expected_tools": ["wikipedia", "web_search"],  # Both background + current context
        "expected_answer_contains": ["encoder", "decoder", "pre-training", "bidirectional"],
        "optimal_steps": 3,
        "category": "comparison"
    },
    {
        "id": "TC004",
        "query": "What is the current state of the art accuracy on ImageNet?",
        "expected_tools": ["web_search"],
        "expected_answer_contains": ["percent", "accuracy", "top-1"],
        "optimal_steps": 2,
        "category": "benchmarks"
    },
    # ... 16 more test cases
]

设计你的测试用例以覆盖：

• 不同的任务类型（事实型、分析型、比较型、当前事件）
• 不同的复杂度层级（1 次工具调用 vs 3+ 次）
• 边界情况（模糊查询、多部分问题）
• 失败情况（超出智能体能力范围的查询）

自动化评估框架

import json
from dataclasses import dataclass
from typing import Optional

@dataclass
class EvaluationResult:
    test_id: str
    completed: bool
    tool_coverage: float
    unnecessary_calls: list
    answer_keyword_score: float
    trajectory_efficiency: float
    overall_score: float
    notes: str = ""

def evaluate_agent_on_test(
    agent_executor,
    test_case: dict
) -> EvaluationResult:
    """Run a single test case and return metrics."""

    try:
        result = agent_executor.invoke(
            {"input": test_case["query"]},
        )

        completed = measure_task_completion(result)

        tool_metrics = evaluate_tool_calls(
            result.get("intermediate_steps", []),
            test_case["expected_tools"]
        )

        # Check if answer contains expected keywords
        output_lower = result["output"].lower()
        keyword_matches = sum(
            1 for kw in test_case["expected_answer_contains"]
            if kw.lower() in output_lower
        )
        keyword_score = keyword_matches / len(test_case["expected_answer_contains"])

        efficiency = measure_trajectory_efficiency(
            actual_steps=len(result.get("intermediate_steps", [])),
            optimal_steps=test_case["optimal_steps"]
        )

        overall = (
            (1.0 if completed else 0.0) * 0.3 +
            tool_metrics["tool_coverage"] * 0.25 +
            keyword_score * 0.3 +
            efficiency * 0.15
        )

        return EvaluationResult(
            test_id=test_case["id"],
            completed=completed,
            tool_coverage=tool_metrics["tool_coverage"],
            unnecessary_calls=tool_metrics["unnecessary_calls"],
            answer_keyword_score=keyword_score,
            trajectory_efficiency=efficiency,
            overall_score=overall
        )

    except Exception as e:
        return EvaluationResult(
            test_id=test_case["id"],
            completed=False,
            tool_coverage=0.0,
            unnecessary_calls=[],
            answer_keyword_score=0.0,
            trajectory_efficiency=0.0,
            overall_score=0.0,
            notes=f"Exception: {str(e)}"
        )

def run_full_evaluation(agent_executor, test_cases: list) -> dict:
    """Run all test cases and aggregate metrics."""
    results = []

    for test in test_cases:
        print(f"Running {test['id']}: {test['query'][:60]}...")
        result = evaluate_agent_on_test(agent_executor, test)
        results.append(result)
        print(f"  Score: {result.overall_score:.2f} | Complete: {result.completed}")

    # Aggregate metrics
    completion_rate = sum(1 for r in results if r.completed) / len(results)
    avg_tool_coverage = sum(r.tool_coverage for r in results) / len(results)
    avg_keyword_score = sum(r.answer_keyword_score for r in results) / len(results)
    avg_efficiency = sum(r.trajectory_efficiency for r in results) / len(results)
    avg_overall = sum(r.overall_score for r in results) / len(results)

    report = {
        "total_tests": len(results),
        "completion_rate": completion_rate,
        "avg_tool_coverage": avg_tool_coverage,
        "avg_answer_quality": avg_keyword_score,
        "avg_efficiency": avg_efficiency,
        "overall_score": avg_overall,
        "failing_tests": [r.test_id for r in results if r.overall_score < 0.6],
        "raw_results": [vars(r) for r in results]
    }

    return report

LLM-as-Judge 模式

关键词匹配是个钝器。要评估推理质量和答案正确性，用第二个 LLM 作为裁判。它在人工评估跟不上的地方可以规模化。

import anthropic

judge_client = anthropic.Anthropic()

def llm_judge_answer(
    question: str,
    agent_answer: str,
    reference_answer: Optional[str] = None
) -> dict:
    """Use Claude to evaluate the quality of an agent's answer."""

    reference_context = ""
    if reference_answer:
        reference_context = f"\nReference answer (ground truth): {reference_answer}"

    prompt = f"""You are evaluating the quality of an AI agent's response.

Question asked: {question}

Agent's answer: {agent_answer}
{reference_context}

Evaluate the answer on these criteria (score each 1-5):
1. Accuracy: Is the information factually correct?
2. Completeness: Does it fully address the question?
3. Reasoning quality: Is the answer well-reasoned and logical?
4. Conciseness: Is it appropriately concise without missing key points?

Respond in JSON format:
{{
  "accuracy": <1-5>,
  "completeness": <1-5>,
  "reasoning_quality": <1-5>,
  "conciseness": <1-5>,
  "overall": <1-5>,
  "strengths": "...",
  "weaknesses": "...",
  "verdict": "pass" or "fail"
}}"""

    response = judge_client.messages.create(
        model="claude-opus-4-5",
        max_tokens=512,
        messages=[{"role": "user", "content": prompt}]
    )

    try:
        return json.loads(response.content[0].text)
    except json.JSONDecodeError:
        return {"error": "Could not parse judge response", "raw": response.content[0].text}

# Example usage
judgement = llm_judge_answer(
    question="What is Flash Attention and why was it invented?",
    agent_answer="""Flash Attention is a memory-efficient attention algorithm introduced in 2022.
    It reformulates the standard attention computation to work in tiles, keeping data
    in fast SRAM instead of slow HBM (GPU memory). This reduces memory usage from O(n²)
    to O(n) and significantly speeds up training of long-context transformers."""
)
print(json.dumps(judgement, indent=2))

LLM-as-judge 模式很强大，但有一个已知偏差：即便简洁的答案更好，LLM 也倾向于更喜欢更长、更详细的答案。缓解方法是明确纳入一个简洁性标准，并把评分量规（rubric）调到适配你的用例。

回归测试：尽早抓住退化

评估不只是为了衡量质量——也是为了在变糟时抓住它。建立一个在每次代码变更时运行的回归测试：

BASELINE_SCORES = {
    "completion_rate": 0.95,
    "avg_tool_coverage": 0.88,
    "avg_answer_quality": 0.80,
    "overall_score": 0.85
}

def check_for_regressions(current_report: dict, tolerance: float = 0.05) -> list[str]:
    """Return list of regression warnings if any metric dropped significantly."""
    warnings = []

    for metric, baseline in BASELINE_SCORES.items():
        current = current_report.get(metric, 0)
        if current < baseline - tolerance:
            warnings.append(
                f"REGRESSION: {metric} dropped from {baseline:.2f} to {current:.2f}"
            )

    return warnings

在部署智能体更新前，在 CI/CD 里运行这个。如果一次提示变更或模型升级引发了退化，你会在用户之前抓住它。

一套实用的评估工作流

对于一个典型的智能体开发周期：

1. 定义 20-30 个测试用例——在写一行代码前就覆盖你的关键用例
2. 在每次重大变更后运行评估——新工具、更新的提示、不同的模型
3. 开发期间用关键词打分做快速反馈
4. 每周用 LLM-as-judge 做更深入的质量评估（它更慢、成本更高）
5. 标记并人工复核任何得分低于 0.6 的测试用例
6. 随时间追踪分数——把它们画在图表上，使退化在视觉上一目了然

总结

• 任务完成率衡量智能体究竟是否干完了活
• 工具准确性衡量它是否用正确参数调用了正确工具
• 答案质量可用关键词匹配（快）或 LLM-as-judge（准）来衡量
• 轨迹效率衡量智能体采取的步数与最优步数之比
• 在发布前构建 20+ 个用例的测试集——覆盖不同任务类型与边界情况
• 用回归测试抓住变更让智能体变差的时候
• LLM-as-judge 在人工评估跟不上的地方可以规模化——但要针对你的具体质量标准做校准