Custom Scorers and mlflow.genai.evaluate(): Evaluating an Agent from Its Traces

What this deep dive adds

Chapter 49 of the course introduces the @scorer decorator, the four arguments a scorer can read and its return types; Chapter 50 explains why agents are evaluated from their traces. This post joins them in one working example: a traced app, an evaluation dataset, four scorers including one that grades the retrieval step from the trace, the results, and re-scoring stored traces without running the app again.

Both are Section 6 (Evaluation and Monitoring) objectives in the 18 March 2026 exam guide. Which built-in judges need ground truth is Chapter 48 of the course, summarised in the table below.

The complete example

A small support agent retrieves a policy and answers in JSON. Each function carries @mlflow.trace, so every call produces a trace with an agent span and a retriever span inside it:

import json
import mlflow
from mlflow.entities import Feedback, SpanType
from mlflow.genai.scorers import scorer

mlflow.set_experiment("/Shared/support-agent-eval")

POLICIES = {
    "returns": "Items can be returned within 30 days with a receipt.",
    "shipping": "Standard shipping takes 3 to 5 business days.",
}

@mlflow.trace(span_type=SpanType.RETRIEVER)
def retrieve(question: str) -> list[dict]:
    return [{"doc_id": k, "text": v}
            for k, v in POLICIES.items() if k in question.lower()]

@mlflow.trace(span_type=SpanType.AGENT)
def support_agent(question: str) -> str:
    docs = retrieve(question)
    answer = docs[0]["text"] if docs else "I don't know."
    return json.dumps({"answer": answer,
                       "sources": [d["doc_id"] for d in docs]})

eval_data = [
    {"inputs": {"question": "What is your returns window?"},
     "expectations": {"expected_source": "returns"}},
    {"inputs": {"question": "How long does shipping take?"},
     "expectations": {"expected_source": "shipping"}},
    {"inputs": {"question": "Do you price match?"},
     "expectations": {"expected_source": "pricing"}},
]

@scorer
def valid_json(outputs) -> bool:
    try:
        json.loads(outputs)
        return True
    except (TypeError, ValueError):
        return False

@scorer
def concise(outputs) -> Feedback:
    words = len(json.loads(outputs)["answer"].split())
    return Feedback(value=words <= 40,
                    rationale=f"Answer has {words} words; the limit is 40.")

@scorer
def cites_expected_source(outputs, expectations) -> bool:
    return expectations["expected_source"] in json.loads(outputs)["sources"]

@scorer
def retrieval_hit(trace, expectations) -> Feedback:
    spans = trace.search_spans(span_type=SpanType.RETRIEVER)
    retrieved = [d["doc_id"] for s in spans for d in (s.outputs or [])]
    hit = expectations["expected_source"] in retrieved
    return Feedback(value=hit, rationale=f"Retrieved {retrieved or 'nothing'}.")

results = mlflow.genai.evaluate(
    data=eval_data,
    predict_fn=support_agent,
    scorers=[valid_json, concise, cites_expected_source, retrieval_hit],
)
print(results.metrics)

• The dataset. Each row has inputs, whose keys become the keyword arguments of predict_fn (here question), and optional expectations holding the ground truth.
• A scorer names only what it reads. valid_json asks for outputs; cites_expected_source adds expectations; retrieval_hit asks for the whole trace. MLflow supplies each by name.
• Return types. A bool is pass or fail. A Feedback carries a value and a rationale, so the evaluation shows why a row failed, not just that it did.
• Scoring a step. retrieval_hit finds the retriever span and checks what it returned. That separates "retrieval missed" from "the model ignored good context", which no score on the final answer can do.

Run against MLflow 3.18 with a local tracking store. It printed valid_json/mean 1.0, concise/mean 1.0, and cites_expected_source/mean and retrieval_hit/mean of 0.67 each: the price-match question has no policy, so both source checks fail on that row. API from the Databricks "Code-based scorers" pages and the MLflow 3.18 package.

Reading the results

Each scorer becomes a metric named <scorer>/mean, the average across rows with a bool counted as 1 or 0. The evaluation run also stores one trace per row with every scorer's feedback attached, which is where the rationales appear. In the example, the low retrieval_hit and cites_expected_source means point to the same row, and its rationale ("Retrieved nothing") says the fix belongs in retrieval, not the prompt.

Trace once, score again

Re-running an agent for every scorer change is slow and costs tokens. Stored traces can be scored directly: pass them as data and leave out predict_fn.

traces = mlflow.search_traces(run_id=results.run_id)
rescored = mlflow.genai.evaluate(data=traces, scorers=[retrieval_hit])

The expectations travel with the traces, so retrieval_hit scored them again without the original dataset. The same pattern applies to production traces: search them, add the useful ones to an evaluation dataset (a Unity Catalog table created with mlflow.genai.datasets.create_dataset, which needs CREATE TABLE on the schema), and evaluate against it after each change.

Built-in judges next to custom scorers

Custom scorers sit in the same scorers list as the built-in LLM judges, so one run can mix deterministic checks with judgment calls. The column that matters for the exam is whether a judge needs ground truth in expectations:

Choose a code-based scorer for exact rules (valid JSON, a schema, a required disclaimer, a length limit, an expected tool) and a judge when quality needs reading the way a person would. By default each judge uses a Databricks-hosted LLM; the model argument swaps it, written as <provider>:/<model-name>, for example databricks:/databricks-gpt-5-mini.

Judge table from the Databricks page "Built-in LLM judges"; mixing judges and scorers from "Scorers and LLM judges".

From evaluation to monitoring

The same scorer can grade live traffic. Two constraints from the docs decide how to write it: production monitoring accepts built-in judges and @scorer-decorated functions, but not class-based Scorer subclasses, and a decorated scorer used in monitoring must be defined and registered from a Databricks notebook, because the monitoring service serializes the function's code to run it remotely. State a monitored scorer needs belongs inside the function body.

What this means for the exam

Distractors offer a single score on the final answer, endpoint logs, or a judge that needs ground truth on a dataset that has none.

FAQ