Agent Observability & Online Eval SOP: Tracing, Cost, Alerts

The scariest part of running an Agent in production is not a bad prompt--it is not knowing what it is doing. After a week live, users complain about quality drops. You open the logs: every line says print("done"). Which step was slow? Which tool call failed? Which user's answer cost $0.30? You have no idea. An Agent without observability is a black box inside a black box: the LLM is a black box, and the Agent's multi-step tool calls stack another black box on top. When things break, you can only guess.

Observability is not just "adding logs." It is a systems engineering discipline: first define what to measure (latency, token cost, tool-call success rate, error rate), then use distributed tracing to link every step together, then run online evaluation (LLM-as-judge + user feedback), and finally set alerts to catch problems automatically. This SOP walks through the full pipeline with copyable Python code at each step. The tooling is open-source-first: Langfuse (open-source LLM observability platform), Arize Phoenix (open-source tracing + evaluation), and OpenTelemetry (the universal observability standard). LangSmith is mentioned as the LangChain-ecosystem SaaS option. Use this alongside our LangGraph Production Agent SOP--that article covers how to build the Agent, this one covers how to monitor it.

1. Baseline: What to Observe

Before writing any tracing code, define what you are measuring. The four core metrics for Agent observability:

1. Latency: End-to-end response time, tracked by p50/p95/p99 percentiles. Averages hide tail latency--a high p99 means some users wait a very long time.
2. Token cost: Input/output token counts and corresponding cost per call. Multi-step Agents can spiral out of budget quickly.
3. Tool-call success rate: Success/failure ratio for each tool invocation. Tool failures are a leading cause of Agent meltdowns.
4. Error rate: Proportion of Agent runs that fail entirely. Distinguish retryable errors (API 5xx) from non-retryable ones (code bugs).

These four metrics form a complete "health dashboard": latency reflects user experience, cost reflects sustainability, tool-call success rate reflects external dependency stability, and error rate reflects system reliability. Missing any one creates a blind spot--watching latency without error rate might look fast because failed requests return instantly; watching cost without tool success rate might save tokens while tools keep failing in a loop. Production must track all four together and cross-reference them to find real problems.

Define this four-metric system in code:

from dataclasses import dataclass, field

@dataclass
class AgentMetrics:
    latencies: list = field(default_factory=list)
    token_costs: list = field(default_factory=list)
    tool_success: int = 0
    tool_total: int = 0
    errors: int = 0
    total_runs: int = 0

    def record_latency(self, seconds: float) -> None:
        self.latencies.append(seconds)

    def record_cost(self, cost_usd: float) -> None:
        self.token_costs.append(cost_usd)

    def record_tool_call(self, success: bool) -> None:
        self.tool_total += 1
        if success:
            self.tool_success += 1

    def record_run(self, error: bool = False) -> None:
        self.total_runs += 1
        if error:
            self.errors += 1

    @staticmethod
    def percentile(values: list, p: float) -> float:
        if not values:
            return 0.0
        sorted_vals = sorted(values)
        idx = min(int(len(sorted_vals) * p / 100), len(sorted_vals) - 1)
        return sorted_vals[idx]

    def summary(self) -> dict:
        return {
            "latency_p50": self.percentile(self.latencies, 50),
            "latency_p95": self.percentile(self.latencies, 95),
            "latency_p99": self.percentile(self.latencies, 99),
            "cost_avg": sum(self.token_costs) / len(self.token_costs)
                        if self.token_costs else 0,
            "tool_success_rate": self.tool_success / self.tool_total
                                 if self.tool_total else 0,
            "error_rate": self.errors / self.total_runs
                          if self.total_runs else 0,
        }

This AgentMetrics class is the foundation for all subsequent steps--tracing collects data, metrics aggregate it, and evaluation and alerting consume it.

2. Tracing: OpenTelemetry + Langfuse/Phoenix

Metrics tell you "what went wrong." Tracing tells you "where it went wrong." A single Agent run may call 3-5 tools and 2-3 LLMs; without a trace, you cannot pinpoint which step was slow or errored.

Langfuse: @observe Decorator

Langfuse's @observe() decorator is the lightest-weight tracing approach--add one annotation to a function and it automatically captures inputs, outputs, timing, and exceptions:

from langfuse import observe

@observe(name="agent_run")
def run_agent(query: str) -> str:
    research = search_tool(query)
    answer = draft_answer(research)
    return answer

@observe(name="search_tool")
def search_tool(query: str) -> str:
    # Replace with your search API call
    return f"Search results: {query}"

@observe(name="draft_answer")
def draft_answer(context: str) -> str:
    # Replace with your LLM call
    return f"Answer based on {context}"

The outer run_agent automatically becomes a trace; inner search_tool and draft_answer become child spans. In the Langfuse UI you see the complete call tree--each step's input, output, duration, and error status.

Phoenix: register + OpenTelemetry Spans

Phoenix is built on OpenTelemetry. Initialize the tracer with a single register() call:

from phoenix.otel import register
from opentelemetry import trace

# Requires: pip install arize-phoenix-otel
tracer_provider = register(
    project_name="my-agent",
    endpoint="http://localhost:6006/v1/traces",
    auto_instrument=True,
)
tracer = trace.get_tracer("agent")

def run_agent_phoenix(query: str) -> str:
    with tracer.start_as_current_span("agent_run") as span:
        span.set_attribute("input.value", query)
        result = f"Answer: {query}"
        span.set_attribute("output.value", result)
        return result

auto_instrument=True automatically adds tracing for OpenAI, LangChain, and other common libraries. For manual spans, use start_as_current_span and record I/O via set_attribute. Phoenix follows the OpenInference semantic conventions (input.value, output.value, etc.); see the OpenInference specification.

The core difference between Langfuse and Phoenix: Langfuse is more of a "full lifecycle" platform--tracing + prompt management + evaluation + user feedback in one, with simple self-hosting (one Docker command). Phoenix is more "analysis and evaluation"--strongest at trace analysis views, drift detection, and embedding visualization, suited for scenarios requiring deep drill-down. If you are in the LangChain/LangGraph ecosystem, LangSmith has the deepest SaaS integration, but your data leaves your network. The three are not mutually exclusive: a common combination is Langfuse as the primary tracing backend with Phoenix for evaluation analysis.

Parameters for @observe (such as as_type, name) and the return value of register() may vary across SDK versions. Refer to the official docs.

3. Token and Cost Tracking

Multi-step Agent calls can spiral costs out of control--one run calls the LLM 5 times at 2000 tokens each; 1000 runs is 10 million tokens. Without cost tracking, the monthly bill is a nasty surprise.

First, define a pricing table and cost calculation function:

# Pricing table (USD per 1K tokens); update with your model's rates
PRICING = {
    "gpt-4o": {"input": 0.0025, "output": 0.01},
    "gpt-4o-mini": {"input": 0.00015, "output": 0.0006},
}

def compute_cost(model: str, input_tokens: int, output_tokens: int) -> float:
    rates = PRICING.get(model)
    if rates is None:
        return 0.0
    return (input_tokens * rates["input"]
            + output_tokens * rates["output"]) / 1000

Then add tracing to LLM calls to record token usage. Langfuse parses the usage field when as_type="generation":

from langfuse import observe

@observe(name="llm_call", as_type="generation")
def call_llm(model: str, messages: list) -> dict:
    # Replace with your LLM SDK call (e.g., openai.chat.completions.create)
    response = {
        "content": "model response",
        "usage": {"prompt_tokens": 500, "completion_tokens": 200},
    }
    usage = response["usage"]
    cost = compute_cost(model, usage["prompt_tokens"],
                        usage["completion_tokens"])
    return {
        "content": response["content"],
        "cost_usd": cost,
        "input_tokens": usage["prompt_tokens"],
        "output_tokens": usage["completion_tokens"],
    }

For deeper cost optimization strategies (prefix caching, model routing, batch API), see our LLM API Cost Optimization SOP. One production tip: cost tracking should be aggregated per Agent run, not per individual LLM call. A single run may call the LLM 5 times--each call looks cheap, but together they may cost $0.15, and 10,000 runs is $1,500. After recording the cost_usd field in traces, aggregate by trace in the Langfuse dashboard to see p50/p95 per-run cost--that is the real cost from the user's perspective.

4. Tool Calls and Artifact Logging

Tool calls are a high-frequency failure zone for Agents--search APIs return empty, code execution errors out, database queries time out. Every tool call's input and output should be logged as an artifact for post-mortem debugging.

import json
from langfuse import observe

@observe(name="web_search_tool")
def web_search(query: str) -> dict:
    # Replace with actual search API call
    raw_result = {
        "title": "Example Result",
        "snippet": "Relevant content summary",
        "url": "https://example.com",
    }
    return {
        "result": raw_result["snippet"],
        # artifact stores the raw return for debugging
        "artifact": json.dumps(raw_result, ensure_ascii=False),
    }

@observe(name="code_executor_tool")
def execute_code(code: str) -> dict:
    # Replace with sandboxed code execution
    return {
        "stdout": "execution output",
        "exit_code": 0,
        "artifact": code,  # store the executed code
    }

@observe automatically records the function return value in the trace. If the return value includes an artifact field, you can see the full raw tool output in the Langfuse UI. When debugging "why did this search return nothing," just look at the artifact.

5. Online Evaluation: LLM-as-Judge + User Feedback

Offline evaluation runs a fixed test set before deployment (see our AI Agent Evaluation and Benchmarking SOP). Online evaluation runs continuously on live traffic. The two are complementary: offline evaluation catches regressions; online evaluation discovers real-world drift and edge cases. The key design decision for online evaluation is the sampling strategy--you do not run the judge on every trace (too expensive). Instead, sample by rule: 100% of error traces (must review), 10% of normal traces (random sample), 100% of downvoted traces (the most authentic negative signal). This controls judge cost while covering the cases that matter most.

LLM-as-Judge: Automated Scoring

Use one LLM to score another LLM's output, covering a sample (e.g., 10%) of live traces:

from langfuse import observe

@observe(name="llm_judge")
def evaluate_answer(query: str, answer: str) -> float:
    judge_prompt = (
        "Rate the following answer (1-5) for accuracy and helpfulness.\n"
        f"Question: {query}\n"
        f"Answer: {answer}\n"
        "Reply with only a number 1-5."
    )
    # Replace with your judge LLM call
    # Recommendation: use a stronger model than production to avoid self-preference bias
    score_text = "4"  # parsed from judge_response
    return float(score_text)

User Feedback: Thumbs Up/Down

Explicit user feedback is the most authentic quality signal. Use Langfuse's create_score API to attach feedback to a trace:

from langfuse import get_client

def submit_user_feedback(trace_id: str, thumbs_up: bool) -> None:
    langfuse = get_client()
    langfuse.create_score(
        trace_id=trace_id,
        name="user_feedback",
        value=1 if thumbs_up else 0,
        data_type="NUMERIC",
        comment="user thumbs up/down",
    )

After the frontend collects a thumbs up/down, call this function to write the score back to Langfuse. In the dashboard, filter traces by user_feedback score to compare characteristics of downvoted vs. upvoted answers and pinpoint quality bottlenecks.

The create_score parameter signature may vary across SDK versions. Refer to the official docs.

6. Alerting and Iteration: Thresholds and Dashboards

Having data is not enough--you need to detect problems automatically. Define alert rules that trigger notifications when metrics cross thresholds:

from dataclasses import dataclass

@dataclass
class AlertRule:
    metric: str
    threshold: float
    comparator: str  # "gt" (trigger when greater) or "lt" (trigger when less)

    def check(self, value: float) -> bool:
        if self.comparator == "gt":
            return value > self.threshold
        return value < self.threshold

RULES = [
    AlertRule("latency_p95", 10.0, "gt"),        # p95 latency > 10s
    AlertRule("error_rate", 0.05, "gt"),          # error rate > 5%
    AlertRule("tool_success_rate", 0.90, "lt"),   # tool success < 90%
    AlertRule("cost_avg", 0.50, "gt"),            # avg cost > $0.50/run
]

def check_alerts(snapshot: dict) -> list:
    """Check a metrics snapshot and return triggered alerts."""
    triggered = []
    for rule in RULES:
        value = snapshot.get(rule.metric, 0.0)
        if rule.check(value):
            triggered.append(
                f"ALERT: {rule.metric}={value:.4f} "
                f"({rule.comparator} {rule.threshold})"
            )
    return triggered

snapshot is the return value of AgentMetrics.summary() from Step 1. In production, put this check in a scheduled task (every 5 minutes) and send alerts to Slack/DingTalk/email when triggered. Both Langfuse and Phoenix have built-in dashboards--view p95 latency, error rate, and cost trends directly in the UI without building your own Grafana. The core iteration loop for alerting is: set threshold -> alert triggers -> manual investigation -> root cause fix -> adjust threshold. Expect frequent threshold adjustments in the first two weeks; once stable, review monthly.

7. Pitfall Log

Pitfall 1: Looking at averages instead of percentiles. An average latency of 2 seconds looks fine, but the p99 might be 30 seconds--1 in 100 users waits half a minute. Averages get dragged down by many fast requests and hide tail problems. Production must track p95 and p99.

Pitfall 2: Tracing is on but usage is not passed. Langfuse/Phoenix auto-capture depends on SDK integration. If you call the LLM API directly with requests.post instead of the official SDK, token usage is not recorded automatically. Either switch to an instrumented SDK or manually set token attributes on the trace.

Pitfall 3: Deploying LLM-as-judge without calibration. Judge models have their own biases--they may prefer longer answers or certain formats. Without calibrating against human-labeled samples, you might optimize for "what the judge likes" rather than "what users like." Run 50-100 human-labeled samples first, compute judge-human agreement (e.g., Cohen's kappa); do not deploy if below 0.6.

Pitfall 4: Artifact logs ballooning storage. Search results, web HTML, and API JSON responses can be tens of KB or even MB. Logging every artifact in full makes trace storage costs explode. Truncate artifacts over 4KB--keep the first 2000 characters plus an ellipsis.

Pitfall 5: Setting alert thresholds by gut feel. Setting thresholds without baseline data leads to either alert storms (threshold too low) or missed incidents (threshold too high). Collect 1-2 weeks of data first, find where the p95 sits, then set the threshold at 1.5-2x that value.

Pitfall 6: Running Langfuse and Phoenix simultaneously causes interference. Both are built on OpenTelemetry. If you register tracer providers for both without isolation, spans may be duplicated or routed to the wrong backend. Pick one as the primary tracing backend; configure the other with a separate OTEL_EXPORTER_OTLP_ENDPOINT, or distinguish them via resource attributes.

FAQ

Q1: How to choose between Langfuse, Phoenix, and LangSmith? A: Langfuse is an open-source LLM observability platform that can be self-hosted, ideal for teams requiring data to stay on-premise. Phoenix (Arize) is also open-source, with strengths in evaluation and drift detection, suited for evaluation-heavy workflows. LangSmith is LangChain's SaaS, offering the deepest LangChain/LangGraph integration but with vendor lock-in. For self-hosted and framework-agnostic, choose Langfuse; for evaluation-first, Phoenix; for pure LangChain ecosystems, LangSmith. Best used alongside our LangGraph Production Agent SOP.

Q2: Does the @observe decorator impact performance? A: Overhead is minimal--a few milliseconds per span for serialization and async batch upload. Langfuse v4 uses async batch uploading, so traces do not block the main loop. But in high-throughput scenarios (1000+ calls/sec), tune batch parameters (flush_at, flush_interval) or memory will accumulate.

Q3: Which model should LLM-as-judge use? A: Use a stronger model than your production model (e.g., if production uses GPT-4o-mini, judge with GPT-4o) to avoid self-preference bias. For cost-sensitive setups, sample 10% of live traces for judging rather than running on everything. For more evaluation methods, see our AI Agent Evaluation and Benchmarking SOP.

Q4: What if user feedback (thumbs) volume is too low? A: Explicit feedback is naturally sparse--most users do not bother rating. Supplement with implicit signals: answer was copied (satisfied), user re-asked the same question (dissatisfied), session length changed abruptly. Weight implicit signals lower than explicit feedback, but they cover more cases.

Q5: What is the relationship between online and offline evaluation? A: Offline evaluation runs a fixed test set before deployment as a release gate (catching regressions). Online evaluation runs continuously on live traffic as a runtime monitor (catching drift and edge cases). They are complementary: offline tests "known scenarios"; online tests "the real world." Without offline evaluation, regressions slip through; without online evaluation, real-world problems go unnoticed. For budget allocation in cost optimization, see our LLM API Cost Optimization SOP.

Perspective

The core tension of observability is: you want to record everything, but the cost of recording everything (storage, performance, noise) will drag down the system. Good observability is not "record everything" but "record in layers"--traces capture the full chain but at 10% sampling, metrics capture aggregated values at 100%, artifacts capture summaries but not raw large objects. The value of Langfuse and Phoenix is not just the tools themselves, but that they define a set of semantic conventions for LLM observability (what to record, how to record it, at what granularity), so you do not have to design a schema from scratch. One last thought: observability is a prerequisite for an Agent to go to production, not an optional feature. A system you cannot see is a system you dare not let make decisions autonomously.

References

• Langfuse Docs - @observe Decorator and Instrumentation
• Langfuse Docs - LLM-as-a-Judge Evaluation
• Langfuse Docs - Scores (Score API)
• Arize Phoenix GitHub Repository - arize-ai/phoenix
• Phoenix Docs - Tracing Setup (register)
• Phoenix Docs - arize-phoenix-otel SDK
• OpenTelemetry Docs - Python SDK
• OpenInference Semantic Conventions - Arize-ai/openinference
• LangSmith Official Documentation