Why Traditional APM Can't See Your Agents
Traditional APM was built on one assumption: a request follows a predictable path, and failures surface as exceptions or 5xx responses. AI agents break that assumption on every count.
Non-deterministic, multi-step execution: The same question may finish in 3 tool calls or spiral into 17, depending on what the planning step decides. Average latency tells you nothing about what actually happened in the p99 tail.Instrumentation gaps: Tool calls, vector searches, and agent-to-agent handoffs often happen outside the application framework, so auto-instrumentation misses them entirely.The shape of failure: The most dangerous failure isn't an exception — it's a plausible wrong answer returned with HTTP 200. Status-code-based alerting will never catch it.Agent operations need observability aimed at "did it answer correctly?" rather than "is it still alive?"
The Instrumentation Standard: OpenTelemetry GenAI Semantic Conventions
OpenTelemetry defines semantic conventions for generative AI workloads. They're still experimental, but aligning your attribute names to the standard means your data survives a change of observability vendor.
Designing the Span Structure
Treat a single user request as the root span, then nest planning → model inference → tool call → validation → final response as child spans. Each tool-call span should carry the tool name, an input summary, retry count, and failure reason.
Required Attributes
Model identifier and request parameters (temperature, max_tokens)Input, output, and reasoning token counts, plus cache-hit statusSession ID and conversation turn number, so multiple requests roll up into one sessionPrompt and response bodies stored separately, assuming sampling and maskingLocking prompt logs inside a vendor-specific SDK means losing three years of quality history the moment you migrate. Collecting them under standard attributes is the cheapest insurance available.
Building a Continuous Evaluation Pipeline
Passing a 100-case test suite before deployment guarantees nothing about production quality. Real user input follows a different distribution, and a model version bump alone can shift response behavior.
Sampled scoring: Route 5–10% of traffic through an LLM-as-judge and attach accuracy, citation-grounding, and format-compliance scores back onto the trace.Human review queue: Send only the cases below a score threshold or with low judge confidence. At 100,000 requests a day, that keeps the review queue in the dozens.Validating the judge: The scorer itself needs periodic agreement checks against 200–300 human-labeled cases. If the judge drifts, every metric drifts with it.Golden-set regression: Promote every human-confirmed failure into the regression suite immediately so the next release proves it won't recur.Cost, Quality, and Safety on One Screen
Once token counts and tool-call counts live on the trace, you can compute cost per request directly. This unit-economics view usually delivers results fastest in practice.
Find the window where average tool calls jumped from 4 to 9, and you can pinpoint the prompt change behind it.A/B test combinations that cut reasoning tokens while holding quality scores flat.Link policy violations, PII exposure, and prompt-injection detections to the same trace ID in your governance log, and post-incident audits shrink from days to minutes.An Adoption Roadmap
Phase 1 — Tracing: Start by collecting requests, tool calls, and tokens through OpenTelemetry. Baseline visibility typically lands in two to three weeks.Phase 2 — Evaluation sets: Pull 200–500 representative cases from real traffic, build the golden set, and document the scoring rubric.Phase 3 — Automated regression watch: Add a quality gate to the deployment pipeline and alert on score drops and cost spikes.At POLYGLOTSOFT, we design tracing, continuous evaluation, cost dashboards, and governance logging as one standard package whenever we build and operate AI systems. If you want to audit the observability of an agent already in production, or start a new AI project on reliable footing, please reach out. We'll assess your current setup and map out a phased adoption plan with you.
