A logistics group in Kwun Tong has an AI agent running invoice reconciliation. It has been live for four months. Last Tuesday it approved a duplicate payment of HK$180,000.
The operations director asks the obvious question: which step went wrong? Nobody can answer. The reasoning is gone. All that survives is a log line saying the task completed successfully.
That gap has a name. It is called agent observability, and in 2026 it is one of the most common reasons enterprise AI works in a pilot and stalls in production.
What is agent observability?
Agent observability is the practice of capturing structured telemetry at every step of an AI agent's reasoning and execution path, from the prompt it received through each retrieval, model call and tool action, to the final output. It allows you to reconstruct what an agent did, why it did it, and what it cost.
The distinction that matters for a decision-maker is between a log and a trace. A log records that something happened. A trace records the whole decision tree that produced it.
An agent run is not one request and one response. It is a branching sequence: the model decides to search a document store, reads three results, calls an internal API, reconsiders, calls a second API, then writes an answer. Each of those is a separate step with its own inputs, outputs, latency and cost.
Observability means every one of those steps is captured as a linked record you can replay afterwards.
Why does agent observability matter now, in 2026?
Because agent deployment has outrun agent supervision. Organisations are moving agents into live operations faster than they are building the ability to inspect them, and the evidence now shows the supervision gap, not model quality, is what blocks production. Regulators in Hong Kong have moved in the same direction over the past year.
According to McKinsey's State of AI Trust in 2026 report, 62% of organisations are at least experimenting with agents and 23% are already scaling them, but only 30% reach maturity level three or higher in agentic AI governance.
The same report finds that nearly two-thirds of respondents name security and risk as the top barrier to scaling agentic AI, ranking it ahead of regulatory uncertainty.
The production numbers are starker. IDC research cited in July 2026 found that 88% of AI agent proofs-of-concept never reach broad production: for every 33 pilots launched, roughly four enter live operation.
Gartner's 2026 CIO and Technology Executive Survey found only 17% of organisations have fully deployed AI agents, while more than 60% expect to within two years. That is a very large cohort about to discover whether they can audit what they deployed.
How is agent observability different from AI evals and traditional monitoring?
They answer different questions at different times. Evals ask whether an agent is good enough before you ship it. Monitoring asks whether the system is up. Observability asks what a specific agent run actually did after the fact. An organisation can hold all three and still be blind, because each covers a different window.
Traditional application performance monitoring was built for deterministic request and response paths. It measures latency, error rate and throughput. Those metrics are still useful, and they are also close to useless for diagnosing a wrong answer, because a wrong answer usually returns HTTP 200 in normal time.
Evaluation is a pre-deployment and periodic discipline. If you want the framework for that, UD has covered it separately in What Is an AI Eval?
Observability is the runtime layer. It is what makes an incident reconstructable six weeks later when a client, an auditor or a regulator asks.
--- Monitoring answers: is the service healthy?
--- Evals answer: is the agent accurate enough to deploy?
--- Observability answers: what exactly happened in this run, and where did it go wrong?
What does an agent trace actually contain?
A usable trace captures the full span tree of one agent run, with each model call, retrieval and tool invocation recorded as its own node. Without step-level capture you cannot answer the basic forensic question of which step produced the bad output. This is the practical test to apply to any vendor claim.
The elements a trace should contain are consistent across serious platforms:
--- The input: the user prompt, the system instructions, and any context injected at runtime
--- Each retrieval: what was searched, which documents came back, and their relevance scores
--- Each model call: which model version, which parameters, the tokens in and out, and the cost
--- Each tool call: which system was touched, with which arguments, and what it returned
--- The decision points: where the agent chose one branch over another
--- The identity: which user, service account or agent credential authorised the run
That last item is the one enterprise buyers most often omit from their requirements, and it is the one auditors ask about first.
What do Hong Kong regulators expect you to be able to show?
Hong Kong has no single AI statute. Instead, expectations sit inside existing regimes, and those expectations tightened through the first half of 2026. For a regulated institution the practical requirement is an explainable, auditable decision trail that stands up to examination, not a policy document saying one exists.
The Hong Kong Monetary Authority expects authorised institutions to maintain explainable, auditable AI decision trails, meaning the system must be able to show what data informed each decision in a form a supervisor can examine.
The Privacy Commissioner for Personal Data has separately turned attention to agentic AI specifically, noting that agents may reach local devices, files, email, credentials, browser contents and external services, and may execute multi-step tasks without real-time human involvement. Its recommendations include minimum access rights, no administrator privileges, separated runtime environments, and human review of decisions with significant individual impact.
Read together, the message is straightforward. If your agent touches personal data and you cannot reconstruct its actions, you do not have a compliance position. You have an assumption.
UD's separate treatment of the control side is in How to Govern AI Agents.
How do you build an agent observability capability in 90 days?
Start with one agent, not a platform purchase. The sequence that works in mid-market Hong Kong organisations is to instrument a single production agent end to end, prove you can replay one real incident, then extend the same pattern outward. Buying a platform before you can replay one trace usually produces a dashboard nobody trusts.
Days 1 to 30: instrument one agent
Pick the agent with the highest financial or client exposure. Capture the full span tree for every run, including cost per step. Retain traces for at least the period your longest audit cycle requires.
Days 31 to 60: prove replay
Take one real failure from the last month and reconstruct it entirely from the trace. If you cannot, your instrumentation is incomplete, and you have learned that cheaply.
Days 61 to 90: attach it to accountability
Name an owner for each agent. Define which failure classes trigger human review. Publish one monthly report that shows run volume, failure rate by class, cost per successful outcome, and open incidents. That report is what converts telemetry into a management instrument.
What goes wrong when organisations skip observability?
Five failure patterns recur, and all five surface late, after the agent has been trusted with real work. Each one is cheap to prevent at instrumentation time and expensive to fix once an incident is already under review.
--- Silent quality decay. A model version changes, retrieval quality shifts, and accuracy drops for weeks before anyone notices, because uptime stayed at 100%.
--- Unreconstructable incidents. The organisation knows an agent caused a loss and cannot demonstrate how, which turns a technical problem into a legal exposure.
--- Cost drift. Nobody attributes spend to individual steps, so an agent that quietly retries five times per run looks like a general budget overrun rather than a fixable bug.
--- Diffused ownership. The IT team says the business owns the workflow, the business says IT owns the system, and no one owns the output.
--- Governance theatre. A policy states that agents are supervised. No trace exists to prove it. The policy fails at the first audit.
What should you do next?
Ask one question at your next technology review: pick a specific agent run from last month and ask your team to show you the trace. The answer tells you more about your production readiness than any maturity score. If the trace exists, you are in the 30% McKinsey describes. If it does not, you now know your first project.
Agent observability is unglamorous work. It builds no new capability and demonstrates nothing in a board slide. It is also the difference between an AI programme you can defend and one you can only describe.
Twenty-eight years of Hong Kong enterprise technology has taught us that the systems which survive are the ones whose operators can explain them. We understand AI. We understand you. With UD by your side, AI never feels cold.
Reviewed by the UD enterprise AI team.
Ready to run AI agents you can actually audit?
Knowing what a trace should contain is one thing. Standing up agents in an environment where every action is captured, permissioned and reviewable is another. We'll walk you through every step, from readiness assessment and secure sandbox deployment to monitoring, cost attribution and the monthly report your board will ask for. Twenty-eight years serving Hong Kong enterprises, with you the whole way.
