Gartner's Market Guide for AI Gateways projects that by 2028, 70% of software engineering teams building multi-model applications will run them through an AI gateway, up from 25% in 2025. The interesting part is not the growth rate. It is what the number implies: most enterprises did not choose multi-model AI as a strategy. They arrived at it by accident, one team at a time, and are now discovering they have no single place to control it.
A Dataiku and Harris Poll survey of 600 enterprise CIOs found that 81% expect to depend on two or more large language model providers in 2026, and 93% say different models perform better for different use cases. That is the honest state of enterprise AI. Not one vendor, one contract, one dashboard. Several, all billing separately, all holding your data under different terms.
This article explains what an AI gateway is, what it actually controls, how to tell whether your organisation needs one yet, and what goes wrong when the decision is made badly.
What is an AI gateway?
An AI gateway is a single control point that sits between your applications and every AI model your organisation uses. Gartner defines it as an intermediary that simplifies and manages access to AI capabilities, providing one place to enforce security, governance and observability across AI workloads. Every request passes through it before reaching a model.
The comparison most IT directors find useful is the corporate firewall. Nobody argues about whether a 300-person company should route internet traffic through one controlled boundary. An AI gateway applies that same principle to model traffic.
Without one, each application holds its own API keys, picks its own model, and bills to its own budget line. Nobody can answer a simple question at board level: how much are we spending on AI this month, and on what.
Why does an AI gateway matter now, in 2026?
It matters because the multi-model reality arrived faster than the governance to manage it. When 81% of CIOs run two or more providers, the practical outcome is fragmented spend, inconsistent security policy, and no unified record of what any model was asked or answered. The gateway is the layer that re-consolidates that sprawl.
Three forces converged in the last eighteen months.
--- Model churn became normal. Providers deprecate and re-release models on cycles measured in months, not years. Applications hard-wired to one model identifier break when it retires.
--- Pricing stopped being stable. Per-token costs have fallen sharply across the major providers, which means the cheapest correct model for a given task changes several times a year.
--- Regulators started asking for records. Hong Kong's PCPD has been explicit that organisations deploying AI on personal data must be able to show what was processed and on what basis.
Each of those is manageable in isolation. Together, they make per-application AI integration a liability.
How does an AI gateway actually work?
An AI gateway exposes one internal endpoint to your developers and applications. Requests arrive at that endpoint, the gateway applies policy, then forwards the request to whichever model provider the policy selects. The response returns through the same path, where it is logged, filtered and metered before reaching the caller.
In practice that means your applications stop knowing which model they are talking to. They call an internal address. The gateway decides the rest.
That indirection is the whole point. When a provider raises prices, retires a model, or suffers an outage, the change happens in gateway configuration rather than in the source code of every application that touches AI.
The operational difference shows up during incidents. A logistics firm running order-status summarisation through three applications, each holding its own provider key, has three separate outages to diagnose when a provider degrades. The same firm behind a gateway has one configuration change and a fallback model.
Deployment usually takes one of two shapes. A self-hosted gateway runs inside your own network, which keeps prompts and responses on infrastructure you control and suits regulated workloads. A managed gateway is faster to stand up but routes your traffic through a third party, which becomes a data-processing question your privacy team will need to answer before, not after, procurement.
What are the four jobs an AI gateway performs?
An AI gateway performs four distinct functions: routing, security, cost control and observability. Most vendor conversations focus on routing because it demonstrates well. The other three are where the operational value sits, and they are the reason finance and risk teams end up caring about a piece of infrastructure they never asked for.
Routing. The gateway directs each request to the most suitable model based on policy: cost, latency, accuracy, or data-sensitivity classification. A summarisation task can go to a cheap small model while a contract-analysis task goes to a frontier model.
Security. API keys live in the gateway, not in application code or a developer's laptop. Key rotation becomes a configuration change. Input and output guardrails, including checks for prompt injection and for personal data leaving the organisation, apply uniformly instead of being re-implemented by each team.
Cost control. The gateway meters token consumption per team, per application and per use case, enforces quotas, and caches repeated calls. This is the first time most organisations can attribute AI spend to a cost centre.
Observability. Every request and response is logged with the metadata needed to reconstruct a decision later. Without this, an internal audit question about an AI-influenced decision has no answer.
How is an AI gateway different from an API gateway?
An API gateway routes and secures traffic based on endpoints, methods and rate limits. An AI gateway does all of that, then adds the things that only matter for model traffic: token accounting, semantic caching, prompt and response inspection, model fallback, and per-request cost attribution. The difference is that AI traffic is priced by content volume, not by call count.
That single pricing difference breaks most assumptions built into traditional API management. A rate limit of 1,000 calls per hour means nothing when one call can cost two hundred times another.
An AI gateway is also distinct from an orchestration framework. Orchestration frameworks help developers build agent logic. A gateway governs traffic regardless of what built it, including traffic from tools your IT team did not procure.
What does an AI gateway change for a Hong Kong enterprise?
For Hong Kong organisations the gateway converts three recurring compliance problems into configuration. It provides the processing record the PCPD expects, it enforces which categories of data may leave the jurisdiction, and it gives regulated firms a defensible answer when an examiner asks how an AI-influenced decision was reached.
The local adoption context makes this urgent rather than theoretical. The Standard Chartered Hong Kong SME Leading Business Index for Q1 2026, published by HKPC, found that 55% of surveyed SMEs have used or plan to use AI tools within the next year, with Information and Communications at 92%, Professional and Business Services at 72% and Financing and Insurance at 62%.
Adoption at those levels inside a 300-person firm does not arrive through one procurement decision. It arrives through dozens of individual sign-ups, which is precisely the condition an AI gateway exists to correct. You can read more on that specific exposure in our guide to shadow AI and the governance gap it creates.
How do you decide whether you need an AI gateway yet?
Five questions settle the decision. If you answer yes to three or more, the gateway is overdue. If you answer yes to fewer than two, you are still early enough that a gateway adds operating cost without removing meaningful risk, and the honest recommendation is to wait.
--- Do two or more model providers appear on your monthly bills?
--- Can you attribute AI spend to a department without a manual reconciliation exercise?
--- If a provider retired a model tomorrow, how many codebases would need changing?
--- Could you produce, within one working day, a record of every AI request that touched customer data last month?
--- Do teams outside IT hold their own API keys?
The fourth question is the one that changes board conversations. Most organisations discover the answer is no, and that the reason is architectural rather than procedural. Nobody decided against keeping records. The records were simply never centralised, because each team integrated AI the fastest way available to it.
A useful sequencing rule: the gateway should arrive after your first genuine production AI workload and before your third. Earlier and you are governing an experiment. Later and you are retrofitting control onto systems that already carry customer traffic, which costs materially more and usually requires a change freeze nobody wants to schedule.
What goes wrong when organisations get this wrong?
Five failure patterns account for most disappointing AI gateway deployments. All five come from treating the gateway as a procurement item rather than an operating decision about who controls model access and who is accountable for the spend it reveals.
Treating it as a cost-savings project. Routing optimisation typically trims a meaningful share of inference spend, but the durable value is governance. Justify it on control and audit, or the business case collapses the first quarter savings underperform.
Deploying it without mandating it. A gateway that teams may bypass governs nothing. If direct provider access remains open, the shadow traffic simply continues.
Adding latency nobody budgeted for. Every gateway adds a hop. For batch work this is irrelevant. For a customer-facing voice application, it needs measuring before commitment.
Buying a gateway before having a policy. The gateway enforces rules. It does not write them. Organisations that install one before deciding which data classifications may reach which models end up with an expensive proxy.
Ignoring the observability output. The logs are the asset. Teams that never review them keep the risk and pay for the infrastructure. Our guide to agent observability covers what a usable trace must contain.
The strategic takeaway
The AI gateway is not an exciting technology. It is plumbing. But it is the specific piece of plumbing that turns a collection of departmental AI experiments into something a board can be given a number for, and something a regulator can be given a record of.
The decision is not really about software. It is about whether your organisation is willing to say that AI model access is a governed resource rather than a personal one. Most enterprises reach that conclusion eventually. The ones that reach it before their first audit question spend considerably less getting there.
Technology cycles reward the organisations that build the boring layer early. We understand AI. We understand you. With UD by your side, AI never feels cold.
Reviewed by the UD enterprise AI team, Hong Kong.
Where to start
Now that you have the framework, the next step is identifying where model access is already ungoverned inside your organisation. We'll walk you through every step, from mapping current AI usage to policy design, deployment and ongoing cost reporting, backed by 28 years of Hong Kong enterprise experience.
