Whitepaper·01AI LTD · Dublin, Ireland·August 2026

Follow the Prompt

A Data-Flow Analysis of the Enterprise AI Stack, and the Intermediaries You Never Vetted

What looks like one call to the LLM is a chain of intermediaries. Each hop optimises token cost or latency, and each hop reads your prompt in plaintext to do it. Trace a single request and you find parties you never contracted with, retention you never configured, and borders you never authorised it to cross.

Abstract

The economics of enterprise AI now run on token optimisation. Sending every request to the most capable frontier model is expensive, so a layer of intermediaries has grown up to make it cheaper: AI gateways that cache and meter, LLM routers that pick the cheapest model that can do the job, and model marketplaces that present one billing surface over dozens of providers. Each of these is sold as a cost or convenience feature. Each of them is also a party that reads your prompt.

This is the middleman problem. A request that the architecture diagram draws as a single arrow to "the LLM" is in practice a chain of three to five hops, and the data-protection posture of the whole chain is set by its weakest link. Most organisations cannot draw the chain. They know which router they signed up for; they do not know which provider served last Tuesday's traffic, in which country, under whose terms, with what logging enabled.

The method that surfaces the problem is old and boring: data-flow analysis. Follow one prompt, hop by hop, and ask three questions at each hop. Who can read this in plaintext? Where does it get written to disk, and for how long? On which physical server, in which jurisdiction, did the computation actually run? The answers are uncomfortable, because the optimisation layer is built precisely on the ability to read, cache, and relay the traffic it is optimising.

This paper walks the trace and names the five leak surfaces it exposes: the decryption point at every hop, the gap between logging off and data off, residency blindness, the unvetted sub-processor chain behind a marketplace listing, and the attribution gap that makes a subject-access request unanswerable. None of the five is exotic. Each maps to a control the enterprise already applies to its other data processors. The work is extending those controls to a traffic path that was drawn as a single arrow.

The single arrow that is really a chain

Ask an engineering team to draw its AI architecture and you will get a box for the application, a box for "the model," and an arrow between them. The arrow is a lie of omission. It compresses a routing layer that the team assembled deliberately, one convenience at a time, until the direct path to a single provider had been replaced by an indirect path through several.

The layer accumulates for good reasons. Frontier tokens are expensive, so a router is added to send easy requests to a cheaper model and reserve the expensive model for the hard ones. Spend needs to be tracked across teams, so a gateway is added in front to meter it. Latency spikes and provider outages hurt, so a fallback provider is configured to absorb them. A second cheap provider appears on a marketplace at half the price, so it is added to the routing pool. Each decision is locally sensible and individually cheap. The cumulative effect is that the organisation no longer has a direct relationship with the party that runs its inference.

This is the same shape as the intermediary layers that grew up in advertising technology and in payments, and it carries the same governance hazard. The value of the layer is that it abstracts away the messy detail of who actually serves the request. The cost of the layer is that it abstracts away the messy detail of who actually serves the request. For a marketing pixel that abstraction is a convenience. For a prompt that contains a customer record, a draft contract, or a line of proprietary code, the abstraction is the exact thing a data-protection regime is designed to prevent.

The rest of this paper does the un-abstracting. It follows one prompt through the chain and looks at what each hop can see, keep, and move.

Follow the prompt: the trace

Take one concrete request. An internal support copilot answers a staff question, and to ground its answer it pulls in a customer record. That record is now inside the prompt. Here is where the prompt goes, and what each hop can do with it. Every hop below sees the payload in plaintext, because every hop has to.

Your application

Assembles the promptReads: Everything, by definition

The prompt is built here: the user question, the retrieved context, the system instructions, and often a slice of a customer record or an internal document pulled in for grounding. This is the only hop you fully own and fully control.

AI gateway / proxy

Caching, logging, analytics, rate limitsReads: Full plaintext

A self-hosted or managed gateway sits in front of everything to give you a dashboard, spend controls, and a cache. To do any of that it terminates the request and reads the body. If it is a managed gateway, that is your first external party.

LLM router

Picks the cheapest or best-fit modelReads: Full plaintext

The router decrypts the prompt to count tokens, classify the task, run moderation, and decide which downstream model wins. It cannot route what it cannot read. Its routing decision is invisible to you unless it returns one.

Model marketplace

Abstracts the provider and billingReads: Full plaintext

The marketplace presents one API over many providers. It relays your prompt to whichever model the router chose, applies its own terms, and meters the call. The provider identity behind the marketplace listing is often not surfaced to you.

Inference provider

Runs the computationReads: Full plaintext

The actual model host. This is where the payload lands on a physical GPU to do the maths. It may be a first-party lab, a reseller, or a GPU cloud in a different jurisdiction from every hop above it.

Fallback provider

Absorbs errors and overflowReads: Full plaintext

On a timeout, rate limit, or outage at hop 4, the router silently re-sends the same prompt to an entirely different provider. Your data reaches a party that never appeared in the happy-path diagram and may sit under different terms and a different flag.

Six hops. One contract, at hop 1 or hop 2, with the vendor you actually signed up with. The customer record inside the prompt was read in plaintext at all six, may have been written to disk at three of them, and finished its journey on a physical server whose location you were never told. The architecture diagram still shows one arrow.

Leak surface 1

Every hop is a decryption point

Transport encryption protects the prompt between the hops, not at them. TLS secures the link from your application to the gateway, and a second TLS session from the gateway to the router, and a third from the router to the provider. At each junction the traffic is decrypted, processed, and re-encrypted for the next leg. The encryption is hop-by-hop, not end-to-end, and it cannot be otherwise, because the intermediary's entire function is to read the payload.

A router cannot route what it cannot read. To choose a cheaper model it counts the tokens, which means parsing the text. To enforce a moderation policy it inspects the content, which means understanding the text. To serve a cache hit it fingerprints the prompt, which means storing a representation of the text. There is no version of the optimisation layer that operates on ciphertext. The plaintext window at each hop is not a bug or a misconfiguration. It is the feature you are paying for.

This turns every intermediary into an in-memory copy of your most sensitive traffic, and therefore into a breach target. If the router's infrastructure is compromised, the attacker does not need to break your encryption or reach your database. The raw prompts, and the raw completions, pass through that infrastructure in the clear on their way through. The blast radius of a breach at hop 2 is every prompt that every one of the router's customers sent while the compromise was live.

The human-world parallel is a mail-forwarding service that opens and reads every envelope in order to decide which courier to use. The service may be entirely trustworthy and entirely competent. It is still a place where your correspondence exists in readable form, outside your control, and its security is now part of yours.

The corrective

Count the decryption points and treat each one as an entity that holds your data in the clear. For each hop you do not operate yourself, you need the same assurance you demand of any processor with plaintext access: a contract, a security attestation, a breach-notification clause, and a defensible answer to why that party needs to see the payload at all. Hops that exist only for convenience, and hold plaintext they do not need, are the first candidates to collapse out of the chain or bring in-house.

Leak surface 2

Logging off is not data off

To trace a prompt honestly you have to follow it to the point where it stops being traffic and starts being a record: the moment it is written to a disk. Many intermediaries write the full text of prompts and completions to storage by default, because that is what powers the developer dashboard, the analytics, the debugging view, and the evaluation datasets that make the product useful. The logging is the feature, again.

There is a control for this, and it is routinely misunderstood. Toggling off logging or opting in to a zero-retention arrangement changes what the intermediary keeps. It stops the prompt being persisted to the vendor's long-term storage, which is a real and worthwhile reduction. What it does not change is what the intermediary sees. The prompt is still decrypted, still parsed, still resident in memory and in transient buffers for the moments it takes to route and relay. The retention setting governs the hard drive. It does not govern the wire or the RAM.

This distinction matters because organisations treat the retention toggle as if it removed the party from the data-flow diagram entirely. It does not. A zero-retention router is still a plaintext reader of every prompt (leak surface 1); it has merely promised not to keep a copy. The promise is only as good as the vendor's controls, its honesty, and its ability to enforce the same setting across the downstream providers it relays to. A retention guarantee at the router says nothing about retention at the marketplace or the inference provider behind it, unless it has been contractually flowed all the way down.

The failure mode is a false sense of closure. A team enables zero retention on the one vendor it has a dashboard for, records the risk as mitigated, and never asks whether the fallback provider at hop 5, reached only during an outage, honours the same setting. Retention is not a single switch. It is a property of every hop, and it has to be verified at every hop, or it is verified at none.

The corrective

Separate two questions and answer both, per hop. First, who can read the prompt in transit? Second, who writes it to disk, and for how long? A zero-retention setting answers the second question for one hop and says nothing about the first, or about any other hop. Require retention terms to be flowed down the whole chain in writing, including to fallback and overflow providers, and verify them rather than assuming the toggle in your dashboard governs parties you have no dashboard for.

Leak surface 3

Residency blindness

A prompt is not an abstraction. It is a physical payload that travels through cables to a specific machine in a specific building in a specific country, where a processor does the arithmetic and produces a response. Following the prompt to the end forces the question that the optimisation layer is designed to make you stop asking: where did this specific payload land to do the computation?

The router's job is to make that answer vary. It sends easy requests to whichever provider is cheapest right now, and the cheapest provider is often cheap because of where it runs. A marketplace listing that undercuts the incumbent may be served from a GPU cloud in a jurisdiction with no adequacy finding, no equivalent data-protection regime, and a legal environment in which a local authority can compel access to data on servers within its borders. When the router chooses that provider, your prompt has crossed a legal border, and it has done so as an automated side effect of a cost-optimisation policy that no one framed as an international transfer decision.

Under GDPR this is not a technicality. Moving personal data outside the European Economic Area is a regulated act that requires a lawful transfer mechanism, a transfer impact assessment, and, after Schrems II, an honest evaluation of whether the destination country's laws undermine the safeguards on paper. The regulators have been narrowing the room for vagueness. The European Data Protection Board's Opinion 28/2024, on personal data in the context of AI models, treats data-protection obligations as following the data into the AI processing context and turns on a case-by-case assessment, and a controller cannot even begin that assessment without knowing which parties processed the data and where. None of it is possible if you cannot say which country the payload landed in. And you frequently cannot, because the marketplace abstracts the provider and the router abstracts the marketplace, and neither is contractually obliged to tell you where any individual request was served.

Residency blindness is the precise name for this state: the organisation has a data-residency commitment on paper, and no ability to demonstrate that any given request honoured it. The test is concrete. Ask the router for a signed, per-request record of the physical location where the computation ran. If it can produce one, you have residency evidence. If it can only point you to a policy page describing where it might run, you have residency blindness, and every claim you make to a regulator or a customer about where your AI traffic is processed is an assumption rather than a fact.

The corrective

Treat routing to a new provider as what it is: a transfer decision. Pin routing to a set of endpoints whose processing locations you have verified and whose transfer mechanisms you hold. Require per-request residency evidence rather than a policy-page assurance, and refuse to add providers to the pool that cannot produce it. Where the workload touches regulated personal data, constrain the router to region-locked endpoints even at the cost of the cheaper token, because the saving on the token is not worth the loss of the answer to where the data went.

Leak surface 4

The unvetted sub-processor chain

A model marketplace sells the convenience of one integration for many models. What it also sells, without naming it as such, is a sub-processor relationship that you did not individually assess. When you route through a marketplace, the providers it relays to are, in data-protection terms, your sub-processors: parties that process your data on your behalf, at one remove, through your chosen intermediary.

In every other part of the enterprise, sub-processors are governed. A vendor that wants to add one has to disclose it, and often has to give you a window to object. You keep a register of who processes your data and under what terms. The marketplace pattern quietly bypasses this discipline. The list of models behind the listing changes as providers are added and prices move, and the effective sub-processor set for your traffic changes with it, without a disclosure, without an objection window, and without an entry in your register.

The provenance question compounds the terms question. A model offered on a marketplace may be hosted by the lab that made it, or by a reseller, or by a third party that has taken an open-weights model and stood it up on its own hardware. These arrangements carry materially different postures on training use, retention, security, and jurisdiction, and the marketplace listing rarely surfaces which one you are getting. The same model name can be a different processor, in a different country, under different terms, depending on which listing the router selected on price.

This is the AI-specific form of a supply-chain risk that the security community already takes seriously everywhere else. The OWASP Top 10 for LLM applications lists supply chain as a first-tier category precisely because the components behind an AI feature are assembled from parties the deploying organisation did not vet. A marketplace is a supply chain wearing the interface of a single vendor, and the single interface is exactly what makes the underlying multiplicity easy to miss.

The corrective

Maintain a register of the actual providers your traffic can reach, not just the marketplace you contracted with. Require the intermediary to disclose its downstream set and to notify you when it changes, the same standard you hold every other processor to. Vet providers on provenance and terms before they enter the routing pool, and treat the marketplace's convenience layer as a reason for more scrutiny of what sits behind it, not less.

Leak surface 5

The attribution gap: you cannot answer where it went

The previous four surfaces converge on a single operational failure: when someone asks where a specific piece of data went, you cannot tell them. This is the attribution gap, and it is the one that turns an architecture problem into a compliance problem, because the questions that expose it come from outside and carry deadlines.

A data-subject access request asks what personal data you hold and process, and where. A record of processing activities, required under Article 30, asks you to document the categories of recipient and the transfers to third countries. A customer's security questionnaire asks which sub-processors touch their data. A regulator's inquiry after an incident asks for the flow. Each of these presumes that you can reconstruct, for a given request or a given category of data, the path it took and the parties it reached. The middleman chain, left unmapped, makes that reconstruction impossible.

The router chose a provider on the fly and did not return the choice. The marketplace relayed to a host it did not name. The fallback fired during an outage you did not log. Three months later the subject-access request arrives, and the honest answer to "which parties processed this data, and in which countries" is that you do not know, because the layer you added to save on tokens was not built to tell you, and you did not require it to. The saving was real and measurable. The answer you traded for it was neither, until the moment you needed it.

Attribution is what makes every other control auditable. A retention promise you cannot verify per hop, a residency commitment you cannot evidence per request, and a sub-processor register you cannot reconcile against actual traffic are all the same failure seen from different angles: the chain acted, and it did not record what it did in a form you can stand behind later.

The corrective

Log the chain, not just the call. For every request, capture which provider served it, in which region, under which retention terms, and whether a fallback fired. Keep that record at the hop you control (your gateway) so that it survives regardless of what the downstream intermediaries choose to expose. The test is simple and worth rehearsing before a regulator does it for you: pick one request from last month and reconstruct its full path. If you can, you have attribution. If you cannot, you have a data-flow diagram that stops at the first arrow.

What the trace proves

The five surfaces are not independent findings. They are one finding seen at five points along a single path. Decryption at every hop is what makes the traffic readable. Default logging is what makes the readable traffic persistent. Cheapest-provider routing is what makes the persistent traffic mobile across borders. The marketplace abstraction is what makes the mobile traffic reach parties you never assessed. And the absence of chain-level logging is what makes all of it unaccountable after the fact.

Closing any one surface without the others moves the risk rather than removing it. Zero retention with unpinned routing gives you a prompt that is not stored but still crosses borders you cannot evidence. Region-locked routing with no chain-level logging gives you a compliant path you cannot prove you took. A vetted provider list with default logging at the gateway gives you a clean supply chain feeding a plaintext store you forgot you were keeping. The surfaces have to be closed together, because they are the same opening described from different sides.

The token saving that motivates the whole layer is real. Routing easy work to cheaper models is a legitimate and often substantial economy, and this paper is not an argument against it. It is an argument that the saving has a ledger entry on the other side, and that the entry is denominated in a currency (readable traffic, persistent records, cross-border movement, unvetted processors, lost attribution) that does not appear on the invoice the router sends you. The optimisation is worth doing. It is not worth doing blind.

Data-flow analysis as the discipline

The method that closes the gap is the same one that opened it: follow the prompt. Data-flow analysis is unglamorous and it predates AI by decades, which is exactly why it works here. It does not require a novel AI-governance paradigm. It requires drawing the arrow correctly and then governing what the corrected drawing reveals.

Map the real chain

Replace the single arrow with the actual hops. For each request path, list every party that reads, stores, or relays the prompt, including fallback and overflow providers that only appear under load. You cannot govern a chain you have not drawn.

Classify what flows

Not every prompt carries regulated data. Know which paths can carry personal data, confidential material, or source code, and apply the strictest controls to those paths rather than uniformly throttling everything.

Contract every plaintext reader

Every hop that decrypts the payload is a processor. Hold each to the standard you already apply to processors: a data-processing agreement, a flowed-down retention term, a breach-notification clause, and a disclosed sub-processor set.

Pin routing to verified endpoints

Constrain the router to providers whose processing locations and transfer mechanisms you have verified. Treat adding a provider to the pool as a transfer decision that requires assessment, not a pricing update that happens automatically.

Log the chain at the hop you own

Record which provider served each request, in which region, under which terms, and whether a fallback fired, and keep that record at your own gateway. Rehearse the reconstruction of a single request before a regulator or a customer asks you to perform it live.

Five steps. None of them new. The organisations that will run the optimisation layer safely are not the ones that refuse routers and marketplaces, because the token economics make that refusal untenable at scale. They are the ones that keep an honest diagram of where their prompts go, and hold every hop on that diagram to the standard they already hold their other processors to. The saving is kept. The blindness is not.

Conclusion: draw the arrow correctly

The middleman layer in enterprise AI is not going away. Token optimisation is a permanent economic pressure, and routers, gateways, and marketplaces are the rational response to it. The layer will grow, not shrink, and more of every organisation's AI traffic will pass through parties it does not directly control.

That makes the discipline of following the prompt more important over time, not less. The single arrow in the architecture diagram is where the risk hides, because it hides the chain. Every leak surface in this paper lives in the gap between what the arrow shows and what the traffic actually does, and every one of them is closable by the plain act of drawing the chain and governing it hop by hop.

We expect the next wave of AI-related data-protection findings to come not from model behaviour but from the transport layer beneath it: prompts read at intermediaries no one registered, personal data moved across borders by routing policies no one framed as transfers, and subject-access requests that cannot be answered because the chain kept no record of itself. These are not model failures. They are data-flow failures, and data-flow analysis is the fifty-year-old discipline that prevents them.

The optimisation is worth the money. The blindness is the part that is not, and it is optional. Follow the prompt, and it stops being blind.

References

European Parliament & Council (2016). "Regulation (EU) 2016/679: General Data Protection Regulation." Articles 28 (processors), 30 (records of processing), and 44–49 (international transfers).
European Data Protection Board (2024). "Opinion 28/2024 on certain data protection aspects related to the processing of personal data in the context of AI models." Adopted 17 December 2024.
European Data Protection Board (2021). "Recommendations 01/2020 on measures that supplement transfer tools to ensure compliance with the EU level of protection of personal data." Version 2.0.
Court of Justice of the European Union (2020). "Case C-311/18, Data Protection Commissioner v Facebook Ireland Ltd and Maximillian Schrems (Schrems II)."
European Parliament & Council (2024). "Regulation (EU) 2024/1689: Artificial Intelligence Act."
IBM Security & Ponemon Institute (2025). "Cost of a Data Breach Report 2025": shadow AI present in 20% of breaches and adding USD 670,000 to the average cost; 63% of breached organisations had no AI governance policy.
OWASP Foundation (2025). "OWASP Top 10 for Large Language Model Applications, v2025": LLM02 Sensitive Information Disclosure (risen to second) and LLM03 Supply Chain (risen to third).
NIST (2024). "Artificial Intelligence Risk Management Framework: Generative AI Profile (NIST AI 600-1)."

Back to Blog Talk to the team