--- title: "Let a proxy hold your app's AI key" description: "Railway's October attacker searched for AI provider keys first, and about $740,000 was spent with the stolen keys. Notes on why apps on Vibelab, the internal platform I maintain at Miro, don't hold one." pubDate: "2026-11-11" heroImage: "/railway-ai-keys-hero.png" --- On November 10 Railway published [its report on the October 21 attack](https://blog.railway.com/p/incident-report-october-security-incident). It has a minute-by-minute timeline and a dollar figure for the damage, which few companies share. Thanks to my colleague Rodrigo for sending it to me. At Miro I created and maintain **Vibelab**, an internal platform where people deploy apps and prototypes straight from a Git repository. So I read the report to see what we could learn from it. Once the attacker reached the database of customer secrets, they searched for credentials to AI providers first. Railway estimates that **about $740,000 was spent with the stolen AI keys**. I only know Railway's side from what they published. ## What happened at Railway The short version, taken from the report: - **October 18.** The attacker signed up with throwaway accounts and began probing. - **October 20.** A previously unknown flaw in the build system let one build read files on a machine it shared with other customers' builds. That exposed a credential, which opened the stored histories of 353 builds and a copy of Railway's own source code. - **October 21.** A misconfigured cloud permission led to the database of encrypted customer variables and a way to decrypt some of them. Railway puts the upper bound at 1.85% of workspaces. - **October 21, 09:24 UTC.** The attacker was locked out. Cleanup took longer. Railway paused builds twice, logged every user out, invalidated all API tokens and asked providers such as OpenAI and Anthropic to revoke keys. Customers then rotated their own secrets, reconnected GitHub and redeployed. Railway is refunding or covering the unauthorized spend, and each build now runs in its own short-lived virtual machine. ## Why AI keys A provider key needs no second factor and works from any network. Whoever holds it spends on the owner's account until someone revokes it. The report says the stolen keys went to services that resell model access. Many apps now keep one in their environment variables, so a host with thousands of tenants stores thousands of them in one database. ## What our apps hold instead On Vibelab, an app that wants OpenAI or Anthropic gets two environment variables: ``` OPENAI_BASE_URL=https:///openai/v1 OPENAI_API_KEY= ``` The base URL points the SDK at a proxy the platform runs. The proxy checks the app's key, swaps in the real provider key and forwards the call. **The app never sees the real one.** We added the per-app key to track which app spends what. Keeping the real key out of the app came along with it, and this report made me value that part more. What we get from the proxy: - It forwards model calls only. Chat, embeddings and transcription go through. Files, batch jobs and fine-tuning get a 404. - OpenAI and Anthropic reject an app's key. It only opens the proxy, and the proxy isn't on the public internet. - It counts tokens per app and flags one that passes its daily budget. A leaked key shows up as one app's usage jumping. - The real keys live in one encrypted setting. Rotating one is a single change, and app owners do nothing. ## Who can get in Railway's attacker began as a customer. A public host lets anyone sign up, so its build system runs code from strangers. Vibelab has no signup page. It sits behind company login and only builds repositories from the company's own GitHub organizations. An insider or a stolen laptop could still get in. The first step of this attack, a stranger creating an account, has no equivalent here. | | Public host | Internal platform with a proxy | |---|---|---| | Who can deploy | Anyone who signs up | Employees | | AI key inside the app | The real provider key | A per-app key for the proxy | | Where a stolen key works | From anywhere, on every endpoint | Through the proxy, on model calls | | After a breach | Every customer rotates their own keys | One admin rotates the platform key | ## What this doesn't cover This compares one thing, which is where the AI key lives. Railway runs a much larger platform than Vibelab, with a security team and, since the incident, a separate virtual machine for every build. I'm not arguing that Vibelab is safer overall. My claim is that an app on the proxy doesn't hold a provider key, and provider keys were what this attacker searched for. That covers apps on the proxy and no others. Nothing stops someone from pasting a real key into an app's secrets. ## If you host on a public platform So, you can get most of this without running your own platform: - Give each app its own provider key, so you can revoke one without breaking the rest. - Set a spend limit on each key where the provider offers one. - If several apps share a provider account, put a proxy in front of it. An open-source gateway such as [LiteLLM](https://github.com/BerriAI/litellm) hands out per-app keys with budgets, and the real key stays on the proxy. - Rotate any key that sat on a host that had an incident, even if nothing looks wrong. Everything about the attack comes from Railway's report, last updated on November 10, 2026.