Most applications described as AI products are ordinary web applications that make slow, expensive, occasionally-failing HTTP calls - and that description tells you far more about what infrastructure they need than the word AI does.
Search this and you get model-training platforms and enterprise ML suites. Those are real products for teams training and serving their own models. They are the wrong answer for the much larger group of people building applications on top of models that already exist, which is a different problem with a much more boring solution.
Table of contents
- Two different problems wearing the same name
- The four ways AI workloads stress ordinary infrastructure
- Agents need an operating environment, not a chat window
- Cost, which is mostly not infrastructure
- A boring architecture that works
- How this fits the rest of the stack
- FAQ
Two different problems wearing the same name
Training and serving your own models. You need GPUs, experiment tracking, a model registry, dataset versioning, and inference infrastructure. This is a specialist domain with specialist platforms and genuinely high costs, and if this is you, you already know it.
Building applications that call existing models. You need a web service, a database, background workers, a queue, secret management, and logs. This is ordinary application infrastructure with a few specific stresses, and it describes the overwhelming majority of what gets called an AI product.
The confusion is expensive in one direction: teams in the second group shop in the first group’s aisle, evaluate ML platforms they will never use, and end up with something complicated that does not solve their actual problem, which was that a request takes ninety seconds and their host times out at thirty.
The four ways AI workloads stress ordinary infrastructure
The requirements are not exotic, but they are specific, and they are exactly the places where a default setup breaks.
- Long request durations. Model calls take seconds to minutes. Serverless function timeouts and default proxy timeouts are frequently shorter than a single generation. This is the number one reason AI features fail in production and it has nothing to do with AI.
- Streaming responses. Users expect tokens as they arrive. That requires a runtime that holds a connection open and a proxy that does not buffer the whole response before forwarding it.
- Bursty, unpredictable load. One user can trigger a lot of expensive work. Autoscaling between a sensible floor and a ceiling matters more than raw capacity, and so does a queue in front of the expensive part.
- Failure as routine. Upstream APIs rate-limit, time out, and return malformed output. Retries with backoff, idempotency, and a dead-letter path are core requirements rather than polish.
Notice that all four are solved by conventional infrastructure done properly. None of them require a specialised AI platform.
Agents need an operating environment, not a chat window
The term agent covers a lot of ground, but anything that acts rather than just answers needs a specific set of things, and the pile-of-scripts approach runs out fast.
A runtime that persists between calls. Something that keeps running, holds a connection, and can be reached at a URL by other systems. A function that spins up and dies does not fit an agent that has to wait for a tool to return.
Persistence. Conversation state, task state, intermediate results, and a record of what was actually done. In a real database, not in memory, because the process will restart.
Secrets and permission boundaries. An agent with credentials is a program with credentials, and the interesting question is which endpoints, domains, and datasets it can reach. Those boundaries should be visible and deliberate rather than implied by whatever happened to be in the environment.
Event triggers rather than polling. If a workflow should react to something happening, a webhook is the mechanism. A loop checking every thirty seconds is not real-time, it is a nervous refresh button, and it costs money on every iteration where nothing happened.
Observability. What did it decide, what did it call, what came back, and what did that cost. Without this you cannot debug a bad outcome or explain a bill, and both will happen.
Cost, which is mostly not infrastructure
The unusual thing about this category is that the model API bill typically dwarfs the hosting bill, often by an order of magnitude. That inverts the usual optimisation priorities.
Which means the levers that matter are: caching identical requests, keeping prompts short, using a smaller model for the many easy calls and a larger one only where it earns its place, and setting hard per-user and per-day limits so one loop cannot generate an enormous invoice overnight.
It also means the infrastructure decision should optimise for something other than saving a few pounds on compute. Optimise for not timing out, for scaling under a burst, for surviving upstream failures, and for being able to see what happened. Those properties protect the expensive part.
A boring architecture that works
A web service that handles requests and streams responses. A queue and a background worker for anything longer than a few seconds, so the request returns immediately and the work happens elsewhere. A managed relational database for conversation and task state. Object storage for generated artefacts and uploads. Environment variables for API keys. Logs and metrics in one place.
That is the same six pieces as any other application, with the worker doing more of the load-bearing work than usual. AI can generate the interface and the first draft of the code. It still needs somewhere to run, something to remember, and someone able to see what it did.
How this fits the rest of the stack
The infrastructure under an AI application is unglamorous and easy to price, which is useful when the model bill is the unpredictable half. The RunxBuild hosting calculator shows the service, the worker, the database, the storage, and the bandwidth as separate line items so the fixed part of the cost is known. RunxBuild deploys Node, Next.js, Python, Go, Ruby, Java, .NET, and Docker services with environment variables, runtime logs, rollback, and autoscaling between plans you choose, plus managed MySQL and Postgres and managed n8n for event-driven workflows.
Useful related references:
- RunxBuild vs Vercel: Full Cloud Comparison
- RunxBuild vs Railway: App Platform Comparison
- Netlify and SEO: The Platform Settings That Actually Affect Rankings
- Services on RunxBuild
FAQ
What kind of platform do I need for an AI application?
If you are training or serving your own models, a specialist ML platform with GPUs and a model registry. If you are building on models that already exist - which is most people - you need ordinary application infrastructure: a web service, a database, background workers, a queue, secret management, and logs.
Why do AI features time out in production?
Because model calls take seconds to minutes while serverless function timeouts and default proxy timeouts are often shorter than a single generation. The fix is a runtime that holds long connections, plus moving anything slow into a background worker so the request returns immediately.
What does an AI agent need besides a model?
A runtime that persists between calls, a database for conversation and task state, secrets with explicit permission boundaries, event triggers rather than polling loops, and observability showing what it decided, what it called, and what it cost. A chat window is not an operating environment.
How do I control the cost of an AI application?
The model API bill usually exceeds the hosting bill by a wide margin, so optimise there first: cache identical requests, keep prompts short, use smaller models for easy calls, and set hard per-user and per-day limits so a runaway loop cannot generate an enormous invoice overnight.
Should AI work run in the request or in a background job?
In a background job for anything longer than a few seconds. Return immediately with a task reference, do the work in a worker, and let the client poll or receive a webhook. This avoids timeouts, survives restarts, and lets you retry failures without the user waiting.