LLM Inference on AWS: Bedrock or Your Own GPUs, RAG, and Cost Control

An AI prototype is quick to build. The hard part is moving it into production: what infrastructure do you need, how does the model use your own knowledge, and why is the GPU bill growing faster than the value?

Key takeaways

• For most product use cases, managed model access through Amazon Bedrock is the right default.
• Your own GPU infrastructure only pays off when you need a specific model, tighter control over latency, or data residency that Bedrock doesn't offer.
• GPU inference scales on queue depth and latency rather than CPU utilization, and it needs a warm minimum capacity to absorb cold starts.
• In RAG, retrieval quality caps the quality of every answer, so chunking and embeddings matter more than prompt tuning.
• Token throughput and GPU utilization are the two numbers that predict your bill.

The first decision: managed service or self-hosted

The first architecture decision I make on any LLM project is not about the model. I first settle whether to use a managed service or run inference myself.

When Bedrock is enough

Bedrock gives me managed access to a range of foundation models without owning any GPU capacity, patching, or scaling logic. For most product use cases that's the right default. I don't want to operate model-serving infrastructure if a managed API meets the latency and control requirements.

When your own GPU services make sense

I move to containerized GPU services when I need something Bedrock doesn't offer:

• a specific fine-tuned or open-weight model,
• tighter control over batching and latency,
• data residency requirements that rule out a shared managed endpoint.

That's a real infrastructure commitment. I only take it on when the managed path can't meet the requirement.

Autoscaling GPU workloads properly

Self-hosted GPU inference has to scale on the right signal, and CPU utilization is the wrong one. A GPU-bound model can sit at low CPU while the actual bottleneck, the GPU itself, is saturated.

I scale containerized inference services on queue depth and request latency instead. That way the system reacts to the thing that is actually degrading.

Absorbing cold starts

GPU instances also have a real cold-start cost. Loading model weights onto a fresh instance takes noticeably longer than a typical container start.

So I keep a warm minimum capacity sized to expected baseline traffic rather than scaling from zero. Autoscaling handles the bursts above that floor.

RAG: pulling knowledge into the prompt at request time

Retrieval-augmented generation is where I spend most of my design time. RAG is less a feature than an architecture decision about where knowledge lives.

Rather than relying on what a model learned during training, a RAG pipeline retrieves relevant context from a vector store at request time. It feeds that context into the prompt alongside the user's query.

The retrieval step in the request path

Retrieval sits in the request path before generation and has three steps:

1. Embed the query.
2. Search the vector store for the nearest matches.
3. Assemble that context into the prompt.

Retrieval latency and quality therefore directly bound the quality of the final answer. Getting the chunking and embedding strategy right upstream matters more than any prompt tuning downstream.

Cost control from day one

GPU capacity is expensive enough that cost has to be an architecture concern from the start. A cleanup pass later comes too late.

Instance types and spot capacity

I size instance types to the model's actual memory and throughput needs rather than defaulting to the largest available GPU. Interruption-tolerant work such as batch embedding jobs and offline evaluation runs on spot capacity. Only latency-sensitive live inference stays on on-demand instances.

The right metrics

Observability has to cover tokens and GPU utilization specifically, in addition to request count and response time. Those two numbers predict the bill and tell you whether capacity is sized correctly.

What this means for your company

In most cases you don't need to build your own GPU infrastructure to get started. Bedrock lets you run a production AI use case without operating your own model serving. Your own GPUs only come into play when concrete requirements for the model, latency, or data residency call for them, and then with a clear view of the running costs.

The quality of the answers, on the other hand, is decided by your data. How well documents are prepared, split, and made searchable determines whether an assistant gives useful or wrong answers. So plan time and budget for preparing your data, and do it from the start. If you are thinking about how to bring an LLM project on AWS into production with predictable costs, get in touch.