Hosting Considerations for RAG (Retrieval-Augmented Generation) Apps

What RAG Is and Why It Demands Specialized Hosting

Retrieval-Augmented Generation—universally abbreviated as RAG—is the architectural pattern that has quietly become the backbone of production AI applications in 2025 and 2026. At its core, RAG combines two distinct capabilities: a retrieval engine that searches a knowledge base (typically a vector database) for semantically relevant documents, and a generation engine (a large language model) that uses those retrieved documents as grounding context to produce factual, citation-backed responses. Unlike a standalone LLM that relies entirely on its parametric memory—the knowledge frozen into its weights during training—a RAG system queries external data at inference time. This means it can answer questions about your company's internal documentation, recent events that postdate the model's training cutoff, or proprietary datasets that never appeared in any public training corpus. The result is dramatically reduced hallucination, verifiable outputs, and the ability to keep knowledge current without retraining or fine-tuning the underlying model.

However, the architectural elegance of RAG conceals a hosting complexity that catches many teams off guard. A production RAG system is not a single application running on a single server. It is a distributed system comprising at least four distinct components—a vector database, an embedding model, a large language model, and an orchestration layer—each with its own CPU, memory, storage, and often GPU requirements. These components interact along the critical path of every user request, which means that a bottleneck in any one component degrades the end-to-end latency of the entire pipeline. Worse, the components scale differently: your vector index might grow linearly with your document corpus while your LLM throughput requirements scale with user concurrency, and the two growth curves are largely independent. At Hosting Captain, we have observed that the majority of RAG deployments that fail in production do so not because of model quality issues but because of hosting architectures that treat a RAG pipeline as if it were a monolithic web application. This guide provides a comprehensive, component-by-component breakdown of what hosting a RAG application actually requires, grounded in real-world deployment data and informed by our experience provisioning infrastructure for RAG workloads ranging from single-digit-QPS chatbots to enterprise document search systems handling millions of queries per day. If you are new to the broader category of GPU-accelerated infrastructure, our AI hosting fundamentals guide establishes the vocabulary and infrastructure paradigms that underpin everything discussed below.

The RAG Hosting Architecture Deconstructed

Before evaluating hosting options, you need a clear mental model of what a RAG system actually runs on. Conceptually, every RAG query traverses four logical tiers, and each tier maps to a distinct hosting requirement. Understanding these tiers and how they couple is the prerequisite for sensible infrastructure decisions.

Tier 1: The Vector Database (The Retrieval Engine)

The vector database is the persistent knowledge store. During an ingestion process (which may run continuously or in batch), documents are chunked into semantically coherent segments, each chunk is converted into a dense vector embedding by an embedding model, and the resulting vector is stored alongside the original text and any metadata. At query time, the user's question is converted into a query embedding, and the vector database performs an approximate nearest-neighbor (ANN) search to retrieve the k most semantically similar chunks. Popular vector databases in production RAG deployments include Milvus (open-source, distributed, GPU-accelerated indexing), Qdrant (Rust-based, single-binary deployable with excellent query-per-second per core), Pinecone (fully managed, serverless scaling, no operational burden), Weaviate (hybrid vector + keyword search with built-in embedding generation), and pgvector (PostgreSQL extension, ideal when you want vector search colocated with relational data). The hosting requirements vary enormously across these options. A pgvector deployment on a standard VPS hosting instance with 8 vCPUs and 32 GB RAM can comfortably serve 100,000–500,000 document chunks at sub-50ms query latency—sufficient for many internal knowledge base applications. A distributed Milvus cluster ingesting 10 million documents per day, by contrast, demands multiple nodes with NVMe storage, high-memory instances for index construction, and careful network topology to avoid cross-node latency introducing tail-end query delays.

Tier 2: The Embedding Model (The Semantic Encoder)

The embedding model converts text—both documents during ingestion and user queries at runtime—into fixed-length dense vectors (typically 768, 1024, or 1536 dimensions). This model runs on every single RAG query (to embed the user's question) and on every ingested document, which means its throughput directly bounds the system's ingestion rate and contributes to query latency. Popular embedding models include OpenAI's text-embedding-3-large and text-embedding-3-small (accessible only via API), and open-weight alternatives such as BGE-M3, E5-Mistral-7B, and gte-Qwen2-7B. The hosting implications are stark: the smallest embedding models (e.g., BGE-small-en, 384 dimensions, 33M parameters) run on CPU with acceptable latency and cost essentially nothing to host, while the largest and most accurate open embedding models (7B+ parameters) require a GPU for any production-level throughput. A 7B-parameter embedding model run on CPU might take 500–800 milliseconds to encode a single 512-token passage, which is unacceptable for real-time query embedding. On an NVIDIA L40S or RTX 4090, the same model encodes hundreds of passages per second. For ingestion pipelines processing millions of documents, this difference translates to hours versus days of wall-clock time.

Tier 3: The Large Language Model (The Generation Engine)

The LLM receives the user's query and the retrieved context chunks, formats them into a prompt (often with system instructions about how to use the context), and generates the final response. This is the most computationally expensive tier by a wide margin. The hosting choices for the LLM tier map closely to the self-hosted vs API-based AI hosting cost analysis we have covered previously: you can call cloud APIs (GPT-4o, Claude, Gemini) and pay per token, or you can self-host open-weight models (Llama 3, Mistral, Qwen) on GPU infrastructure. For RAG specifically, the LLM hosting decision interacts with the retrieval tier: if your vector database returns high-quality context that makes the generation task easier, you can often use a smaller, cheaper, faster model without sacrificing output quality. A RAG pipeline backed by a well-curated knowledge base can frequently serve a 7B-parameter model where a standalone deployment would require a 70B-parameter model, because the retrieval step offloads factual recall from the model's weights to the external knowledge store. This architectural property is one of the most underappreciated cost levers in RAG hosting: investing in retrieval quality can reduce LLM hosting costs by an order of magnitude.

Tier 4: The Orchestration Layer (The Application Server)

The orchestration layer ties the pipeline together. It receives HTTP requests from the client, calls the embedding model to encode the query, queries the vector database for relevant chunks, constructs the prompt by combining query, context, and system instructions, sends the prompt to the LLM, receives the generated response, and returns it to the client—often with citation metadata mapping each claim back to a source document. This layer is typically a standard web application (FastAPI, Flask, Next.js API routes) running on a CPU server. Its hosting requirements are modest relative to the GPU tiers: 2–4 vCPUs and 4–8 GB of RAM can handle significant concurrency, because the orchestration layer itself does minimal compute—it mostly waits on I/O from the embedding model, vector database, and LLM. However, the orchestration layer is where reliability patterns live: retry logic, circuit breakers, fallback to a simpler model if the primary LLM times out, and caching of frequent queries and their retrieved contexts. These patterns impose state management and observability requirements that influence the hosting topology. At Hosting Captain, we recommend co-locating the orchestration layer with the vector database when possible, to eliminate network round-trips on the critical retrieval path, and using asynchronous I/O frameworks (async Python or Node.js) to maximize throughput per vCPU.

Hosting Requirements for Each RAG Component

With the architecture established, we can specify concrete hosting requirements for each tier. The table below summarizes the resource profiles for a mid-scale RAG deployment serving 50 queries per second with a knowledge base of 5 million document chunks. Treat these as planning baselines; your specific requirements will vary with document volume, query complexity, model size, and latency targets.

Vector Database Hosting: Milvus, Qdrant, Pinecone, and pgvector Compared

The vector database is the most operationally stable component in a RAG stack—it runs continuously, accumulates data, and must never lose state. Milvus excels at billion-scale vector collections and supports GPU-accelerated index construction, but it is the most operationally demanding option, requiring etcd for metadata coordination, MinIO or S3 for object storage, and Pulsar or Kafka for streaming ingestion. Running a production Milvus cluster demands at least 4–6 nodes and a team comfortable with distributed systems operations. Hosting Captain recommends Milvus only when your vector corpus exceeds 50 million chunks and you have dedicated infrastructure engineering capacity.

Qdrant is the pragmatic default for most RAG deployments. It compiles to a single binary, runs efficiently on a single VPS or dedicated instance, and delivers excellent ANN search performance through its Rust-based HNSW implementation. A Qdrant instance with 8 vCPUs, 32 GB RAM, and 200 GB NVMe storage comfortably handles 10 million 768-dimensional vectors at sub-20ms p95 query latency. Its on-disk indexing mode allows vector collections larger than available RAM at the cost of moderately higher query latency, making it the most cost-efficient option for large-but-not-huge knowledge bases. Pinecone eliminates all operational burden in exchange for a pricing premium: its serverless offering scales automatically based on vector count and query volume, and its pod-based architecture provides predictable performance at higher cost. For teams that lack infrastructure expertise or whose RAG workloads are highly variable, Pinecone's operational simplicity justifies its cost. pgvector with PostgreSQL is the right choice when you already operate a Postgres database and your vector workload is modest (under 1M vectors). The IVFFlat and HNSW index types in pgvector have matured substantially, and the ability to run vector search and relational filtering in a single query simplifies the orchestration layer. The trade-off is that ANN search performance degrades beyond roughly 1–2 million vectors compared to purpose-built vector databases.

Embedding Model Hosting: GPU, CPU, or API?

The embedding model hosting decision follows a simple heuristic that we have validated across dozens of RAG deployments at Hosting Captain. If your query volume is under 10 queries per second, run the embedding model on CPU. Modern sentence-transformers models like BGE-M3 and all-MiniLM-L6-v2 are fast enough on a 4-vCPU instance to handle this throughput without GPU acceleration. If your volume exceeds 10 QPS, or if your ingestion pipeline needs to process more than 500 documents per second, add a GPU. A single L40S or RTX 4090 with an open embedding model (BGE-M3, E5-Mistral-7B) can encode 500–1,000 passages per second, which is sufficient for all but the highest-volume RAG deployments. The third path—using embedding APIs (OpenAI text-embedding-3-small/large, Cohere Embed, Voyage AI)—eliminates the embedding hosting question entirely at a per-token cost. OpenAI's text-embedding-3-small costs $0.02 per 1 million tokens, which translates to roughly $0.0002 per typical 512-token query embedding. At 10,000 queries per day, that is $2 per month—a rounding error compared to the LLM API cost. The embedding API path is almost always the right choice during prototyping and for low-to-moderate volume deployments, because it removes an entire tier of infrastructure management for negligible cost. Only self-host the embedding model when data privacy requirements prohibit sending text to external APIs, or when your volume is high enough that self-hosting's fixed cost falls below the API's variable cost—typically above 500 million embedded tokens per month, equivalent to approximately 30 million daily queries.

LLM Hosting for RAG: Special Considerations

The LLM hosting calculus for RAG differs from standalone LLM deployments in one critical respect: context length. Every RAG query includes the user's question plus the retrieved chunks, and those chunks can easily total 2,000–8,000 tokens depending on how many documents you retrieve and how large each chunk is. This matters because the LLM's time-to-first-token and tokens-per-second are both sensitive to prompt length—specifically, the attention mechanism's computational complexity scales quadratically with sequence length in most transformer architectures. A model that generates 50 tokens per second with a 500-token prompt might generate only 25 tokens per second with a 4,000-token RAG context. When sizing GPU instances for RAG LLM hosting, you must benchmark with realistic prompt lengths—not the short single-sentence prompts that appear in most model performance tables. At Hosting Captain, our RAG-specific GPU benchmarks have shown that a Llama 3 8B model running on an L40S serves approximately 35–40 tokens per second with a 3,000-token RAG prompt, which translates to roughly 15–20 concurrent users before p95 latency exceeds 2 seconds. For context on how these GPU hosting costs compare to API-based alternatives, our comprehensive AI hosting cost comparison provides break-even analyses across GPU types and volume tiers.

Self-Hosted vs Managed RAG Hosting

The RAG hosting landscape in 2026 offers a spectrum from fully self-managed open-source deployments to fully managed platforms that abstract away every infrastructure decision. Between these poles lies a rich set of partial-management options—managed vector databases with self-hosted LLMs, managed LLM APIs with self-hosted vector stores, and everything in between. The right choice depends on your team's expertise, your data privacy requirements, and your cost structure.

Fully Self-Hosted RAG

A fully self-hosted RAG stack—where you run the vector database, embedding model, LLM, and orchestration layer on infrastructure you control—provides maximum data sovereignty and, at sufficient scale, the lowest unit cost. This path requires GPU instances for the LLM (and optionally the embedding model), a reliable instance for the vector database, and a web server for orchestration. On Hosting Captain's infrastructure, a capable fully self-hosted RAG stack for moderate workloads (10–50 QPS, 5 million document chunks) can be assembled from: one GPU server with an L40S or A10 for the LLM ($350–600/month), one VPS or dedicated server for Qdrant and the orchestration layer ($60–180/month), and optionally a second GPU instance for the embedding model if you choose to self-host that tier ($150–350/month). The total monthly infrastructure cost for a fully self-hosted mid-scale RAG deployment ranges from $500 to $1,200, with engineering operational overhead adding an estimated 8–15 hours per month for monitoring, updates, and scaling adjustments. This compares favorably to managed alternatives at sustained query volumes above roughly 50 million output tokens per month for the LLM tier.

Managed RAG Platforms

At the other end of the spectrum, managed RAG platforms bundle the entire pipeline. Pinecone Assistant and LangChain Cloud offer end-to-end RAG-as-a-service, where you upload documents and receive a query endpoint without touching a single server. These platforms handle chunking, embedding, vector storage, retrieval, and LLM generation, typically charging based on document volume, query count, and token consumption. The pricing premium over self-hosting is substantial—expect to pay 3–5× the raw infrastructure cost—but the operational simplicity can be transformative for teams without ML infrastructure expertise. Other platforms occupy middle positions: LlamaIndex Cloud provides managed ingestion and retrieval with configurable LLM backends (bring your own API key or use their managed models), while Vercel AI SDK with managed vector stores abstracts the RAG pipeline into frontend-adjacent infrastructure. The W3C standards for web interoperability increasingly influence how these managed platforms expose their functionality, particularly around structured output formats and streaming protocols that RAG applications depend on.

Hybrid RAG Hosting: The Pragmatic Middle Path

The hybrid model—managed components for some tiers, self-hosted for others—is the most common production RAG architecture we observe at Hosting Captain. A classic hybrid configuration uses a managed vector database (Pinecone or Zilliz Cloud) to eliminate the operational burden of distributed state management, while keeping the LLM self-hosted on a GPU server for cost control and data privacy. In this setup, the vector database is an operational expense that scales linearly with your knowledge base, and the LLM is a capital-like investment whose unit cost decreases with utilization. Teams that adopt this pattern typically spend $200–600 per month on the managed vector tier and $350–1,200 on the GPU server, with the orchestration layer running on a modest VPS. The total monthly cost sits between the fully self-hosted and fully managed extremes, while preserving data locality for the component that handles the most sensitive information: the LLM that sees both the user query and the retrieved context.

RAG Hosting Cost Breakdown and Latency Considerations

Understanding the cost structure of a RAG deployment requires modeling it as a queuing system where latency compounds across serial stages. Each user query triggers a pipeline where total latency equals the sum of embedding time, vector search time, prompt construction time, and LLM generation time, plus network round-trips between tiers. The costs, meanwhile, divide into fixed infrastructure costs (GPU servers, database instances) and variable usage costs (API tokens, managed service fees). The interplay between latency and cost determines the economically optimal hosting configuration.

Latency Budgeting Across the RAG Pipeline

A well-tuned RAG system targeting a 2-second end-to-end response time for a typical Q&A workload allocates its latency budget roughly as follows: 50–100 milliseconds for query embedding, 20–50 milliseconds for vector search (assuming an HNSW index in memory), 100–300 milliseconds for LLM time-to-first-token plus prompt processing, and 1,000–1,500 milliseconds for token generation (assuming 30–40 tokens of output at 25–40 tokens per second). Network latency between tiers should be under 5 milliseconds, which means all components must be hosted in the same data center region. Deploying the vector database in one cloud region and the LLM GPU server in another adds 20–80 milliseconds of cross-region latency—a 10–40% increase on the retrieval step alone. At Hosting Captain, we provision all RAG components within the same physical data center (or same availability zone within a cloud region) as a default architectural constraint, because network-induced latency cannot be optimized away at the application layer. The embedding model's placement relative to the vector database also matters: if you use an API-based embedding service (e.g., OpenAI embeddings), the query must travel from your orchestration server to the API endpoint, back to your server, and then to the vector database. This round-trip adds 50–100 milliseconds for typical API latency. Self-hosting the embedding model on the same server as the orchestration layer or vector database eliminates this hop entirely, contributing directly to faster end-to-end response times.

Cost Modeling: Fixed vs Variable Cost Intersection

The cost break-even between self-hosted and API-based RAG components follows the same logic we detailed in our AI hosting cost comparison, but with RAG-specific multipliers. For the LLM tier, the key variable is context length: RAG prompts are longer than typical standalone LLM prompts (3,000–6,000 tokens versus 500–1,000 tokens), which means input token costs dominate the API bill. A RAG query with 4,000 input tokens and 500 output tokens on GPT-4o costs approximately $0.035 per query ($0.01 for input plus $0.0075 for output). At 10,000 queries per day, the monthly API bill reaches $10,500. A self-hosted Llama 3 8B on a single L40S GPU instance at $500 per month can serve the same volume at roughly $0.0017 per query—a 20× cost reduction. The break-even point for this workload is approximately 480 queries per day: below that, API costs are under $500 per month and self-hosting does not justify its fixed GPU cost; above that, self-hosting delivers progressively larger savings.

For the vector database tier, the cost driver is storage volume and query throughput. A self-hosted Qdrant instance on a $60/month VPS handles up to 5 million vectors and 50 QPS comfortably. Pinecone's equivalent pod-based plan for the same workload costs approximately $200–350 per month. The managed premium—roughly 3–5×—buys automatic scaling, zero-downtime upgrades, and a team that handles index optimization. Whether that premium is justified depends entirely on whether your organization has personnel who can maintain a Linux server, monitor disk usage, configure backups, and respond to a 3 a.m. disk-full alert. The vector database is stateful; losing it means losing your entire knowledge base's vector representation. For many organizations, the managed premium for the vector tier is the cheapest insurance policy in the RAG stack.

Scaling RAG Applications: From Prototype to Production

RAG prototypes are deceptively easy to build. A Jupyter notebook with ten lines of LlamaIndex or LangChain code, an in-memory vector store, and an OpenAI API key can answer questions over a hundred PDF documents in an afternoon. The gap between that prototype and a production system that handles thousands of concurrent users, ingests millions of new documents per day, and maintains sub-2-second p95 latency is where hosting considerations become existential. Scaling a RAG application means scaling each tier independently according to its specific bottleneck, and recognizing that the bottlenecks shift as the system grows.

Scaling the Vector Database

The vector database scales along two axes: data volume (number of vectors) and query throughput (queries per second). These axes are largely independent and require different scaling strategies. Data volume scaling is about storage and index construction: as your document corpus grows from thousands to millions to billions of chunks, the vector database must store more vectors and maintain searchable indices over them. Most production-grade vector databases support sharding—partitioning the vector collection across multiple nodes—which allows linear scaling of both storage capacity and index build parallelism. Qdrant supports horizontal scaling via its distributed deployment mode, where a cluster of nodes shares a Raft-based consensus protocol for metadata and distributes vector segments across nodes. Milvus was architected from the ground up for horizontal scaling, with separate components for data nodes (storing vector data), query nodes (executing searches), and index nodes (building ANN indices). Query throughput scaling, by contrast, is about replication: adding read replicas that each hold a complete copy of the index. Qdrant, Milvus, and Pinecone all support read replicas. The practical hosting implication is that a RAG deployment serving 10 QPS on 1 million vectors can run on a single VPS; a deployment serving 1,000 QPS on 100 million vectors requires a multi-node cluster with dedicated data, query, and index nodes. The infrastructure cost growth is approximately linear with data volume and sub-linear with query throughput due to the efficiency of ANN index structures that compress vector representations.

Scaling the LLM Tier

Scaling the LLM tier for a RAG application is a GPU provisioning problem. A single L40S serving Llama 3 8B at FP16 can handle roughly 15–20 concurrent RAG queries before queuing delays push p95 latency above the acceptable threshold. Adding concurrent capacity requires either scaling vertically (moving to a larger GPU—an A100 or H100 can serve the same model at 3–5× the throughput) or scaling horizontally (adding more GPU instances behind a load balancer). Horizontal scaling of LLM inference is non-trivial because each GPU instance must load the full model weights into VRAM, and request routing must be session-aware if the RAG application maintains conversation state. At Hosting Captain, we recommend provisioning LLM GPU capacity at 1.5× your measured peak QPS to provide headroom for traffic spikes, because GPU cold starts—launching a new instance and loading model weights—take 30–90 seconds, during which incoming requests either queue or fail. For RAG workloads with predictable diurnal patterns (e.g., a customer-support chatbot that is quiet from midnight to 6 a.m.), scheduled auto-scaling that provisions additional GPU instances before the morning traffic ramp is more reliable than reactive auto-scaling that triggers on CPU or GPU utilization thresholds.

Scaling the Embedding Tier

The embedding tier scales differently depending on whether you use an API or self-host the embedding model. API-based embeddings (OpenAI, Cohere) scale automatically—the provider absorbs throughput fluctuations—but are subject to rate limits that may cap your ingestion or query throughput. OpenAI's tiered rate limits range from 500 RPM for free-tier users to 10,000 RPM for enterprise users, and hitting the limit during a bulk ingestion job can stall a document processing pipeline for hours. If you self-host the embedding model, scaling means adding more replicas of the embedding service behind a load balancer. Each replica needs either a GPU (for large embedding models) or sufficient CPU cores (for small models under 100M parameters). The embedding tier is stateless—no model weights are modified during inference—so horizontal scaling is straightforward compared to the vector database. A practical hosting pattern we recommend at Hosting Captain is to run a small, fast embedding model (e.g., BGE-small-en or all-MiniLM-L6-v2) on CPU for real-time query embedding, while using a larger, more accurate model (BGE-M3 or E5-Mistral-7B) on GPU for the offline document ingestion pipeline. This bifurcated approach maximizes query throughput while maintaining high retrieval quality on the ingestion side, where latency is less critical.

Recommended Hosting Setups for Different RAG Use Cases

RAG deployments exist on a wide spectrum, from internal documentation Q&A bots to customer-facing product search engines. The hosting setup that is optimal for one use case can be wasteful or underpowered for another. Drawing on deployment data from Hosting Captain's managed RAG customer base, we have identified four archetypal RAG use cases and their corresponding recommended hosting configurations.

Use Case 1: Internal Knowledge Base Q&A (Low Volume, High Privacy)

Profile: A company wiki or internal documentation Q&A bot serving 50–200 employees, handling 500–2,000 queries per day, with a knowledge base of 5,000–50,000 documents. Data must never leave the corporate network.
Recommended Setup: Fully self-hosted on a single dedicated server or VPS. Run Qdrant (or pgvector if a PostgreSQL instance already exists) for the vector database, BGE-M3 on CPU for query and ingestion embedding, and Llama 3 8B (INT4 quantized) on a single L40S or RTX 4090 for generation. The orchestration layer (FastAPI) runs on the same server. Total infrastructure: one GPU server at approximately $400–600 per month. This setup keeps all data on-premises or within a controlled VPS environment, eliminates all external API dependencies, and provides sub-3-second response times for the typical retrieval-plus-generation workload. For teams that want to start simpler, substitute the self-hosted LLM with a local API proxy to OpenAI or Anthropic during prototyping, then migrate to self-hosted when data privacy requirements are confirmed.

Use Case 2: Customer Support Chatbot (Medium Volume, Moderate Latency Sensitivity)

Profile: A public-facing support chatbot integrated into a SaaS product, handling 5,000–50,000 queries per day, with a knowledge base of 10,000–200,000 support articles and product documentation pages. Latency target is sub-2 seconds p95.
Recommended Setup: Hybrid hosting. Use Pinecone or Zilliz Cloud for the vector database (managed, eliminating scaling concerns as document volume grows). Self-host the LLM tier: Llama 3 8B or Mistral 7B fine-tuned on support interaction data, running on 1–2 L40S GPU instances behind a load balancer for concurrency. Use OpenAI's text-embedding-3-small API for query embedding (cost is negligible at this volume) and for ingestion embedding. The orchestration layer runs on a 4-vCPU VPS with async I/O. Total monthly cost: $200–400 for the managed vector database, $400–800 for GPU instances, $30–100 for embedding API calls, and $40–80 for orchestration—approximately $700–1,400 per month. This configuration balances cost efficiency with operational simplicity: the vector database's operational burden is outsourced, the LLM's costs are fixed and predictable, and the embedding tier is so cheap on API that self-hosting is not worth the engineering effort. For guidance on provisioning GPU servers for this setup, see our small AI model hosting guide which covers the GPU instance types, VRAM requirements, and benchmarking methodology applicable to both training and inference workloads.

Use Case 3: Enterprise Document Search (High Volume, Multi-Tenant)

Profile: A document search and Q&A platform serving multiple enterprise tenants, each with its own document corpus totaling 1–50 million documents, handling 10,000–100,000+ queries per day across all tenants. Latency target is sub-1.5 seconds p95. Data isolation between tenants is mandatory.
Recommended Setup: Multi-node self-hosted vector database cluster (Milvus or distributed Qdrant) with tenant-level namespace isolation. Three to six GPU servers (L40S or A100) for the LLM tier, running Llama 3 70B or Mixtral 8×7B with continuous batching via vLLM to maximize throughput per GPU. The embedding model is self-hosted on dedicated GPU instances to eliminate API rate limits and data egress concerns at enterprise volume. The orchestration layer runs on a Kubernetes cluster with horizontal pod autoscaling based on request queue depth. Total monthly infrastructure: $2,500–6,000 for the vector database cluster, $2,400–8,000 for the LLM GPU fleet, $600–1,200 for embedding GPU instances, and $200–600 for orchestration—approximately $5,700–15,800 per month. At this scale, the per-query cost is on the order of $0.002–0.005, which compares favorably to the $0.03–0.05 per query that a fully API-based RAG stack would cost at equivalent volume. The self-hosted investment pays back within 6–9 months, and the data isolation guarantees satisfy enterprise compliance requirements that API-based architectures cannot provide.

Use Case 4: E-Commerce Product Search with RAG (High Volume, Extreme Latency Sensitivity)

Profile: An e-commerce product search and recommendation engine using RAG to ground product descriptions, reviews, and specifications in a conversational interface, handling 100,000–1,000,000+ queries per day, with latency requirements below 800 milliseconds end-to-end because every 100ms of delay measurably reduces conversion rates.
Recommended Setup: Fully self-hosted, latency-optimized stack. The vector database runs on a small cluster of high-CPU, high-memory instances with NVMe storage and the vector index pinned entirely in memory—no disk reads on the query path. The embedding model is the fastest available small model (BGE-small-en or gte-small) running on CPU instances co-located with the vector database to minimize network hops. The LLM tier uses the smallest model that delivers acceptable response quality—typically Llama 3 8B or Phi-3-medium at INT4 quantization—running on A100 or H100 GPUs with TensorRT-LLM for maximum throughput and minimum time-to-first-token. A Redis cache sits between the orchestration layer and the vector database, caching frequent query embeddings and their retrieved context sets to bypass the full retrieval pipeline for common queries. Total monthly infrastructure: $1,500–4,000 for the vector database cluster, $1,200–4,000 for the LLM GPU instances, $80–200 for embedding (CPU-based, marginal cost), and $300–800 for the orchestration and caching layer—approximately $3,100–9,000 per month. At 500,000 queries per day, the per-query cost is under $0.001, and the end-to-end latency target of 800ms is achievable only because the entire stack is co-located within a single data center with no external API calls on the critical path. This is the RAG hosting configuration where every millisecond matters and every dollar of infrastructure investment must be justified against a latency SLA.

Frequently Asked Questions

What is the minimum hosting setup needed to prototype a RAG application?

A functional RAG prototype can run on a single $20–40 per month VPS hosting instance with 4 vCPUs and 8 GB RAM. Use pgvector (PostgreSQL extension) for the vector store, a small open-source embedding model (all-MiniLM-L6-v2) running on CPU, and the OpenAI API for the LLM tier. This setup handles up to 10,000 document chunks and 100–500 queries per day comfortably. It is the fastest path from idea to working RAG endpoint, and it defers all GPU infrastructure decisions until you have validated that RAG is the right architecture for your use case. Hosting Captain offers VPS plans specifically configured for this prototyping profile, with PostgreSQL and the pgvector extension pre-installed.

Do I need a GPU for the vector database in a RAG system?

For the vast majority of RAG deployments, no. Vector databases like Qdrant, Milvus, and pgvector execute approximate nearest-neighbor search efficiently on CPU using optimized index structures (HNSW, IVF). GPU acceleration for vector search becomes relevant only at extreme scale—typically above 100 million vectors or above 10,000 queries per second—where the parallelism of GPU-based brute-force or ANN search justifies the additional cost. Milvus supports GPU-accelerated index construction (building the ANN index from raw vectors) and GPU-accelerated search, but even Milvus deployments serving 50 million vectors commonly run CPU-only search with GPU reserved for index building. Unless your RAG application is performing search over a billion-scale vector corpus, invest your GPU budget in the LLM and embedding tiers, not the vector database.

How do I choose between self-hosting the LLM and using an API for my RAG application?

The decision follows the same break-even logic we detailed in our self-hosted vs API cost comparison, with the added consideration that RAG context windows are longer and therefore input token costs are proportionally higher. Calculate your expected monthly token volume (including both input and output tokens at realistic RAG prompt lengths), compare the API cost at that volume to the fixed cost of renting a GPU instance, and factor in your in-house GPU-operations expertise. If your volume exceeds approximately 15,000–20,000 RAG queries per day with 3,000–4,000 token prompts, self-hosting an 8B-parameter model on a single L40S GPU ($350–600/month) is almost always cheaper than paying API rates—even before accounting for data privacy benefits. Below that threshold, the operational simplicity of APIs usually wins. Organizations handling sensitive data should self-host regardless of cost, because even zero-retention API policies do not satisfy the most stringent contractual and regulatory data residency requirements.

Which vector database is best for a small RAG application just getting started?

For a small RAG deployment with under 1 million document chunks, pgvector (PostgreSQL extension) or Qdrant are the most pragmatic choices. pgvector is ideal if you already operate a PostgreSQL database—it adds vector search to your existing infrastructure with zero new services to manage. Qdrant is the better choice if you are starting from scratch, because it compiles to a single binary, consumes fewer resources than a full PostgreSQL instance, and provides a purpose-built API for vector search with built-in payload filtering. Both options are open-source, run comfortably on modest VPS instances, and can be migrated to managed services (Zilliz Cloud for Milvus compatibility, Qdrant Cloud, or Pinecone) if your scale outgrows the single-instance deployment. Avoid distributed systems like Milvus for small deployments—the operational overhead of coordinating etcd, MinIO, and Pulsar components is not justified for sub-million-vector workloads.

How does RAG hosting differ from hosting a regular AI chatbot?

A regular AI chatbot (LLM-only) has one hosting concern: the GPU server or API endpoint that runs the language model. A RAG chatbot adds at least two additional hosting concerns: the vector database (which is stateful, grows with your document corpus, and must be backed up) and the embedding model (which runs on every query and on every ingested document). The pipeline architecture of RAG also introduces serial latency that a standalone chatbot does not experience: the embedding step and the vector search step must both complete before the LLM even begins processing. This means RAG hosting must optimize not only the LLM tier but also the retrieval tier, and must keep all components within the same data center region to avoid network latency compounding across pipeline stages. Additionally, RAG ingestion pipelines—periodic or continuous re-indexing of updated documents—represent a background workload that consumes embedding throughput and vector database write capacity, and this workload must be provisioned for alongside the real-time query path. Standalone chatbots have no equivalent ingestion workload.

What are the most common RAG hosting mistakes you see teams make?

The most frequent and costly mistake we observe at Hosting Captain is underestimating the memory requirements of the vector database. Teams provision a VPS with adequate storage but insufficient RAM, and when the HNSW index grows beyond available memory, query latency spikes from 20 milliseconds to 500+ milliseconds as the database thrashes between RAM and disk. A related mistake is failing to budget for ingestion throughput: a team builds a RAG system that handles query traffic comfortably, then ingests 500,000 new documents and discovers that the embedding model and vector database cannot keep up, causing ingestion to fall behind and the knowledge base to go stale. Other common errors include deploying the vector database in a different cloud region from the LLM (adding 30–80ms of cross-region latency that no application-level optimization can fix), using an embedding model that is too large for the available hardware (a 7B-parameter embedding model on CPU takes 500ms+ per encoding), and failing to implement caching for frequent queries—a simple Redis cache on the orchestration layer can reduce both vector database load and embedding API costs by 40–60% for applications with repetitive query patterns.

Can I run a production RAG application on shared hosting?

No. Shared hosting environments lack the resource isolation, persistent storage guarantees, and software installation flexibility required by any tier of a RAG stack. A vector database needs consistent RAM allocation for index performance; an LLM inference engine needs dedicated GPU access; and even a lightweight orchestration layer benefits from guaranteed CPU and the ability to install system-level dependencies like libtorch or CUDA libraries. For production RAG workloads, the minimum viable hosting tier is a VPS hosting instance or dedicated server, where you have root access, guaranteed resources, and the ability to install and configure the specialized software—vector databases, inference engines, and embedding runtimes—that a RAG pipeline depends on. Hosting Captain's VPS and dedicated server plans are designed for exactly this class of workload, with pre-configured GPU options for the LLM and embedding tiers and CPU-optimized instances for vector database and orchestration workloads.