SQLite RAG Without a Vector Database: When It Works

SQLite RAG is having a deserved moment with builders who are tired of deploying an entire vector database for an agent that searches a few thousand notes. A recent r/SaaS post argues that a local SQLite file plus native Go cosine similarity can deliver sub-2 ms retrieval on roughly 1,000 chunks—without Docker, a separate database server, or background indexing.

That conclusion is directionally right, but the useful lesson is more precise: a vector database is not the default requirement for RAG; it is a scaling and operations choice. For personal agents, desktop apps, internal tools, and early-stage products with modest corpora, a deliberately simple SQLite RAG design can be the better engineering decision. The trap is treating “works at 1,000 chunks” and “works below 100,000 chunks” as interchangeable claims.

The original post, shared by Reddit user u/Sea-Lettuce-255, describes a local AI-agent setup on an 8GB M2 MacBook Air. Rather than run Chroma, Qdrant, or Postgres with pgvector in Docker, the author keeps text, metadata, and embeddings in SQLite, loads a namespace into memory, and scores vectors with a hand-written cosine-similarity function in Go. The reported result is a native binary with no Docker overhead and very fast retrieval at the tested scale. (reddit.com)

The real argument for SQLite RAG: eliminate unnecessary moving parts

The appeal of this architecture is not that SQLite is secretly a high-scale vector-search engine. It is that many RAG systems begin with a small, local, single-user workload, while their infrastructure is selected as if they already serve millions of documents and concurrent queries.

A self-contained SQLite file has attractive properties for that first category of product. SQLite describes itself as a small, fast, self-contained, high-reliability SQL database engine. For a local agent, that translates into one portable file, simple backups, ordinary SQL for metadata, and no network hop between the application and its retrieval store. (sqlite.org)

The Reddit author’s design also removes a practical pain point for low-memory laptops: Docker Desktop adds a virtualization layer and a second operational environment to manage. Docker provides configurable settings intended to control Desktop behavior and resource use, and current Mac installations use Apple’s virtualization framework by default, but that does not make every containerized local stack lightweight in real-world use. (docs.docker.com)

That distinction matters. “Docker takes 2GB” is not a universal benchmark; memory consumption varies by Docker Desktop version, VM configuration, running containers, images, extensions, caches, and the database process itself. Still, on an 8GB machine, avoiding an always-on virtualized stack can be a legitimate product requirement rather than premature optimization.

The stronger framing is this:

• Use SQLite RAG when retrieval is embedded inside one application and the corpus is bounded.
• Use a vector service when search itself becomes a shared, independently operated system.
• Treat containers as a deployment convenience, not a prerequisite for semantic search.

What the proposed architecture actually does

The implementation described in the post is straightforward. SQLite acts as the durable memory layer. Go acts as the retrieval engine. The application filters records by namespace, computes vector similarity locally, ranks the results, and returns the top matches to the agent.

A useful conceptual schema looks like this:

On a request, the application embeds the user’s question, fetches candidate vectors for the active namespace, scores every candidate, sorts by score, and selects the top k. That is exact nearest-neighbor search: every candidate is considered rather than approximated.

The source post reports under-2 ms similarity search across approximately 1,000 chunks. That is plausible for a warm, in-process brute-force scan, especially when vectors are already available as float arrays. It should be understood as an application-specific benchmark, not a portable promise: embedding dimension, CPU state, allocation behavior, SQLite read strategy, filtering selectivity, and whether decoding happens in the request path can all change the result dramatically. (reddit.com)

Why cosine similarity is enough at small scale

Cosine similarity compares the angle between two vectors. For embeddings, it is commonly used to estimate semantic proximity: a score closer to 1 indicates vectors pointing in more similar directions.

For vectors a and b, the calculation is:

cosine(a, b) = dot(a, b) / (||a|| × ||b||)

A basic implementation iterates over every dimension, accumulating the dot product and the two vector norms. That means one query requires work proportional to:

number of candidate chunks × embedding dimensions

At 1,000 chunks and 768 dimensions, the scoring loop performs roughly 768,000 multiply-add-style operations before sorting. Modern laptop CPUs can do that quickly. At 100,000 chunks, the same query performs roughly 76.8 million dimension comparisons—before accounting for fetching, decoding, filtering, sorting, garbage collection, and other application work.

The math itself is not the problem. The question is whether the amount of work remains comfortably inside your latency and memory budget as data grows.

The community’s criticism is the crucial caveat

The leading community response to the post identifies the central limitation: the reported speed is brute-force cosine similarity over a small in-memory working set, and the vectors themselves eventually become the memory problem the design was intended to avoid.

That is exactly the right critique. A local SQLite RAG implementation can avoid the baseline overhead of Docker and a database server, but it cannot repeal the storage cost of dense embeddings.

OpenAI’s embeddings documentation, for example, lists default vector lengths of 1,536 dimensions for text-embedding-3-small and 3,072 dimensions for text-embedding-3-large; its API also supports choosing reduced dimensions for the version 3 embedding models. (platform.openai.com)

A float32 takes four bytes. So the raw vector memory alone is approximately:

• 1,000 vectors × 1,536 dimensions × 4 bytes = 5.86 MiB
• 100,000 vectors × 768 dimensions × 4 bytes = 292.97 MiB
• 100,000 vectors × 1,536 dimensions × 4 bytes = 585.94 MiB
• 100,000 vectors × 3,072 dimensions × 4 bytes = about 1.14 GiB

Those numbers exclude the original text, metadata, database pages, Go slice headers, result buffers, query vectors, application memory, the embedding model if it runs locally, and operating-system cache. They also describe raw binary float32 values—not JSON arrays.

This is why the phrase “under 100,000 chunks” needs qualification. A 100,000-chunk collection might be perfectly comfortable on a desktop with ample RAM, but it is no longer automatically lightweight on an 8GB laptop, especially if the application has other agent tools, browser tabs, an IDE, and a local model competing for memory.

JSON embeddings are convenient, but not the best storage format

The original implementation stores JSON-encoded float32 embeddings in SQLite. It is an understandable starting point: JSON is inspectable, easy to debug, language-neutral, and fits naturally alongside text metadata. SQLite includes JSON functions and supports a binary JSONB representation intended to be somewhat smaller and faster to process than text JSON. (sqlite.org)

But JSON is usually the wrong long-term representation for dense numeric vectors.

A float32 vector is binary data. When converted into JSON, every number becomes a decimal string, with commas, brackets, and formatting overhead. Reading it back requires parsing text and allocating a native numeric slice. At a few thousand chunks, this may be irrelevant. At tens of thousands, repeated decoding can become more expensive than the cosine loop itself.

A better pattern: binary vector storage, SQL metadata

For a small embedded RAG system, keep SQLite—but store vectors as packed little-endian float32 bytes in a BLOB column. This produces a predictable footprint of dimensions × 4 bytes per vector and avoids JSON parsing during retrieval.

The practical pattern is:

1. Serialize the embedding into a fixed-width float32 BLOB during indexing.
2. Store the dimension count and embedding-model identifier in separate columns.
3. Query SQLite for IDs, content, metadata, and BLOBs constrained by the relevant namespace.
4. Decode each BLOB into a reusable buffer or use a safe zero-copy approach only if your language and driver make that reliable.
5. Score candidates in native code and retain only the best k values with a small heap.

Do not casually mix embeddings from different models or dimensions in the same search space. A migration from one embedding model to another should either create a new collection/namespace or version every record so queries only compare compatible vectors.

Normalize at write time when possible

If vectors are normalized to unit length at index time, cosine similarity becomes a dot product at query time, assuming the query vector is normalized too. That saves norm calculations for every stored vector and makes the hot path simpler.

There are guardrails: reject zero vectors, preserve the original model’s recommended similarity behavior, and validate that your embedding provider’s output is suitable for normalization. But for a homogeneous dense-vector store, pre-normalization is one of the cleanest low-level optimizations available.

Exact search versus approximate nearest-neighbor search

Brute-force search is often described as naive, but for small corpora it has real advantages. It is exact, straightforward to test, has no index-build phase, and has none of the recall tradeoffs introduced by approximate nearest-neighbor (ANN) methods.

ANN systems trade a small amount of retrieval certainty for much lower query work at larger scales. HNSW, one of the most common approaches, builds a graph over dense vectors to make approximate nearest-neighbor retrieval fast. Qdrant’s documentation also notes that its payload indexes consume additional memory and disk, and its indexing, vector storage, and memory settings can be tuned independently. (qdrant.tech)

Neither option is inherently superior. They solve different constraints.

The important detail is that SQLite can still do the first half of retrieval very well: metadata filtering. If a user belongs to one workspace, one repository, one project, or one date range, let SQL reduce the candidate set before native vector scoring begins.

That turns the question from “How many vectors exist in total?” into “How many vectors are eligible for this query?” For many product workflows, that second number stays small.

When SQLite RAG is the right choice

A simple embedded retrieval system is especially strong when the workload has natural limits. Think personal knowledge bases, coding assistants that index one repository, customer-success copilots scoped to a single account, offline document tools, and local AI agents that search recent notes and memories.

SQLite RAG is a strong default when most of these statements are true:

• The app is single-user, desktop-first, self-hosted, or has only a small number of concurrent users.
• The typical query searches a narrow namespace rather than every tenant’s data.
• Your corpus is measured in thousands or low tens of thousands of chunks per active search space.
• You need simple offline backups and easy portability.
• Exact top-k matches are useful, and a full scan fits your latency budget.
• The engineering team would rather spend time improving chunking, prompts, evaluation, and permissions than operating another database.
• You are validating a product hypothesis and expect the data model to change repeatedly.

For founders, this can be a competitive advantage. A one-binary app with an embedded database is easier to demo, easier to distribute, and easier to run in privacy-sensitive environments. It also makes the local-first story more credible because the user can keep documents and retrieval data on-device.

The simplicity is valuable only if you preserve an exit path. Design the storage model so an eventual migration to a dedicated engine does not require rewriting every product feature.

When a dedicated vector database becomes worth it

The original post’s broad “no heavy infrastructure below 100k chunks” message is too optimistic if read as a general rule. Scale is not only a record count; it is the interaction of dimensions, filters, update rate, concurrency, latency objectives, hardware, and retrieval quality requirements.

Move beyond a basic embedded scan when one or more of these conditions appears:

1. Active candidate sets are consistently large. If each request compares against tens or hundreds of thousands of vectors, linear work will eventually show up in tail latency and CPU cost.
2. You need predictable performance under concurrency. A single laptop benchmark does not represent 50 simultaneous user requests, ingestion jobs, and reranking.
3. You require sophisticated filtering at scale. Many metadata fields, high-cardinality filters, and permission-aware search can justify a system with dedicated payload indexes.
4. Your corpus changes continuously. High ingestion volume, deletes, re-embedding operations, and index maintenance create a different operational problem than a mostly static personal archive.
5. Memory is tight but the corpus must grow. Mature vector systems offer features such as on-disk vector storage, quantization, and memory-tier controls. Qdrant, for instance, documents memory-mapped vector files and configurations that balance RAM use against speed. (qdrant.tech)
6. You need hybrid retrieval or advanced ranking. Dense vectors are only one signal. Combining lexical retrieval, metadata filtering, multiple vectors, reranking, and analytics can exceed what an intentionally minimal implementation should own.

This is not a case for starting with infrastructure just because it may be needed someday. It is a case for measuring the signals that tell you “someday” has arrived.

The overlooked bottleneck: retrieval quality, not retrieval speed

Builders frequently optimize vector infrastructure before they have shown that their RAG pipeline retrieves the right source material. A 0.5 ms search result that contains the wrong chunks is still a product failure.

Before moving from SQLite to a heavier system, evaluate these higher-leverage variables:

• Chunk boundaries: Do chunks preserve a coherent thought, API procedure, policy section, or code unit?
• Chunk size and overlap: Is context split too aggressively, or are chunks so large that they dilute relevance?
• Metadata: Can the system filter by customer, repository, document freshness, locale, access role, or content type before semantic ranking?
• Embedding model: Does it reflect the language, domain, and query style your users actually use?
• Top-k selection: Are you retrieving too little context or flooding the LLM with weak matches?
• Reranking: Would a lightweight reranker improve the final ordering more than an ANN index would?
• Answer grounding: Does the final model cite, quote, or otherwise visibly connect its response to retrieved source material?

A practical evaluation set should contain real user questions, expected source passages, and pass/fail judgments. Track retrieval recall separately from answer quality. That separation helps identify whether the failure is in chunking and search, prompt construction, or the generation model.

The larger point: the infrastructure discussion should follow the retrieval evaluation, not replace it.

A pragmatic implementation blueprint in Go

If you want the portability benefits of the Reddit approach without creating a dead end, build an embedded retrieval layer with clear interfaces and instrumentation.

Indexing flow

1. Extract source text and normalize it consistently.
2. Split content into chunks with stable identifiers and source offsets.
3. Generate an embedding for each chunk.
4. Validate dimension count and reject malformed output.
5. Normalize the vector if that matches your similarity design.
6. Write text, metadata, model version, hash, and packed vector bytes in one SQLite transaction.
7. Record indexing timestamp and source revision so stale chunks can be replaced safely.

Query flow

1. Authenticate the request and derive its permitted namespace or tenant filters.
2. Embed and normalize the query.
3. Use SQLite to fetch only eligible candidates.
4. Score vectors in Go.
5. Keep a fixed-size min-heap for the best results rather than sorting every candidate when the set grows.
6. Apply a minimum-score threshold or score-gap heuristic where appropriate.
7. Optionally rerank the top 10 to 50 candidates.
8. Send the selected chunks to the generation model with source metadata.

Engineering details that matter

Avoid allocating a new large slice for every row on every request. Reuse buffers where safe, keep request-scoped data bounded, and benchmark with realistic namespaces rather than synthetic single-document tests.

Use SQLite transactions for batch ingestion, enable an appropriate journaling mode for your read/write pattern, and test concurrent access under the exact driver and operating system you will ship. SQLite is robust, but an embedded database does not remove the need to understand locking, write contention, backups, and crash recovery.

Finally, expose a narrow Retriever interface in your application. The rest of the product should ask for Search(query, filters, k), not care whether results come from brute-force SQLite, an embedded vector extension, or a remote vector service. That small abstraction is what makes the initial simple choice reversible.

Benchmark the system you actually plan to ship

The source post’s sub-2 ms number is a useful proof that simple code can be extremely fast at a small scale. It is not enough to make a capacity decision.

A credible benchmark should report:

• Number of chunks and number of vectors actually searched after filters
• Embedding dimension and data type
• Whether vectors are cached, fetched from SQLite, or JSON-decoded during each query
• Cold-start versus warm-cache performance
• Median, p95, and p99 latency rather than a single best case
• CPU and peak RSS memory use
• Concurrent query count
• Ingestion activity during search
• Top-k size, reranking cost, and end-to-end latency including embedding generation
• Retrieval-quality metrics, not only speed

Run the test at 1,000, 10,000, 50,000, and 100,000 candidates—or at the closest distributions your product expects. If namespaces are central to your design, benchmark both your average namespace and the largest plausible namespace. A giant global corpus is irrelevant if every user query is scoped to 2,000 documents; conversely, an average of 1,000 hides a few enterprise tenants with 500,000 chunks.

This discipline also prevents false precision. “Sub-2 ms vector search” can coexist with a 400 ms remote embedding call, a 1.5-second reranker, or an 8-second LLM response. Optimize the path that affects user experience, not the easiest microbenchmark.

The sensible scaling path: start simple, migrate intentionally

The best takeaway from the Reddit discussion is not “never use Chroma, Qdrant, or pgvector.” It is “earn the complexity of those tools.”

A staged approach looks like this:

This path avoids both extremes. You do not launch a personal tool with distributed-search infrastructure. You also do not let a clever 1,000-vector demo become an unmeasured production bottleneck.

Conclusion: simple RAG is a feature, not a compromise

SQLite RAG is a practical answer to a common problem: modern AI tutorials often recommend infrastructure designed for a later stage than the builder has reached. For a local agent on constrained hardware, an embedded SQLite store plus in-process cosine similarity can provide fast, exact retrieval with fewer dependencies and less operational friction.

The community’s objection is equally important: vector memory and linear search costs do not disappear merely because Docker does. A 100,000-vector dataset can be hundreds of megabytes to more than a gigabyte in raw float32 embeddings, depending on dimensionality, and JSON encoding adds avoidable overhead.

So adopt the simple architecture—but make it disciplined. Store vectors efficiently, filter aggressively with metadata, normalize where appropriate, measure realistic p95 latency and memory, evaluate retrieval quality, and isolate the retriever behind an interface. That gives small teams the simplicity they need today and a credible migration route when their RAG workload truly outgrows it.

FAQ

Is SQLite RAG good enough for production?

Yes, for bounded workloads such as local-first apps, small-team tools, tenant-scoped knowledge bases, and early products. Production readiness depends on backup strategy, access control, concurrency testing, retrieval evaluation, and observability—not whether the database has a separate server process.

How many vectors can SQLite RAG handle?

There is no universal limit. The meaningful measure is the number of vectors searched per request, multiplied by embedding dimension and concurrent query load. A few thousand candidates are usually trivial; tens or hundreds of thousands require benchmarks on your hardware and workload.

Should embeddings be stored as JSON in SQLite?

JSON is acceptable for a quick prototype and debugging, but packed float32 BLOBs are generally more space-efficient and avoid repeated text parsing. Keep model version and dimension count in separate fields so you can safely validate and migrate embeddings.

When should I switch from brute-force cosine similarity to ANN?

Switch when linear scans threaten your latency, CPU, or memory budgets; when active filtered candidate sets become consistently large; or when you need high concurrency, advanced filtering, rapid ingestion, and predictable tail performance. ANN is a scaling tool, not a prerequisite for every RAG app.

Does avoiding Docker mean avoiding vector databases forever?

No. You can run a vector engine without Docker, use an embedded library, or move to a managed service later. The important decision is whether the retrieval problem currently justifies the operational and memory overhead of a dedicated search system.