Caching embeddings locally for a private retrieval layer

The first version of every local retrieval setup I’ve built makes the same mistake: it re-embeds everything, every time, because embedding one document feels cheap enough not to think about.

It’s cheap once. Run it against a folder of a few thousand files on a schedule, or on every startup, and the “cheap” operation is now the thing your laptop’s fan spins up for. The fix is boring and worth doing on day one rather than after you’ve noticed the fan.

The thing worth caching, and the key that makes it correct

Cache the embedding vector, keyed by a hash of the document’s content, not its filename or path.

That distinction matters more than it looks. Keying by filename means a renamed file re-embeds for no reason and a silently edited file with the same name serves a stale vector forever, which is the worse failure of the two because it’s invisible. Keying by content hash, a SHA-256 of the text you’re about to send to the model, means the cache is correct by construction: same content, same hash, same vector, returned instantly. Different content, even by one character, is a cache miss and a fresh embedding.

import hashlib, json, sqlite3

def content_hash(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()

def get_embedding(text: str, model: str, db: sqlite3.Connection):
    key = content_hash(text) + ":" + model
    row = db.execute("SELECT vector FROM embeddings WHERE key = ?", (key,)).fetchone()
    if row:
        return json.loads(row[0])
    vector = ollama_embed(text, model)  # the actual call
    db.execute("INSERT INTO embeddings (key, vector) VALUES (?, ?)", (key, json.dumps(vector)))
    db.commit()
    return vector

Include the model name in the key. This is the detail that bites people later: swap nomic-embed-text for a newer model and every cached vector from the old one is silently wrong for the new one’s vector space, unless the model tag is part of what you’re keying on. SQLite is enough for this. It’s a key-value lookup with a hash key; you don’t need a vector database until you’re actually running similarity search at scale, which is a separate concern from caching.

Why local, specifically

The obvious answer is cost, and it’s real: embedding is the highest-volume call in most retrieval pipelines, run once per chunk of every document, and paying an API per call for something you’ll frequently recompute anyway adds up fast.

But the reason I actually reach for Ollama here isn’t cost. It’s that embedding is where the content leaves your machine. A chat completion sends a prompt. An embedding call sends the full text of every document you’re indexing, which for a lot of the material worth building retrieval over, internal docs, contracts, anything with a name attached, is exactly the content you didn’t want on a third party’s logs in the first place.

Running the embedding model locally means that question doesn’t need an answer. Nothing leaves. The cache then becomes not just a performance optimisation but the thing that makes re-indexing cheap enough that you’ll actually do it when the source documents change, instead of leaving a stale index running for months because rebuilding it is annoying.

Where this breaks

Two failure modes worth knowing before they surprise you.

Chunking strategy is part of the cache key whether you think about it or not, because if you change how you split documents into chunks, every chunk’s text changes and every hash changes, and the entire cache silently invalidates itself on the next run. That’s correct behaviour, but it means a chunking change looks like a full re-embed even though nothing about the model changed, and it’s worth knowing that in advance rather than being confused by the CPU spike.

And the cache has no opinion about whether a document should still be in the index at all. Deleted or moved source files leave orphaned embeddings behind forever unless something explicitly prunes entries whose source no longer exists. A cache answers “have I computed this before,” not “should this still be here.” Those are different questions and conflating them is how an index quietly accumulates content nobody meant to keep searchable.

What this actually buys you

On a real corpus, re-embedding after this change goes from every document to only the ones that changed since last run, which in practice for most personal or team knowledge bases is a handful of files, not a few thousand. The first index build is still slow, there’s no way around embedding everything once. Every build after that is nearly instant, because you’re doing a hash lookup for content that hasn’t moved and a real embedding call only for the paragraph someone actually edited this morning.

That’s the whole trade: thirty lines of caching logic against never wondering again whether hitting re-index is going to lock up the machine for ten minutes.

More field notes on local LLMs

This piece is one entry in a running series on how AI coding tools change day-to-day engineering work. For more practical notes on local LLMs specifically, browse the full set at /blog/tag/local-llm/. For the wider view across every tool in the stack, the AI coding tag collects the whole archive in one place.