Vector database vs markdown for agent memory

For the memory an AI coding agent needs between sessions, plain markdown files are the better default, and a vector database is usually the wrong first reach. That memory is a few hundred short records: decisions you made, facts about the system, follow-ups nobody has closed. A vector database stores text as embeddings, which are lists of numbers that capture meaning, and retrieves them by similarity. That machinery is built for a large pile of documents you will never read yourself, and agent memory is not that.

The more useful framing is that the two sit at different layers. Markdown files are a store of record, the thing you edit, review, and correct. A vector database is an index, a derived structure you can rebuild from the source at any time.

Good setups keep the truth in files and add embeddings once the collection outgrows exact search. Going vector first means your agent’s memory lives somewhere you cannot open, diff, or fix by hand.

What each option actually is

Markdown memory is a folder of .md files, one decision or fact per file. A small block of YAML metadata at the top records its kind, project, and status, and the body runs a few sentences. Retrieval is a text search, or the agent reads a file you pointed it at.

A vector database stores each piece of text alongside its embedding. To build one you split documents into chunks, embed each chunk, and store the resulting vector. Most keep the original chunk text next to the vector, so the words are not lost. That text lives as a database row rather than a file you can open, and changing it means updating the row and re-embedding.

The options span a wide cost range. pgvector and sqlite-vec add vector columns to a Postgres or SQLite database you already run, so the operational cost is close to nothing. Qdrant, Weaviate, Chroma, and Pinecone are dedicated stores, and each brings a service to keep alive.

Agent memory is smaller than you think

Scale is the intuition that pushes people toward a vector database, and the scale usually is not there. An agent logging every decision, fact, and follow-up tends to land in the hundreds of notes across a year, not the hundreds of thousands. That is well under a megabyte of text, and listing your own memory folder will give you a real number rather than an estimate.

At that size, exact search is not just adequate, it is more predictable. You know why a result came back and why one did not. A vector database here adds an embedding model, an index to keep fresh, and a new way for retrieval to be quietly wrong, in exchange for very little.

Where a vector database genuinely wins

The first case is vocabulary mismatch at volume. If your agent asks “why is checkout slow” and the note says “cart latency regression after the pricing rollout”, keyword search misses and similarity search hits. That gap matters once you hold thousands of documents with no shared vocabulary across them.

The second is corpora nobody wrote for retrieval: years of support tickets, a scraped documentation set, meeting transcripts, imported PDFs. Similarity search is usually the fastest way in, because you cannot guess the words the writer used. Strong setups run it alongside keyword search, which still wins on names, error codes, and identifiers.

The third is genuine scale, though that word covers two problems. A slow full text scan calls for a full text index such as Lucene or SQLite FTS5, not for vectors. Vector indexes earn their place when the queries themselves are fuzzy and the results are too many to read by hand.

Where markdown wins

Markdown wins on everything that involves a human. You can read a memory file, see that it is out of date, and fix it in the editor you already have open. A vector gives you nothing to look at.

Correction matters more for memory than it does for search. Agent memory accumulates mistakes: a decision that got reversed, a fact that was true for one release. With files you edit the note or mark it superseded.

Vector stores can do this too, by replacing the record under its identifier or deleting every chunk from one document. That work runs through a client rather than an editor, though, and no diff shows you what changed.

Auditing is where the gap is widest. You can list every memory the agent holds, read it in a few minutes, and put the folder in version control so changes arrive as diffs. That is the difference between knowing what your agent remembers and assuming you know.

Durability is the quiet one. Markdown needs nothing but a text editor, so a note written today reads the same in ten years on any machine. Change your embedding model and every vector in the store is meaningless until you rebuild.

Markdown has one real weakness, and it is the mirror of that win. Exact search finds a note only if the agent guesses your vocabulary. If it searches “auth” and your note says “session token rotation”, that memory may as well not exist, and this bites at a hundred notes as readily as at a hundred thousand.

Consistent titles and a fixed set of values in that metadata block push the problem back a long way. Once your searches start missing anyway, an index is the answer, which is the setup below.

Side by side

Dimension	Markdown files	Vector database
Best at	Structured recall you also read	Fuzzy recall over large unread corpora
Human readable	Yes, as files	Yes, as database records
Editing a wrong memory	Open the file, change it	Update the record and re-embed it
Auditing everything stored	List the folder	Enumerate the collection through its API
Version control	Line by line diffs	Not meaningfully diffable
Infrastructure	A search index your editor maintains	A service or embedded engine you run
Recall when wording differs	Misses unless you guessed the words	Strong
Finding an exact phrase	Guaranteed hit	Needs the keyword leg of a hybrid setup
Model change	Nothing to do	Requires a full rebuild

The hybrid most setups end up with

The answer that holds up is layered. Keep every memory as a markdown file, because that is what you correct and audit. Build embeddings over those same files when the collection outgrows exact search, and treat the index as disposable: if it breaks, delete it and rebuild from the files.

Adding an index later is easy. Extracting readable memory back out of a vector store is not.

How this works in Jotura

Jotura is a local-first markdown notes app, so the memory an agent writes is ordinary .md files in an ordinary folder. Keyword search over 10,000 of them takes about 8 milliseconds in release builds. The bundled jotura command line tool gives agents structured memory operations on top of those files:

jotura memory log --kind decision \
  --title "Use Argon2id for key derivation" \
  --project "Billing" \
  --body "Ruled out scrypt after the security review. Revisit if hardware changes."

jotura memory recall "key derivation" --limit 5
jotura context "harden the auth flow"

memory recall ranks by relevance, freshness, and importance. context prints the open follow-ups and the decisions most relevant to the task you name, so the agent starts with them in view. Superseding a decision does not silently delete the old note:

jotura memory supersede Memory/2026-03-11-argon2id.md --by Memory/2026-08-02-argon2id-params.md
jotura memory list --kind followup --status open

The embedding layer runs on your machine too. Turn on semantic search in desktop Settings, which downloads a local embedding model once and needs no API key. The command line tool queries that index rather than building its own:

jotura search "what did I decide about hosting" --mode semantic --json
jotura search "cert renewal" --mode smart --json

Two limits are worth stating. The desktop app maintains the semantic index, so --mode semantic from the command line exits with code 13 until you have enabled it there. Jotura is also closed source, so if you need to audit the retrieval code itself rather than the stored memories, this is not the tool for you. The memories themselves are plain files you can read without us.

For more on the underlying pattern, see give AI agents persistent memory and audit what your AI agent remembers. The CLI reference covers the full command surface.

Questions people ask

Is semantic search the same thing as a vector database? It uses the same machinery. The difference is which copy counts as the truth. Jotura derives its semantic index from your markdown files and rebuilds it from them, so the files stay the record.

I have 5,000 notes. Do I need embeddings? Probably not for exact lookups, since full text search handles that size comfortably. You may still want them for questions phrased differently from how you wrote the notes. Semantic search offline covers what that feels like in practice.

What about a memory server over the Model Context Protocol? That is a different axis: protocol, not storage. A memory server can be backed by either files or vectors. MCP memory server vs plain markdown files works through the tradeoff.

Can the two disagree? Yes, in one way that matters. The index can go stale after you edit a file, so it can return text the file no longer contains. The rule that fixes it is that the file always wins: when they differ, reindex, because the index is the disposable copy.

Start with files. Add vectors when your own searches start missing things, and keep the files either way.