Hindsight AI: The Open-Source Memory Layer Teaching AI Agents to Learn, Not Just Remember
Hindsight AI: The Open-Source Memory Layer Teaching AI Agents to Learn, Not Just Remember
Every AI agent has the same quiet flaw: the moment a conversation ends, everything it just learned about you disappears. Ask it something today, come back tomorrow, and it treats you like a stranger. For the past two years, the industry’s answer has been retrieval-augmented generation, or RAG — storing old messages and pulling up whichever ones sound similar to the current question. That works for a quick lookup. It falls apart the moment an agent needs to genuinely remember a person, track how their preferences shift over time, or connect one fact to another the way a human colleague would.
A project called Hindsight, built by Vectorize, has quietly set out to solve this differently — and it is doing so in the open. It is fully open source under an MIT licence, you can run it on your own infrastructure for free, and it recently posted the highest accuracy ever recorded on LongMemEval, the toughest public benchmark for agent memory. In under a year, it grew from roughly 1,000 GitHub stars to more than 30,000, with commits landing almost daily. This article breaks down what Hindsight actually is, how it works under the hood, and whether it deserves a place in your own agent stack.
Why Agent Memory Has Been Broken Until Now
There are broadly three ways to give a language model a past, and each has a specific failure mode.
The first is the transcript: replay the whole conversation every turn. It’s simple and exact, but it dies on arithmetic — every replayed token costs money and latency, and no context window is infinite.
The second is RAG: chop history into chunks, embed them as vectors, and retrieve whatever is closest in meaning. This is the default in almost every agent framework today, and its flaw is subtle but serious — it retrieves text, not knowledge. If you mentioned a preference once in March and contradicted it in July, RAG will happily hand the model both chunks, ranked purely by how the sentences happen to be phrased, and leave the model to guess which one is still true.
The third is a knowledge graph of entities and relationships. More structured, but considerably harder to build and keep consistent as facts change.
Hindsight’s position is that none of these actually solve the real problem, because none of them learn. Remembering is storing a sentence. Learning is deciding what that sentence means, noticing when it conflicts with something already known, and updating the underlying belief. One of those is a database operation. The other is a process — and that process is what Hindsight is built around.
How Hindsight Works: Retain, Recall, Reflect
Everything in Hindsight reduces to three operations.
Retain
This is how information enters the system. Rather than dumping raw text into storage, Hindsight uses a language model to extract the key facts, entities, relationships and timing from whatever is passed in, then normalises all of it into canonical entities, time series and search indexes. If you tell it “Alice got promoted to senior engineer,” it doesn’t just store the sentence — it extracts Alice as an entity, the promotion as an event, “senior engineer” as a new attribute, and the exact timestamp, all of it searchable and linkable to anything else the system already knows about Alice.
Recall
This is retrieval, and it’s where Hindsight separates itself most clearly from basic vector search. Instead of one method, Recall runs four retrieval strategies in parallel: semantic search for conceptually related memories, BM25 keyword search for exact terms and proper nouns, graph-based retrieval that follows entity and causal links, and temporal filtering for time-bound questions. The results are merged with reciprocal rank fusion, re-ranked with a cross-encoder model, and trimmed to fit a token budget — so the agent gets the most relevant context rather than a flood of loosely related memories.
Reflect
This is the operation that earns the word “learn.” Where retain stores and recall retrieves, reflect reasons across existing memories, forming new connections that were never explicitly stated. A project-management agent can reflect on everything it knows to flag emerging risks. A sales agent can reflect on every outreach attempt to work out why some messages land and others don’t. A support agent can reflect across every conversation to spot gaps the documentation doesn’t cover. In each case, the agent isn’t just recalling what happened — it’s drawing a conclusion from it.
Four Memory Types, One Learning System
Hindsight doesn’t treat all information equally. It organises memory into four categories: world facts (objective, agent-independent knowledge, like “the stove gets hot”), experiences (what actually happened to the agent — “I touched the stove and it hurt”), observations (consolidated, evidence-backed beliefs that form automatically as related facts pile up), and mental models (a broader, synthesised understanding built from observations and facts together).
The clever part is how observations behave. When new information relates to an existing belief, that belief isn’t deleted and replaced — it’s refined. Evidence can strengthen it, soften it, or extend its scope, with a running record of exactly which facts support it. Mental models take this further: they’re a standing, pre-written answer to a question you define once — “what are this user’s preferences?” — that updates quietly in the background as the system learns more. Reading one is a plain database read, with no search and no extra model call, so an agent can start a session already equipped with settled knowledge instead of rebuilding it from scratch.
All of this lives inside isolated banks — one memory store per user, agent or project, with strict separation between them, which matters enormously for anyone running an agent across multiple customers.
The Benchmark Numbers Worth Taking Seriously
The headline figure is a score between 91% and 94% on LongMemEval, a benchmark widely used to test memory systems across realistic conversational scenarios — and Hindsight is reportedly the first memory system of any kind to cross the 90% mark. What gives that number real weight is who checked it: the result was independently reproduced by researchers at Virginia Tech’s Sanghani Center for Artificial Intelligence and Data Analytics, and separately by a team at The Washington Post. In a field where most vendors simply self-report their own numbers, having two outside, unaffiliated groups reproduce the same result is unusual, and worth noting explicitly.
Vectorize also publishes its own benchmark suite, the Agent Memory Benchmark (AMB), designed to test agentic tasks that older benchmarks never covered — tracking memory across tool calls, building knowledge from ongoing research, and applying learned preferences across multi-step decisions. The full methodology, prompts and scoring logic are published openly, and live, continuously updated results — including per-model accuracy, latency and cost — sit at a public leaderboard rather than a static chart.
Getting Hindsight Running
The simplest path is a single Docker command, which spins up a full server with an embedded Postgres database, an API and a web UI in a couple of minutes. For production, there’s a Docker Compose setup built around an external Postgres database, Oracle AI Database support for enterprise storage, a pip package for bare-metal installs, an official Helm chart for Kubernetes, and a fully embedded Python mode that needs no separate server at all. There’s also Hindsight Cloud, a managed version running the identical codebase, billed on usage with free credits to start.
Model support is unusually broad: more than 25 LLM providers, from the major hosted options to fully local setups through Ollama or LM Studio, plus any OpenAI-compatible endpoint. Notably, it can also run directly off subscriptions many developers already pay for — Claude Pro or Max, ChatGPT Plus or Pro, Cursor, and GitHub Copilot — removing the usual first hurdle of creating a separate billing account just to try something out.
On the integration side, a lightweight LLM wrapper (built on LiteLLM) adds memory to an existing agent with no rewrite required, and Hindsight ships with more than 60 ready-made integrations, including Claude Code, Cursor, LangChain, LangGraph, CrewAI, the OpenAI Agents SDK, n8n and Zapier, plus a built-in MCP endpoint per memory bank so any MCP-compatible client can use retain, recall and reflect as tools immediately.
For production use, Hindsight also includes an opt-in security layer called Memory Defense, which scans every retained item against 45 patterns for secrets and personally identifiable information, either redacting or blocking matches before they reach long-term storage — alongside Prometheus monitoring, an admin CLI, and webhooks for retain, consolidation and refresh events.
Where Hindsight Fits — and Where It Doesn’t
The simplest use case is personalising conversational agents: tagging memories by user so that one customer’s history never leaks into another’s context. The more ambitious use case, and the one Vectorize is explicit about, is agents that handle open-ended work over weeks or months, adjust their behaviour based on ongoing feedback, and gradually get better at their job — closer to how a human employee operates than a chatbot answering isolated questions.
To its credit, the team is equally direct about where this is overkill: for a simple automation built in a tool like n8n, this level of architecture is probably more than the task needs. That kind of honesty about a product’s own limits is rare, and worth taking at face value.
Final Thoughts
Three things are worth remembering about Hindsight. First, the retain, recall and reflect loop handles getting information in, pulling it back out through four parallel retrieval strategies, and reasoning over what’s been learned so far. Second, observations refine rather than overwrite, which is the actual engineering behind the word “learn.” Third, the whole thing is MIT-licensed and free to self-host — everything described here can be running on your own machine within minutes.
For teams building agents meant to have a long-term relationship with a user — a support agent, a coding assistant, a tutor that should know by week three that you hate meetings before 10am — memory has historically been the part everyone hand-rolls badly. Hindsight is a serious, independently verified attempt to turn it into infrastructure instead.
Frequently Asked Questions
What is Hindsight AI used for?
Hindsight is an open-source memory system for AI agents. It lets agents retain information over time, recall it accurately through multiple search strategies, and reflect on it to draw new conclusions — rather than starting from zero in every conversation.
Is Hindsight free to use?
Yes. The core project is open source under an MIT licence and can be self-hosted for free via Docker, pip or Kubernetes. A managed, usage-based option, Hindsight Cloud, is also available with free credits to start.
How accurate is Hindsight compared with other memory systems?
Hindsight reports a score of 91–94% on LongMemEval, the leading agent memory benchmark, independently reproduced by researchers at Virginia Tech and by The Washington Post — reportedly the first memory system to cross the 90% mark.
Does Hindsight work with Claude Code, Cursor and other coding agents?
Yes. Hindsight has over 60 built-in integrations, including Claude Code, Cursor, GitHub Copilot, LangChain and CrewAI, plus a built-in MCP endpoint that any MCP-compatible client can use directly.
Do I need a separate API key to run Hindsight?
Not necessarily. Hindsight supports more than 25 LLM providers and can run directly off existing subscriptions, including Claude Pro or Max, ChatGPT Plus or Pro, and GitHub Copilot, without a separate API key.
Ready to give your agents real memory? Explore Hindsight on GitHub and see how a learning memory layer could change what your own AI tools are capable of.
