I spent a year building agent memory on knowledge graphs. Here are my 5 mistakes

3 points

6 days ago

I spent the past year building a unified memory layer for my AI agents using knowledge graphs and ontologies on top of MongoDB. I followed every trend first. I reached for the shiny frameworks and tried to design the perfect upfront ontology. I made basically every mistake possible.

Naive memory fails at scale. File search bloats the context window once memory gets big. Claude Code handles it this way out of the box. Even semantic search over history can't traverse the relationships between people, topics, objects, locations, and preferences. The fix was to stop treating memory as a retrieval problem and treat it as a *data-modeling problem*.

Here are the 5 mistakes I made:

1. I reached for frameworks first. I tried LangGraph and CrewAI. The moment I needed custom ontology constraints, immutable observation logs, composite IDs, and multi-hop traversal, I was fighting the framework. Lesson: Own the memory and the harness yourself because frameworks encode assumptions your system rarely matches.

2. I overthought the ontology. Knowing it's a data-modeling problem, I tried to design the perfect ontology upfront. This froze projects for months. Lesson: Ontology design is a data-exploration loop. Start with POLE+O (Person / Object / Location / Event / Organization) and extend only on collisions. For example, I had a run tagging "Claude Code" as a Person when it's an Object.

3. I confused resolution with deduplication. Naming is not identity. Confusing them corrupts the graph. Resolution normalizes names, while deduplication decides identity from the entities' context. Lesson: Use specific thresholds: ≥0.95 auto-merges, >0.85 triggers human review, and ≤0.85 creates a new node. This stops "Apple" the company from merging with "Apple" the fruit.

4. I only built short-term and long-term memory. The agent repeated failed strategies because I skipped reasoning memory. This is a trace per run including the strategy, tools used, and the success or failure. Lesson: Reasoning memory is like RL at the database layer instead of the weights. Honest caveat: it can backfire because bad traces reinforce bad strategies, and it's overkill for one-off tasks.

5. I tried to build an immutable log layer before materializing the graph into the database because it sounded fancy, as it adds versioning and temporality to the graph. The con is that it puts a ton of pressure on your VM's RAM, which is crazy expensive. Lesson: Do that ONLY if you really need it.

The schema decides everything about the system's performance. Settling on edges as first-class documents in MongoDB allowed for native `$graphLookup` and finally let the system scale. This approach avoids relationship duplication and makes writes much simpler.

If you want to understand the whole reasoning behind these mistakes supported by the system of my agentic memory via KG and ontologies, consider going over my latest 6 LinkedIn posts:

1. 3 ways to model your ontologies for GraphRAG → https://www.linkedin.com/feed/update/urn:li:share:7446856909179027456

2. LangGraph/CrewAI or from scratch? → https://www.linkedin.com/feed/update/urn:li:share:7449362677560221696

3. A year building GraphRAG from scratch → https://www.linkedin.com/feed/update/urn:li:share:7449366886603128833

4. The third memory type: reasoning memory → https://www.linkedin.com/feed/update/urn:li:share:7454454641939034113

5. Building a production-grade personal AI assistant → https://www.linkedin.com/feed/update/urn:li:share:7456973563858821120

6. Designing Your Agents' Unified Memory → https://www.linkedin.com/feed/update/urn:li:share:7464580605327060992