Context Engineering: The Thing That Actually Makes AI Agents Work
I've been building with AI agents for months now - RAG pipelines, multi-agent systems, coding assistants, personal tools. And the thing I keep coming back to is this: the model matters way less than I thought. What actually matters is memory.
Not memory in the traditional computer science sense. Memory as in: what does the agent know right now? What did it know yesterday? What will it remember tomorrow? How does it decide what to keep and what to throw away?
This is context engineering. And once you start seeing it, you realize it's the thing that separates agents that feel magical from agents that feel like fancy autocomplete.
The Problem Nobody Talks About
Here's the dirty secret of AI agents: they're stateless. Every time you send a message, the model processes the entire conversation from scratch. It doesn't "remember" your previous message - it re-reads the whole transcript and generates the next response. The illusion of memory is just the growing list of messages in the context window.
This works fine for a 10-message chat. It breaks down completely when you're:
- Working on a codebase for 3 hours straight
- Building up knowledge about a topic over weeks
- Running an agent that needs to maintain state across sessions
- Using tools that generate thousands of tokens of output
The context window is finite. On a long Claude Code session, the input can easily hit 100,000-200,000 tokens. Without caching, the server processes all of those tokens fresh on every single message. That's slow and expensive. With caching, the unchanged prefix is served at 1/10th the cost. But caching is fragile - change anything in the prefix and everything after it is invalidated.
This is why the Anthropic team builds their entire Claude Code harness around prompt caching. They run alerts on cache hit rates and declare incidents when the rate drops too low. The cache isn't an optimization. It's the thing that makes the product economically viable.
Memory Is the Harness
There's a great framing from Sarah Wooders at Letta: "Asking to plug memory into an agent harness is like asking to plug driving into a car."
Memory isn't a feature you bolt onto an agent. It IS the agent, or at least the most important part of it. The harness - the scaffolding around the model - makes hundreds of invisible decisions about memory:
- How is the system prompt structured and loaded?
- What survives when the context window fills up and gets compacted?
- How is conversation history managed and summarized?
- What information persists across sessions?
- How does the agent decide what's relevant right now?
Different harnesses answer these questions differently. Claude Code has a multi-level memory hierarchy: a static system prompt, CLAUDE.md files for project context, session memory, and conversation messages. Letta Code takes it further with a git-backed memory filesystem that agents can read and write to directly. Codex has its own approach with AGENTS.md and encrypted compaction summaries.
The choice of harness is a choice about memory. And if you use a closed harness - especially one behind a proprietary API - you're giving up control of your agent's memory to a third party. That's a bigger deal than most people realize.
The Context Engineering Stack
After working with several different approaches, here's how I think about the layers of context engineering:
Layer 1: Prompt Architecture
The most basic level. How you structure the prompt determines what gets cached and what doesn't.
Claude Code's ordering is instructive:
1. Tool definitions (static, globally cached)
2. System prompt + CLAUDE.md (cached within a project)
3. Session context (cached within a session)
4. Conversation messages (grows each turn)Static content first, dynamic content last. This maximizes prefix sharing across requests. But it's surprisingly fragile. Putting a timestamp in the system prompt, shuffling tool definitions non-deterministically, toggling features on/off - any of these can break the cache and make everything downstream get reprocessed from scratch.
The counterintuitive lessons:
- Don't update the system prompt to change state. Instead, insert a
<system-reminder>in the next message. The system prompt stays cached. - Don't add or remove tools mid-session. Use tools to model state transitions instead. Claude Code's plan mode doesn't swap out tools - it calls an
EnterPlanModetool while keeping all tools in the request. - Don't switch models mid-conversation. Caches are per-model. Switching from Sonnet to Opus means rebuilding the entire cache from scratch. It's actually cheaper to let Opus answer a simple question than to switch to Haiku and pay for a full cache rebuild.
Layer 2: Compaction and Summarization
What happens when you hit the context window limit? You can't just truncate - you'd lose crucial context from earlier in the conversation. You can't keep everything - there isn't room.
The answer is compaction: summarize the conversation so far and continue with the summary. But the naive implementation (separate API call with a different system prompt) causes a complete cache miss. You pay full price for all those tokens.
Claude Code's approach: use the exact same system prompt, tools, and conversation history as the parent conversation, then append the compaction prompt at the end. The cached prefix is reused. The only new tokens are the compaction instruction itself.
This is a detail that matters enormously for cost and latency but is invisible to the user. It's exactly the kind of thing that lives in the harness layer.
Layer 3: Session Memory
Within a single session, you're working with the conversation history. But across sessions, you need something persistent. This is where things get interesting.
Claude Code uses CLAUDE.md files - markdown files that get loaded into every session for a project. You can put project context, coding conventions, architectural decisions in there. It's simple and it works.
Here's what my setup looks like in practice: my CLAUDE.md points to an AGENTS.md file that contains the real context - project architecture, domain knowledge, coding patterns, things the agent needs to know to be effective. I also use graphify to build knowledge graphs of my codebases, so the agent can navigate by structure instead of grepping through every file. And for giving instructions, I use Whisprflow to dictate what I want - no structured prompts, just talking naturally and trying to give as much context as possible. The more context you front-load, the better the agent performs. It's that simple.
This is context engineering at the coding level: setting up your project so that every time the agent spins up, it already knows what it needs to know. The CLAUDE.md and AGENTS.md files are your persistent memory. graphify gives you structural memory. Your conversation gives you session memory. Layer them together and the agent feels like it actually understands your codebase.
The more advanced version is what Letta calls "system prompt learning" - the agent modifies its own system prompt over time based on what it learns. User preferences, corrected assumptions, behavioral guidelines. The system prompt becomes a living document that the agent curates.
This is a genuinely new idea. Historically, system prompts were static things written by humans. The idea that the agent should evolve its own prompt based on experience is powerful but raises obvious questions about drift, stability, and trust.
Layer 4: Long-term Knowledge
This is where I've been spending the most time personally. Not for code - for learning.
I've been experimenting with Andrej Karpathy's LLM Wiki pattern. The idea is simple but powerful: instead of using RAG (retrieve chunks at query time, forget everything between queries), you have the LLM incrementally build and maintain a persistent wiki. When you add a new source - an article, a paper, notes from a podcast - the LLM doesn't just index it. It reads it, extracts key information, and integrates it into the existing wiki. It updates entity pages, revises topic summaries, flags contradictions with existing content, strengthens or challenges the evolving synthesis.
The difference from RAG: the wiki is a persistent, compounding artifact. The cross-references are already there. The contradictions have already been flagged. The synthesis already reflects everything you've fed it. It gets richer with every source you add.
I'm using it as a personal knowledge base - the stuff I'm learning about AI, engineering, things I'm interested in. Every article I read, every interesting idea I encounter, it goes into the wiki. The LLM does all the grunt work - summarizing, cross-referencing, filing, maintaining consistency across pages. I just direct the exploration and ask questions.
It's the difference between a search engine and a library. RAG is a search engine - it finds relevant chunks when you ask. The wiki is a library - it's organized, cross-referenced, and the librarian (the LLM) keeps it maintained.
For navigating the wiki as it grows, I use qmd - a local search engine for markdown files that does hybrid BM25/vector search with reranking, all on-device. It has both a CLI and an MCP server, so the LLM can use it as a tool. At small scale (under 100 pages) the wiki's own index.md file is enough. Beyond that, you need real search.
Why RAG Isn't Enough
I built a RAG chatbot (AskBase - wrote about it in a previous post). It works. But every query starts from scratch. There's no accumulation. Ask a subtle question that requires synthesizing five documents, and the system has to find and piece together the relevant fragments every time. Nothing is built up between queries.
RAG is retrieval. The wiki pattern is comprehension. They solve different problems.
RAG is great for: "find me the document that answers this specific question." It's a search problem.
The wiki is great for: "what's the current state of my understanding of this topic, given everything I've read?" It's a synthesis problem.
In practice, you probably want both. RAG for precise retrieval, the wiki for accumulated understanding. The wiki itself can be the corpus that RAG searches over - and because the wiki is already synthesized and cross-referenced, the retrieval quality is much better than searching raw source documents.
The Knowledge Graph Angle
There's another layer beyond flat wiki pages: knowledge graphs. Instead of just writing prose, the LLM extracts structured entities (people, projects, concepts, libraries) and typed relationships between them ("uses", "depends on", "contradicts", "caused").
graphify is a tool that does this for codebases. It runs AST analysis on code files and uses LLM extraction on docs, images, and papers, then builds an interactive knowledge graph. The interesting part is that it reduces the tokens needed per query by 70x+ compared to reading raw files. The graph is a compressed, navigable representation of the codebase's structure.
The same idea applies to any knowledge domain. A wiki page about "attention mechanisms" is useful. A graph node for "attention" with typed edges to "transformer", "self-attention", "cross-attention", "scaled dot-product", each with confidence scores and source citations - that's more useful for precise queries.
But you don't need to choose. The wiki gives you readable prose. The graph gives you structured navigation. They complement each other.
The Open Memory Problem
Here's the thing that concerns me about where the industry is heading.
Memory creates lock-in. If your agent has been learning your preferences, building up context about your codebase, accumulating project knowledge for months - and all of that lives behind a proprietary API - you can't leave. Switching to a different model or platform means starting from zero.
Model providers know this. It's why Anthropic launched Claude Managed Agents (everything behind an API). It's why Codex generates encrypted compaction summaries that only work within the OpenAI ecosystem. They're incentivized to make memory proprietary because memory is the stickiest part of the experience.
Without memory, your agents are easily replicable by anyone with access to the same tools. With memory, you've built up a proprietary dataset of user interactions and preferences that creates a genuinely differentiated experience.
This is why open harnesses matter. If you own your harness, you own your memory. Your CLAUDE.md files, your wiki, your knowledge graph - these are portable. They're just files. You can take them to any model, any platform. The memory isn't locked into a specific provider's infrastructure.
I think this is going to be one of the most important debates in AI over the next year. Not which model is best - they're converging. But who owns the memory that makes agents actually useful.
What I've Learned So Far
After months of working with different memory approaches, here's what's stuck:
1. The cache is the foundation. For real-time agent interactions, prompt caching determines cost and latency. Design your entire system around prefix matching. Static content first, dynamic last. Never change tools or models mid-session. Monitor cache hit rates like uptime.
2. RAG is a building block, not the destination. It solves retrieval. It doesn't solve comprehension or accumulation. For anything where you need knowledge to compound over time, you need something more.
3. The wiki pattern is underrated. Having an LLM maintain a persistent, structured knowledge base that compounds with every source you add is genuinely transformative. The maintenance burden that kills human-maintained wikis is exactly what LLMs are good at.
4. Memory should be open and portable. Files on disk, markdown you can read, formats any model can process. Don't let your agent's memory get locked into a platform.
5. Context engineering is the new prompt engineering. Prompt engineering was about crafting the right instruction for a single query. Context engineering is about designing the entire information architecture around your agent - what it knows, when it knows it, how long it remembers, what gets evicted, what persists. It's a systems design problem, not a copywriting problem.
The models will keep getting better. Context windows will keep growing. But the fundamental challenge of memory - what to keep, what to forget, how to organize, how to retrieve - that's not going away. If anything, it's becoming the central problem.
The agent that remembers best, wins.
I'm still early in this journey. If you're working on agent memory, context engineering, or personal knowledge management with LLMs, I'd love to compare notes.