The complete source code for Lyte is available on GitHub.
In Part 2, we gave the AI an Agentic loop and reigned in the massive token costs of recursive tool-use.
For small apps, it worked beautifully. But as the product matured and users began building more complex React applications (30+ files), generation quality plummeted.
The LLM was giving inconsistent answers. It would modify the wrong files, ignore existing functionality, and hallucinate imports.
The Root Cause: We were sending the entire codebase in the prompt.
Our HeuristicContextBuilder was incredibly naive:
if (files.length <= 15) {
return files; // Just send everything
} else {
return files.filter(f => !f.path.endsWith('.lock'));
}At 30+ files, the context window contained 80,000+ tokens of mostly irrelevant code.
LLMs suffer from the "Lost in the Middle" phenomenon. Relevant information placed in the middle of a long context window gets a significantly lower attention weight than information at the beginning or the end. The AI's attention was literally stretched too thin.
Context window management in AI is analogous to database query optimization. Sending a 30-file project into a prompt is like doing a SELECT * full table scan when you only need 5 rows. You need to know what you're looking for before you fetch it.
We needed RAG (Retrieval-Augmented Generation).
Codebase RAG with pgvector
We wanted to retrieve only the relevant file chunks based on the user's prompt.
Design Decision 1: Zero New Infrastructure
We already had Postgres deployed. We simply enabled the pgvector extension. No new services, no separate vector databases (like Pinecone or Weaviate) to manage.
Design Decision 2: HNSW over IVFFlat When building vector indexes, IVFFlat is faster to build but its centroid lists go stale as data changes, requiring periodic rebuilding. Project files change constantly (the LLM is literally writing them). HNSW (Hierarchical Navigable Small World) requires no retraining — you just insert new embeddings.
Design Decision 3: Hybrid Search (RRF)
Vector search is semantic. It finds your "auth controller" when you search for "login". But keyword search (BM25) is exact. It finds the auth-controller.ts file when you search for exactly auth-controller.ts.
We used Reciprocal Rank Fusion (RRF) to combine both. It's parameter-free and consistently outperforms trying to manually weight vector scores against keyword scores.
Design Decision 4: Incremental Sync Embedding files takes time. We only re-embed changed files by comparing a content hash. The sync overhead is 200–500ms, which is completely negligible compared to a 5–30s LLM generation time.
The Crucial Insight: Bridging RAG and Tool Use
We set a strict RAG retrieval budget of 20,000 tokens. We greedily fill this budget with the top-ranked files from our Hybrid Search.
But this introduced a fatal flaw.
If a file is excluded by RAG (because it ranked too low), it completely disappears from the LLM's reality. If the LLM is writing a new component and needs to import a Button, but the Button.tsx file was excluded by RAG, the LLM will hallucinate a fake Button component or assume it doesn't exist.
We solved this with a hybrid approach: Path Listings.
After RAG fills the 20,000 token budget, we take all remaining excluded files and inject a lightweight manifest into the prompt:
[Excluded Files (Available to read)]
- components/ui/Button.tsx (45 lines, 1.2KB)
- utils/formatters.ts (120 lines, 3.4KB)
- hooks/useAuth.ts (85 lines, 2.1KB)By explicitly listing the excluded files by path, the LLM knows exactly what exists in the codebase. If it realizes it needs useAuth.ts, it can use its native read_file tool to fetch it on the very next agent iteration.
This bridges the gap between RAG's approximate retrieval and deterministic file access. We keep the context window tight, but we don't blindfold the AI.
What I Actually Learned
- Context is a database, not a dump. You cannot just append strings to an array and expect an LLM to reason about it. You have to query, filter, and rank the context you provide.
- RAG is not a silver bullet. Semantic search is notoriously bad at exact code syntax matching. You must use Hybrid Search (Vector + BM25) for codebases.
- Provide maps, not just destinations. If you hide data from an LLM to save tokens, you must leave a map (like a file tree manifest) so it can find the data via Tool Use when necessary.
What's Next in Part 4
Our Agent has a tight context window and tools to explore the codebase.
But it still generates React Navigation v5 syntax in a world where Expo Router exists. And even with perfect context, it still makes TypeScript errors.
In Part 4, we will explore how treating prompts like version-controlled code fixed our hallucinations, and how we built the Self-Healing Loop — a deterministic TypeScript compiler feedback loop that forces the AI to fix its own bugs before the user ever sees them.
Up Next: Building Lyte (Part 4): Prompt Engineering and Self-Healing Code
