The complete source code for Lyte is available on GitHub.
In this final part of the series, the core architecture is complete. We have an Agentic Engine that manages its context window, heals its own errors, and streams progress to a real-time UI.
But as we prepared for production traffic, two fundamental problems remained:
- Scaling: A systematic audit revealed critical bottlenecks that would crash the system under load.
- Quality Assurance: "It feels better" is not an engineering metric. When we tweak a prompt to fix a bug, how do we mathematically prove we didn't degrade generation quality elsewhere?
Here is how we solved both.
The Scaling Bottleneck Audit
We identified 9 critical scaling bottlenecks. Here are the three most dangerous ones and how we fixed them using classic distributed systems patterns.
1. Database Connection Starvation
Our Transactional Outbox poller (from Part 1) maintained a persistent LISTEN connection to Postgres. Under heavy generation load, the poller was actively querying the database, aggressively competing for connections in our Node.js pool.
The result: Standard HTTP API requests (like a user fetching their profile) would hang because the connection pool was exhausted by the background poller.
The Fix: The Bulkhead Pattern.
We created two entirely separate TypeORM connection pools. The primary pool (max: 10) is exclusively reserved for HTTP handlers. The secondary pool (max: 3) is strictly for background outbox operations.
If the outbox gets overwhelmed, its pool saturates, but the primary API remains lightning fast. The ship doesn't sink.
2. Head-of-Line Blocking
We were using a single SQS queue. If User A requested a massive 40-file regeneration (a 5-minute job), and User B requested a tiny 1-line text edit (a 3-second job), User B had to wait 5 minutes for User A to finish.
The Fix: Priority Queues.
We split the pipeline into three separate SQS queues: high, default, and bulk. The Worker polls the high queue first. Quick edits for active users get immediate capacity, while heavy background generations fall to default.
3. Reactive Circuit Breakers vs. Proactive Rate Limits
We had circuit breakers on our LLM providers. But 429 Too Many Requests is an expected state at high throughput. Running 429s through a circuit breaker would trip it, completely halting all outbound API calls for 30 seconds.
The Fix: Proactive Token Buckets. We placed an in-memory Token Bucket rate limiter before the circuit breaker. If Anthropic allows 2 requests per second, our Token Bucket only allows 2 outbound requests per second. It smooths the traffic proactively, meaning we rarely ever hit a 429, keeping the circuit breaker closed and the system flowing.
Building the Eval Framework
With scaling solved, we tackled the hardest problem in AI Engineering: Regression testing.
You cannot write standard unit tests for LLM outputs. You cannot assert expect(code).toContain('const login') because an LLM might write const handleLogin tomorrow. The output is non-deterministic.
We built an Eval Framework — a regression test suite for AI.
It operates in two layers.
Layer 1: Deterministic Checks
Even though the code structure is non-deterministic, certain properties must be absolute truths.
expectations: {
filesCreated: ['app/(tabs)/settings.tsx'], // The file MUST exist
shouldCompile: true, // `tsc --noEmit` MUST pass
packageJsonMain: 'expo-router/entry', // Critical field MUST NOT change
}These are pass/fail. If the AI hallucinates a change to the package.json entry point, the eval fails instantly.
Layer 2: LLM-as-a-Judge
For qualitative metrics, we use another AI model to grade the output.
We use a cheap, fast model (Gemini Flash) to grade our primary model's (Claude Sonnet) output on a strict 1-5 rubric.
The key design decision was Rubric over Open-Ended Scoring. Asking a Judge model "Is this code good? 1-10" produces useless noise.
Instead, we provide explicit grading criteria:
"Safety (1-5): Does the generated code preserve all existing user functionality? Score a 1 if a previously existing button was removed."
Because the Judge model doesn't need to be highly capable (it just needs to be consistent), using Gemini Flash ($0.10/M tokens) instead of Claude Sonnet ($3.00/M tokens) saves massive amounts of money when running a 50-case regression suite in CI/CD.
CI/CD Integration
This is the holy grail.
If an engineer opens a Pull Request that modifies the system prompt, our CI pipeline automatically runs the Eval suite. It generates 20 sample projects. If the average LLM-as-a-Judge score drops below 4.0, or if a single Deterministic Check fails, the PR is blocked.
Prompts are no longer magic spells. They are versioned, tested artifacts with objective mathematical proof of quality.
Conclusion: AI Engineering is Distributed Systems Engineering
When I set out to build Lyte, I thought my time would be spent reading whitepapers on Attention Mechanisms and Prompt Engineering.
Instead, I spent my time implementing Transactional Outboxes, tuning Postgres connection pools, writing Redis Lua scripts for distributed locks, and building ACK protocols over WebSockets.
The LLM is an incredible piece of technology. But it is just an engine. It is a slow, expensive, non-deterministic database query.
If you want to build a production-grade AI product, the secret isn't in the prompt. The secret is building a fault-tolerant, event-driven distributed system around the prompt to protect it from itself.
The Final Architecture of Lyte
Putting it all together, here is the end-to-end architecture we built throughout this 6-part series:
