Building Lyte (Part 2): The Agentic Engine and Token Economics

The complete source code for Lyte is available on GitHub.

In Part 1, we hardened our infrastructure. We implemented Transactional Outboxes, distributed Redis locks, and Sidecar containers. The backend was bulletproof.

The AI, however, was a mess.

Our initial AI pipeline treated the LLM as a text generator. We instructed it to output structured XML containing the file paths and the code, which we would parse with regex and write to the filesystem.

<lyteArtifact>
  <lyteAction type="file" filePath="app/(tabs)/index.tsx">
    // The LLM writes JSX here...
    <View> 
      // Wait... this is XML. The <View> tag just broke the parser.
    </View>
  </lyteAction>
</lyteArtifact>

It was a disaster. JSX inside XML requires strict escaping (<View>), which LLMs frequently forget. When the parser failed, the entire 60-second generation was lost. Worse, the LLM was operating entirely blind. It couldn't read existing files, check installed packages, or verify if its code actually worked.

We had to stop treating the LLM like a text generator, and start treating it like a frontend client calling an API.

The XML Parser to Tool Use Migration

We ripped out the XML parser entirely.

Instead, we migrated to Native Tool Use (Function Calling) using a ReAct (Reason + Act) agent loop. We provided the LLM with strict JSON schemas for tools like read_file, write_file, and run_command.

What changed architecturally:

1. The LLM now receives a JSON schema for tools (guaranteed valid JSON — no fragile regex parsing).
2. The LLM calls tools sequentially: inspect → install → write → verify.
3. The Worker executes each tool call, feeds the results back to the LLM in the next iteration.
4. The loop continues until the LLM emits an end_turn signal.

What the LLM gained: Before writing _layout.tsx, it can now read_file('app/(tabs)/_layout.tsx') to see the current routes and ensure it doesn't accidentally delete existing tabs. Before installing a package, it can run_command('npm list <package>') to check if it's already installed.

The transition from XML structured output to native tool use mirrors the industry's evolution (e.g., Cursor, Devin, Claude Computer Use). Give the AI hands, and let it drive.

The "Loop Stuck" Circuit Breaker

This freedom introduced new failure modes.

Sometimes an LLM will call run_command('npm start') (a blocking command) over and over again, expecting a different result.

We had to build a Circuit Breaker for the loop itself: If the LLM calls the exact same tool with the exact same arguments 3 times consecutively, we force-exit the loop. Never let a while-loop driven by probability run unbounded.

The LLM Provider Coupling Problem

Initially, our worker.service.ts directly called anthropicClient.messages.create(...).

In the AI space, providers change pricing, rate limits, and capabilities on a weekly basis. Being tightly coupled to Anthropic meant switching to GPT-4o or Gemini Flash would require rewriting the core pipeline.

The Fix: The Provider Adapter Pattern.

We built a normalized interface:

worker.service.ts → LlmClientService → AnthropicProvider
                                     → OpenAIProvider
                                     → GeminiProvider

Each provider implements a toolCompletion() method that returns a standardized ToolCompletionResult { stopReason, textContent, toolCalls, tokens }.

The translation layer handles each vendor's proprietary format:

• Anthropic: tool results as { role: 'user', content: [{ type: 'tool_result', ... }] }
• Gemini: tool results as { role: 'function', parts: [{ functionResponse: ... }] }
• OpenAI: tool results as { role: 'tool', tool_call_id, content }

Our worker.service.ts is now completely provider-agnostic. We can switch LLMs via an environment variable with zero code changes.

This also enabled per-provider circuit breakers and rate limiters. Each provider gets its own token bucket. Anthropic hitting 429 Too Many Requests doesn't block Gemini. One provider down doesn't take down the product.

Learning Token Economics the Hard Way

After implementing the agent loop, we built an observability dashboard. The results were terrifying.

A single complex generation (e.g., a 30-file project, a 5-iteration agent loop, and 2 heal iterations) was costing $0.80–$1.20 in LLM API calls per request.

Why? The Agent Loop Multiplier Effect.

In a tool-use loop, you must re-send the entire conversation history—including all previous tool results—on every single iteration. Iteration N costs more than Iteration N-1:

• Iter 1: 10k input tokens
• Iter 2: 10k + 1.5k tool results = 11.5k
• Iter 3: 11.5k + 1.5k = 13k

After 7 iterations: 10k + (6 × 1.5k) = 19k tokens for the input prompt alone.

A stuck agent can easily burn 200,000+ tokens for a single user request. We needed strict token economics.

The Three Layers of Protection

1. Accurate Token Counting: We replaced naive text.length / 4 heuristics with exact BPE encoding (tiktoken). We check the token count before every LLM call. If we approach 80% of our limit, we aggressively compact the history.
2. Explicit Budget Allocation: We explicitly allocate tokens across prompt sections. If the history is short, we give more tokens to file context. If there are few files, we give more tokens to the output buffer.
3. USD Cost Tracking: Every ai_model_calls row in Postgres stores the exact dollar cost. This rolls up to a total_cost_usd per job. You cannot optimize what you do not measure in dollars.

Token economics is the new compute cost. In traditional backends, you monitor CPU and memory. In AI backends, you monitor tokens and USD per operation.

What's Next in Part 3

We gave the AI hands (Tool Use), and we stopped it from bankrupting us (Token Economics).

But as users started building more complex apps with 30+ files, generation quality plummeted. The LLM's attention was split across 80,000 tokens of mostly irrelevant code, exhibiting the dreaded "lost in the middle" phenomenon.

In Part 3, we will look at the Context Window Crisis. We'll explore how we used pgvector, HNSW indexes, and Hybrid Search (RRF) to retrieve only the exact files the AI needs to see.

Up Next: Building Lyte (Part 3): The Context Crisis and Codebase RAG