Skip to main content

Case Study: How CodeLeft Achieved Production-Ready Reliability with ObjectWeaver

· 4 min read
ObjectWeaver Team
ObjectWeaver Team
AI Infrastructure Platform

CodeLeft is an AI-powered code quality tool built as a VS Code extension. Their product performs intelligent static analysis on your code against metrics such as SOLID, Clean Code and OWASP Top Ten along with generating comprehensive documentation including requirements tables, data flow analysis, and Mermaid diagrams directly from codebases. The challenge: standard LLM function calling had failure rates of 35-65% for complex structured outputs, making the product unusable for enterprise customers.

ObjectWeaver's guaranteed JSON generation solved this reliability crisis, transforming CodeLeft from prototype to production-ready solution.

The Challenge​

CodeLeft required reliable generation of complex structured outputs:

  • Requirements Tables: Extracting functional and non-functional requirements from codebases
  • Data Flow Analysis: Mermaid diagrams visualizing system data flows
  • Tech Debt Reports: JSON structures identifying SOLID principle violations

Standard LLM function calling produced malformed outputs 35-65% of the time. The issues were classic but catastrophic: missing commas, unclosed curly braces, or the model hallucinating Markdown code blocks around the JSON. A single syntax error in a 20-row requirements table would crash the entire operation.

This unreliability blocked enterprise adoption. Customers required 99%+ success rates, and the product couldn't scale beyond simple use cases. CodeLeft needed guaranteed structured outputs while maintaining on-device processing for data privacy.

The Solution​

ObjectWeaver is an LLM orchestration service designed for guaranteed structured JSON generation. CodeLeft integrated the ObjectWeaver binary directly into their VS Code extension to ensure reliable documentation generation.

Architecture & Privacy:

CodeLeft runs the ObjectWeaver binary locally via the VS Code Terminal API. This architecture ensures that orchestration and validation happen entirely on the user's device.

Crucially, this setup supports a "Bring Your Own Model" approach. Users can connect to local models (like Ollama) for complete air-gapped privacy, or route to secure enterprise cloud endpoints (Azure OpenAI, Anthropic) if preferred. In all cases, the structural guarantee is enforced locally by ObjectWeaver.

Solving Context Pollution with Field Overrides:

One of the biggest breakthroughs came from ObjectWeaver's ability to define specific instructions per field. For complex Mermaid diagrams, standard prompts often confused the model with irrelevant context from the rest of the codebase.

By using ObjectWeaver's schema structure, CodeLeft could isolate the context for the diagram generation:

"dataFlow": {
  "type": "string",
  "instruction": "Generate a Mermaid flowchart focusing ONLY on data movement between services. Ignore implementation details.",
  "overridePrompt": "You are a system architect. Output only the Mermaid syntax."
}

This isolation ensured the model received only the necessary context for that specific field, eliminating the "context pollution" that previously caused hallucinations.

The Results​

Integrating ObjectWeaver enabled CodeLeft to achieve:

  • From 30 Seconds to Seconds: Before, the system relied on multiple internal retry loops to catch and fix malformed JSON, often taking 30+ seconds for a single analysis. Now, with guaranteed output, SOLID analysis and code reviews complete in just a few seconds.
  • 99.9% Reliability: Zero malformed JSON errors in production.
  • Enterprise Adoption: The ability to support custom endpoints (Azure/Local) opened the door to enterprise customers with strict data policies.
  • Complex Use Cases: Successful generation of multi-file architecture documentation that was previously impossible.

ObjectWeaver's guaranteed structured output generation transformed CodeLeft from a promising prototype to a production-ready tool. By handling the orchestration complexity, ObjectWeaver allowed CodeLeft to focus on delivering high-quality code analysis rather than writing regex parsers for broken JSON.

Acknowledgement​

The author is a co-founder of CodeLeft. ObjectWeaver was first used there as part of its proof of concept, originating from a dissertation at the University of Bath.