Ollaya: Open Source Decision Models Locally

Ollaya provides a powerful, open-source solution for executing decision models on your own hardware, rivaling hosted services in performance and accuracy.

Binary Buddies · 5 min read · 7 views

The Challenge of Local Decision Model Inference

Deploying and running sophisticated decision models, particularly those mimicking the structure of Jev-style engines, traditionally presents a significant hurdle. High-performance hosted APIs offer convenience but come with inherent costs, privacy concerns, and potential latency issues. For organizations requiring granular control over their data, or seeking to optimize inference speed without relying on external infrastructure, a viable on-premises solution is essential. This need becomes acute when dealing with sensitive information or when predictable, low-latency responses are critical for user experience and operational efficiency.

The drive towards localized AI solutions is fueled by a desire for greater autonomy and cost predictability. While many open-source large language models (LLMs) can be run locally, specialized decision models, which operate differently from generative text models by outputting calibrated answers in a single pass, have lacked robust, user-friendly frameworks for local deployment. This gap has limited the adoption of these powerful tools outside of highly specialized environments.

The specific problem Ollaya addresses is the lack of an accessible, high-performance, open-source framework for running Jev-style decision models locally. Existing solutions often involve complex setups, compromise on accuracy or speed, or are not genuinely open source. Ollaya aims to fill this void by providing a drop-in compatible, performant, and private alternative for local decision model inference.

Introducing Ollaya: Local Decision Models

Ollaya is an open-source project designed to bring Jev-style decision model inference directly to your own hardware. Unlike traditional LLMs that generate text token by token, decision models like those supported by Ollaya operate in a single forward pass. This fundamental difference allows them to provide calibrated answers to typed questions about text or JSON with remarkable speed, often within milliseconds. Ollaya enables users to run these models privately, securely, and cost-effectively on their local machines, whether they possess high-end GPUs or more modest CPUs.

The project emphasizes its independence, clearly stating it is not affiliated with Ollama or TypeSafe, though it does offer compatibility with TypeSafe's API. This independence allows Ollaya to focus on its core mission: providing the most accurate and fastest local inference for decision models. The project provides concrete examples, such as running the winnow:e4b model, which answered five questions in 87 ms on an RTX 4090, demonstrating its practical performance capabilities.

Ollaya's commitment to being an open-source solution means transparency in its development and accessibility to its codebase. This allows for community contributions, custom modifications, and a deeper understanding of how the models operate, fostering trust and innovation in the field of localized AI decision-making.

Performance Benchmarks: Ollaya vs. Ollama

Direct comparisons highlight Ollaya's significant advantages in both accuracy and speed over platforms like Ollama when running decision models. In tests using Bespoke Labs' public benchmark, which comprises 3,880 human-labeled questions across 13 datasets, Ollaya's winnow:12b model achieved an accuracy score of 0.773. In contrast, Ollama's best performing model, Nimble, scored 0.749.

Latency is another critical differentiator. winnow:12b on Ollaya processed questions in approximately 60 ms per question, whereas Nimble on Ollama required 210 ms. This represents a more than 3.5x speed improvement. Furthermore, Ollaya exhibits a dramatically lower calibration error (ECE), scoring 0.022 compared to Ollama's 0.122 for the same Nimble weights. This indicates Ollaya's outputs are more reliably calibrated, meaning their confidence scores more accurately reflect the true probability of correctness.

These benchmark results, conducted on the same RTX 5090 GPU using Ollaya 0.8.0 and Ollama 0.35.0, clearly position Ollaya as the superior choice for performance-critical decision model inference. The data also shows that while Ollama can be faster with certain configurations (like running Q8_0 on llama.cpp), Ollaya's ability to apply author-defined temperatures and return calibrated softmax probabilities provides a more accurate and trustworthy output, even if the raw inference on Ollama is sometimes quicker due to its specific implementation choices.

Accuracy and Speed: A Deeper Dive

Ollaya consistently delivers decision model performance that is both fast and accurate, often rivaling hosted Jev accuracy while operating in under 100 ms. For instance, the winnow:e4b model, when run on an RTX 4090, can complete a five-question request end-to-end in just 89 ms, achieving an accuracy of 0.722 on typed decisions. This is remarkably close to the 0.738 accuracy achieved by TypeSafe's hosted Jev API, which requires 236-276 ms per median request.

Smaller models, such as laya, can perform even faster, completing requests in about 10 ms, and are capable of running efficiently on a CPU. This scalability ensures that Ollaya can accommodate a wide range of hardware capabilities and performance requirements. The benchmark data also reveals a spectrum of models, with jeb and laya:typed-decisions showing particularly high accuracy when trained on specific datasets, underscoring the flexibility and power of Ollaya's platform.

The latency figures provided are based on median requests over an HTTP API on an NVIDIA RTX 4090. While acknowledging that setups can differ and direct comparisons must be made cautiously, the order-of-magnitude improvements offered by Ollaya are undeniable. The project’s ability to process typed decisions, score options directly from logits, and avoid token-by-token generation is key to its speed advantage.

Drop-in Compatibility with TypeSafe API

A significant advantage of Ollaya is its compatibility with TypeSafe's decision model API. Ollaya serves requests using the /v1/systemone and /v1/models endpoints, mirroring the request and response shapes defined by TypeSafe. This adherence to established API standards means that existing applications built using the official TypeSafe Python SDK, for example, can be seamlessly redirected to use a local Ollaya instance without requiring code modifications.

To leverage this compatibility, users can set environment variables such as TYPESAFE_BASE_URL to point to their local Ollaya server (e.g., http://localhost:11435) and TYPESAFE_API_KEY to local. The TYPESAFE_DEFAULT_MODEL can then be set to a specific Ollaya model, like winnow:e4b. This simple configuration change allows developers to switch from a hosted API to a local, private inference engine with minimal effort.

The example provided demonstrates a typical API call to /v1/systemone with a customer's query about an invoice. The expected response format includes the model used, the identified answer (e.g., intent: invoice), the confidence level, and detailed probabilities for each choice. This interoperability is crucial for adoption, as it lowers the barrier to entry for integrating powerful, localized decision models into existing workflows.

Open Models for Diverse Needs

Ollaya provides access to a diverse range of open models, each fine-tuned for specific use cases and offering different balances of accuracy, speed, and capability. These models are available to pull directly, allowing users to select the best fit for their particular application. For instance, the winnow series, fine-tuned from Google's Gemma 4, offers a good balance of accuracy and speed, with variants like winnow:e4b and winnow:12b.

For users prioritizing speed and CPU compatibility, the laya models from Convai Innovations are excellent choices, supporting typed, calibrated answers for choice, score, and yes/no questions in multiple languages. The decider models, based on Qwen3.5, provide strong performance in typed decisions, with models like decider:4b scoring 0.680. For more specialized tasks, models like von can handle longer contexts up to 8,192 tokens, while qwen3guard is designed for text safety screening.

These models are often distributed in formats like GGUF and run using optimized backends such as llama.cpp. Ollaya's ability to leverage these formats and backends on various hardware, including NVIDIA GPUs, Apple Silicon, and CPUs, ensures broad accessibility. The project's models page offers detailed information on each model's accuracy and speed, empowering users to make informed choices for their specific inference needs.

Running Models: Practical Examples

Executing models with Ollaya is designed to be straightforward, often involving simple command-line invocations that specify the model and the input query. For example, to run the winnow:e4b model and ask it to process a customer complaint about double-charging, the command might look like this:

ollaya run winnow:e4b \
"Third time this year you've
double-charged me. Refund it
today or I'm cancelling and
moving to a competitor."

The output provides not only the model's answer but also performance metrics, such as the time taken and the hardware used. For winnow:e4b, this could be: winnow:e4b answered five questions in 87 ms on an RTX 4090. This immediate feedback on performance is invaluable for tuning and optimization.

For programmatic use, particularly when leveraging the TypeSafe API compatibility, setting up environment variables is key. As previously mentioned, setting TYPESAFE_BASE_URL, TYPESAFE_API_KEY, and TYPESAFE_DEFAULT_MODEL allows tools like curl or SDKs to interact with the local Ollaya server. The structure of the request, as shown in the API example, includes the model, the current state, and specific questions with defined intents and criteria, demonstrating how structured queries yield precise, calibrated answers.

Ollaya's Role in the Open Source Ecosystem

Ollaya contributes significantly to the open-source AI landscape by providing a dedicated framework for running specialized decision models locally. While many projects focus on general-purpose LLMs, Ollaya carves out a niche by optimizing for the unique requirements of decision-making systems. Its open-source nature ensures that it remains accessible and auditable, fostering trust and encouraging community involvement, much like other successful open-source tools such as ILSpy or Mono Cecil that offer alternatives in different domains (as seen in discussions on finding open-source alternatives to proprietary software).

By offering compatibility with established APIs like TypeSafe's, Ollaya lowers the barrier to entry for developers and organizations looking to adopt local inference. This interoperability allows for gradual migration and experimentation, reducing the risk associated with adopting new technologies. The availability of a variety of open models, each with documented performance characteristics, further empowers users to tailor solutions to their specific needs without vendor lock-in.

The project's commitment to private, on-device execution aligns with growing concerns about data privacy and security. By keeping data and model inference local, Ollaya enables users to maintain full control over their sensitive information, a critical factor in regulated industries or for any application handling personal data.

Where Ollaya Does Not Fit

While Ollaya excels at running Jev-style decision models locally, it is not a general-purpose LLM inference engine for tasks like creative writing, complex code generation, or open-ended conversational AI. Its strength lies in structured, rapid, and calibrated responses to specific, typed questions. If your primary need is for a model that can brainstorm ideas, write poetry, or engage in free-form dialogue, Ollaya and its current model offerings may not be the most suitable solution.

Furthermore, achieving optimal performance, particularly for larger models, typically requires a dedicated GPU. While Ollaya supports CPU inference, the speed gains may not be sufficient for real-time applications with very low latency requirements. Users with only basic CPU hardware might find that models run too slowly for their intended use cases. The benchmarks provided are largely based on high-end NVIDIA GPUs, setting an expectation for performance that might not be universally met.

Finally, Ollaya is an independent project, and while it strives for compatibility, it is distinct from both Ollama and TypeSafe. Users should be aware of this distinction and understand that while the API compatibility is a key feature, the underlying inference engine and model support are specific to Ollaya. Reliance on bleeding-edge features exclusive to TypeSafe's hosted services might require careful evaluation before transitioning entirely to a local Ollaya setup.

  • #AI
  • #Ollama
  • #Ollaya
  • #Open Source
  • #decision models
  • #local inference

Comments

Add a comment
Sign in with Google to join the conversation