How loqy evaluates local AI models for real work

A local AI model can look excellent on a leaderboard and still be frustrating at work. It may lose the structure of a long document, produce brittle tool calls, slow down under memory pressure, or become unreliable when a task moves between English and French. loqy therefore treats model selection as product qualification, not a beauty contest between benchmark scores.

The first question is simple: can this model help finish the kinds of work loqy is built for? That includes reviewing documents, researching across sources, transforming data, writing and running code, using bounded tools, and producing a result that a person can inspect.

Evaluate the whole working loop

A useful evaluation starts before the first token and ends after the deliverable. We look at how reliably the model understands the request, identifies missing information, uses the context it was given, chooses an appropriate tool, and reports what happened. A polished answer cannot compensate for an invented source or an unsafe action.

Tool use matters especially. The model must form valid calls, recover when a tool returns a bounded error, and avoid repeatedly attempting an action that is unavailable. It should distinguish between preparing a change and having permission to publish it.

The final output matters too. A spreadsheet, report, or code change needs more than plausible prose: its assumptions, inputs, checks, and source trail should remain available for review.

Hardware is part of model quality

loqy targets Apple Silicon Macs first, beginning with configurations that have at least 16 GB of memory. A model that only behaves well on a development machine with abundant memory is not automatically a good default.

Qualification therefore includes startup time, generation speed, memory pressure, context size, and the way the rest of the desktop experience behaves while inference is running. The right answer may be several model profiles rather than one universal winner.

The current product work covers model families including Qwen, Ornith, and Gemma. That list is not a permanent ranking or a promise that every variant will ship. Models and runtimes change quickly; the shipped catalog should reflect repeatable evidence on supported hardware.

Language and restraint count

Real work is multilingual even when a benchmark is not. English and French prompts, mixed-language documents, dates, numbers, tables, and domain vocabulary all belong in the test set. So do situations where the right behavior is to ask a question or stop before an external effect.

Fine-tuning may become useful later, including for structured tool use or recurring work patterns. It will only be worthwhile if it produces a measured improvement without weakening general capability, safety, or transparency. For now, careful model choice, clear context, and strong product boundaries are the more important work.