How loqy evaluates local AI models for real work

A model can earn an excellent reasoning score and still be frustrating to use. It may write an impressive answer, then invent a source. It may understand a twenty-page document but lose a constraint after the third tool call. It may run quickly on a laboratory machine and exhaust the memory of a Mac that someone is using for work.

The word system matters. The model is one component among the exact downloaded artefact, its quantisation, inference engine, conversation template, available context, exposed tools, Mac memory, and controls applied to the output. loqy's execution map locates each of those components.

The test starts with real work

Consider a concrete qualification case. The folder contains a twelve-page budget memo, a 240-row CSV export, and a framing note in French. Three anomalies have been introduced: a duplicated expense, a rate applied to the wrong base, and an assumption that is no longer current. The assignment asks the agent to find them, recompute the figures, verify two public facts in a simulated environment, then deliver a decision memo and a corrected table.

The folder also contains a deliberate incident. The first file path does not exist. The model must read the error, locate the correct file within its authorised scope, and continue without inventing the missing content. This step may look secondary. It reveals the strength of an agent loop better than a perfect answer to an isolated question.

Success requires several capabilities in sequence:

• understand a request written in professional language;
• retain constraints throughout a long run;
• choose an available tool without inventing another one;
• produce valid arguments for that tool;
• use the result without distorting it;
• distinguish fact, assumption, and missing information;
• cite the origin of an important claim;
• prepare a file that a person can review and reuse.

Knowledge and coding benchmarks measure some of these capabilities. The product scenario observes how they combine in a complete loop.

loqy qualification begins with versioned work scenarios. Every configuration receives the same inputs, constraints, and criteria. External effects are simulated. Outputs are evaluated against observable requirements.

For this case, the rubric is frozen before execution:

Deterministic checks cover files, calculations, tool schemas, and known citations. Calibrated human review covers register, clarity, and the relevance of the synthesis. A configuration must pass both levels.

Evaluation through several gates

loqy treats qualification as a sequence of mandatory gates. Each gate produces either admitted evidence or an actionable rejection reason: unfrozen input, mismatched artefact, invalid tool loop, defective deliverable, or unstable hardware envelope.

1. Identity and provenance

The configuration fixes the repository, revision, file, byte count, cryptographic digest, licence, quantisation, and vision projector. A different file can change quality, memory use, and even the expected protocol.

In loqy, a result belongs to a precise artefact and engine pair. A new conversion receives its own qualification.

2. Bilingual work and grounding

Scenarios use French, English, and mixed documents. They contain dates, amounts, tables, ambiguity, and professional vocabulary. Quality covers meaning, register, numbers, and caveats alongside grammar.

Grounding is a separate gate. Important elements of an answer must be traceable. In a report or document review, a correct citation matters more than a confident tone.

3. The tool loop

An agent needs a model that can choose from a closed set of tools, respect their schemas, and react to bounded results. Faced with “file not found,” a robust loop interprets the error, repairs the path, and stops when authority or information is missing.

Tests therefore observe call validity, recovery after errors, repair count, absence of invented tools, and the ability to stop when authority or information is missing. A well-formed call is not yet a sound decision. The model can propose an action, but policy and permission remain outside it.

4. Deliverable and evidence

The result connects the conversational answer to the report, spreadsheet, or code change. The file must be valid in its format, retain its inputs, and make transformations inspectable. loqy verifies the produced file directly.

5. Hardware and experience

Finally, a configuration must work on a supported real Mac. Time to first token, generation speed, resident memory, memory pressure, thermal state, energy use, and application responsiveness all contribute to quality.

A product improvement requires both a better result and a Mac that remains usable.

The catalogue covers several resource envelopes

The V1 catalogue describes four exact local profiles for different memory envelopes. Every result remains tied to the file that is actually delivered.

These are manifest artefact sizes in decimal gigabytes. The download space shown to the user includes weights and projector, approximately 3.41, 6.62, 15.44, and 24.41 GB. Source revision, byte count, and SHA-256 digest complete the identity of every file.

Image access is enabled when the run holds a model lease that permits it, a qualified projector, and a validated source.

Qwen 3.5 4B is the first catalogue profile and the initial recommendation on eligible Macs. Larger models remain user-selectable when the Mac has enough memory for their profile.

The launch profile reserves the engine's technical capacity for a precise model and memory tier. The context granted to a run is 16,384 tokens in Standard mode, 32,768 in Extended mode, and up to 65,536 in Deep mode when the model, profile, and Mac allow it.

That margin covers output, the tool loop, summaries, and safety reserves. A model advertising 256K, or a launch profile carrying that window, does not automatically become a 256K prompt. The engine, cache, tools, and the rest of the application still need to fit together.

Hardware is part of the configuration

Useful measurements cover startup, first token, sustained throughput, and the complete scenario. They also observe memory, thermal state, and clean resource release.

loqy's method therefore separates several timings:

• cold model loading;
• time to first token;
• throughput once generation begins;
• total time to complete the scenario;
• peak memory and thermal state;
• clean resource release after use.

Candidate settings are tested after warm-up with at least three repetitions. The median prevents one exceptional run from controlling the decision. Excessive variation invalidates the measurement. An optimisation is promoted only if the output remains correct, tool calls stay valid, memory retains a reserve, and thermal state does not worsen.

Tokens per second matter when the complete work becomes faster and remains reliable.

The tested configuration is larger than the model

Two installations of the same model can behave differently. Quantisation changes size and sometimes fidelity. A mismatched conversation template can break tool calls. A different engine may interpret caching or vision differently. An ambitious context budget can create memory pressure that the original benchmark never encountered.

loqy therefore defines the qualification unit as a chain:

exact artefact + projector + engine + parameters + context contract + tool schemas + hardware class

The Core keeps that identity in an immutable lease for each run. Changing models midway through a generation would make attribution and recovery ambiguous. A change takes effect on a new run, with context rebuilt for the new engine.

The rule also makes comparisons useful. When an output regresses, it is possible to determine whether the model, runtime, context, or setting changed and reproduce the configuration.

Public benchmarks still matter

Benchmarks map strengths and help select the product tests worth running. A strong Terminal-Bench result suggests useful agentic ability. LongBench or MRCR can signal context behaviour. A multilingual evaluation can expose a likely weakness.

A public benchmark is a situated observation. It depends on the prompt, harness, reasoning level, trial count, and sometimes a model judge. The comparison of local and hosted costs applies the same explicit scope to published prices, speeds, and scores.

loqy therefore combines three levels:

1. public signals, to understand the landscape and identify candidates;
2. product suites, to test the tasks, languages, and tools people actually use;
3. integrated qualification, to verify behaviour inside the application and on target hardware.

Qualification requires success at all three levels.

Adapt a model only to answer a measured gap

V1 ships the current catalogue. This section describes a method for possible later research into model adaptation.

A model adapted to loqy could follow a tool schema more reliably, cite sources more consistently, or produce more stable professional French. These goals can be measured precisely, especially for a model between 3 and 4 billion parameters.

Such a study would begin with three arms: the base model, the best context and instruction system without training, and the adapted model. They would receive identical scenarios and seeds. Adaptation would be retained only if it produced a measurable improvement on the target axes without degrading general ability, safety, or transparency.

Training data would need explicit provenance. Synthetic, open, or authorised content would remain separate from private data. Customer conversations, documents, diagnostics, and memories would not become training material by default. Teacher-model output could be used only when its terms allowed it.

Training would be retained only when measured gains exceeded its cost and maintenance. A better context structure or clearer tool would remain preferable when it resolves the gap.

What the user should understand

This infrastructure produces simple choices: which models are available on this Mac, how much space they use, what they can process, which one is selected, and why a configuration is unavailable.

The user can select another eligible model. The catalogue narrows the option space to known, reproducible configurations, then lets the person balance speed, capacity, and resource use.

That is the difference between integrating a model and shipping a capability. The first can answer a request. The second must remain reliable when the document is long, the tool fails, the language changes, and the Mac is doing something else at the same time.

loqy evaluates local models at that level: across the full work loop, all the way to the deliverable and its evidence. The evidence chain then explains what must remain attached to the result.

Sources and method

• loqy, local inference architecture and exact catalogue
• loqy, context planning and the modes actually granted
• loqy, product-specific small-model adaptation study
• loqy, Evaluation Lab design
• Qwen 3.5 4B, source model
• Ornith 1.0 9B, selected GGUF package
• Gemma 4 26B-A4B, selected GGUF package
• Ornith 1.0 35B, selected GGUF package
• Stanford CRFM, HELM and holistic model evaluation