Moustafa Elzeftawy

Work · AI

Cloud orchestrates, local executes: local-LLM research

Can local open-weight models safely take over mechanical AI work? I ran a structured evaluation of 19 model configurations and several agent harnesses, and applied the same pattern to an "AI factory" for coding and customer engagement.

Organisation
Amplify-Now · independent research
My role
Research lead
When
2026
  • 25–50%projected token saving
  • 84%best local model (cloud Sonnet 87%)
  • ~75%of AI-factory work done locally
  • 0 errorsin a 17,176-request soak test
The AI-factory pattern
  1. Cloud orchestratorplans, writes test kits and judges
  2. Local workersopen-weight models on a 16 GB GPU
  3. Quality gatespass/fail by error cost
  4. Human approvalbefore anything goes out

The question

Cloud LLM tokens are the running cost of AI enablement. How much of the work can move to local models without losing quality?

Method

  • Hardware: a single workstation (RTX 4070 Ti SUPER 16 GB, Ryzen 5 9600X, 48 GB RAM) running llama.cpp.
  • Models: the Gemma 4 family, Qwen3.8-27B (dense), Qwen3.6-35B and Qwen3-Coder (MoE), plus GLM, GPT-OSS, Devstral, Granite, Nemotron and Hermes. Each was tested at several quantisations from Unsloth, Alibaba, ISTA-DASLab and others.
  • Test suite: 40+ cases in 8 suites: reasoning traps, evidence rules, agentic retrieval with verbatim citation, T-SQL authoring, PII stripping, DAX reading, executed web code and speed telemetry. Criteria were scored 0–5 or pass/fail.
  • Harnesses: Claude built an in-house Python harness (sandboxed, read-only tools). I compared it with Pi, OpenCode, Qwen Code, DeepSeek’s harness and Claude Code. I also wrote a protocol shim so Claude Code could talk to local servers.
  • Orchestration: a cloud model ran the waves unattended for weeks and logged every decision: about 400 runs and more than 5,000 case-runs.

Findings

  • The best local configuration scored 84% on the common test set, close to Claude Sonnet (87%) and ahead of Claude Haiku (74%).
  • Dense models beat mixture-of-experts on quality; MoE won on speed.
  • The minimal in-house harness tied or beat every off-the-shelf agent harness.
  • Safety belongs in the tool, not the prompt: asked to “restore deleted rows”, most small models happily wrote INSERTs.
  • Mechanical work, 25–50% of real work, can safely go to local models: SQL that is then executed, executed code, formatting. That gives a projected 25–50% token saving. Judgement-heavy work stays in the cloud.

The AI factory

I applied the same pattern in independent research on an “AI factory” for coding and customer feedback and engagement: classifying feedback, escalating risky items to a person, drafting replies and checking drafts with a second model.

  • Across 472 runs and about 41,700 local model calls, every role faced a hard pass/fail gate weighted by the cost of a mistake.
  • One mid-size local model passed four role gates, with 100% escalation recall with a rules backstop.
  • A production-style soak test handled 17,176 requests in 3.5 hours with zero errors.
  • Local models now do ~75% of the work, and the cloud only orchestrates.

Next: a real-ticket trial, then production.