Pick the wrong open-weight model for an on-premises deployment and you spend the next three quarters fighting it. Pick the right one and the operator forgets the model is there. We have shipped open-weight LLM stacks into healthcare, banking, government, retail, and defence-adjacent operators across the United Kingdom, European Union, Americas, GCC, MENA, Africa, and Asia over the last 18 months. The model landscape moves every quarter; the decision-making framework does not.
Why the question matters more in 2026 than it did in 2024
Two years ago, "use open-weight" was largely a sovereignty argument — the operator did not want their data in a third-party cloud. The capability gap to the leading closed-weight models was real and you had to accept it. In 2026, that gap closed enough for most operator workloads that the question stopped being "is open-weight good enough" and started being "which open-weight is the right fit for my hardware, my languages, and my regulator". Different question. Different answer.
The shortlist
For an on-premises deployment in 2026, we work from a shortlist of four model families. Llama 3.x — Meta's open-weight line — remains the safest default. Strong English. Solid licensing posture. Good ecosystem support. Mistral and Mixtral — Mistral's dense and mixture-of-experts variants — excel at multilingual workloads, particularly French, Spanish, German, Italian, and Arabic. Qwen — Alibaba's open-weight line — leads on Chinese and East Asian languages, with a Qwen-VL variant that is the best open vision-language model we have deployed for OCR and document workflows. DeepSeek — open-weight from the DeepSeek team — currently leads on long-context reasoning and code, with the trade-off of a more carefully read license clause.
There are dozens of other open-weight models that are competent. These four cover roughly 95% of the operator workloads we actually see.
GPU envelope by model size
The hardware envelope is the first filter — not because operators do not have budget, but because the wrong model on the right GPU silently caps your throughput and the right model on the wrong GPU silently caps your latency. For 2026:
- 7B–8B class (Llama 3.1 8B, Mistral 7B, Qwen 2.5 7B) — runs comfortably on a single RTX 5090 (32 GB) or L40S (48 GB). Right for clinic-scale or single-site deployments doing chat, summarisation, drafting, and light RAG.
- 13B–22B class (Mixtral 8x7B at quantised precision, Qwen 2.5 14B, Llama 3.2 11B vision) — needs an L40S (48 GB), a pair of RTX 5090s with NVLink, or a single A100 40 GB. Right for multi-site operators or single-site deployments with heavier RAG and bilingual / multilingual coverage.
- 70B class (Llama 3.1 70B, DeepSeek 67B) — needs an H100 80 GB or twin L40S in tensor-parallel. Right for regional-scale deployments where the quality bar is enterprise-grade and the operator can justify the GPU spend.
- 100B+ MoE (Mixtral 8x22B, DeepSeek V3) — needs multi-H100 or MI300X. Reserve for operators with genuine load and a serious AI roadmap, not pilots.
The key trap is over-provisioning at pilot stage. A clinic of 50 beds does not need a 70B model. Start with 8B, prove the workflow, scale up only when the limit is the model, not the prompt.
vLLM vs Ollama vs TGI vs llama.cpp
Different serving stacks serve different purposes. We deploy all four depending on the operator. vLLM is our default for production inference — it leads on throughput, supports continuous batching, and handles the kind of concurrent-clinician or concurrent-agent load that healthcare and banking generate. Ollama is the right pick for a single-operator workstation deployment, a developer environment, or an air-gapped site where ops simplicity beats raw throughput. TGI (Text Generation Inference, Hugging Face) is a strong middle path — solid on multi-model serving and easy to operate. llama.cpp is the right pick for CPU-only edge deployments — a kiosk, a branch terminal, a hardware-constrained POS lane.
The decision is not religious. It is workload-shaped. We have shipped all four in the same year, into different operators.
RAG, mode-based prompts, and the operator's perimeter
The model is half the answer. The other half is how you build the prompt and the RAG context that wraps it. Every on-prem deployment we ship uses mode-based prompts — Inquiry, Differential, Summariser, Audit, etc. for healthcare; KYC, Reconciliation, Risk-Review, Customer-Reply for banking; Citizen-Inquiry, Permit-Review, Audit-Letter for government. Each mode constrains the model to evidence-based, short-by-default replies anchored on retrieved context from the operator's knowledge base.
Embeddings, vector store, and retrieval all stay inside the operator perimeter. Reference stack: pgvector or Qdrant for the vector store, BGE-M3 or E5 for embeddings, a reranker on top for higher-quality retrieval, audit logging on every prompt and completion. Zero data leaves the perimeter. The regulator can read every prompt that was ever issued.
Picking the right model for healthcare vs banking vs government
Our rule-of-thumb shortlist for the operators we ship to most often:
- Healthcare (bilingual EN/AR clinic, 30–80 beds): Llama 3.1 8B on a single L40S or a pair of RTX 5090s. Mistral 7B if the Arabic load is heavy. Mode-based prompts. RAG against the formulary and the operator's prior consult notes. The MediCare clinic management system ships with a 7-mode AI Clinical Assistant as the reference deployment for this profile.
- Banking (mid-tier retail bank, multi-branch): Mixtral 8x7B at quantised precision on L40S or A100. Mode-based prompts for KYC, reconciliation, customer-reply. RAG against the bank's product catalogue and prior tickets.
- Government (ministry-scale citizen services): Llama 3.1 70B on H100 if budget allows; Llama 3.1 8B at multi-site otherwise. Mode-based prompts for citizen-inquiry, permit-review, audit-letter. RAG against the ministry's policy library.
- Retail / hospitality: Llama 3.1 8B or Qwen 2.5 7B (if bilingual EN/AR or East-Asian language load). Lightweight RAG against product / menu / loyalty data.
Locale coverage that survives a regulator review
One final selection criterion that operators in multilingual markets care about: locale coverage that survives a regulator review. English and Arabic (with full RTL rendering across the whole UI surface) is the production baseline we ship as standard. French, Spanish, German, Portuguese, Italian, Dutch, Turkish, Urdu, Hindi and others are added per engagement. The model has to render the user's language end-to-end without translation artefacts that the regulator can flag as misleading. Mistral and Mixtral lead the open-weight pack on European multilingual coverage; Qwen leads on East Asian; Llama is the safe default for English-dominant deployments. Pick for the actual language mix the operator's customers speak, not the language mix the brochure shows.
Quantisation, fine-tuning, and the trade-offs operators actually make
Two further decisions sit underneath the model-selection question and shape the deployment more than operators often realise.
Quantisation — running the model at reduced precision (typically 4-bit or 8-bit) to fit a larger model on smaller hardware — is a standard tactic. The quality loss at 8-bit is usually negligible for the workloads operators run; the loss at 4-bit is visible on reasoning-heavy tasks but acceptable for chat, summarisation, and basic RAG. Quantisation is what makes Mixtral 8x7B fit comfortably on an L40S rather than needing an H100. We default to 8-bit unless the hardware envelope forces 4-bit; we benchmark both against the operator's actual workload before committing.
Fine-tuning is the second question. The operators we ship to fall into two camps. The first camp — most of them — does not fine-tune at all and instead invests the same effort into better RAG, better mode-based prompts, and a richer retrieval index. This is almost always the higher-leverage path; the model has the capability, the system just needs to put the right context in front of it. The second camp — usually banking and government operators with a stable, high-volume, narrowly-scoped workload — fine-tunes a base model on their own data using LoRA or QLoRA adapters. The fine-tuning stays inside the operator perimeter; the adapter is a small artefact that can be versioned, rolled back, and audited. We deploy the fine-tuning pipeline as part of the engagement scope when it is the right answer, and we say so when it is not.
The operator's operating-model implications
Running an on-premises LLM is not just a procurement decision. It is an operating-model decision. The operator needs at minimum a small ML-ops capability — typically one or two engineers — who can monitor the inference stack, manage the model versions, tune the RAG retrieval, and handle the inevitable production incidents (a noisy GPU, a stuck token, a misbehaving prompt). For operators who do not want to build that capability in-house, we offer Care Plan tiers that cover the ML-ops layer alongside the application layer. The cost of the Care Plan is materially lower than the cost of either a managed cloud LLM or a closed-weight API at the volumes we typically see in production.
The deeper truth: in 2026, the right open-weight model is rarely the most impressive one — it is the one the operator's hardware can comfortably serve and the regulator can comfortably accept. Pick for fit, not for benchmark. If the operator wants help running that selection against their actual workload, our enterprise development services and digital transformation consultation lines exist for exactly that scoping conversation.
