A local model can be a candidate for offline work or greater control over data flows. Its suitability is not determined by execution location alone. This guide explains parameters, memory and tests to run before buying hardware.
A model with downloadable weights can run on your hardware under its licence. Local inference describes where computation happens; it does not guarantee privacy of the entire application. Check cloud features, search, tools, telemetry, logs and backups. Assess suitability on a concrete task: quality, speed, costs and data management depend on both model and deployment.
What an open model is and how it differs from ChatGPT
When you type into ChatGPT, Claude or Gemini, your text goes to the provider's servers, where it's processed by a model you know only as much about as the provider tells you, and the answer comes back. The model is closed, you can't download it, inspect it or run it elsewhere.
An open-weights model publishes learned parameters under a specific licence. This does not automatically mean open training data or unrestricted use. Qwen3-30B-A3B, for example, uses Apache 2.0; follow the exact version's terms and those of associated components.
Important: "open" doesn't mean "worse". It means you can run the model yourself, with everything that follows from that.
What "runs locally" means
A downloaded model can run on a laptop, workstation or company server. Before using real data, inspect the whole application and its connections, not just the model server address:
- verify the data flow, local computation does not rule out sending a query through tools, cloud fallback, telemetry or synchronisation,
- you don't pay per token, you pay for hardware and electricity,
- test offline operation, required models and dependencies must be available locally; remote tools do not work offline,
- you are responsible for operation, updates, security and for the model not answering nonsense.
The most common route for individuals and small teams is Ollama (a command line plus a local interface, running on macOS, Windows and Linux) and LM Studio (a graphical application with a model catalogue). Both download the model, run it and offer an interface that a chat or your own application connects to. For company deployment with more users there are server tools, but that's already a job for someone who manages servers.
How to read model size: parameters and quantisation
Parameters are learned numbers; B means billions. Their count is not a universal scale of intelligence or speed. Architecture, training, quantisation and the task also matter.
Quantisation stores values at lower precision. It can reduce memory use; quality and speed effects depend on the method and task. A 4-bit label does not determine the memory of the entire application.
Theoretically, 4 bits per parameter is 0.5 GB per billion parameters in decimal units. That covers values alone, not a complete run: quantisation metadata, context cache, work buffers and runtime add overhead. Measure the exact file at the target context and concurrency.
MoE (mixture of experts) activates only some experts per token. Qwen3-30B-A3B specifies 30.5 billion total and 3.3 billion active parameters, with thinking/non-thinking modes. A smaller active portion does not promise a particular speed. Depending on the runtime, weights may be split between GPU and system memory; offloading affects latency.
What hardware you need
Base a purchase on the actual file, runtime, context length and concurrency. Ollama reports CPU/GPU placement; LM Studio documents offline requirements. A large model need not require multiple GPUs if another loading configuration works acceptably, but speed must be measured.
Practical advice: start with what you have. Download Ollama or LM Studio, try a small model, then a medium one, and you'll find out whether the quality is enough for the given task before you spend a crown on hardware.
Worked example: What to check before buying a GPU
pilot worksheet
| Item | Pilot record |
|---|---|
| Hardware | GPU: 12 GB VRAM; RAM: 32 GB |
| Model | Exact file + quantization: to record |
| Workload | 1 user; 8,000-token document |
| Verification | Loading, memory, latency, correctness |
Compatibility is not confirmed. Test VRAM use and response time with the specific model file.
Where local models genuinely help today
The following are pilot candidates. A 30B label alone does not establish quality; define expected answers and failure cases:
- Summarisation and extraction, summarise a document, pull data out of a contract, compile minutes from a transcript.
- An internal assistant over company documents, answering questions from internal guidelines, manuals, order history, without the documents leaving the company.
- Classification and sorting, sorting enquiries, tickets, reviews; labelling sentiment; pre-processing data for another system.
- Personal-data redaction, proposing removal of identifiers. Check completeness and re-identification risk before external transfer; model rewriting alone does not prove anonymisation.
- High-volume simple tasks, thousands of short operations a day, where a cloud bill would grow and a small model's quality is enough.
Worked example: Extraction should point to its source
ORIGINAL EXAMPLE · FICTIONAL DATA
| Field | Draft output | Source check |
|---|---|---|
| Document | DEMO-017 | Fictional purchase order |
| Product | Alba lamp | "Item" line |
| Quantity | 2 units | "Quantity" line |
| Missing field | Date: not stated | Do not invent a date |
The example does not prove where a model ran or where data went.
What to compare with a cloud alternative
This is the part people don't like to hear, but you need to know it before you buy servers:
- Quality. Compare specific local and cloud models on identical cases. Location does not determine capability; assess accuracy, appropriate refusals and correction work.
- Maintenance. Models, tools and libraries change every month. Someone has to update, test and make sure the model answers just as well after an update.
- Hardware and costs. Include purchase or rental, power, administration, backups and utilization. Cloud may cost less or more depending on load and pricing; there is no universal winner.
- Deployment security. Check network requests, API access, tools, log retention and backups. Ollama documents local operation and disabling cloud features; runtime settings do not automatically constrain a connected application. LM Studio distinguishes offline capabilities from features that require a network.
- Model behaviour. Open models have different safety settings and different tendencies to hallucinate; nobody watches on your behalf what the model tells users.
Base selection on a pilot and operational requirements. Local and cloud systems can both succeed or fail. Combining them needs explicit boundaries for which tasks and data may enter each environment.
The GDPR angle
Local computation can reduce disclosure to other parties only when actual operation matches that design. A downloaded-weight provider need not process your documents; external hosting, support, backups or cloud tools may create other relationships and transfers. The EDPB explains controller and processor roles. Legal basis, minimisation, access and security still matter locally. The label "local" does not establish GDPR compliance.
How to find out whether your company needs them
Answer four questions:
- Do we work with data that must not leave the company, health, legal, financial, trade secrets, and are the contractual guarantees of a cloud provider not enough for us?
- Do we need AI where there's no reliable internet (production, field work, closed networks)?
- Do we have a high volume of simple tasks where a cloud bill would grow into tens of thousands a month and a smaller model's quality is enough?
- Do we have someone to look after it, internally or externally?
Do not turn answers into a universal score. A single offline requirement may decide the choice; without an owner, a local service may be unsuitable despite several reasons for it. Test one task on available hardware first.
That exact comparison, when cloud, when local, how to test it and how to set rules for the team, is what I do in corporate AI training. And anyone who wants to get a feel for it themselves first can start with the open course, where you run a local model live.
Summary
- Downloadable weights have their own licence; local inference means computation on your hardware. Verify privacy across the whole application and data flow.
- Parameters, quantisation, context and concurrency affect memory. Measure latency and quality for the actual configuration.
- Ollama and LM Studio are the usual route for individuals and small teams; start with what you have.
- Summarisation, extraction and classification are pilot candidates, not guaranteed abilities of every model of a certain size.
- Compare cloud and local execution by task, data handling and total cost.
- Local deployment can reduce data transfers, but does not itself establish GDPR compliance.
Frequently asked questions
Is a local model free?
Licence cost depends on the exact terms and use. Also account for hardware, power, software and administration. Free weights do not mean free operation.
What's the smallest computer enough to try it?
Choose an exact model file and supported runtime. Check loading, actual RAM/VRAM use, context and latency. Parameter count or laptop memory alone is insufficient for a reliable recommendation.
Can I connect a local model to company documents?
Yes, with local retrieval and enforced access. Separately verify that embeddings, tools, logs and backups do not send documents elsewhere. Connecting a folder alone proves neither quality nor privacy.
Will local models replace ChatGPT and Claude?
Local models may replace cloud for some tasks and not others. Quality on your cases, required tools, hardware and administration decide; cloud models are not universally better.
Sources and links
- Qwen3-30B-A3B on Hugging Face, 30.5 billion parameters in total, 3.3 billion active, Apache 2.0, thinking and fast mode
- Ollama, running open models locally on macOS, Windows and Linux
- LM Studio, graphical application for downloading and running local models
- GDPR Regulation (EU) 2016/679, processing of personal data