During AI training sessions one question comes up more and more often: "Can't we run AI ourselves so our data doesn't leave?" The answer is: you can, I do it myself, and it's easier today than people think. But at the same time, for most companies it isn't a replacement for ChatGPT, Claude or Gemini – at least not yet. In this article I'll explain without technical jargon what local models are, how to read their parameters, what hardware you need, where they genuinely help today and where you'd get burned.
An open model is an AI model whose "weights" (learned parameters) you can download and run on your own hardware – typically through tools like Ollama or LM Studio. Runs locally means that neither queries nor documents leave your computer or server, which is a strong argument with sensitive data and GDPR. Current open models around 30 billion parameters (for example the Qwen3 series) are good at summarisation, internal assistants over company documents and classification. For most companies, however, they aren't yet a replacement for cloud models: on complex tasks they lag in quality, and they need hardware, maintenance and secure deployment. They make sense for sensitive data, offline operation and high volumes of simple tasks; elsewhere a business plan of a cloud service is the more sensible choice.
What an open model is and how it differs from ChatGPT
When you type into ChatGPT, Claude or Gemini, your text goes to the provider's servers, where it's processed by a model you know only as much about as the provider tells you, and the answer comes back. The model is closed – you can't download it, inspect it or run it elsewhere.
An open model (more precisely an "open-weights model") differs in one fundamental way: its learned parameters are published for download, usually on the Hugging Face platform, under a licence that says what you may do with it. Some have the Apache 2.0 licence, which allows commercial use with practically no restrictions (for example Alibaba's Qwen3 series); others, like Meta's Llama, have their own licence with terms you have to read. Open models are also released by Mistral, Google (Gemma), DeepSeek and others; the line-up changes every month.
Important: "open" doesn't mean "worse". It means you can run the model yourself – with everything that follows from that.
What "runs locally" means
You download the model (files ranging from a few to tens of gigabytes) and run it on your own hardware: a laptop, a workstation or a company server. From that moment:
- no data leaves – queries, documents and answers stay on your machine,
- you don't pay per token – you pay for hardware and electricity,
- it works without internet,
- you are responsible for operation, updates, security and for the model not answering nonsense.
The most common route for individuals and small teams is Ollama (a command line plus a local interface, running on macOS, Windows and Linux) and LM Studio (a graphical application with a model catalogue). Both download the model, run it and offer an interface that a chat or your own application connects to. For company deployment with more users there are server tools, but that's already a job for someone who manages servers.
How to read model size: parameters and quantisation
Parameters are the learned numbers inside the model; their count is given in billions (7B, 30B, 70B). Simplified: more parameters, smarter model, but higher memory demands and slower answers.
Quantisation is compression: parameters are stored with less precision (for example 4 bits instead of 16), the model shrinks several times over and loses a little quality. It's thanks to quantisation that a model with tens of billions of parameters fits into an ordinary computer.
A rough rule of thumb for memory: with common 4-bit quantisation, count roughly half to six tenths of a gigabyte of memory per billion parameters, plus a reserve for context (the length of text the model works with). A model around 30 billion parameters therefore wants roughly 18–20 GB of free graphics or unified memory; a 7–8B model fits into 5–6 GB. Exact requirements differ by model and settings – always check them in the description of the specific model.
One more abbreviation you'll see: MoE ("mixture of experts") and labels like "30B-A3B". The model has 30 billion parameters in total, but uses only about 3 billion for each step. It has to fit into memory whole, but computes with only a part – so on ordinary hardware it's noticeably faster than an equally large "dense" model. That's exactly how Qwen3-30B-A3B is built (30.5 billion parameters in total, 3.3 billion active, Apache 2.0 licence), which can switch between a "thinking" mode for more complex tasks and a fast mode for ordinary dialogue.
What hardware you need
Order-of-magnitude, for orientation:
- Small models (up to ~8B): a current laptop with 16 GB of memory. Suitable for classification, simple summaries, text completion.
- Medium models (~14–32B): a graphics card with 16–24 GB of memory, or a Mac with 32 GB or more of unified memory. This is where useful quality for internal assistants and longer documents begins.
- Large models (70B and up): several graphics cards or a server – an investment in the higher tens to hundreds of thousands of crowns and someone to look after it.
Practical advice: start with what you have. Download Ollama or LM Studio, try a small model, then a medium one, and you'll find out whether the quality is enough for the given task before you spend a crown on hardware.
Where local models genuinely help today
Current open models around 30 billion parameters (the Qwen3 series and newer generations, comparable models from other publishers) are, in my experience, good at:
- Summarisation and extraction – summarise a document, pull data out of a contract, compile minutes from a transcript.
- An internal assistant over company documents – answering questions from internal guidelines, manuals, order history, without the documents leaving the company.
- Classification and sorting – sorting enquiries, tickets, reviews; labelling sentiment; pre-processing data for another system.
- Anonymisation – removing personal data locally before sending to a cloud model.
- High-volume simple tasks – thousands of short operations a day, where a cloud bill would grow and a small model's quality is enough.
Where you'd get burned – and why they aren't yet a replacement for the cloud
This is the part people don't like to hear, but you need to know it before you buy servers:
- Quality on complex tasks. Long reasoning, demanding analysis, quality writing, complex code: top cloud models are still noticeably ahead. A local model that looks great in a demo summarising a paragraph starts making mistakes on a more complex brief that you won't notice until you check.
- Maintenance. Models, tools and libraries change every month. Someone has to update, test and make sure the model answers just as well after an update.
- Hardware and costs. Graphics cards age, electricity is paid for, the server has to run somewhere. A cloud business plan for a few dozen users often works out cheaper than one decent server plus human labour.
- Deployment security. "Data doesn't leave" holds only when the server is well secured. A badly deployed local model reachable from the internet is worse than a good cloud provider with a data processing agreement.
- Model behaviour. Open models have different safety settings and different tendencies to hallucinate; nobody watches on your behalf what the model tells users.
That's why my recommendation for most companies is: a cloud model with a business plan and contractually covered data processing as the base, a local model as a complement for sensitive tasks, offline scenarios and high-volume simple operations. Whoever starts from a local model because "it's free and safe" usually ends up with an expensive server and average quality.
The GDPR angle
Local operation has one big advantage from a GDPR perspective: personal data doesn't leave the company, so you don't need a data processing agreement with the model provider and the question of transfers outside the EU disappears. But everything else remains: you must have a legal basis for processing, inform data subjects, secure the server, manage access and handle deletion. A local model doesn't solve GDPR – it solves one part of it. If you're deploying AI on personal data, I also recommend reading the article on the AI Act for Czech companies, which covers the obligations around AI systems in practice.
How to find out whether your company needs them
Answer four questions:
- Do we work with data that must not leave the company – health, legal, financial, trade secrets – and are the contractual guarantees of a cloud provider not enough for us?
- Do we need AI where there's no reliable internet (production, field work, closed networks)?
- Do we have a high volume of simple tasks where a cloud bill would grow into tens of thousands a month and a smaller model's quality is enough?
- Do we have someone to look after it – internally or externally?
Two or more "yes" answers mean it's worth trying a local model – first on a laptop, on one specific task, comparing quality against a cloud model. One or no "yes" means you're better off with the cloud and well-set usage rules.
That exact comparison – when cloud, when local, how to test it and how to set rules for the team – is what I do in corporate AI training. And anyone who wants to get a feel for it themselves first can start with the open course, where we run a local model live.
Summary
- Open model = downloadable weights under a licence; local operation = data doesn't leave your hardware, you pay with hardware and your own responsibility.
- Read size through parameters and quantisation: a 4-bit model needs roughly half to six tenths of a GB of memory per billion parameters; MoE models (e.g. 30B-A3B) are faster on ordinary hardware.
- Ollama and LM Studio are the usual route for individuals and small teams; start with what you have.
- Models around 30B are good today at summarisation, internal assistants, classification, anonymisation and high-volume simple tasks.
- For most companies they aren't yet a replacement for cloud models due to quality on complex tasks, maintenance, hardware and deployment security – they're a complement for sensitive data, offline operation and volume.
- Local operation solves part of GDPR (data doesn't leave), not all of GDPR.
Frequently asked questions
Is a local model free?
The licence is usually free even for commercial use (with Apache 2.0 yes; with other licences read the terms). But you pay with hardware, electricity and above all the time of someone who deploys, updates and monitors the model. For a small team a cloud business plan often works out cheaper.
What's the smallest computer enough to try it?
An ordinary laptop with 16 GB of memory handles small models (up to about 8 billion parameters). For models around 30 billion parameters you want a graphics card with 16–24 GB of memory or a computer with 32 GB or more of unified memory. Check exact requirements for the specific model.
Can I connect a local model to company documents?
Yes, that's one of the most useful scenarios: an internal assistant that answers from your guidelines, manuals and order history without the documents leaving the company. But it requires proper setup of document search and quality control of the answers – it's not just "upload and done".
Will local models replace ChatGPT and Claude?
Not yet. On complex tasks top cloud models are noticeably ahead and local deployment requires maintenance and hardware. The sensible approach is to combine: cloud with a business plan as the base, a local model for sensitive data, offline operation and high-volume simple tasks.
Sources and links
- Qwen3-30B-A3B on Hugging Face – 30.5 billion parameters in total, 3.3 billion active, Apache 2.0, thinking and fast mode
- Ollama – running open models locally on macOS, Windows and Linux
- LM Studio – graphical application for downloading and running local models
- GDPR Regulation (EU) 2016/679 – processing of personal data