Why run a model locally
Three reasons, and any one can decide it.
Your data does not leave. Nothing is sent to a third party. For confidential documents, client code, or anything under a data protection obligation, this is not a preference — it is the requirement.
No per-call cost. An API bill grows with usage. A local model costs the hardware and the electricity.
It works offline. No dependency on somebody else’s service being available.
Against that: a model you can run locally is smaller than the largest hosted ones, so it is less capable in general. The question is whether it is capable enough at your task — and for a fine-tuned model on a narrow subject, often it is.
GGUF, briefly
A file format for running models efficiently on ordinary hardware.
A model published for training is stored one way. A model published for running on your laptop is converted to GGUF — packaged into a single file that local tools can load directly.
You will see a suffix like Q8_0 or Q4_K_M. That is the quantisation level — how much precision was traded for size.
Higher number, larger file, slightly better quality. Q8_0 is high quality and larger. Q4 variants are smaller and faster with a modest quality cost.
Both AboutKnowledge quality models publish Q8_0 GGUF files — the 1B model at around 1.3 GB, the 3B at around 3.4 GB.
Which puts them on a normal laptop, not a server.
FIGURE 1: WHAT THE FILE SIZES MEAN
A 1B model at Q8_0
- Roughly 1.3 GB. Runs comfortably on a laptop.
A 3B model at Q8_0
- Roughly 3.4 GB. Still a laptop, with reasonable memory.
Lower quantisation
- Smaller and faster, with a modest quality cost.
The trade-off
- Local models are smaller. The question is whether they are good enough at your task.
Ollama
The easiest way to run a model locally.
Install it, and running a model is one command. It handles downloading, loading and serving.
What it gives you:
A command-line chat.
A local API that looks like the OpenAI one — which means tools built to call OpenAI can be pointed at your machine instead, often with only a URL change.
Model management, so you can keep several and switch.
That local API is the important part for business use. It means a workflow, a script or an application can use a local model exactly as it would a hosted one.
The other options
llama.cpp. The engine underneath much of this. More control, more setup.
LM Studio and Jan. Desktop applications with a graphical interface. Good for trying models without a terminal.
Docker Model Runner. For running models as containers alongside other services.
Most people should start with Ollama or a desktop app, and move to something more controlled if they need it.
What a model card offers
A well-published model gives you several routes in, and both AboutKnowledge models do.
The adapter, for loading in code with a framework.
A GGUF file, for running locally.
A Modelfile, which is the recipe Ollama uses to create a named local model.
Working example code, showing the system prompt and a sample question.
That last one is more useful than it sounds. A fine-tuned model expects to be addressed a certain way — the compliance analyzer’s own example sets it up as a quality-management auditor and asks for a verdict with clauses and recommendations.
Using a model without its intended prompt is how people conclude it does not work.
FIGURE 2: FROM PUBLISHED MODEL TO RUNNING LOCALLY
Download the GGUF
- One file, from the model page
Point the Modelfile at it
- Edit the path
Create the local model
- One command
Run it
- Chat, or call the local API
Hardware
Roughly, and it depends on quantisation:
A 1B model runs on almost any modern machine.
A 3B model needs a few gigabytes of memory available. Comfortable on a normal laptop.
Larger models — 7B and up — want more memory and benefit substantially from a GPU.
A model runs on CPU alone. It is slower, and for a small model answering short questions it is often fast enough.
The practical test: download it and try. It costs nothing but time, and it tells you more than any specification comparison.
Where local models fit in a business
Four patterns.
Confidential document analysis. Contracts, client material, code. The data never leaves.
Narrow specialist tasks. A fine-tuned model on your subject, called by a workflow.
High volume. Where per-call API pricing would add up.
Offline or restricted environments. Where an internet-dependent service is not an option.
Where they do not fit: anything needing broad general knowledge or the strongest available reasoning. A small local model is good at its subject and unremarkable outside it.
Running one in production
If a local model becomes something a process depends on, four things apply.
A dedicated machine. Not somebody’s laptop.
Monitoring. Know when the service stops.
Version control. Know which model version is running, so results are reproducible.
A fallback. What happens if it is unavailable — queue, retry, or a manual path.
Same discipline as any production service. A model is not exempt because it is new.
FIGURE 3: LOCAL OR HOSTED
Local
- Data never leaves your infrastructure
- No per-call cost
- Works offline
- Smaller, so narrower capability
Hosted
- The strongest available models
- Nothing to run or maintain
- Data goes to the provider
- Cost grows with usage
The short version
Running a model locally means your data stays where it is, there is no per-call cost, and it works offline.
GGUF plus Ollama makes this genuinely easy — a download and one command.
A small fine-tuned model on a laptop can be better at its own subject than a much larger general model, which is what makes this practical rather than merely private.
And use the model’s intended prompt. A fine-tuned model addressed the wrong way looks worse than it is.
Data that cannot go to a third-party AI service?
Get in touch. We build and deploy local models — including fine-tuned ones for narrow business tasks that run entirely on your own hardware.
