Three reasons, and any one can decide it.<\/p>\n
Your data does not leave.<\/strong> Nothing is sent to a third party. For confidential documents, client code, or anything under a data protection obligation, this is not a preference \u2014 it is the requirement.<\/p>\n No per-call cost.<\/strong> An API bill grows with usage. A local model costs the hardware and the electricity.<\/p>\n It works offline.<\/strong> No dependency on somebody else’s service being available.<\/p>\n Against that:<\/strong> a model you can run locally is smaller than the largest hosted ones, so it is less capable in general. The question is whether it is capable enough at your task<\/strong> \u2014 and for a fine-tuned model on a narrow subject, often it is.<\/p>\n A file format for running models efficiently on ordinary hardware.<\/strong><\/p>\n A model published for training is stored one way. A model published for running on your laptop is converted to GGUF \u2014 packaged into a single file that local tools can load directly.<\/p>\n You will see a suffix like Q8_0 or Q4_K_M.<\/strong> That is the quantisation level \u2014 how much precision was traded for size.<\/p>\n Higher number, larger file, slightly better quality.<\/strong> Q8_0 is high quality and larger. Q4 variants are smaller and faster with a modest quality cost.<\/p>\n Both AboutKnowledge quality models publish Q8_0 GGUF files<\/strong> \u2014 the 1B model at around 1.3 GB, the 3B at around 3.4 GB.<\/p>\n Which puts them on a normal laptop<\/strong>, not a server.<\/p>\n FIGURE 1: WHAT THE FILE SIZES MEAN<\/p>\n A 1B model at Q8_0<\/p>\n A 3B model at Q8_0<\/p>\n Lower quantisation<\/p>\n The trade-off<\/p>\n The easiest way to run a model locally.<\/strong><\/p>\n Install it, and running a model is one command. It handles downloading, loading and serving.<\/p>\n What it gives you:<\/strong><\/p>\n A command-line chat.<\/p>\n A local API<\/strong> that looks like the OpenAI one \u2014 which means tools built to call OpenAI can be pointed at your machine instead, often with only a URL change.<\/p>\n Model management, so you can keep several and switch.<\/p>\n That local API is the important part for business use.<\/strong> It means a workflow, a script or an application can use a local model exactly as it would a hosted one.<\/p>\n llama.cpp.<\/strong> The engine underneath much of this. More control, more setup.<\/p>\n LM Studio and Jan.<\/strong> Desktop applications with a graphical interface. Good for trying models without a terminal.<\/p>\n Docker Model Runner.<\/strong> For running models as containers alongside other services.<\/p>\n Most people should start with Ollama or a desktop app<\/strong>, and move to something more controlled if they need it.<\/p>\n A well-published model gives you several routes in, and both AboutKnowledge models do.<\/p>\n The adapter<\/strong>, for loading in code with a framework.<\/p>\n A GGUF file<\/strong>, for running locally.<\/p>\n A Modelfile<\/strong>, which is the recipe Ollama uses to create a named local model.<\/p>\n Working example code<\/strong>, showing the system prompt and a sample question.<\/p>\n That last one is more useful than it sounds.<\/strong> A fine-tuned model expects to be addressed a certain way \u2014 the compliance analyzer’s own example sets it up as a quality-management auditor and asks for a verdict with clauses and recommendations.<\/p>\n Using a model without its intended prompt is how people conclude it does not work.<\/strong><\/p>\n FIGURE 2: FROM PUBLISHED MODEL TO RUNNING LOCALLY<\/p>\n Download the GGUF<\/p>\n Point the Modelfile at it<\/p>\n Create the local model<\/p>\n Run it<\/p>\n Roughly, and it depends on quantisation:<\/strong><\/p>\n A 1B model<\/strong> runs on almost any modern machine.<\/p>\n A 3B model<\/strong> needs a few gigabytes of memory available. Comfortable on a normal laptop.<\/p>\n Larger models<\/strong> \u2014 7B and up \u2014 want more memory and benefit substantially from a GPU.<\/p>\n A model runs on CPU alone.<\/strong> It is slower, and for a small model answering short questions it is often fast enough.<\/p>\n The practical test:<\/strong> download it and try. It costs nothing but time, and it tells you more than any specification comparison.<\/p>\n Four patterns.<\/p>\n Confidential document analysis.<\/strong> Contracts, client material, code. The data never leaves.<\/p>\n Narrow specialist tasks.<\/strong> A fine-tuned model on your subject, called by a workflow.<\/p>\n High volume.<\/strong> Where per-call API pricing would add up.<\/p>\n Offline or restricted environments.<\/strong> Where an internet-dependent service is not an option.<\/p>\n Where they do not fit:<\/strong> anything needing broad general knowledge or the strongest available reasoning. A small local model is good at its subject and unremarkable outside it.<\/strong><\/p>\n If a local model becomes something a process depends on, four things apply.<\/p>\n A dedicated machine.<\/strong> Not somebody’s laptop.<\/p>\n Monitoring.<\/strong> Know when the service stops.<\/p>\n Version control.<\/strong> Know which model version is running, so results are reproducible.<\/p>\n A fallback.<\/strong> What happens if it is unavailable \u2014 queue, retry, or a manual path.<\/p>\n Same discipline as any production service.<\/strong> A model is not exempt because it is new.<\/p>\n FIGURE 3: LOCAL OR HOSTED<\/p>\n Local<\/p>\n Hosted<\/p>\n Running a model locally means your data stays where it is, there is no per-call cost, and it works offline.<\/strong><\/p>\n GGUF plus Ollama makes this genuinely easy<\/strong> \u2014 a download and one command.<\/p>\n A small fine-tuned model on a laptop can be better at its own subject<\/strong> than a much larger general model, which is what makes this practical rather than merely private.<\/p>\n And use the model’s intended prompt.<\/strong> A fine-tuned model addressed the wrong way looks worse than it is.<\/p>\n Data that cannot go to a third-party AI service?<\/p>\n Get in touch. We build and deploy local models \u2014 including fine-tuned ones for narrow business tasks that run entirely on your own hardware.<\/p>\n<\/div>","protected":false},"excerpt":{"rendered":" Why run a model locally Three reasons, and any one can decide it. Your data does not leave. Nothing is sent to a third party. For confidential documents, client code, or anything under a data protection obligation, this is not a preference \u2014 it is the requirement. No per-call cost. An API bill grows with […]<\/p>\n","protected":false},"author":3,"featured_media":9438,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[26],"tags":[],"class_list":["post-9437","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai"],"yoast_head":"\nGGUF, briefly<\/h2>\n
\n
\n
\n
\n
Ollama<\/h2>\n
The other options<\/h2>\n
What a model card offers<\/h2>\n
\n
\n
\n
\n
Hardware<\/h2>\n
Where local models fit in a business<\/h2>\n
Running one in production<\/h2>\n
\n
\n
The short version<\/h2>\n