Skip links

Case Study: A Compliance Analyzer

What it is

A LoRA adapter fine-tuned on Llama 3.2 3B, built to analyse statements about how an organisation operates and return a structured compliance assessment.

Published openly on Hugging Face by AboutKnowledge (Hong Kong) Limited.

Give it a statement — “We do not document management review meetings” — and it returns:

A verdict. Compliant, partially compliant, or non-compliant.

The relevant clause. Which requirement this touches.

An explanation. Why that verdict.

Recommendations. What to do about it.

That structure is the point of the model, and it is what separates it from asking a general model the same question.

Why structured output matters

A general model asked this question gives you a paragraph. Useful to read, awkward to act on.

A structured answer can be used by a system.

Three things become possible:

Sorting. Show me everything non-compliant. Filter by clause.

Counting. How many findings, of which severity, against which standard.

Routing. Non-compliant findings go to one queue, partially compliant to another.

None of that works on prose. A downstream system needs fields, and a fine-tuned model produces them consistently — without a long instruction on every call.

FIGURE 1: PROSE VERSUS STRUCTURE

A general model

  • Returns a readable paragraph
  • Format varies between calls
  • A person must extract the verdict
  • Cannot be sorted or counted

A fine-tuned analyzer

  • Verdict, clause, explanation, recommendations
  • Same shape every time
  • A system can filter and route
  • Findings can be counted and tracked

Two models, two jobs

AboutKnowledge published two related models, and the split is deliberate.

The 1B quality model answers questions and drafts documents. What does a clause require? Write me a checklist. Suggest some KPIs.

The 3B compliance analyzer makes judgements. Here is what we do — is it compliant, and against what?

Why not one model for both?

Different jobs need different training. One is trained to explain; the other to assess and cite. Training a single model on both produces something that does neither cleanly.

And different jobs justify different sizes. Judgement with citation is a harder task than answering a question, which is why the analyzer is the larger of the two.

That is a design decision worth borrowing. Rather than one model that does everything adequately, two models each doing one thing properly.

The training

From the published model card:

| | |

|—|—|

| Base model | Llama 3.2 3B Instruct |

| Framework | Unsloth |

| LoRA rank / alpha | 16 / 16 |

| Dropout | 0.0 |

| Quantised training | 4-bit NF4 |

| Training records | 1,000,000 |

| Training steps | 3,000 |

| Final train loss | 0.1742 |

| Sequence length | 2048 |

A million training records, each an example of a statement and the structured assessment it should produce.

Which is the real work. Not the training run — assembling a million consistent examples of what a good compliance assessment looks like.

What it covers

Standards — ISO 9001, 14001, 45001, 27001, 13485, 50001, plus IATF 16949 for automotive and AS9100D for aerospace.

Verdicts — compliant, partially compliant, non-compliant.

Output — verdict, cited clauses, explanation, recommendations.

The three-way verdict is worth noting. Compliance is rarely binary. “Partially compliant” is the honest answer for most real findings — the thing is being done, just not documented, or done inconsistently.

A model that only said yes or no would be less useful, because it would force a wrong answer on the most common case.

FIGURE 2: WHY THE DESIGN CHOICES MAKE SENSE

Three-way verdict

  • Most real findings are partial. Binary would force a wrong answer.

Cited clause

  • A verdict without a reference cannot be checked or acted on.

Recommendations included

  • The next question is always “so what do we do”.

3B rather than 1B

  • Judgement with citation is harder than answering a question.

How it is used

Both routes are published, as with the sibling model.

In code, loading the adapter with the intended system prompt — establishing it as a quality-management auditor and ISO consultant, and asking for a verdict, clauses, reasoning and recommendations.

Locally, using the published Q8_0 GGUF file — about 3.4 GB — with Ollama.

The system prompt matters more here than usual. The model was trained to produce a specific structure in response to a specific framing. Address it differently and the structure degrades.

Where it fits

Triage. Run a set of statements about your operations through it and see what comes back flagged. That gives an audit team a starting list rather than a blank page.

Preparation. Before an internal audit, check what is likely to come up.

Drafting findings. A first version of a nonconformity write-up, with the clause already identified.

Consistency. Different auditors assess the same thing differently. A model applies the same standard to everything, which is useful as a cross-check even when a person makes the final call.

Where it does not fit

The model card is explicit, and this is the important part.

It should not replace human auditors or regulatory review.

Always verify against the official standard text for compliance decisions.

Why: certification depends on precise clause wording. A model trained on synthetic data can produce a plausible clause reference that is subtly wrong — right standard, wrong sub-clause, or right idea in slightly wrong words.

For triage and drafting, that is acceptable. For a decision a certification body will examine, it is not.

FIGURE 3: HOW TO USE IT PROPERLY

Appropriate

  • Triage — what should we look at first
  • A first draft of a finding
  • A consistency cross-check across assessors
  • Preparation before an internal audit

Not appropriate

  • The compliance decision itself
  • Quoting clause wording as authoritative
  • Anything a certification body reviews
  • Replacing a qualified auditor

What this shows

Three transferable points.

Structured output is a training objective, not a prompting trick. If a downstream system needs fields, train for fields.

Split tasks across models rather than overloading one. Two models doing one job each beat one doing both adequately.

Publish the limitations. Both these model cards say plainly where the output should not be trusted. That is what a professionally published model looks like, and its absence elsewhere is a warning.

The short version

A 3B model fine-tuned to return a verdict, a clause, an explanation and recommendations — consistently, in a shape a system can use.

Trained on a million examples, which is the real work behind it.

A three-way verdict, because most real compliance findings are partial rather than pass or fail.

For triage and drafting, not for the decision — and the model’s own documentation says so, which is how it should be.

Audit preparation starting from a blank page every time?

Get in touch. We build models that produce structured, consistent output your systems can act on — running locally, so nothing confidential leaves your infrastructure.

Leave a comment

Drag