Responsible AI: keeping hold of our data and our independence
21/10/2026 Article
We want to do interesting things with AI. We also want to retain positive control of the sensitive data we handle. Both requirements need to survive contact with the actual implementation though and as we rush towards the use of AI, this is becoming an increasingly large challenge. In any case, this is one of our primary goals for how we employ AI and features quite heavily in our upcoming “Responsible AI Usage Statement”. Basically, client information needs the same protection whether we are examining it ourselves or asking a language model to help.
Alongside that, we want to remain operationally independent of multinational AI providers. We would quite like our ability to do our jobs to survive someone else changing a subscription plan.
Privacy and independence fit together here. Being able to run suitable workloads on infrastructure we control gives us more choice over how information is handled. Being able to replace a model or supplier helps us preserve that choice over time.
So, we regularly find ourselves asking the question about how much capability we can achieve when “only” playing with constrained resources. This regularly leads us to asking how we give smaller models better prepared work and still get the quality we need.
Paper: Locating Standards References Using Within Document Repetition
Giving a smaller model a fighting chance
Our recent paper investigates extracting references from cyber security standards and configuration benchmarks. Admittedly, this sounds like an excellent way to make a room go quiet or at least, start snoring. Bear with us.
A control heading might appear in the contents, above the actual requirement and again in an appendix. All three occurrences contain the right words. Only one of those may contain the explanation we need about the control.
A person reading the document has layout to help them. Once it has been converted into chunks of indexed text, some of that structure disappears. The contents page and the substantive requirement can start looking inconveniently similar.
There is useful information left, though. The repetition itself tells us something.
The method in our paper measures repeated sequences within the document, identifies a likely contents group and ranks candidate passages for each suspected heading. Conventional software does that preparation. A language model then examines one heading and one candidate passage at a time.
It can reject the passage, request more context or propose a supported reference. Checks on source identity and quotations keep the proposed result tied to the document. This is the hybrid approach we have been investigating: use deterministic processing to narrow the search, then ask the model to interpret the evidence.
The preparation does not need to choose the right passage first every time. If an appendix entry comes first and the actual description comes second, the reviewer can still get there. Losing the description altogether is a much bigger problem. Even a very impressive model cannot review a passage we have helpfully hidden from it.
More than 2,500 references, one rejection
Subsequent testing covered 33 documents containing more than 2,500 references. Only one reference was missed by the complete system. That result gives us plenty of reason to keep digging. The failure is interesting too.
Initially it was frustrating to find that the deterministic extraction process had actually found the reference. Unfortunately, a downstream validation rule then rejected it as implausible because it consisted of a numerical identifier and a single acronym. The equivalent of a perfectly legitimate reference arrived at the standards compliance party’s door, but our software bouncer did not like its shoes.
It still counts as a failure of the complete system. But we can locate the fault: the evidence was found, then discarded by validation logic that was too restrictive. In an attempt at being positive about it, we are looking at this as a useful reminder to examine the routine and ordinary software surrounding the model helping routine. It can make mistakes with considerable confidence too, along side its non-deterministic AI cousins.
The paper documents the earlier development experiment, involving 44 labelled recommendations in one document. A valid description ranked first for 43 recommendations and second for the remaining one. Later testing extends the evidence for coverage beyond that initial document.
There are separate questions to answer about incorrect accepted references and total computational cost. After all, preparing the evidence takes resources, so any efficiency claim needs to include that work alongside AI inference and retries.
Keeping the capability within reach
The prospect we are pursuing is useful AI that smaller organisations can operate within their own constraints, including their obligations to protect sensitive information.
Smaller models could help make that practical. Better preparation could make those models more useful. But apparently…. neither removes the need to test the complete system properly.
We are publishing Locating Standards References Using Within-Document Repetition so others can examine the method, challenge it and build upon it. It is one part of our work towards keeping useful AI capabilities within reach of smaller companies while retaining control of the information entrusted to us.
Paper: Locating Standards References Using Within Document Repetition