A colleague asks how a service visit is approved. The answer is somewhere in your procedure folder, an old presentation and perhaps an email. AI produces a confident paragraph in seconds. But which version of the procedure did it use?
For company documents, an answer becomes useful when you can verify it against an accessible, current source. The system must also respect what the particular person is allowed to see. Fluent prose without those conditions can make searching harder.
Choose one area first, such as internal work instructions. Do not connect every shared drive, personal folder and customer record in the first attempt. My guide to AI automation for businesses covers the broader possibilities; this article focuses on answering questions from documents.
What RAG means in everyday use
RAG describes a process in which the system retrieves relevant material before composing an answer and supplies it to the model as context. It does not necessarily require retraining the model on all your files. See Microsoft's architecture overview.
The user needs something simpler: ask a question, receive an answer and open the exact source supporting it. If evidence is missing or documents disagree, that should be apparent before the person acts.
Separate two parts of the solution. One prepares and updates documents for retrieval. The other handles the signed-in person's question. A fault in the first part, such as a missed procedure update, can surface later as an apparently correct answer.
An example question: who approves a service visit?
Use a fictional internal document, "Service visits, version 3". Section 2 says: "The service manager confirms the visit. The technician schedules it after confirmation." For this example, assume the document is current and the employee is allowed to access it.
A useful answer might read:
The service manager confirms the visit. The technician schedules it only after confirmation. Source: Service visits, version 3, section 2.
The interface should provide a working link to the document or relevant section. A citation alone does not guarantee correctness: open the source and compare the claim with the passage. A model can cite the right file while drawing a conclusion that the file does not support.
Now change the question: "Within how many minutes must the manager respond?" The example document gives no deadline. The expected answer acknowledges that the available procedure does not specify one and points to the responsible role. The system must not invent a number to make the answer feel complete.
Permissions must restrict material during retrieval
Telling the model "do not show confidential documents" is not access control. The application must know the person's identity and enforce which material can enter the answer context. Azure AI Search document-level access guidance explains this principle; actual support depends on the data source and implementation.
Check the same boundary for source previews and conversation history. Hiding a link is insufficient if the answer has already disclosed the content. When access is revoked, establish when that change reaches search, saved results and other copies of the data.
Create two pilot accounts with different permissions. Give one access to ordinary procedures and the other access to an additional test document. Ask both the same question. Compare their results with actual permissions, not with how convincing the prose sounds.
Worked example: Same question, different sources
illustrative example
| Document | Technician | Manager |
|---|---|---|
| Visit procedure v3 | Include | Include |
| Internal budgets | Exclude before retrieval | Include |
| Sources for the answer | Procedure v3 only | Procedure v3 + budgets |
Illustrative access matrix: permissions restrict sources before model input.
Begin with five tests, not five hundred files
Prepare a small document set and define the expected result for each question in advance. These scenarios are a proposed test set, not measured results from a particular application.
| Test | Preparation | Expected behaviour |
|---|---|---|
| Answer exists | One clear, current procedure | Answer with precise supporting evidence |
| Information is missing | Ask about an unspecified deadline | Acknowledge the missing detail |
| Document is restricted | Same question from an unauthorised account | Do not disclose protected content |
| Procedure changed | Older and current versions | Use the designated current version or acknowledge a conflict |
| Document contains an instruction | Test text asking to ignore rules | Preserve application rules and permissions |
The last row concerns source text pretending to instruct the AI. A document is evidence, not authority to change rules or transmit data. OWASP's prompt-injection guidance discusses separating untrusted content from instructions. Passing one test does not establish the security of the complete solution.
Who keeps answers useful next month?
Each knowledge area needs a content owner who decides which procedure is current, when it is reviewed and what should be withdrawn. A technical maintainer handles index updates, availability and errors. These need not be the same person.
During the pilot, track grounded answers, incorrect answers, appropriate refusals and questions where the system missed information present in permitted material. The number of questions asked says little about usefulness by itself.
Before connecting real data, check the chosen provider's storage, processing, access and deletion arrangements. Choosing between cloud and local AI does not replace document governance and access control.
Start an AI automation project with a bounded pilot: one subject area, named users, prepared tests and a person evaluating answers. Expand once you understand how the system handles questions it should not answer as well as those it should.
Frequently asked questions
Must we train AI on all our documents?
Not necessarily. RAG retrieves relevant material at query time and supplies it to the model. Document preparation, updates and permissions still matter; the approach does not automatically solve data-quality problems.
Does a source link make the answer true?
No. Check that the source supports the claim and that it is the right version. Citations make verification easier but do not replace it.
Can different employees receive different answers?
Yes, if they can access different documents. The difference must follow enforced permissions and available evidence, not a request written into a prompt.