Choose an internal AI assistant with evidence

Evaluate a knowledge assistant against your documents, access rules, and real employee questions before rolling it out.

2 min read

Define the questions it should answer

Begin with a specific team and a bounded collection of knowledge. Collect questions employees ask repeatedly, including questions that require several documents and questions the assistant should decline. Identify which sources are authoritative, how frequently they change, and who maintains them. Old policies and competing versions will affect answer quality even when the underlying model performs well. Agree whether the assistant should locate information, explain it, or carry out an action. These are different responsibilities, with different testing and access requirements, so a first release benefits from a clear purpose.

Test answers and access together

Use a small evaluation set that reflects real work. Check whether each answer is correct, supported by an appropriate source, and useful enough to act on. Include incomplete questions, contradictory documents, and requests for information outside the approved collection. Test with accounts that have different permissions. An assistant should preserve the source system’s access rules and avoid revealing restricted information through summaries or citations. Make uncertainty visible: when the evidence is insufficient, the assistant should explain the gap and point the employee toward a responsible person or an appropriate source.

Plan for everyday operation

Compare options on document refresh, permission handling, evaluation tooling, operating cost, and maintenance effort. Estimate cost using expected usage and representative questions instead of a single demonstration. Decide what conversation data is retained, who can inspect it, and how employees report incorrect answers. Assign owners for source updates and answer-quality reviews. Introduce the assistant to a small group, keep the original knowledge channels accessible, and review failures before widening access. A useful launch decision combines answer quality, clear ownership, and a workable process for correcting the system when company knowledge changes.