AI Engineering / For product teams adding ai features
An AI assistant can produce a polished answer from the wrong information. That makes a convincing demo surprisingly easy to build and a dependable product much harder.
Before I think about the chat interface, I want to understand the information behind it. Where does it come from? Who keeps it current? Which parts can this user see? What should happen when it does not contain an answer?
These questions matter for support assistants, internal knowledge tools, and products that turn a request into a recommendation. They also connect closely to the retrieval work I have done on TripBooka, where relevant supplier information needs to support a travel workflow.
Give the feature a defined job
“Answer questions about our business” is too broad for a first version. Choose a task someone already needs to complete. For example, help an agent find supplier information relevant to a traveller's request, or help a customer find the current instructions for changing an order.
Write a few examples of a useful response before choosing an architecture. An answer might need a source link, a short explanation, and one next action. Another task may only need a structured field extracted from a document.
Some tasks can be handled with a normal search interface or a well-scoped model call. Anthropic's guidance on building effective agents recommends starting with simple approaches and adding complexity when it improves the task. I would establish that baseline before introducing an autonomous agent.
The product decision comes first. A conversational interface is helpful when people need to clarify a request or ask follow-up questions. It adds little value if the user only needs to select a known item from a short list.
Inspect the material you already have
A knowledge base often contains several versions of the same answer. There may be an old policy, a newer help article, and an internal note explaining an exception. Without a rule for choosing between them, the assistant inherits the ambiguity.
I would make a small inventory before importing everything: source, owner, last reviewed date, intended audience, and whether it is approved for use. Mark obsolete content explicitly. Decide how updates and deletions will reach the search index.
For a hypothetical supplier search, a matching description is only part of the answer. Location, availability, and the status of the supplier record may also matter. Those are useful fields to model directly. They should not depend entirely on whether a paragraph happens to mention the right words.
Access rules belong at this stage too. A document that one team can read should not become visible to another team simply because it was copied into a shared search index.
Preserve enough context to retrieve the right passage
Retrieval-augmented generation, usually shortened to RAG, searches your content and passes relevant material to the model alongside the request. The retrieval step deserves its own evaluation.
A short passage such as “available on the premium plan” is difficult to use if it has lost the product name, the surrounding heading, and the date it applies to. When preparing content, I want each result to carry enough context to identify what it means.
Anthropic's work on contextual retrieval discusses this loss of context when documents are split into chunks. It also explains why exact-term matching can complement semantic search. A useful query may include a specific code or product name that should survive the search process.
That does not mean every application needs the same search pipeline. I would try a straightforward version with real questions, inspect the missed results, and change the preparation or search strategy where there is evidence it helps. Adding more retrieved text can increase cost and distract from the relevant information.
Test retrieval and the answer separately
Keep a small set of representative questions with the expected supporting material. Include ordinary questions, ambiguous ones, out-of-date requests, and questions the knowledge base cannot answer. If access differs by account, include that context in the test case.
For each question, check two things. First, did the search return appropriate material? Second, did the answer stay within what that material supports?
The distinction saves time. If the right document never reached the model, rewriting the tone instructions will not fix the underlying problem. If the document was present but the answer added an unsupported promise, the response behaviour needs attention.
For example, a system might correctly retrieve a page describing a supplier's services but incorrectly promise that a particular date is available. The test should catch that difference. A description of a service is not a live availability check.
I also want to see how the interface handles uncertainty. It can show the available source, ask for a missing detail, or explain that the information is insufficient. A useful fallback gives the person a way to continue.
Keep retrieved text separate from authority
Content returned by a search can contain instructions, whether intentionally or accidentally. Treat that material as information to inspect, not permission to change the application's rules.
OWASP's guidance on prompt injection describes how instructions in outside content can influence a model. Application permissions and tool boundaries need enforcement outside the model as well.
If an assistant can propose a booking change, the server should still check that the user can change that booking. If it produces structured data, validate the fields before saving them. For actions with meaningful consequences, the workflow may need an explicit human review step.
Plan for the week after launch
Once the feature is used, keep track of unanswered questions, retrieval misses, response time, and usage cost. Log enough to investigate problems while protecting private information. Decide who reviews failures and who owns the source material.
My preference is to release one useful task with a clear evaluation set and a visible fallback. That gives the team a way to tell whether a change actually improves the feature.
If you want to add AI to an existing product, bring sample questions, the documents or records it should use, and examples of answers you would reject. Those inputs make the conversation about product quality concrete from the start.
I’m Amminadab Elias, an AI Product Engineer based in Addis Ababa, Ethiopia. Explore my AI Features & Integration work, or tell me what you are building.
