I’ve been reading quite a bit about retrieval Augmented Generation (RAG), and it seems like almost AI Discussion now recommends it instead of relying solely on Large Language Models.
On Paper it makes a lot of sense. Healthcare applications deals with constantly changing clinical guidelines, internal documentation, and highly sensitive patient information, so retrieving verified information before generating a response seems like a more reliable approach.
Are Team builds RAG system directly on top of FHIR resources and Clinical documentation, or are they maintaining separate database that needs continuous synchronized? It also makes me wonder how organization handle versioning when medical knowledge changes or new clinical guidelines is released.
Another thing Im curious about is evaluation. It’s relatively easy to build a Prototype that returns relevant document, but how are teams Measuring whether response are actually enough for Clinical Workflow?
I’d be interested to hear from anyone who’s worked on Production healthcare system or experimented with RAG in this Space?
I’m currently working on a small, private application that generates speech therapy reports using Langflow + 3 Chroma DB (RAG). I provide the chatbot with assessment results as attachments, along with some information; it then queries the RAGs, returns a Markdown + JSON result, and a .docx document is generated.
Short answer to your question – tl;dr
Use a RAG only for resources that do not change, or change very little, over time (primarily documentation)
Use the FHIR API for everything else.
A RAG is essentially static; you have to ingate each document, and re-ingate to update it, assuming this is managed properly and that obsolete data is properly deleted. This is not trivial; there are risks if there is a large volume of data, it comes from numerous sources, and needs to be updated regularly. Every new observation or lab result requires re-indexing. For clinical data that changes in real time, this delay is prohibitive and potentially dangerous.
The FHIR API already has its own powerful query engine, with filters that enable precise and structured retrieval.
A RAG processes more or less independent chunks of text; it easily breaks relationships, whereas a well-constructed API query preserves them natively.
Since you’re using the FHIR API, how do you manage latency and authentication for real time queries? Also do you combine API Response with RAG content before sending the final prompt to the LLM?
I’ve been reading quite a bit about retrieval Augmented Generation (RAG), and it seems like almost AI Discussion now recommends it instead of relying solely on Large Language Models.
This does not say why you should use RAG. Here is the why.
An LLM’s knowledge is frozen in time the moment its training finishes. If you ask a base model about your company’s private API documentation, yesterday’s stock prices, or a user’s recent purchase history, it will either admit ignorance or—much worse—hallucinate (I call it Deceive) a highly convincing lie.
To solve this, we use Retrieval-Augmented Generation (RAG). Instead of retraining the model (which is slow and expensive), we turn the LLM into an open-book test taker
One challenge I don’t see discussed enough is keeping the retrieval layer fresh without introducing inconsistencies. In healthcare, clinical guidelines and internal documentation can change frequently, so the ingestion and versioning strategy becomes just as important as the LLM itself. A RAG system is only as reliable as the quality and freshness of the information it retrieves, which makes evaluation and monitoring critical in production.
The following is my response that has been enhanced a bit by ChatGPT.
Use RAG if your requirements call for it, and don’t use it if they don’t. “Retrieval Augmented Generation” sounds more complicated than the underlying concept. At its core, RAG is simply retrieving relevant information and providing it to the LLM as additional context before it generates a response.
A simple example is giving the AI a document or web page to use. Production RAG systems usually automate that process by searching a document collection or vector database for the most relevant information and supplying it to the LLM. The sophistication lies in how the information is retrieved, not in the basic idea of giving the model additional relevant context.
And ChatGPT helped me with:
That is a common implementation of RAG, but it is not the only implementation. RAG is simply retrieving relevant information at query time and providing it to the LLM. If the underlying data has changed, the retrieval can simply be run again for a new query, such as by querying a database or a live API like FHIR. In that case, the latest data is retrieved for each query, so RAG is not limited to static documentation.