The clinical argument for AI in healthcare stopped being interesting a while ago. Imaging models flag findings a tired radiologist can miss. Predictive models spot patients heading for readmission. Transcription drafts a note in the time it takes to say it out loud.
Very little of this is in daily use, and the reason is almost never accuracy.
Projects stall somewhere between "the model works on our data" and "a clinician uses it on a Tuesday". That gap is an integration problem, a governance problem and a workflow problem, and it is the part nobody budgets for.
The data is not where the model expects it
A hospital does not have a dataset. It has an EHR, a lab system, a PACS, a scheduling system, a billing platform, and several spreadsheets that somebody maintains personally.
Extracting a training set from that is a project. Doing it repeatedly, in production, at the speed a clinical workflow needs, is a bigger one. HL7 and FHIR help enormously and are still only a common vocabulary — they do not make two systems agree on what a "visit" is, or reconcile the same patient existing three times because a name was entered differently at reception.
Before any model discussion, someone has to answer: which system is authoritative for each field, and how does a correction in one propagate to the rest? Get that wrong and you have an accurate model reading stale data, which is worse than no model, because now it is confident.
Somebody has to be accountable for the output
Regulators are clear that a person remains responsible for clinical decisions. That has consequences the technical design has to carry.
The system needs to record what it recommended, on what inputs, at what time, and who saw it. Not for analytics. For the conversation eighteen months later about why a particular decision was made.
It needs to fail visibly. A model that silently degrades because the input distribution shifted is dangerous in a way an outage is not. An outage is noticed.
And it needs a defined behaviour for uncertainty. A confidence score is not a plan. Below whatever threshold, does the software stay quiet, escalate, or show its answer with a caveat? That is a clinical policy decision, and it belongs to the organisation, not the vendor.
Privacy is an architecture decision, not a checkbox
HIPAA in the US, GDPR in Europe, and their equivalents elsewhere all push the same way: minimise what you collect, control who sees it, prove you did both.
For AI systems that turns into concrete constraints. Where does inference run, and does patient data leave your infrastructure to reach it? If a third-party API is involved, what does its data retention policy actually say? Can you produce an audit trail showing which records the model touched?
These questions change the architecture, so they need answering before it is built rather than during a security review afterwards.
Bias is a data problem you inherit
A model trained on one hospital's population learns that population. Deployed somewhere with a different demographic, it is quietly less accurate for the people it was not trained on — and nothing in the interface says so.
The mitigations are unexciting and they work: know what your training population looked like, measure performance across subgroups rather than in aggregate, and re-measure after deployment. Aggregate accuracy is the number that hides the problem.
Where the money actually goes
In practice the budget splits roughly like this, and only the first line is what people expect.
The model, or the API subscription if you are using someone else's.
Integration with clinical systems, which is usually the largest line and the one most often underestimated.
Workflow design, so the output appears where a clinician is already looking rather than in a separate application nobody opens.
Governance and monitoring, forever, because a deployed model is a system that needs operating.
Four questions before you start
Which system is authoritative for each piece of clinical data, and how do corrections propagate?
Who is accountable for acting on an output, and what does the audit trail need to show?
What happens when the model is uncertain — silence, escalation, or a caveat?
How will you measure performance across patient subgroups after go-live, not just before?
Clear answers and the build is tractable. Without them you get an accurate model that nobody uses, which is the most common outcome in this field and the most expensive one.
We build the systems this runs on — healthcare CRM and patient-facing platforms, and the integration layer that connects them. If you are working through where AI fits in yours, get in touch.
