Responsible Healthcare AI

Why Responsible AI Matters More in Healthcare

When a recommendation algorithm on a shopping site gets it wrong, you see an irrelevant product ad. When a healthcare AI gets it wrong, a patient might receive the wrong treatment, miss a critical diagnosis, or be denied care they need. The stakes are categorically different, and so the standards must be higher.

Responsible healthcare AI is not a nice-to-have ethics module tacked onto the end of a course. It is the foundation that determines whether AI in healthcare helps or harms. In this chapter, we will examine the specific ways healthcare AI can go wrong, the regulatory frameworks designed to prevent that, and the technical and organisational practices that make AI trustworthy.

Racial and Demographic Bias in Healthcare AI

Healthcare AI has already produced documented examples of bias that harmed real patients. These are not hypothetical risks — they are failures that have been published in peer-reviewed journals and covered by mainstream media.

Case Study 1: Pulse Oximetry Bias

A pulse oximeter is a small clip placed on a patient's finger that measures blood oxygen saturation (SpO2). It works by shining light through the skin and measuring how much is absorbed by oxygenated vs deoxygenated haemoglobin. It is one of the most common devices in medicine — used in every hospital, ambulance, and increasingly in consumer wearables (Apple Watch, Fitbit).

In 2020, a landmark study in the *New England Journal of Medicine* found that pulse oximeters overestimate oxygen levels in patients with darker skin by 3-8 percentage points. The device might read 95% (normal) when the true level is 88% (dangerously low and requiring supplemental oxygen).

This is not an AI problem per se — it is a sensor calibration problem. But it becomes an AI problem when:

The FDA issued guidance in 2023 requiring manufacturers to test pulse oximeters on diverse populations, but the installed base of biased devices remains enormous.

Case Study 2: Dermatology AI on Dark Skin

AI models that diagnose skin conditions from photographs perform significantly worse on darker skin tones. A 2021 study in *JAMA Dermatology* found that leading dermatology AI tools had accuracy rates of 80-90% on light skin but dropped to 55-70% on dark skin.

The root cause is training data imbalance. The datasets used to train these models — including widely used research datasets like ISIC (International Skin Imaging Collaboration) — are overwhelmingly composed of images of light-skinned patients. The AI learned what melanoma looks like on white skin but never saw enough examples on dark skin to learn the different visual patterns.

This has real consequences. Melanoma in Black patients is more often diagnosed at later stages (when survival rates are much lower), and AI tools that cannot detect lesions on dark skin will widen this gap rather than narrow it.

Case Study 3: The Optum Algorithm

In 2019, researchers at UC Berkeley published a study in *Science* revealing that a widely used algorithm by Optum (a UnitedHealth Group subsidiary) for identifying patients who need extra care was systematically biased against Black patients. The algorithm used healthcare spending as a proxy for health needs — but Black patients historically spend less on healthcare due to barriers like insurance coverage gaps, distrust of the medical system, and unequal access. As a result, the algorithm identified healthier white patients as higher-need than sicker Black patients.

At equal levels of illness, Black patients were assigned lower risk scores. The study estimated that fixing this bias would increase the percentage of Black patients flagged for extra care from 17.7% to 46.5%.

This case illustrates a critical principle: the choice of proxy variable determines whether the AI is fair. Healthcare spending is not the same as healthcare need. An AI system that conflates the two inherits and amplifies the structural inequities of the healthcare system it was trained on.

Case Study 4: Chest X-Ray AI and Underserved Populations

A 2022 study in *Nature Medicine* found that chest X-ray AI models trained primarily on data from academic medical centres performed significantly worse on images from community hospitals, rural clinics, and safety-net hospitals (hospitals that serve a high proportion of uninsured and Medicaid patients). The X-ray machines, image quality, patient positioning, and disease prevalence all differ between these settings.

An AI model that works beautifully at Massachusetts General Hospital may fail at a community health centre in rural Appalachia — and the patients at that health centre are the ones who need diagnostic AI the most.

> Look at data/bias-case-studies.json for the detailed bias case studies and mitigation strategies used in the sandbox exercises.

FDA Regulatory Pathway for AI/ML

The US Food and Drug Administration has been the global leader in regulating AI as a medical device. As of 2025, the FDA has authorised over 900 AI/ML-enabled devices — primarily in radiology (75%), cardiology (10%), and ophthalmology (5%).

The Three Pathways

The Predetermined Change Control Plan (PCCP)

Traditional medical device regulation assumes that a device is fixed — you validate it, clear it, and it does not change. But AI models are designed to learn and improve. A model trained on 100,000 images today might be retrained on 500,000 images next year, with different performance characteristics.

The FDA's 2023 guidance on Predetermined Change Control Plans allows manufacturers to pre-specify the types of changes their AI will undergo (new training data, algorithm updates, expanded indications) and the validation protocols they will follow. If the changes fall within the approved plan, the manufacturer does not need to submit a new 510(k) for each update.

This is a genuinely novel regulatory framework — no other product category has anything like it. It acknowledges that AI is fundamentally different from traditional devices and needs a regulatory model that accommodates continuous improvement.

EU AI Act: High-Risk Classification

The European Union's AI Act, which began phased implementation in 2025, takes a different approach from the FDA. Instead of regulating AI as a medical device (product-specific), the EU AI Act regulates AI by risk category (horizontal regulation).

Healthcare AI is classified as high-risk, which triggers mandatory requirements:

Non-compliance with the EU AI Act can result in fines of up to 35 million EUR or 7% of global annual turnover — whichever is higher.

HIPAA De-Identification: Safe Harbor Method

Training AI models requires data. In US healthcare, patient data is protected by HIPAA. To use patient data for AI development without individual patient consent, the data must be de-identified — stripped of all information that could identify a specific person.

HIPAA defines two methods for de-identification. The more commonly used is the Safe Harbor method, which requires removal of 18 specific identifiers:

Additionally, the covered entity must have no actual knowledge that the remaining information could identify an individual.

The Re-Identification Risk

De-identification is not as simple as removing names and dates. Research has shown that:

AI developers must use additional protections beyond Safe Harbor:

> Look at data/deidentification-checklist.json for the Safe Harbor compliance checklist and re-identification risk assessment used in the sandbox exercises.

Algorithmic Accountability

Who is responsible when a healthcare AI makes an error? This question does not have a clean answer, and the legal and ethical frameworks are still evolving.

The Accountability Stack

In current US law, the physician retains ultimate liability for clinical decisions. An AI recommendation does not transfer responsibility. But this creates a tension: if a physician overrides a correct AI recommendation and the patient is harmed, was the physician negligent? If a physician follows an incorrect AI recommendation, is the AI developer liable?

These questions are working their way through courts and legislatures. In the interim, best practices include:

A Framework for Evaluating Healthcare AI

Before deploying or purchasing a healthcare AI tool, ask these five questions:

If the vendor cannot answer these questions clearly, the tool is not ready for clinical use.

Key Takeaways