“If you’re training your AI on existing datasets, we know the evidence we have is disproportionately skewed to represent certain populations, namely middle-aged white men,” says Krubiner.
“Without mitigation, machine learning will reproduce and amplify those biases, with potentially disastrous effects for populations underrepresented in the data.”
Improving representation and accounting for lived experience in datasets is crucial.
Anna Studman, Senior Researcher at the Ada Lovelace Institute, is leading work on the impacts of data-driven systems and AI on healthcare.
Interviews with people experiencing poverty or chronic health conditions showed that “the nuance of lived experience doesn’t come through in clinical datasets,” says Studman.
But it’s not always as simple as collecting more data.
Studman says that lack of trust in healthcare systems and institutions is particularly evident in marginalised populations. Organisations using health data need to earn trust from the communities they serve.
Studman believes that greater transparency is needed. “Explaining to people why and how that data would be shared and used is important, especially for people who are digitally excluded or feel that innovations in digital health are being pushed on them.”
Getting AI right is particularly important in health care settings.
The unprecedented speed of the digital transformation of healthcare systems often means that “time-strapped clinicians have to quickly get on board with new technologies that are parachuted in,” Studman explains.
