From predicting who will be a repeat offender to who's the best candidate for a job, computer algorithms are now making complex decisions in lieu of humans. But increasingly, many of these algorithms are being found to replicate the same racial, socioeconomic, or gender-based biases they were built to overcome.
This racial bias extends to software widely used in the healthcare industry, potentially affecting access to care for millions of Americans, according to a new study by researchers at the University of California, Berkeley, the University of Chicago Booth School of Business, and Partners HealthCare in Boston.
The new study, published October 25 in the journal Science, found that a type of software program that determines who gets access to high-risk healthcare management programs routinely lets healthier whites into the programs ahead of blacks who are less healthy. Fixing this bias in the algorithm could more than double the number of black patients automatically admitted to these programs, the study reveals.
"We found that a category of algorithms that influences healthcare decisions for over 100 million Americans shows significant racial bias," says Sendhil Mullainathan, the Roman Family University Professor of Computation and Behavioral Science at Chicago Booth and senior author of the study.
"The algorithms encode racial bias by using healthcare costs to determine patient 'risk,' or who was mostly likely to benefit from care management programs," says Ziad Obermeyer, acting associate professor of health policy and management at UC Berkeley and lead author of the paper.
"Because of the structural inequalities in our healthcare system, blacks at a given level of health end up generating lower costs than whites," Obermeyer says. "As a result, black patients were much sicker at a given level of the algorithm's predicted risk."
By tweaking the algorithm to use other variables to predict patient risk, such as costs that could be avoided by preventative care, researchers were able to correct much of the bias that was initially built into the algorithm.
"Algorithms by themselves are neither good nor bad," Mullainathan says. "It is merely a question of taking care in how they are built. In this case, the problem is eminently fixable––and at least one manufacturer appears to be working on a fix. We would encourage others to do so."
More generally, Obermeyer says, incorporating routine audits into algorithm developers' workflows would help. "For algorithms, just as for medicine, we'd prefer to prevent problems instead of curing them."
Digging up the roots of algorithmic bias
Uncovering algorithmic bias––be it in the criminal justice system, in hiring decisions, or in healthcare––is often hindered by the fact that many of the prediction algorithms in use today are designed by private companies and are proprietary, making it difficult for data scientists and researchers to analyze them.
To tackle this problem, Mullainathan and Obermeyer partnered with researchers at an academic hospital that was using a risk-based algorithm to determine which patients were getting preferential access to a high-risk care management program. Programs like this are designed to improve care for patients with complex medical needs by providing them with additional attention and resources.