Engineered Racism

By Gilbert Borman

temp-post-image


The headline says it all: a study found that blacks received less medical care than whites because the software used in the health care system told doctors to do so. Yes, this is actually happening in 2019.

The study was just announced in the respected journal, Science.

The software told the doctors what care was recommended, meaning unless the doctor made the effort to override the system, the doctor gave a lower level of care to blacks. The doctors might not have even known this was happening.
Remember one thing about software: it is engineered. It functions exactly as it is designed to do. Someone engineered this. All software, especially for large organizations, is created through a process. The process starts with the needs of its owners, then determines the who, what, where, when and how of each function; such a process had to be used to design, build, test and deploy this health software.

There is good news and bad news in this story. We do not yet know why the software was designed to function as it did. Right now, we can only speculate. We will learn more about this as it is further investigated.

The likelihood a bunch of guys wearing Klan robes were designing the software for a major American medical group is low. Think about it: the Klan are not famous for their intellectual prowess, most of their members can only dream of getting a degree in computer science.

It is far more likely the health care providers assigned cost control factors into their system. Those controls are what was wrong. Could those controls deliberately have targeted African American medical conditions? Maybe. We do know that software engineers typically test to make sure the product works as designed, they do not currently test to see if a product improperly affects black patients. This is about to change fast.

At the Beautiful Machine, we look at this as a parable of the age we live in: computers run everything in our lives, that means evil can slip in without our even knowing it. It means that we must learn to never accept what the machines tell us regarding our children’s and personal medical care. Always ask: “doctor, if this was you, or your child, what would you do?”

We should be mindful that software can impact our lives in ways both subtle and great. From social media, to medical insurance, to banking, the machines are everywhere. As tools, they can help or harm, but we must always be sure we are asking what they are doing and watch to see if the software is hurting us- whether the code’s engineers intended to, or not.

To read more on this topic, visit: https://arstechnica.com/tech-policy/2019/12/senators-want-answers-about-algorithms-that-provide-black-patients-less-healthcare/