THE MAGAZINE FOR THE FUTURE BY TÜV SÜD

“THERE WILL NEVER BE AN UNBIASED AI”

—— Vivienne Ming is one of the world’s most distinguished experts on artificial intelligence (AI). Ming, an American tech founder and neuroscientist, explains why algorithms aren’t objective and why the use of AI is already encroaching on people’s civil rights.

TEXT ANNA GAUTO
PHOTO DAMIEN MALONEY

Ms. Ming, algorithms provide recommendations for action based on cool mathematical calculations. Does this make them neutral?
———— Many experts and professors actually believe it does. As if artificial intelligence consists solely of simple equations where you enter one number and another one pops out at the end. Sometimes I ask myself if people can actually be that naive or if they’re just pretending to be.
Why would they do that?
———— To more easily sell their AI. Especially for human resource departments, AI is often advertised as providing objective support for selecting new employees. But it’s downright ridiculous to suggest that algorithms are neutral.
Why is that?
———— There are a number of examples that prove the opposite. Let’s look at facial recognition. Twenty-two years ago, when the technology was first being developed, it didn’t recognize the faces of people of color. At the time, the explanation was that the faces from the internet used to train the algorithm were mostly white people and that this error would be corrected. This was understandable at the time, but nothing has changed to this day.
Why not?
———— Because the people who are programming the algorithms feed them — albeit unconsciously — with racist information.
How do you accidentally build algorithms that treat minorities worse?
———— By forgetting or ignoring those people who aren’t white and male like the majority of programmers. And by assuming that mathematics and truth are the same. Of course, taking the technical perspective, algorithms are made up of equations. But they are fed with assumptions about the world—and those are based on human prejudices.
For instance?
———— For an educational start-up, I once built a small AI system to reunite refugee children with their missing families. To train the system, I used scientific data for facial recognition and enriched it with faces from the internet, for instance from Facebook and Twitter. At first, I just wanted to make sure that my neural network understood what faces are. Then it learned a few really interesting things.
Namely?
———— The majority of smiling faces belonged to women. So the system assumed that women always smile. If a woman wasn’t smiling, the network believed that the face must belong to a man. We didn’t teach the network about gender roles. It learned them itself.
You’re alluding to a phenomenon that science calls “social smiling”: many women smile even if they don’t feel like doing so because they believe it is desirable to be friendly.
———— Exactly. And the AI couldn’t help but notice it.
Even Amazon was quite surprised when they discovered the conclusions reached by a seemingly neutral algorithm that was supposed to select personnel. It’s said that you warned Amazon about the algorithm.
———— Yes. Amazon wanted to hire me as its chief scientist. I was told: “In seven years, a million people will be working for us. Your job will be to make their lives better.” It was actually a fascinating job offer. Everything that I’m doing today with Socos, my think tank — crazy, data-driven science — I could have had there.
You turned them down. Why?
———— One of my projects would have been to build an AI to make more diverse personnel selections. It was supposed to find the best software developers based on historic Amazon data. That meant the algorithm should determine the best candidates for the job using information about who received promotions during their first years at Amazon.
That initially sounds plausible…
———— But only at first glance. I had already built an AI for recruiting as the head scientist at another company and knew that this method wouldn’t work. I told this to the people at Amazon as well, but they still wanted to do it their way.
Why didn’t it work?
———— Like other tech companies, Amazon believed that they just needed the smartest programmers and enough data to be able to program anything. However, what these first-class computer engineers didn’t realize was that their data sets did not contain objective indicators for performance, but rather twenty years of subjective personnel decisions and promotion practices. The system very accurately read from the data who the company had predominantly hired and promoted: men. So it consistently eliminated women from consideration.
They stopped using the algorithm at some point?
———— Yes, because it was still screening out women even after the company removed all the obvious references to gender from the data. The system had already internalized too much of what had long been crucial for a successful career at the company—being male. The algorithm then found a way to determine if an application was likely to be from a woman, even without gender markers, and then rejected it. By the way, the same thing could have occurred at many other tech companies.
Why is that?
———— When students today are getting a degree in machine learning, in most cases they are improving existing systems. For instance, they’re trying to make a certain photo recognition program a little bit better. In order to do this they are given the program, the data, the problem and even the answers. Based on this experience, they are suddenly supposed to program an unbiased AI later on. No wonder it doesn’t work.
The computer engineers are following a recipe and that’s the very reason they’re failing?
———— Of course. They’ve never had to compile their own data set or develop their own sets of questions. They’ve only learned how to make systems more accurate and more powerful. In the twenty years that I’ve been working on really tricky problems, not once did I receive a data set and even less often the appropriate question about it. I had to find out both of these myself. Many software developers who go to the big tech firms dutifully perform their assignment: What’s the best possible algorithm based on my data? They don’t question whether the data set itself is suitable, whether the assumptions in it reproduce biases, or where such biases might be hidden in the data. They build outstanding tools. But they keep building more tools, when it’s really houses, not tools, that are needed. They haven’t learned to develop real solutions.

“It’s downright ridiculous to suggest that algorithms are neutral.”

Vivienne Ming
And how do you build an algorithm that doesn’t discriminate?
———— My advice is not to use any historical data and instead to incorporate scientific findings. Psychology, sociology and economics have known for a century what is crucial for someone to be good at their job. Simply ignoring this knowledge is insane.
You’ve built AI for recruitment yourself: what approach did you take?
———— We programmed small AI systems that simulated scientific experiments. We know from psychology that a core characteristic for quality on the job is, for example, resilience—that is, whether or not you are discouraged by failure. Our AI then searched for such indicators for resilience or queried for them in the form of a psychological questionnaire on a website. We know a lot about the world. If we enrich machine learning with it, we can achieve a lot.
AI seems to be holding a mirror up to us as people, in which we can see our own prejudices. Couldn’t AI be used in a very targeted way to make such unconscious biases visible to us?
———— Absolutely. In the end, the systems that we build are really only a reflection of ourselves. After training a language AI only with American literature, it connected positive concepts to men and all things negative to women and people of color. The media turned this into “AI is racist.” But AI isn’t racist, we are. AI is just a reflection of us.
Do neutral algorithms even exist?
———— I don’t believe they do. But there are significant differences. For instance, it’s important how complex the equations are. In modern vehicle engines, machine learning can help use energy more efficiently. In those sorts of simple, closed systems, biases play hardly any role. But when a cognitive system has to make decisions with many uncertain variables, it gets difficult: How fast do I drive heading into a curve? What move do I make next in this game of chess? Who do I consider criminal—and how do I prevent discrimination in hiring? There are so many possibilities that prejudices always play a role. There aren’t any unprejudiced people, which means there will never be an unbiased AI.
Should AI be banned from particularly sensitive areas?
———— It’s something worth thinking about, at the very least. San Francisco has used the flaws of facial recognition technology as a reason to ban the technology in public institutions. On the other hand, I’ve also used that sort of AI to help autistic children understand facial expressions. Or to reunite refugee children with their families. Bans will always also affect AI that helps people or even saves lives. Despite this, I find facial recognition technology highly problematic, for instance in police work.
The use of AI seems to require a continual balancing of pros and cons.
———— This is important, since AI is working its way deeper and deeper into our lives. AI doesn’t just influence who gets which jobs. It decides about loans, the type of news we see, social connections, who the police check and many other facets of our lives. We ask ourselves if AI discriminates, and it does. But the problem runs much deeper.
Where, for example?
———— AI is undermining people’s civil rights. Not because it invariably discriminates, but because it shifts a tremendous amount of power to a very small group of people that AI ultimately serves.
There are guidelines for using AI. What do you suggest to prevent the large data collectors from becoming too powerful?
———— AI must be established as a civil right in order to restore the balance of power between citizens and tech companies. Each individual must have access to AI that is focused on that person’s well-being, like a person’s doctor or lawyer. This also requires having access to the AI infrastructures and algorithms. After all, how else will we know whether there isn’t some hidden system responsible for our not being promoted at work? In the United States, the model of the “public option” for AI has emerged, a sort of voluntary alliance between the state and the citizens. There are some ideas about how this would work, for instance public funding for machine learning or a type of escrow function for private data. None of this means that I think badly of Jeff Bezos, Jack Ma or Larry Page. But because the interests of some billionaires ­aren’t likely to be congruent with those of the users, we should have the right to an AI that is committed to our well-being.

MORE ARTICLES