Well constructed machine learning models can give a business a competitive advantage by helping make more nimble decisions but ML models should not be left to themselves. Companies that can make decisions faster can capture new opportunities quicker and compete with and take share from larger rivals. While ML can be very valuable, there are also risks when models don’t perform as expected or perpetuate past mistakes.
To work, ML models need to be trained and how we train them will dictate if outcomes are good or bad. Machine learning models are like children they learn from what they see around them and don’t differentiate between good and bad examples. Models that malfunction or make decisions that cut against society’s ethics can cause serious damage to brands or create legal problems. ML algorithms need rules or guardrails to ensure that they don’t cause irreversible damage. With rules defined by humans to monitor ML models, we can have more confidence that they will not make unscrupulous decisions that can cause issues.
How can ML get into trouble?
Well by learning from our bad habits or unethical past behavior. Models are developed using historical data and if that data includes biases, these prejudices can unwittingly be incorporated into the model. In many cases, ML models can even amplify our past poor decisions. Specifically, bias around gender and race can lead ML models to make discriminatory decisions. For example, a model used to hire software developers may detect a trend in the data that shows that the majority of software developers are male and learn to disqualify applicants that are not male. Using this variable to discriminate against females is not only unethical but it is illegal.
An astute data scientist should be able to spot this error and exclude sex as a factor in the model, but there are secondary and tertiary variables that may correlate with sex that can lead to an ML model to make a sexist decision. The fact that an applicant might be a member of a women’s organization may lead to rejection. While not specifically disqualifying women, females are much more likely to be a member of a women’s group or attending a women’s college leading to discriminatory practices. This is not a hypothetical example but is what happened to a recruiting bot built by Amazon.
Even when the most diligent data scientist removes biases from models, data drift and new data incorporated into the model can introduce new biases. This requires someone to monitor models in production to make sure that they are operating properly. Unfortunately in some cases, models can get orphaned with no owner to keep track of them. Algorithms can operate for an extended period of time before anyone notices that they are discriminating.
