How do people react to AI failure? Automation bias, algorithmic aversion, and perceived controllability

Abstract

AI can make mistakes and cause unfavorable consequences. It is important to know how people react to such AI-driven negative consequences and subsequently evaluate the fairness of AI’s decisions. This study theorizes and empirically tests two psychological mechanisms that explain the process: (a) heuristic expectations of AI’s consistent performance (automation bias) and subsequent frustration of unfulfilled expectations (algorithmic aversion) and (b) heuristic perceptions of AI’s controllability over negative results. Our findings from two experimental studies reveal that these two mechanisms work in an opposite direction. First, participants tend to display more sensitive responses to AI’s inconsistent performance and thus make more punitive assessments of AI’s decision fairness, when compared to responses to human experts. Second, as participants perceive AI has less control over unfavorable outcomes than human experts, they are more tolerant in their assessments of AI. As artificial intelligence (AI) is replacing important decisions that used to be made by human experts, it is important to study how people react to undesirable outcomes caused by AI-made decisions. This study aims to identify two critical psychological processes that explain how people evaluate AI-driven failures. The first mechanism is that people have high expectations of AI’s consistent performance (called automation bias) and then are frustrated by unsatisfactory outcomes (called algorithmic aversion). The second mechanism is that people perceive that AI has less control over negative outcomes, compared to humans, which in turn, reduces negative evaluations of AI. To demonstrate these two ideas, we used two online experiments. Participants were exposed to several scenarios where they experienced undesirable outcomes from either AI or human experts.