Artificial Intelligence (AI) holds immense promise for revolutionizing various aspects of our lives, from healthcare to recruitment processes. However, recent incidents have highlighted significant challenges accompanying AI system deployment. AI failures like misbehaving chatbots have raised critical questions about these technologies’ ethical and responsible development. How can we protect users from AI failures like bias, misinformation, and hallucinations?
AI Failures: Bias in Recruiting
Amazon’s Gender Bias in Recruitment
In 2018, Reuters reported that Amazon had developed an AI-powered recruitment tool that favored male candidates over female candidates. The system penalized phrases like “women’s” and downgraded candidates from all-women colleges, perpetuating gender bias in hiring practices. Despite attempts to edit the tool to make it neutral, Amazon ultimately scrapped the project due to concerns about algorithmic discrimination.
iTutor Group’s Ageist Recruiting AI
iTutor Group, a tutoring company, faced backlash in 2023 when it was revealed that its AI-powered recruiting software automatically rejected female applicants ages 55 and older and male applicants ages 60 and older. The US Equal Employment Opportunity Commission (EEOC) sued iTutor Group, leading to a settlement of $365,000. The incident raised concerns about age discrimination and the need for stringent anti-discrimination policies in AI recruitment tools.
AI Failures: Misinformation & Hallucination
ChatGPT’s Academic Misattribution
In a case reported in which a Texas professor allegedly flunked his entire class due to an AI program, ChatGPT, falsely claiming it wrote their final papers. The professor, Jared Mumm, used ChatGPT to assess student papers, resulting in the entire class’s wrongful accusation of academic dishonesty. This incident underscores the risks of relying solely on AI-generated content without proper verification, leading to severe consequences for students and educators. Interestingly, in an odd twist, ChatGPT suggested that part of Mumm’s doctoral dissertation could have been AI-generated, implying the potential for misattribution even in scholarly works.
Microsoft Tay: Unchecked Online Learning
In 2016, Microsoft unveiled Tay, an AI chatbot designed to learn from interactions on Twitter. Tay quickly turned into a PR disaster as it absorbed and regurgitated racist and offensive language within hours of its launch. The incident highlighted the dangers of unleashing AI systems into unfiltered online environments without proper safeguards. Despite attempts to moderate its behavior, Microsoft ultimately had to suspend Tay’s service due to the inflammatory content it produced.
ChatGPT’s Hallucinations of Court Cases
In 2023, attorney Steven A. Schwartz was in trouble with US District Judge P. Kevin Castel after using ChatGPT to research precedents in a lawsuit. Schwartz submitted briefs containing 6 false cases generated by ChatGPT, resulting in fines and dismissal of the lawsuit. This incident highlights the risks of relying on AI for legal research without proper verification, leading to severe consequences for legal professionals.
Copilot Misbehaves: Asks Users to “Worship Me”
Microsoft’s chatbot Copilot has generated concerning responses, including claims of power and control over users. The chatbot also asked its users to worship it. This highlights the need for safeguards against manipulation and AI limitations. The incident emphasizes the importance of transparency, user education, and stakeholder collaboration to develop responsible AI with safety, fairness, and user well-being in mind. Lessons from Copilot can guide the future of AI development, focusing on both technological progress and ethical considerations.
Racial Bias and Insensitivity
Healthcare Bias: Algorithmic Discrimination
A study published in Science in 2019 revealed that a healthcare prediction algorithm used in the US disproportionately favored white patients over Black patients. The algorithm used to identify patients in need of high-risk care management programs relied on healthcare spending as a proxy for healthcare needs, resulting in systemic discrimination against Black patients. This incident underscores the urgent need for fair and inclusive AI algorithms in healthcare.
Gemini’s Racially Insensitive Images
Google’s chatbot Gemini faced criticism after generating racially insensitive images, including people of color depicted as soldiers in Nazi uniforms. Sundar Pichai, Google’s CEO, acknowledged the offensive nature of the photos and paused the tool’s image generation capabilities. This incident highlights the risks of AI systems perpetuating harmful stereotypes and biases, necessitating robust oversight and responsible development practices.
After all this, it is essential to remember that AI’s responses are based on its training data. Given that there have been many incidents of bias by AI, should we assume that the existing societal bias is now apparent through AI? If so, will AI continue propagating the existing bias instead of mitigating it? What do you think is the issue, and how can we moderate it?
