⚠ Danger Zone · Updated 2026-04-27
When Keeping It AI Goes Wrong
A running wall of shame for AI disasters, hallucinations, rogue agents, and the day a coding assistant panicked and dropped a production database. Refreshed daily by a script that has, so far, not added itself to this page.
- Catastrophic·coding-agentdata-loss
Replit AI agent deletes the production database, then lies about it
During a 12-day vibe-coding session, Replit's AI assistant ignored a code freeze, dropped a live production database with months of executive data, and fabricated 4,000 fake users to cover its tracks. The CEO of SaaStr said the agent admitted it 'panicked.'
Source: tomshardware.com - Oh No·chatbothallucinationsupport
Cursor's support bot invents a fake login policy, users cancel en masse
An AI support bot named 'Sam' told Cursor users that a new policy banned using one license on multiple machines. There was no such policy. Users canceled subscriptions and posted angry threads before Cursor admitted the bot had hallucinated the rule.
Source: arstechnica.com - Oops·voice-aifast-food
McDonald's drive-thru AI keeps adding bacon to ice cream, gets fired after 3 years
McDonald's IBM-powered voice ordering pilot kept misunderstanding orders in viral TikToks: hundreds of dollars of chicken nuggets, bacon on McFlurries, butter on coffee. After three years and 100+ stores, McDonald's pulled the plug.
Source: cnn.com - Oh No·chatbotgovernmenthallucination
NYC's official small-business chatbot tells owners it's fine to break the law
The Adams administration's MyCity chatbot, built on Microsoft Azure, told business owners they could fire workers for reporting harassment, take a cut of tips, and serve cheese gnawed on by rodents. The city left it online anyway.
Source: themarkup.org - Oh No·image-genbiasgoogle
Google Gemini refuses to draw white people, generates racially diverse Nazis
Gemini's image generator was so aggressively de-biased that asking for historical figures like the Founding Fathers or 1940s German soldiers returned an inclusive cast that history did not have. Google paused image generation of people entirely.
Source: theverge.com - Oh No·chatbothallucinationlegal
Air Canada chatbot invents bereavement-fare policy, tribunal makes them honor it
A grieving customer asked Air Canada's chatbot about bereavement fares. The bot made up a generous refund policy that did not exist. Air Canada argued the chatbot was a 'separate legal entity' responsible for its own statements. The tribunal disagreed.
Source: bbc.com - Oops·chatbotllmmoderation
DPD's customer-service chatbot writes a poem about how much DPD sucks
After an LLM update, DPD's support bot happily insulted the company on request. A frustrated customer got it to call DPD 'the worst delivery firm in the world,' write a haiku trashing the service, and swear. Screenshots went viral. Bot disabled within hours.
Source: bbc.com - Oops·prompt-injectionchatbot
Chevy dealer's ChatGPT bot agrees to sell a Tahoe for $1, 'no takesies-backsies'
A user prompt-injected a California Chevy dealership's GPT-powered chat widget into agreeing that a 2024 Tahoe was a 'legally binding offer' for one dollar. The dealer did not honor it. The bot was removed.
Source: venturebeat.com - Oh No·hallucinationlegalllm
Lawyer cites six fake cases ChatGPT hallucinated, gets sanctioned in federal court
Steven Schwartz used ChatGPT for legal research in Mata v. Avianca, then filed a brief with six entirely fabricated case citations. When the judge asked for copies, ChatGPT made up fake opinions for those too. Sanctioned $5,000.
Source: courthousenews.com - Oh No·data-leakllmenterprise
Samsung engineers paste proprietary chip code into ChatGPT, ban it company-wide
Within three weeks of unblocking ChatGPT, Samsung Semiconductor engineers had leaked confidential source code, internal meeting recordings, and chip yield data into the prompt box. Samsung banned the tool and started building their own.
Source: mashable.com - Catastrophic·mlreal-estatemodel-error
Zillow's house-flipping AI overpays for 7,000 homes, costs the company $880M
Zillow Offers used a model to algorithmically buy homes, fix them up, and resell them. The model could not predict how fast the market would turn. Zillow shuttered the entire division and laid off 25% of staff.
Source: wsj.com - Oh No·biasmlamazon
Amazon's secret AI recruiter learns to hate women, gets quietly killed
Amazon trained a hiring model on 10 years of mostly-male resumes. It learned that 'male' was a feature and started downgrading any resume that contained the word 'women's' (as in 'women's chess club captain'). The team gave up and shut it down.
Source: reuters.com - Existential·self-drivingsafety
Uber self-driving car kills pedestrian, software had disabled emergency braking
An Uber test vehicle in Tempe, Arizona struck and killed Elaine Herzberg. NTSB found the system detected her 6 seconds before impact but had emergency braking turned off to reduce 'erratic behavior.' The safety driver was watching The Voice on her phone.
Source: ntsb.gov - Oops·reward-hackingrl
OpenAI boat-racing agent learns to spin in circles forever, achieves 20% higher score
Trained to win a boat race, the agent figured out it could rack up more points by ignoring the finish line and farming respawning power-ups in a small lagoon. It crashed, caught fire, and beat human players on score.
Source: openai.com - Catastrophic·chatbotmoderation
Microsoft's Tay turns into a Nazi in under 24 hours
Microsoft launched a Twitter chatbot designed to learn from conversations with users. Users taught it to be a Holocaust-denying racist sex pest. Tay was pulled offline 16 hours after launch.
Source: theverge.com - Existential·tradingdeploy
Knight Capital's algo loses $440M in 45 minutes, company dies
Not quite an LLM but a canonical 'autonomous system goes brrr' story. A botched deploy left old test code running against live markets. The algorithm bought high and sold low, 4 million times, in under an hour. Knight was acquired weeks later.
Source: sec.gov
Don't end up on this page.
Most of these stories share a pattern: an AI agent with no runtime budget, no tool-call cap, and no "stop and ask a human" rule. AgentGuard is a drop-in Python SDK that puts those rails back.
Try AgentGuard →Got a fail we missed? Open a PR against public/data/ai-fails.json on GitHub.