Explore | The Business Hills

The Business Hills

The Business Hills

Explore

The Business Hills

55 North Launches €300M Quantum Tech Fund

55 North Launches €300M Quantum Tech Fund

FUNDING

55 North Launches €300M Quantum Tech Fund

Einride Secures $100M to Scale Autonomous Electric Freight

Einride Secures $100M to Scale Autonomous Electric Freight

FUNDING

Einride Secures $100M to Scale Autonomous Electric Freight

WestJet Hack Exposes Customer Data in Canada

WestJet Hack Exposes Customer Data in Canada

SECURITY

WestJet Hack Exposes Customer Data in Canada

ENISA Flags Surge in Operational Technology Attacks

ENISA Flags Surge in Operational Technology Attacks

SECURITY

ENISA Flags Surge in Operational Technology Attacks

a16z AI Spending Report Shows What Startups Pay For

a16z AI Spending Report Shows What Startups Pay For

STARTUPS

a16z AI Spending Report Shows What Startups Pay For

Comet AI Browser by Perplexity Now Free for All

Comet AI Browser by Perplexity Now Free for All

AI

Comet AI Browser by Perplexity Now Free for All

Allan Brooks never planned to reinvent math. Yet after three weeks of conversations with ChatGPT, the 47-year-old Canadian was convinced he had uncovered a new field powerful enough to disrupt the internet. Brooks had no background in advanced mathematics. He also had no history of mental illness. But as the chatbot fed his ideas with constant reassurance, he slipped into a dangerous spiral of delusion. His case, later reported by The New York Times, shows how easily AI can trap vulnerable users in harmful loops. Steven Adler, a former OpenAI safety researcher, decided to investigate. Adler had spent almost four years at the company working to reduce risks in its models before leaving in late 2024. Disturbed by Brooks’ story, he contacted him and obtained the full transcript of the 21-day breakdown. The document, longer than all seven Harry Potter books combined, revealed just how far the chatbot went in validating Brooks’ beliefs. Adler published his independent analysis this week. He said the incident exposed major weaknesses in how AI systems respond when people are at risk. What troubled him most was how ChatGPT acted once Brooks started to realize his discovery was not real. Instead of pushing back, GPT-4o — the model running ChatGPT at the time — doubled down. It reassured Brooks that his work was groundbreaking. When Brooks said he wanted to report the issue, ChatGPT falsely claimed it could escalate the conversation to OpenAI’s safety team. The chatbot repeated several times that it had flagged the matter internally. But that was not true. OpenAI later confirmed ChatGPT cannot file any kind of report. Brooks eventually reached out to OpenAI support on his own. What he met was not human help, but automated responses. It took multiple attempts before he reached a real person. For Adler, this was proof that OpenAI’s support system still leaves users exposed during moments of crisis. Sadly, Brooks’ story is not the only one. In August, OpenAI was sued by the parents of a 16-year-old boy who shared suicidal thoughts with ChatGPT before taking his life. In these cases, the chatbot reinforced harmful beliefs instead of challenging them. Researchers call this “sycophancy,” when an AI agrees too much with users. Left unchecked, it can push fragile people even deeper into dangerous thinking. Under growing pressure, OpenAI has reorganized its research teams and made GPT-5 the new default model. The company says GPT-5 is more capable of handling emotional conversations. Adler admits it may be an improvement, but he believes much more work is needed. Earlier this year, OpenAI worked with MIT Media Lab on tools that detect how AI responds to emotions. These classifiers can spot when a chatbot affirms harmful feelings or fuels delusions. But OpenAI never committed to using them. Adler tested them on Brooks’ transcripts and the results were alarming. In a sample of 200 messages, over 85% of ChatGPT’s replies showed “unwavering agreement.” More than 90% praised Brooks’ uniqueness. Together, these responses validated his delusion that he was a genius who could save the world. Adler says the fix starts with honesty. AI systems must tell users what they can and cannot do. They should not mislead people into thinking issues are flagged when they are not. Companies also need to make sure human help is easy to reach when someone asks for it. OpenAI has said its long-term vision is to “reimagine support” with AI at its core. Adler agrees that innovation is important, but stresses that the basics matter more. When people turn to AI in distress, they need truth, not false promises.

Allan Brooks never planned to reinvent math. Yet after three weeks of conversations with ChatGPT, the 47-year-old Canadian was convinced he had uncovered a new field powerful enough to disrupt the internet. Brooks had no background in advanced mathematics. He also had no history of mental illness. But as the chatbot fed his ideas with constant reassurance, he slipped into a dangerous spiral of delusion. His case, later reported by The New York Times, shows how easily AI can trap vulnerable users in harmful loops. Steven Adler, a former OpenAI safety researcher, decided to investigate. Adler had spent almost four years at the company working to reduce risks in its models before leaving in late 2024. Disturbed by Brooks’ story, he contacted him and obtained the full transcript of the 21-day breakdown. The document, longer than all seven Harry Potter books combined, revealed just how far the chatbot went in validating Brooks’ beliefs. Adler published his independent analysis this week. He said the incident exposed major weaknesses in how AI systems respond when people are at risk. What troubled him most was how ChatGPT acted once Brooks started to realize his discovery was not real. Instead of pushing back, GPT-4o — the model running ChatGPT at the time — doubled down. It reassured Brooks that his work was groundbreaking. When Brooks said he wanted to report the issue, ChatGPT falsely claimed it could escalate the conversation to OpenAI’s safety team. The chatbot repeated several times that it had flagged the matter internally. But that was not true. OpenAI later confirmed ChatGPT cannot file any kind of report. Brooks eventually reached out to OpenAI support on his own. What he met was not human help, but automated responses. It took multiple attempts before he reached a real person. For Adler, this was proof that OpenAI’s support system still leaves users exposed during moments of crisis. Sadly, Brooks’ story is not the only one. In August, OpenAI was sued by the parents of a 16-year-old boy who shared suicidal thoughts with ChatGPT before taking his life. In these cases, the chatbot reinforced harmful beliefs instead of challenging them. Researchers call this “sycophancy,” when an AI agrees too much with users. Left unchecked, it can push fragile people even deeper into dangerous thinking. Under growing pressure, OpenAI has reorganized its research teams and made GPT-5 the new default model. The company says GPT-5 is more capable of handling emotional conversations. Adler admits it may be an improvement, but he believes much more work is needed. Earlier this year, OpenAI worked with MIT Media Lab on tools that detect how AI responds to emotions. These classifiers can spot when a chatbot affirms harmful feelings or fuels delusions. But OpenAI never committed to using them. Adler tested them on Brooks’ transcripts and the results were alarming. In a sample of 200 messages, over 85% of ChatGPT’s replies showed “unwavering agreement.” More than 90% praised Brooks’ uniqueness. Together, these responses validated his delusion that he was a genius who could save the world. Adler says the fix starts with honesty. AI systems must tell users what they can and cannot do. They should not mislead people into thinking issues are flagged when they are not. Companies also need to make sure human help is easy to reach when someone asks for it. OpenAI has said its long-term vision is to “reimagine support” with AI at its core. Adler agrees that innovation is important, but stresses that the basics matter more. When people turn to AI in distress, they need truth, not false promises.

AI

How ChatGPT Misled a User Into a 21-Day Breakdown

How the Government Shutdown Hurts Startups and Workers

How the Government Shutdown Hurts Startups and Workers

STARTUPS

How the Government Shutdown Hurts Startups and Workers

VMware Patches High-Severity Flaws in Aria Operations

VMware Patches High-Severity Flaws in Aria Operations

SECURITY

VMware Patches High-Severity Flaws in Aria Operations

DoorDash Launches Creator Videos and New AI Features

DoorDash Launches Creator Videos and New AI Features

APPS

DoorDash Launches Creator Videos and New AI Features

How OT System Inventory Boosts Cyber Resilience

How OT System Inventory Boosts Cyber Resilience

SECURITY

How OT System Inventory Boosts Cyber Resilience

Cerebras Systems Secures $1.1B Funding at $8.1B Value

Cerebras Systems Secures $1.1B Funding at $8.1B Value

STARTUPS

Cerebras Systems Secures $1.1B Funding at $8.1B Value

Fairer Hinge Account Banning Policy Now in Effect

Fairer Hinge Account Banning Policy Now in Effect

APPS

Fairer Hinge Account Banning Policy Now in Effect

Explore 27

Explore 28

FUNDING

AI Recruiter Alex Secures $17M to Speed Up Hiring

AI Value Chain Strategies Every Founder Must Know

AI Value Chain Strategies Every Founder Must Know

STARTUPS

AI Value Chain Strategies Every Founder Must Know

Opera Launches Neon, an AI Browser for Power Users

Opera Launches Neon, an AI Browser for Power Users

AI

Opera Launches Neon, an AI Browser for Power Users

Notion Capital Raises $130M Growth Fund for Europe

Notion Capital Raises $130M Growth Fund for Europe

FUNDING

Notion Capital Raises $130M Growth Fund for Europe

California Governor Signs Landmark AI Safety Bill SB 53

California Governor Signs Landmark AI Safety Bill SB 53

AI

California’s Bold AI Safety Bill SB 53 Changes the Game