Cal Fire Chatbot Fails to Answer Basic Questions

An AI chatbot designed to provide information about wildfires can’t answer questions about evacuation orders, among other problems.

2 minute read

July 10, 2025, 11:00 AM PDT

By Diana Ionescu @aworkoffiction


Red and white "Wildfire Evacuation Route" sign on signpost.

MichaelVi / Adobe Stock

California’s state fire protection agency has launched an AI chatbot in an effort to improve efficiency — and the results are troubling, reports Malena Carollo in The Markup. “The chatbot uses the Cal Fire website and the agency’s ReadyForWildfire.org to generate answers. It can tell users about topics such as active wildfires, the agency, fire preparedness tips and Cal Fire’s programs.”

CalMatters found that the chatbot, which is meant to offer “critical fire prevention resources and near-real-time emergency information,” “fails to accurately describe the containment of a given wildfire, doesn’t reliably provide information such as a list for evacuation supplies and can’t tell users about evacuation orders.”

The chatbot had trouble answering questions about fire preparedness depending on how questions were phrased. “And while the chatbot didn’t generate incorrect answers in any of the queries CalMatters made, it doesn’t always pull the most up-to-date information” — for example, showing fire containment information that was six days out of date. The chatbot is also unable to provide information about evacuation orders.

Experts say testing of the bot should have happened long before it was launched for public use. “Agencies interested in using chatbots should identify the questions the public is likely to ask the AI tool ahead of time, ensure those are representative of the expected population the agency serves and refine the chatbot by having members of the public pilot the system to ensure it provides the kind of information they seek.”

Wednesday, July 9, 2025 in The Markup

portrait of professional woman

I love the variety of courses, many practical, and all richly illustrated. They have inspired many ideas that I've applied in practice, and in my own teaching. Mary G., Urban Planner

I love the variety of courses, many practical, and all richly illustrated. They have inspired many ideas that I've applied in practice, and in my own teaching.

Mary G., Urban Planner

Cover CM Credits, Earn Certificates, Push Your Career Forward

Logo for Planetizen Federal Action Tracker with black and white image of U.S. Capitol with water ripple overlay.

Planetizen Federal Action Tracker

A weekly monitor of how Trump’s orders and actions are impacting planners and planning in America.

July 16, 2025 - Diana Ionescu

Green vintage Chicago streetcar from the 1940s parked at the Illinois Railroad Museum in 1988.

Chicago’s Ghost Rails

Just beneath the surface of the modern city lie the remnants of its expansive early 20th-century streetcar system.

July 13, 2025 - WTTV

Blue and silver Amtrak train with vibrant green and yellow foliage in background.

Amtrak Cutting Jobs, Funding to High-Speed Rail

The agency plans to cut 10 percent of its workforce and has confirmed it will not fund new high-speed rail projects.

July 14, 2025 - Smart Cities Dive

Worker in yellow safety vest and hard hat looks up at servers in data center.

Ohio Forces Data Centers to Prepay for Power

Utilities are calling on states to hold data center operators responsible for new energy demands to prevent leaving consumers on the hook for their bills.

July 18 - Inside Climate News

Former MARTA CEO Collie Greenwood standing in front of MARTA HQ with blurred MARTA sign visible in background.

MARTA CEO Steps Down Amid Citizenship Concerns

MARTA’s board announced Thursday that its chief, who is from Canada, is resigning due to questions about his immigration status.

July 18 - WABE

Rendering of proposed protected bikeway in Santa Clara, California.

Silicon Valley ‘Bike Superhighway’ Awarded $14M State Grant

A Caltrans grant brings the 10-mile Central Bikeway project connecting Santa Clara and East San Jose closer to fruition.

July 17 - San José Spotlight