ANALYST PANEL / FILE 02 — SIGHTINGS AND HOAXES

AI vaporware: The questions that break a demo

What you'll discover

"Is it doing the work, or just talking about it?" How to tell an AI that actually runs multi-step investigations from a chat window sitting on top of your data, and where a tool really falls on the four-level maturity spectrum.
"What did the AI look at to reach that decision?" Context engineering is the biggest differentiator in the research. You'll learn how to test it in a demo with one real alert.
"What is it being graded against?" Why vendors should be able to show test sets built from real analyst decisions, and what it tells you when they can't.

‍

About this panel

Every AI demo looks impressive until someone asks the right question. Chartis Research Director Sean O'Malley joins The AI Files to share the questions that separate AI that does the work from AI that only talks about it. The questions come from Chartis and Unit21's joint research on agentic AI maturity. For each one, you'll hear what a strong answer sounds like and what vaporware sounds like.

‍