The timely question that saved $2M

The $2M question nobody asked

Your CEO just announced in the all-hands: "We need an AI strategy by Q2." Your product team scrambles. Someone suggests, "What if AI could automatically detect and fix security rule problems?" Everyone nods. It sounds innovative. The board will love it. Engineering starts scoping.

Six months and $2 million later, you discover the AI confidently suggests solutions that are completely, spectacularly… wrong.

This almost happened to a cybersecurity company. They had the vision, the budget approval, and engineering ready to build. They were weeks from starting development.

Then they brought me in to run a Snowball AI Strategy Workshop.

Three hours into the first day, I asked one question that changed everything. By the end of 3 days, we'd completely redesigned their approach.

Not because the vision was wrong.

But because they were about to build Phase 2 before Phase 1.

Simply put, they didn’t have the data to make AI work.

Here's what happened, and why your team is probably making the same mistake right now.

The plan that sounded perfect

The security company had identified a real customer pain point. Security rules frequently misfire, creating alert fatigue and operational chaos.

Classic example: an "impossible travel" rule detects suspicious logins when someone appears in New York, then Tokyo an hour later. But when a customer gets a new VPN? The rule goes haywire—triggering hundreds of false alerts because VPN traffic makes locations jump constantly.

Their solution: Build an AI system that automatically detects misbehaving rules and recommends fixes. When the impossible travel rule starts firing incorrectly, AI analyzes the pattern and suggests a fix.

Beautiful. Customers would love it. Competitors weren't doing it. The board approved the budget.

What could go wrong?

The pattern I keep seeing

I've run this workshop with AI-savvy teams at Series B-D SaaS companies, Fortune 500 enterprises, and PE-backed software firms. The pattern is always the same: smart teams starting development on AI initiatives that sound great in a corporate deck, in reality, contain a ticking time bomb — a fatal flaw no one's identified.

At a payments company: They wanted AI to detect fraudulent transactions. Made perfect sense—fraud is costing them millions. Except they didn't have cleanly labeled examples of fraud patterns → detection → verification that the pattern was actually fraud. They had sporadic data coverage from one-off overseas transactions, and trained only on that, AI would have just blocked the transactions from their best and most high-volume customers. (Outcome: Together, we pivoted to a new data collection strategy to get the missing data to make this their most successful AI project to date.)

At an Industrial IoT company: They wanted AI to optimize an important industrial process, but were missing key sensor data only available to a human operator. They assumed they had real-time process data. They didn't. AI would have optimized for accuracy, not safety, leaving money on the table and risking catastrophic cleanup. To add to their troubles, all of the installations were highly customized, making scaling impossible. (Outcome: Together, we converted the project from “human operator replacement” into a “blind spot indicator,” offering, which reduced risk and provided instant access to training data.)

At a Smart Irrigation company: The team wanted AI to precisely administer the minium amount of water the plants needed to thirve and maximize crop yeild, by measuring evapotraspiration. However, they lacked sophisticated soil models to make this possible out of the box. (Outcome: Together, we pivoted to sell the message of “complete visibility and control” and deployed a “human in the loop” solution in Phase 1, while also creating a sophisticated patent-pending soil modeling UI to start collecting the missing data the company needed for Phase 2.)

These teams weren't stupid. Their ideas weren't bad. But they all shared one critical flaw: they were building sophisticated AI systems without the foundational data those AI systems needed to actually work.

But let’s get back to our story — here’s what happened in that 3-day workshop.

Hour 3: The question that changes everything

We're in the workshop. I've got product leaders, engineers, and customer success on the whiteboard. We're mapping their systems—what I call the digital twin exercise in my book UX for AI: A Framework for Designing AI-Driven Products. In this exercise, we map inputs, outputs, data flows, what gets handed off where, and so on — in other words, we create a complete blueprint of the desired AI-driven system.

Looking at the diagram, I ask: "Do you have the data showing customers took rules with specific activation patterns, made specific changes, and then those changes fixed the problems?"

Silence.

The head of engineering speaks first: "No. We don't have that data."

"We have rule configurations and activation logs," the product lead adds, "but we don't capture what changes customers made or whether those changes actually worked."

I let that sink in for a moment.

"Then you cannot build this system. Not yet. The AI will hallucinate every single time."

Why AI is a bullsh*t generator (and when that matters)

Here's the thing people don't understand about large language models: they're incredibly sophisticated pattern-matching engines. Give them the right patterns to match against, and they're remarkable.

But—as I describe in my book—they're also fundamentally bullsh*t generators. (One of my 12 co-authors, Chris Nossel, gets the credit for that quote.)

When LLMs don't have the pattern in their training data, they don't say "I don't know." They confidently fabricate an answer that sounds plausible.

They bullsh*t.

For creative writing or brainstorming? Fine. The impact of bullsh*t is low — it can be attributed to a “creative licence”, in many cases, one of the most appealing characteristics of the LLMs.

For security rule recommendations that customers will implement in production environments? This kind of “Creative Licence” bullsh*t is catastrophic.

Without real examples of:

Rule problems → human-identified fixes → verified outcomes

The LLM has nothing to match against. It will always give you an answer. That answer will sound technical and authoritative. It will also be mostly just fabricated nonsense.

The company would have spent $2M building a confident bullsh*t generator that destroys customer trust.

The room got very quiet.

"So what do we build then? This is a real problem."

That's when the VP Product spoke up: "Okay, so we can't build what we planned. But this is a genuine customer pain point. What do we actually build?"

This is the moment where most teams give up or waste months evaluating alternatives.

Instead, we spent the rest of day one storyboarding what could actually work. Walking through the user journey frame by frame. Where does the customer have information that the AI doesn't? Where does AI need human context?

The breakthrough: The customer has the context of the problem that the AI can't see.

When that impossible travel rule starts firing constantly, the customer knows:

• "We just implemented a new VPN last Tuesday."

• "The rule started going crazy right after."

• "We think the VPN is causing it."

The AI only sees: "Rule activated 473 times in 24 hours."

It's missing the context. That's the gap.

The solution: Give AI what it's missing

Phase 1: Build the human-in-the-loop system first

Customer says: "Look at this specific rule. We made this change (new VPN), and now every time this rule activates incorrectly, here's the pattern (constant false alerts from these IPs)."

Now the AI has the critical piece it was missing: context.

Now AI can do what it's actually good at: agentic research.

• Search outside the system for similar documented problems

• Find VPN-related security rule exemption patterns

• Cross-reference vendor documentation

• Analyze configuration best practices

• Synthesize findings into specific recommendations tied to best practices and customer data

AI suggests: "Add exception: when source_IP in [VPN_IP_range], apply relaxed_travel_rule."

Now, the suggested fixes are no longer “creative license” bullsh*t.

With the missing context the customer provides, the suggestions are specific, research-backed, best-practice recommendations to solve the exact problem the customer has, with virtually no effort on their part.

Rule configuration + problem context + AI recommendation + customer implementation + thumbs up/down feedback = exactly the pattern you need for Phase 2.

Phase 2: Six months later, build the autonomous system

After deploying Phase 1 and collecting real-world problem-solving patterns from actual customers, you now have the training data that didn't exist on day one.

Now you can build the original vision: AI that autonomously detects rule problems and suggests fixes. This version will now work because it's trained on real data patterns, not bullsh*t.

Without Phase 1, Phase 2 will always fail. With Phase 1, Phase 2 becomes inevitable.

By the end of day two, the team had a completely different roadmap. Not because I talked them out of their vision—but because we found the path that would actually get them there.

Why smart teams keep getting this backwards

If this sequencing seems obvious now, why do so many teams start with Phase 2?

Hype pressure. "Our competitors announced AI features. What's ours?"

Hammer-first thinking. "We have access to GPT-4. What problems can we solve with it?"

Optimism bias. "We'll figure out the data challenges during development."

No time pressure. Or more accurately, the wrong time pressure. Teams feel pressure to START building, but no pressure to validate WHAT they're building.

The Arctic Wolf CPO, Dan Schiappa, calls this "tabloid headline-type AI adoption, as opposed to building it to solve the right problems"—building AI because it's trendy, not because you've validated it will work.

That's what the workshop does. It creates a structured environment where uncomfortable questions surface early, when changing direction is cheap instead of catastrophic, saving you $2 million and 9 months of building the wrong thing.

The urgency you're ignoring

Here's what's happening right now: Q1 planning cycles are locking budgets in the next 6-8 weeks.

If your AI initiatives make it into the committed roadmap without validation, you'll spend 2025 building them—whether they make sense or not.

As an in-house AI Product Strategy Leader, I’ve seen this pattern repeat many times. Once engineering headcount is allocated, once the OKRs are set, once you've presented the roadmap to the board, pivoting becomes politically fraught. You'll build what you committed to building, even if you discover halfway through that it can't work.

The cost of getting it wrong

I've seen this pattern repeat across dozens of companies:

Months 1-3: Development. Demos look promising. Board updates show progress.

Months 3-7: Reality surfaces. Data quality issues. Edge cases multiply. "We need more time."

Months 7-8: Launch to beta. Recommendations aren't reliable. Customers stop using it.

Month 9: Retrospective. "We didn't have the training data we needed."

The truth: That problem existed in Month 1. But no one asked the question that would have exposed it.

Total cost: $2M in engineering time, 9 months of calendar time. But most importantly, the opportunity cost of what else the team could have built, and board skepticism about your next AI proposal.

What the workshop actually delivered

Investment: $65K | 2 weeks pre-work (I handle 90%) | 3-day intensive

Before we even entered the room, I spent two weeks doing the heavy lifting: interviewing stakeholders, reviewing their proposed AI initiatives, mapping their data infrastructure, and identifying likely gaps. The team contributed maybe 3-4 hours total in pre-work interviews.

By the end of our brief engagement, the team had:

• A realistic roadmap: Phase 1 (8 weeks to build) → collect data (6 months) → Phase 2 (12 weeks)

• Clear success metrics that would satisfy both engineers and the CFO.

• Identification of the "missing training data" gap before spending $2M discovering it.

• Validation of the Phase 1 system that delivers real customer value and collects the data Phase 2 needs.

• Complete stakeholder alignment on why Phase 1 unlocks Phase 2.

What changed: Not the vision. The sequence.

What was prevented: $2M spent building an AI bullsh*t generator that customers would hate. Reputation damage and 9-month opportunity cost.

The three exercises that expose the truth

The workshop uses three key problem framing exercises I developed and documented fully in my book:

Digital Twin: Map your actual data infrastructure against what the AI requires. The gaps become immediately visible. "Do you have this data?" "No." "Then you can't build this yet."

Storyboard: Walk through the user journey frame by frame. Where does the customer have information that the AI doesn't? Where does AI need human input? Where could it fail catastrophically?

Value Matrix: Force-rank every AI prediction by business impact vs. cost instead of data science metrics like accuracy (accounting for real-world constraints, data gaps, and infrastructure reality). Most teams discover their exciting ideas are riskier and have lower ROI than the simpler but more valuable alternatives they did not consider.

They are based on 34 AI-driven projects I delivered since 2009, 13 years before ChatGPT made AI irresistible. These exercises are designed to do one thing really well: have a quality discussion surfacing uncomfortable truths before you've spent millions learning them.

The security company learned in 3 hours what would have taken them 9 months and $2M to discover on their own.

Which option is right for you?

Look, I partnered with 12 of the best and brightest in the business to write a book that gives you the complete framework. I am very proud of the book — it’s my first #1 Amazon New Release. If you have time, a skilled facilitator on your team, and no political complexity, start with the book.

The workshop is for when you DON'T have those things—when you're 6 weeks from a board presentation, when your engineers are optimists who'll say "we'll figure out the data," when you need someone with 34-AI-project experience to ask "Do we actually have what this needs?" and get down into the weeds to get the honest answers.

The workshop's value isn't the framework—that's in the book. It's having someone who's been in the trenches and seen these failure patterns 34+ times to facilitate the hard conversations your team won't have on their own.

Here's how to decide:

📖 Start with the book if:

• AI budget < $500K

• You have 8+ weeks before decisions lock

• You have a strong facilitator on your team

• You want to test the framework first

The book gives you all three exercises, templates, and implementation playbooks. Best for teams with experienced facilitators who can drive tough conversations.

💡Book a diagnostic call if:

• AI budget > $1M

• Board presentation in the next 4-8 weeks

• Engineering starts building in the next 30 days

• You need external validation of your approach

This is not a pressure sales call with a junior sales associate — it is a discovery call with an expert who literally wrote the book. We'll discuss your specific AI initiatives, and you'll leave with 2-3 actionable insights, whether or not we work together.

What you should do right now

Regardless of whether or now we work together, ask yourself:

Do you have normalized training data to “do the thing” that you want your AI system to do? Not "could we theoretically collect it"—do you have it now?

Have you storyboarded the user journeys? Do you know where AI needs human context?

Have you force-ranked your AI decision points by their true cost: risk and ROI calculations? Accounting for real-world constraints and likely outcomes, including data gaps, not just the cost of engineering effort?

Or are you assuming those questions will get answered during development?

These uncomfortable questions will get asked eventually.

Either during a 3-day workshop, when changing direction is cheap.

Or 9 months into development, after you've shipped a system that doesn't work.

The clock is ticking. Q1 budgets lock in 6-8 weeks. After that, you're building whatever made it into the roadmap—whether it makes sense or not.