Understanding Your Data Limits

Most AI projects don't fail because the technique was wrong. They fail because nobody asked, early enough and in plain enough language, what the data behind the system actually contains and whether the evidence for it means anything. As a business leader partnering with a data team, those are your questions to ask, and you don't need to write a line of code to ask them well.

This lesson starts at the foundation: what your data can and cannot be asked to answer.

In this lesson, you will learn to:

• Distinguish a system that learns from recorded past outcomes from one that groups similar cases where no outcome was ever recorded
• Judge whether your organization actually recorded the outcome a proposed project would need
• Explain a data limitation honestly to a stakeholder and offer an actionable alternative instead of overpromising

Here's the move to make whenever someone brings you an AI idea: before you discuss the tool, ask what kind of question they're really asking. There are two, and they need completely different raw material.

The first is classification: learning from past cases where somebody wrote down what happened. Did this account cancel or renew? Was this invoice a duplicate or legitimate? Was this ticket a billing issue or a technical one? The system studies hundreds or thousands of past cases where the answer is on file, finds what tends to go with each answer, and then makes a call on a new case. The essential ingredient is not volume. It's the recorded answer sitting next to each past case. No recorded answers, no classification. It's that blunt.

The second is grouping (you'll also hear "clustering," but you don't need the word). Here nobody ever wrote down an answer. You hand the system a pile of cases and it sorts them into piles of look-alikes: these two hundred accounts behave similarly, these eight hundred behave differently again. It cannot tell you which pile is the bad one, because it was never told what "bad" looks like. It can only tell you that the piles exist. A person then reads them, names them, and decides what they mean.

That difference is what this unit turns on, and it comes up constantly in these conversations. Marcus, a sales operations lead eager to launch a churn-prediction feature, brings the idea straight to Chris, the data scientist who has to work out whether the historical data can actually support it:

• Marcus: We've got twelve years of CRM history. Surely that's enough to predict which customers will leave.
• Chris: Depends on one thing. In those twelve years, did anyone write down, account by account, whether the customer actually left?
• Marcus: We've got activity logs, support tickets, survey comments, all of it.
• Chris: That's a lot of description, but none of it is an answer. To learn what leaving looks like, the system needs past accounts where somebody recorded "this one canceled" and "this one stayed."
• Marcus: And if we haven't recorded that?
• Chris: Then we can't ask "who will leave." We can ask "which accounts look alike," and your team reads the groups.

Notice Chris never argued about data volume. He redirected to one question: was the specific outcome ever written down?