Don't Accept Automation as the Goal
Most AI consultants will automate your workflow and hand it back to you.
That's not enough.
The bar shouldn't be "it works as well as before, just faster." The bar should be "outcomes are measurably improving over time." Those are very different things, and the gap between them is where most applied AI engagements fall short.
Automation vs. Improvement
Automation is a process change. You replace a manual step with a system. The system does it faster or more consistently. That has value.
But what you actually want is a system that gets smarter. One that actively drives better results than what you were achieving before (including better than your best human judgment on a good day) and keeps improving from there.
The first type of engagement ends when the system is delivered. The second type has no end. It compounds.
What This Looks Like in Practice
Say you hire someone to build an AI system that recommends titles and thumbnails for your YouTube channel.
Done poorly: the AI suggests titles, you post them, nobody tracks whether they actually perform better than what you were doing before. Six months later, you can't tell if the engagement went up because of the AI or in spite of it.
Done well: every recommendation is logged. A/B tests are run when possible (same content, different title formats, different thumbnail styles) and the results are recorded. Not summaries. The actual raw data: views, click-through rate, watch time, tied to the specific variant that was used. When the practitioner refines the system, they pull from real outcomes, not secondhand analysis.
Six months in, you have a clear chart. Here's where you started. Here's where you are now. Here's what drove the change.
The difference between these two outcomes isn't the technology. It's whether the person you hired thinks like a scientist or a plumber.
What to Look For When Hiring
A practitioner with an experimental mindset will show certain signals early. Look for these in scoping conversations:
- They ask how you currently measure the thing they're about to automate
- They want to establish a baseline before building anything
- They bring up A/B testing or iterative refinement before you do
- They ask about data retention: "can we store the raw results of each run?"
- They have a clear picture of what happens after the initial delivery
These aren't advanced technical concerns. They're signs that the practitioner has thought past the delivery milestone and is thinking about whether your outcomes actually improve.
A practitioner who can't answer "how will we know in 90 days if this is working better?" is not the practitioner you want for work that matters.
Questions to Ask in Scoping
Use these in your first conversation with any applied AI practitioner:
- "How will we know if this is actually working better?" Push for specifics. What metric, measured how, over what timeframe?
- "What does the baseline look like?" They need to know your starting point to measure improvement.
- "What data will we retain to refine the system over time?" If they haven't thought about this, the system will be static.
- "Have you done work like this before? What did the improvement trajectory look like?" Past work is the best predictor.
- "What happens after the initial delivery?" Is there a plan for iteration, or does the engagement end when the system is handed over?
What This Implies for How You Structure the Engagement
The best applied AI engagements are ongoing, not project-based.
Initial delivery gets you to a baseline. The system is running. You have a measurement framework in place. That's month one or two.
The real value comes after. The practitioner runs experiments. Outcomes improve. They show you the data. You decide what to refine next.
This is a fundamentally different engagement structure from "here's your deliverable, good luck." Budget and timeline accordingly. If a practitioner is only quoting you for the build, ask them what the ongoing optimization looks like.
Build Institutional Knowledge
The companies that win with AI aren't the ones who deploy the most models. They're the ones who learn the fastest.
After every pilot, every experiment, every failed attempt: document what worked, what didn't, and what you'd do differently. Make that knowledge accessible to the rest of your organization. This is how one successful pilot becomes a company-wide capability instead of a one-off project.
The practitioner engagement ends. The institutional knowledge compounds.
Before You Engage Anyone
The three-stage path exists to protect you. The Quick Check confirms you're ready. The Situation Map ensures your workflows, data, and team are mapped honestly. The Pilot Scope asks you to define a concrete first experiment with clear success metrics before you hire anyone.
That's not administrative overhead. It's the prerequisite to knowing whether anything actually improved. Without a clear metric defined at the start, you have no way to evaluate whether the engagement delivered real value, and neither does the practitioner.
Work through all three stages first. It will also tell you a lot about which practitioners take it seriously and which ones just want to start building.