Quick summary
The AI consulting market is full of people who can demo well but struggle to deliver in production. This post gives you a practical checklist to separate the two: what to look for in their background, what questions to ask, how to read their proposals, and the specific things that separate a builder from a talker.
A lot of people entered the AI consulting world in 2023 and 2024. Some of them are very good. A lot of them are not. And in a sales call, it's genuinely hard to tell the difference because everyone has learned the same vocabulary.
I'm writing this from the consultant side — which means I might be biased. Take it with that grain of salt. But I've also talked to a lot of founders who got burned on AI projects, and the patterns are pretty consistent.
Start with what they've actually built
The most reliable signal is deployed work. Not POCs. Not demos. Not "we built a prototype for a client." Something that runs in production, that real users depend on, that the client didn't cancel.
Ask specifically: "Can you show me something in production or walk me through a project that shipped?" If they struggle to answer or pivot to capabilities and frameworks instead of specific outcomes, that tells you something.
The follow-up question is just as important: "What broke, and how did you fix it?" Good consultants have specific stories about things going wrong. People who haven't actually shipped things have vague stories about challenges and solutions that sound rehearsed.
Check when they started building with LLMs
Someone who started experimenting with GPT-3 in 2021 or 2022 has lived through multiple waves of hype and correction. They've seen what LLMs can and can't do. They've made mistakes when the technology was new and learned from them.
Someone who started after ChatGPT launched in late 2022 has a shorter track record. That doesn't automatically make them bad — people learn fast — but it does mean their experience is compressed and they haven't seen as many failure modes.
This isn't gatekeeping. It's just a signal about depth of experience. Ask when they started and what they were building.
Watch for the strategy consultants who don't code
There are two types of AI consultants. Some build systems. Some advise on AI strategy, help you define your approach, produce frameworks, and write recommendations.
Both are legitimate services. But they're not the same thing and you need to know which one you're talking to.
If you want something actually built, you need someone who writes code, designs systems, integrates APIs, debugs production issues, and does the handoff themselves. Ask directly: "Do you write the code or do you manage engineers who do?" If they manage others, ask how many engagements they've done that way and what happens when their engineering resource falls through.
Read the proposal carefully
A good proposal has specific deliverables with specific acceptance criteria. "Build an agentic pipeline that processes invoices from folder X, extracts fields Y and Z, and writes to database table W with 95% field accuracy" is a deliverable. "Implement AI-powered document processing solution" is not.
Vague deliverables protect the consultant, not you. If you can't tell from the proposal whether the work is done, you can't hold anyone to it.
Also check: does the proposal include a handoff plan? What happens at the end? What documentation do you get? Can your team maintain it without them? If the answer is unclear or they need ongoing retainers to keep the system running, that's worth asking about.
Questions worth asking in the first call
Here are the ones I'd ask if I were hiring someone:
- "What's a project you turned down, and why?" Good consultants have standards. If they've never said no to a project, they either haven't been doing this long or they take everything regardless of fit.
- "What's the last thing you built that didn't work as expected, and what happened?" Same as the "what broke" question. Listen for specificity.
- "How do you handle it when a client's scope changes mid-project?" This tells you about their process and communication style, which matters more than you'd think.
- "What does a typical handoff look like?" If handoff is an afterthought, the system will probably be an orphan after the engagement ends.
Red flags that are easy to miss
They use "AI" as the solution before understanding your problem. Good consultants ask a lot of questions before proposing anything. If someone is pitching you a solution in the first 10 minutes, they're selling, not consulting.
Everything in their portfolio looks like a demo. Demos are great. But demos don't break in production. If every case study ends at the "we showed the client it works" stage, ask what happened after that.
They can't explain a technical decision simply. You don't need to understand every line of code they write. But if you ask "why LangGraph over a simpler approach?" and get a wall of jargon, that's a sign they either don't have a good reason or they can't communicate with non-technical stakeholders. Both are problems.
They guarantee outcomes that depend on your team. "We'll increase your revenue by 30% using AI" — ask how. If the revenue depends on your sales team using the system correctly, your data being clean, and your processes being consistent, those aren't guarantees they can make.
What actually matters most
After all the questions and proposals, the most reliable indicator is still: did someone similar to you, with a similar problem, get something that actually worked?
Ask for a reference call with a past client where you can ask directly: "Would you hire them again, and why or why not?" If they won't give you a reference, that's an answer in itself.
Good consultants don't mind this question. They're proud of the work. They've stayed in touch with the clients. The reference call takes 20 minutes and can save you months of wasted time and budget.