Choosing a conversational AI partner is harder than it should be, because almost everyone in the category sounds the same. Every deck has the same promises. Every demo works flawlessly. The hard part is telling the difference between a vendor who will build you something that runs for years and one who will build you a beautiful demo that quietly dies in its second quarter.
This guide is the set of questions that exposes that difference. Ask them in your evaluation calls. The answers, and how comfortably they come, will tell you more than any case study slide.
First, decide what you are actually buying
Before you compare partners, get clear on which of three things you want, because they are not the same purchase.
The first is a tool: a platform you and your team will use to build and run agents yourselves. You are buying capability and taking on the operating burden.
The second is a project: a one-off build delivered and handed over. You own it afterwards, including keeping it alive.
The third is a service: an agent that is designed, built, hosted, and continuously run for you, with the partner staying embedded. You own the outcome, they own the operation.
A lot of operators think they want a project and actually want a service. The reason is the part nobody enjoys discovering. A conversational AI agent is not a thing you ship once. It is a thing you operate. Its knowledge goes stale, its integrations break against system updates, its edge cases need tuning. A project that is handed over and forgotten becomes the half-dead bot every business has a story about. Know which of the three you are buying before you start comparing, or you will compare the wrong things.
The twelve questions
On capability and proof
1. Show me an agent you run in production, and tell me its resolution rate. Demos are designed to work. Production is where reality lives. A partner who actually operates agents can show you a live one and tell you what proportion of real enquiries it resolves without a human. A partner who only has demos will pivot to talking about the technology. The pivot is the answer.
2. What does your agent do when it does not know? This is the single most revealing question in the list. The honest answer describes a confidence threshold and a clean escalation to a human with full context. The dishonest answer is some version of "our AI handles everything." No agent handles everything, and the ones that claim to are the ones that strand customers in loops. You want the partner who is comfortable telling you where their agent stops.
3. What is your resolution rate honestly, and how do you calculate it? Watch for the denominator. A figure like 95% accuracy on the questions it chose to answer is not the same as resolving 95% of all enquiries. A trustworthy partner will tell you both the number and exactly what it is a percentage of. Vague proof is usually inflated proof.
On integration and timeline
4. How will you integrate with our existing systems, and what do you need from us? The agent is only as useful as its connection to your CRM, billing and helpdesk. A serious partner will ask about your stack early and talk specifically about APIs and access. A weaker one will gloss over integration, which is exactly where their timeline will later collapse.
5. What is your realistic timeline, and what determines it? The right answer ties the timeline to your systems' accessibility, not to a fixed sales-deck number. Four to eight weeks for a first agent integrated into existing systems is a credible range. "Live next week" is either a toy or a fib. "Six months" usually means a consultancy and a six-figure invoice.
6. Will you work with our existing tools, or do you need us to replace anything? The honest partner integrates with what you run. The one who wants to rip out your helpdesk and install their preferred stack is optimising for their convenience, not your outcome. Rip-and-replace is slow, expensive, and rarely necessary.
On the ongoing relationship
7. What happens after launch? This separates partners from vendors more cleanly than anything else. The vendor's answer trails off after go-live. The partner's answer describes weekly check-ins, ongoing tuning, knowledge maintenance and a direct line when something breaks. Continuous operation should be the model, not an upsell.
8. Who maintains the knowledge base when our prices and policies change? If the answer is "you do," understand that you have bought a project, not a service, and budget internal time accordingly. If the answer is "we do, continuously," that is the service model, and it is what keeps the agent accurate past month three.
9. How do you handle our data and compliance? For any regulated or data-sensitive operation, a partner should talk fluently about hosting, data handling, identity verification flows and the relevant regulations for your sector without being prompted. Hesitation here is a serious flag.
On commercial structure and risk
10. How is pricing structured, and what triggers it going up? Look for a clear split between a one-off build fee and an ongoing operating fee, with transparency about what causes the latter to rise: more agents, more channels, more volume. Opaque pricing now is opaque pricing forever.
11. What is the cost of leaving? Ask it directly. A confident partner answers without flinching: notice period, what you keep, how the offboarding works. Evasiveness here tells you they are relying on lock-in rather than results to keep you.
12. How many other clients are you running right now, and who will own our account? Capacity is a real risk with smaller partners and attention is a real risk with larger ones. You want to know whether you will get genuine focus or be account number forty. Neither extreme is automatically wrong, but you should know which one you are buying.
Reading the answers
You are not grading these answers for polish. You are grading them for comfort and specificity. The pattern to look for is consistent across all twelve. A real operator answers concretely and is comfortable naming limits. They will tell you what their agent does not do, where timelines slip, what leaving costs. That comfort comes from having actually run these systems and having nothing to hide.
A weaker partner answers in abstractions, reaches for the technology when you ask about outcomes, and gets vague precisely where specifics would expose a gap. The vagueness is information. Trust it.
The one question underneath all the others
If you only ask one thing, ask this. Will you still be responsible for this agent working in twelve months, or will it be ours to keep alive?
Everything else is downstream of the answer. A conversational AI agent that nobody owns the operation of will degrade, not because the technology failed, but because no system runs itself. The right partner is the one whose incentives are tied to your agent still working a year from now, not just to it launching on time.
Choose for the operating, not the build. The build is the easy part. It always was.
Fiveleaf builds and runs AI agents inside mid-market and enterprise businesses. We stay embedded as your AI specialist rather than handing over a build and leaving. If you are evaluating partners, book a call and ask us all twelve.
