A dealership can buy very capable AI—and still deliver a much less capable customer experience when integration, configuration and human execution break down.
I recently had the opportunity to be involved in a large mystery-shopping study examining how AI sales assistants are performing inside automotive dealerships.
The research, conducted in partnership with Heartbeat Labs, covered more than 250 dealership interactions across four automotive AI sales-assistant providers. We wanted to understand the relative strengths and weaknesses of the platforms, but also something potentially just as important: what happens when AI has to operate inside a dealership’s existing systems, processes and people—and when an AI conversation is handed off to a human.
That produced one of the more interesting findings for me.
The quality of the underlying AI clearly matters. But the technology is only one component of the customer experience. A dealership can buy very capable AI and still configure and operate it into a much less capable experience.
As we dug deeper, the issues tended to fall into three broad areas: integration, configuration, and dealership process and training.
Integration determines whether the AI, CRM, marketing automation and other dealership systems work together. Configuration determines what the AI can see, what it is permitted to say and what actions it is allowed to take. Process and training determine what happens when your people enter the conversation.
Across all three is a larger operational question: has the dealership built an environment around the technology that allows it to perform the way management thinks it performs?
One clue came from looking beyond the average platform scores.
Across the four providers studied, the difference between the highest- and lowest-scoring platform averages was approximately 14 points on a 100-point scale. Yet within the same platforms, the gap between the strongest and weakest dealership implementations ranged from roughly 57 to 75 points.
That does not mean configuration caused all of that difference. Other variables can contribute. But it raises an important question: if the underlying technology is the same, why can the customer experience look so different from one dealership to another?
In one platform’s sample, roughly half of the mystery shops were classified as having configuration issues that suppressed what the system could otherwise do. One setup substituted a generic “we’re looking into this and will get back to you” response for useful vehicle information the system was capable of presenting. The study estimated the performance impact at roughly 30 points compared with stronger implementations of the same platform.
The AI may have the capability. The dealership may have purchased it. But that does not mean the customer receives it.
Some of the strongest dealership interactions we reviewed shared a simple characteristic: they answered the shopper’s question.
When someone asked about a vehicle, the response could immediately confirm availability and provide useful information such as price, VIN and stock number. Stronger deployments moved the conversation forward rather than merely acknowledging that a lead had been received.
A shopper asking whether a vehicle is available wants to know whether it is available. Someone asking about price, a feature or a dealership policy generally values an answer more than an acknowledgment immediately followed by an invitation to schedule an appointment.
That creates some basic management questions. Can the AI see current inventory? Can it access relevant vehicle information? Can it answer dealership policy questions? And critically, what is it authorized to say about price? In many dealerships, the limitation isn't the AI's ability to answer the pricing question—it's dealership policy that prevents the system from answering it directly. What else has the dealership deliberately or inadvertently prevented the AI from doing?
Those are not questions about the sophistication of the language model. They are configuration decisions around it.
And those decisions can turn a highly capable system into something that sounds remarkably similar to the old automated lead response:
Thanks for your interest. Someone will get back to you.
A dealership’s AI rarely operates alone. It sits inside an ecosystem that may include the CRM, marketing automation, third-party lead-provider emails, automated texting, appointment reminders, call workflows and multiple employees.
Each can technically function as designed while the collective customer experience is dysfunctional.
We saw examples of exactly that. A shopper could still be waiting for an answer to the primary question while another system sent unrelated promotions. A shopper could cancel an appointment and still receive reminders. A consumer could say a vehicle had already been purchased elsewhere while marketing messages continued weeks later. In another case, dealership personnel had already communicated with the shopper, yet the automated assistant repeatedly asked whether anyone from the dealership had contacted them.
Nothing necessarily “broke” in the traditional technology sense. The systems simply did not appear to share the same understanding of the customer.
The shopper is not thinking, That’s the CRM talking now rather than the AI platform. The shopper sees one dealership.
The study measured synchronization with dealership personnel and systems. All four providers clustered in a narrow range, earning only 49% to 53% of the available Sync points. Those scores are not the percentage of dealerships with a synchronization failure; they represent the share of available points earned. But the consistency across four providers is noteworthy.
One of the industry’s larger opportunities may not be making AI dramatically smarter. It may be making the dealership around the AI better connected.
When a customer is still waiting for the answer to the question that started the conversation, another automated marketing message is not follow-up. It is noise.
Most of us have experienced this elsewhere.
You call your cell phone provider, authenticate yourself and explain a problem. You are transferred. The next person asks for the same account information and then asks, “So, what seems to be the problem today?”
From the company’s perspective, the customer was successfully transferred. From the customer’s perspective, the company was not listening.
That is exactly the risk dealerships face as AI handles more of the initial conversation. A handoff is not successful simply because a salesperson enters the exchange. Context has to make the handoff too.
In one of the stronger interactions we reviewed, the AI handled the initial inquiry. When an employee later entered, that person picked up on what the shopper had already been discussing—space, comfort and how the vehicle would fit the shopper’s needs. The conversation continued to a successful appointment. It did not restart.
In weaker interactions, the opposite happened. One shopper asked whether a vehicle would accommodate golf clubs and luggage; later outreach moved toward a test drive without answering the question. Another asked about a trade-in; the question disappeared from subsequent human communication. A third asked a vehicle-performance question; weeks of consent requests, availability messages and follow-up followed, but the original question was never answered.
There was plenty of activity. There were no appointments set. Activity is not the same thing as continuity.
Michael Markette, co-founder of Heartbeat Labs, and his team evaluate dealership customer interactions at scale. They see the same issue beyond the mystery shops we examined.
Mike described one interaction in which the AI answered a shopper’s questions accurately within six minutes. A salesperson then entered without absorbing the prior conversation and spent weeks asking about trade-ins and financing and leaving voicemails. Meanwhile, the AI continued asking whether anyone from the dealership had contacted the shopper.
The result was two separate conversations coming from the same store.
As Markette put it:
“The AI is only as informed as the least diligent person updating the CRM.”
A salesperson may be able to see in seconds what the shopper and AI have already discussed, but that information only creates value if the salesperson looks at it. The reverse is also true: a phone call or showroom conversation may never make it into the CRM, leaving the AI to operate as though it never happened.
So AI training cannot simply mean teaching employees how the software works. Before jumping in, a salesperson or BDC representative needs to understand what the shopper has asked, what has been answered, what the AI has learned and whether an unresolved question remains.
Managers also need rules for when people take over, what they review first, how ownership transfers, when AI should resume and when other automated communications should stop.
This is not just AI training. It is process training.
Automotive retail has spent decades emphasizing persistence: response time, attempt count, calls, texts, emails and follow-up.
Those measures still matter. But AI gives us an opportunity to ask a better question:
Did the next communication make sense based on everything the customer had already told us?
A dealership can respond in seconds and still fail to answer the question. It can generate ten follow-ups and fail to carry context forward. It can route a lead to a salesperson while forcing the shopper to begin again.
Traditional reporting may count all of that as activity. The consumer experiences it differently.
One way we began framing this in the study was through a KPI we called “Noise-to-Value”: how much of the dealership’s outreach is simply noise versus communication that advances what the shopper is actually trying to accomplish?
Did the shopper get what they needed? Did the dealership remember what had already happened? Did the human improve the conversation? Did the systems behave as though they belonged to one company?
Those are customer-experience questions.
There is a positive side to these findings.
If an unsatisfactory AI experience is caused by a fundamental limitation in the technology, solving it may require a better product. But many of the problems we observed can potentially be addressed through better integration, thoughtful configuration, clearer permissions, stronger handoff rules, suppression of conflicting automation, better CRM discipline and better training.
Most importantly, they require ongoing management.
Dealers should not assume that the AI experience they purchased is the AI experience their customers are receiving.
Mystery shop it. Read the conversations. Look at what happens when a human joins and what the CRM sends next. See what happens after a shopper cancels an appointment, buys elsewhere or speaks with someone at the dealership. Ask whether the customer’s primary question actually got answered.
Do not evaluate the AI in isolation. Evaluate the experience.
Automotive retail is moving rapidly from AI experimentation into everyday operations. The technology is becoming remarkably capable, and the strongest interactions in this study showed what is possible when the pieces work together.
But dealerships do not deploy AI into a vacuum. They deploy it into an environment filled with technology, processes, policies and people. That environment becomes part of the product the customer experiences.
So dealers need to ask more than, How good is the technology?
Is it integrated with the rest of our technology stack? Have we configured it to use the capabilities we are paying for? Have we trained our people to enter and continue an AI-led conversation? Are we managing the experience as one continuous customer journey?
The next phase of dealership AI will not be determined solely by who has the smartest technology. It will also be determined by which dealerships learn how to integrate it, configure it, train people to work alongside it and manage the customer experience as one conversation.
That may be what separates AI investments that genuinely improve the customer experience from those that simply create more automation—and more noise.
