The lights are on but nobody’s home. The engine’s running but no one’s driving. The wheel is spinning but the hamster is dead. Pick your preferred idiom for unattended automations run amok. (“The Wi-Fi’s on, but there’s no Internet.”)
Over the past weeks, I encountered AI agents and automation at two separate martech companies — as a customer of one and a pseudo-prospect for the other — that leaned toward the “annoyingly efficient” upper-left quadrant of the 2×2 below. Efficient for the company. But for the customer? Not so much.
I’ll share what happened, as examples of how these things can slip off the rails, and then offer a few ideas for avoiding such outcomes. I won’t name names, because I generally admire both companies and think these kind of issues are not uncommon.
That said, yeah, it’s ironic that these are martech companies.
Customer service on the route to nowhere
A product I subscribe to has been emailing me helpful onboarding tips for weeks. It’s been a good cadence, not too frequent, and the content has been genuinely useful. They come from an email address that we’ll call friends@acme.com (not the actual company or email address).
However, last week I had a billing question. I searched for how to contact someone about it, and everything pointed me to emailing friends@acme.com. Simple and easy, right? So I emailed them my question.
No reply. Not even an automated acknowledgement of receipt.
Not a big deal. I wait a day, and then email again to check back.
Again, no reply.
A couple of days later though, friends@acme.com sends me another cheerful onboarding message. Really?
I reply once more asking about my billing issue. I volunteer that maybe I need to contact a different email address about this? Again, no reply.
I go back to hunting around their app to see if somehow this was my fault, that I missed a “for billing questions, contact billing@acme.com” instruction. I search FAQs, account screens, help docs. Everything just points to friends@acme.com.
I abandon the effort for the time being, as I had other things to do.
While it’s impossible to debug this from where I sit, it seems like an automated routing problem. Multiple teams sharing the same inbox with some automation rules to direct incoming messages to the right place. But it’s broken. Maybe it’s a bug in the routing logic. Maybe the destination I got routed to is unmanned. Maybe an LLM is trying to interpret my message but it’s erroring out. Perhaps a RAG look up of my account is failing?
Automations can be fragile, especially complex ones. As a martech person, I appreciate that. I empathize with this certainly unintentional fail. But better safeguards could and should have been implemented.
Funny enough, as I’m writing this, I just received yet another onboarding email from friends@acme.com, clearly still oblivious to my prior attempts to engage with them. I laugh out loud, causing the person next to me on the plane to give me a sideways glance.
To err is human, but to really mix things up use AI
Meanwhile, in another part of the martech galaxy, I downloaded a new report that a vendor was offering on a subject of great interest to me.
Now, in fairness to the vendor, I am probably an outlier responding to this classic demand-gen tactic. I don’t think of myself as a prospect for their product, at least not right now. I’m just interested in the subject matter.
Within a couple of minutes of filling out the form, I get a personalized email from a rep.
You want to talk hyperpersonalization? This message nailed it. The opening paragraph was crafted about the nature of my work on chiefmartec. It raised a concern about AI that was very relevant to me. It asked a genuine-sounding question about what sparked my interest in the report. It was a very human sounding message. No simple Hello, $FIRST-NAME template. This was production-quality generative AI personalization.
The only AI tell? The near immediate arrival. No human could have researched me and crafted that message that quickly. But, hey, that’s not a bad thing. A fast reply. A well-personalized message. I admire the fact that it was AI-powered.
So I reply, complimenting them on their response time — my wink-wink that I know how they did that — and letting them know what my interest was and why I’m probably not a real prospect for them. I assume that will make it to the real rep.
They didn’t reply back.
That’s fine. I had intentionally disqualified myself. But it felt a little odd. Their original email was very conversational, and I had engaged with a remark that I would have expected to elicit a response back. In exchanges I’ve had with reps at other martech companies under similar circumstances, they almost always replied. Those threads sometimes led to other beneficial connections and outcomes over time. (The funnel is perennially oversimplified.)
But no worries. They’d moved on, and I moved on.
Until a couple hours later when I got another email from the same rep — but formatted very differently. It opened with old-school Hello, $FIRST-NAME personalization, referenced something unrelated to what I downloaded, and offered calendar options for scheduling a demo. Huh? It was a total non sequitur to the exchange we just had.
Now, being a martech nerd, I can call this: I got thrown into an automated sales sequence.
I shake my head. A week later I get another message from the rep, seemingly sent by their automation engine, as it’s formatted the same as that last one. This one makes a generous offer, but for something pretty irrelevant to me. It’s also Hello, $FIRST-NAME personalized — but now also with a $COMPANY-NAME personalization plugged into the first paragraph. Except the company name inserted is most definitely not my company.
Yikes.
Cross-contaminated data? Misconfigured automation sequences? An LLM hallucination inside an AI agent? I suspect there may have been compounding errors along this journey.
My guess as to what happened? An AI SDR agent crafted the first message. The rep never saw it. Nor my reply, which the AI SDR intercepted and didn’t know what to do. So it just enrolled me in a sequence. It added some fields to my record through a data enrichment service that were just flat-out wrong. The sequence had no “memory” of my exchange with the AI SDR. These were probably two different products with a weak integration at best.
Learning how to leverage hybrid human + AI teams
Humbling moments like these are good to keep things real.
For the record, I am in favor of leveraging AI and automation in martech. That’s arguably been the raison d’être of this blog for the past 17 years. But for nearly as long, I’ve seen Martec’s Law at play: technology changes quickly, but organizations change slowly. As a corollary: we almost always underestimate the people and process changes required to successfully leverage fast-changing technologies.
With all the vague-but-urgent, top-down edicts to adopt AI, teams sometimes feel the need to rush new capabilties into production. The move-fast-and-break-things ethos. For real AI transformation though, I recommend a go-slow-to-go-fast ethos instead.
That doesn’t mean the teams driving this transformation are lollygagging about. It’s a high-gear intensity for everyone in this period of disruption. But you can’t skip over major stages of the journey. You need to take it a step at a time to learn what the technology can and can’t do. But more importantly, you need to understand its dependencies and ripple effects. You need to figure out which legacy processes need to be updated — and which ones need to be entirely reinvented or gracefully put to pasture.
Move fast with pilot projects and experiments! But contain them as narrowly as possible at first. And absolutely keep a human-in-the-loop — even if it’s just post-hoc review of what happened — watching closely. Don’t scale prematurely. You want to make sure that your foundation is solid because that’s what makes it possible to reliably move faster in the next stage. Chasing ghosts and wrangling tech debt in production really gums up the works.
Here are a few concrete suggestions:
Always have a way to reach a real human. AI and automation can be efficient for customers as much as it is for the company — when it addresses the customer’s needs. But it’s never going to be 100%. The edge cases are infinite. And while not all of them will be valuble opportunities lost, some of them will be. If a customer wants to reach a human, you probably want them to as well. The lower-right corner in the 2×2 at the top of this article sums it up: go the extra mile for loyalty & love.
Run secret shopper exercises. Have people who take the journey as prospects or new customers, carefully recording their experience. Doing this with your own employees is a decent place to start. But ideally you want someone external with fresh eyes and no bias for internal politics to run the gauntlet. This is a little more difficult in B2B, given the data enrichment industrial complex that hones in on one’s identity, albeit with varying degrees of accuracy.
Regularly inspect every active AI agent or automation. Just as elevators and escalators require regular inespection, anything that moves your prospects and customers along in their journey should be periodically reviewed. Are the rules and data definitions being used still valid? Is the content and branding still relevant? What are the contexts in which it works — i.e., what could have happened before it, what might happen next? Timestamp and sign these inspections. Repeatable test harnesses and heuristics are good. But you also want the benefit of human judgment in that check-up.
If you use a rep’s name in communications, loop them in. I know, the whole point of AI agents and automations in such scenarios is to reduce the manual work for reps. But they can use inbox rules to reduce the noise until there’s a response from the recipient. Then it’s probably worth at least peeking at it, even if they let the automation carry on. It also makes it easy for them to occasionally review what is being sent in their name. After all, it’s their reptuation as well.
Distinguish between the “laboratory” and the “factory.” My frequent collaborator Frans Riemersma has made the insightful observation that there are two different missions with martech stacks. One is serving as the steady and reliable factory in production use cases to protect current revenue. The other is serving as a laboratory to run experiments with new or adapted use cases. Keep these two things distinct in how you manage them! Be wary of lab leaks.
I also think there’s a tremendous opportunity for existing martech platforms and new martech entrepreneurs to provide more governance and monitoring capabilities to address these issues. This is a big part of what good “orchestration” needs in our new AI age of marketing.
Building an automation or deploying an AI agent is getting easier by the day. Keeping them all aligned and working properly? Without good operational discipline though, that’s going to be like herding cats.
