Why marketers need to push back against AI

When you hear “compliance,” do you immediately think of regulatory frameworks like GDPR, data privacy, and legal restrictions? Well, that’s the standard definition. Another is behavioral. I’m talking about the habit of accepting what AI produces without question. This “cognitive compliance” happens when formal-looking, polished AI outputs hit your screen faster than your instinct to question them.

I once worked for a manager who used a heavily biased LLM to discredit his team. This LLM was known to source its answers from unverified user posts rather than primary research, verified reference sites, or documented reporting. After months of observing how AI was being used across both workplace and personal settings, I noticed three distinct patterns (all of which are supported by scientific research):

• The illusion of accuracy: Given AI’s polished presentation, people immediately believed what was on the screen.
• The erosion of skill: As a person’s reliance on AI grew, their active critical thinking skills diminished.
• The confidence trap: The more confident people became in using AI, the less able they were to maintain independent thinking.

Looking at these patterns, it became clear to me that as AI gets smarter, human judgment matters even more.

The hidden cost of unchecked automation

Martech companies are racing to integrate generative tools into their tech stacks and tracking AI adoption rates as a key performance metric. It takes the form of corporate mandates and performance scorecards that capture the number of AI tools deployed, the percentage of staff trained, and the sheer volume of investment.

So, what’s the problem?

The more we push teams into AI adoption without guardrails, and the more automated campaign creative and copy proliferate, the less prepared we are to direct, supervise, and judge those outputs. Without this critical skill set, we face a reality where AI silently sets the direction of our strategic thinking, creative output, and brand futures.

Here are two high-profile examples in which a lack of human judgment in using AI had devastating consequences.

Hallucination and the law

Attorney Steven Schwartz of the firm Levidow, Levidow & Oberman used ChatGPT to conduct legal research and drafted a brief citing six cases in opposition to a motion to dismiss. The problem? None of those cases actually existed. ChatGPT hallucinated them entirely — fabricating names, citations, and judicial opinions.

If that wasn’t bad enough, when confronted with evidence that the research might be wrong, the lawyer went back to the exact same AI tool to verify it, blindly accepting the AI’s confirmation as sufficient.

Though the resulting court fine was only $5,000, this case paved the way for strict disclosure of AI use in legal filings. Attorneys must now certify either that generative AI was not used in preparing briefs or that all AI-generated content was verified by a human. As we can see, intentionally embedding a human into the supervisory role is the only way to keep critical thinking in the loop.

• The Martech Parallel: If a seasoned lawyer can blindly trust a hallucinated legal brief, it is terrifyingly easy for a marketer to blindly trust hallucinated competitive intelligence, flawed persona data, or fabricated performance metrics when building a high-stakes campaign.

The black box and healthcare claim denial

Imagine finding out that an algorithm, not your doctor, is deciding how long you get to stay in rehab after a major surgery. That is the core of an ongoing class-action lawsuit against UnitedHealth Group. The lawsuit claims that UnitedHealth used an AI tool called “nH Predict” to automatically deny necessary medical care to elderly Medicare Advantage patients.

Instead of evaluating what each patient needed, the AI used a pre-determined, generalized timeline to cut off coverage for critical services like nursing homes and physical therapy. Even worse, the lawsuit highlights a complete lack of meaningful human intervention. Rather than having medical professionals review the cases, UnitedHealth allegedly used AI to systematically override the explicit recommendations of the patients’ treating physicians.

The consequences are devastating. Patients are forced out of care facilities before they are fully healed, leading to serious medical setbacks — which, ironically, often cost the company more than the original course of treatment. Families who refuse to leave are hit with massive, unexpected medical bills. As if all that isn’t bad enough, the Senate Permanent Subcommittee on Investigations found that as UnitedHealth aggressively automated its processes, its post-acute care denial rate more than doubled — jumping from 10.9% in 2020 to 22.7% by 2022.

The families suing UnitedHealth are asking the court to force the company to stop using AI for automated medical decisions and to completely overhaul how claims are reviewed. With any luck, it will not only force the company to bring physician judgment back into the loop but also expose how its AI works behind the scenes.

• The Martech Parallel: While the stakes in healthcare are life and death, the underlying technical flaw is identical to what we risk in marketing: setting an algorithmic model to aggressively optimize for a single metric (like immediate cost-cutting or short-term conversion spikes) while completely ignoring human nuance, long-term brand health, and customer trust.

What’s happening to critical thinking?

We’ve seen what happens when people take automated data at face value, but why is it happening so easily?

A 2025 global study by the University of Melbourne and KPMG surveyed 48,000 people across 47 countries and found that two-thirds of AI users do not evaluate the accuracy of AI outputs before acting on them. It’s not because they don’t know how; it’s because they choose not to. The output looks clean. It looks confident. It looks like something a competent colleague produced. So, they hit launch.

To find out what’s happening underneath the surface, MIT Media Lab researchers placed EEG sensors on participants’ heads while they wrote with AI assistance. They found measurable reductions in brain connectivity compared to people who wrote without tech intervention. While this study is preliminary, they coined a powerful term for what they observed: “cognitive debt.”

The idea is that we are borrowing mental effort from the future—outsourcing the heavy cognitive lifting that builds our judgment, creativity, and critical thinking. In doing so, we accumulate a mental cost that persists even after we close the AI application.

Lastly, Microsoft Research and Carnegie Mellon surveyed 319 knowledge workers and found that the more people trust AI, the less critical thinking effort they apply. The confidence in the tool actively suppresses confidence in oneself. Additionally, researchers found that relying heavily on generative AI tools yields a significantly less diverse set of outcomes than human-led, independent thought.

In modern marketing, where differentiation is your only real defense against market noise, this algorithmic homogenization is a quiet killer. If everyone uses the same prompts and the same models, we sentence our brands to a “sea of sameness”—where every blog post, ad headline, and email campaign sounds exactly like the competitor next door.

The modern marketer’s playbook

Rather than succumbing to automation bias and outsourcing your strategy to an algorithm, you can maintain your expertise, edge, and creativity by practicing a few core principles:

• Do: Direct AI using your industry expertise, unique brand voice, and professional training.
  • Don’t: Allow the AI to lead the strategy. You’ve built up years of hard-won knowledge—use it as the anchor.
• Do: Rigorously pressure-test both your AI inputs and outputs.
  • Don’t: Assume the machine is correct. Question AI outputs critically by adopting a skeptical stance. Reviewing data through a critical lens will save your team from public rework and brand embarrassment.
• Do: Build the first draft, mapping out your core hypotheses and proof points yourself.
  • Don’t: Hand AI a blank-slate topic and allow it to do the fundamental thinking for you.

The post Why marketers need to push back against AI appeared first on MarTech.