Most people testing AI for content stop at a handful of blog posts. They generate ten, skim them, decide it reads fine, and start selling the service. That is where the testing ends, and it is not far enough to learn anything useful.
A friend and I run a publishing project called Oliver Ahn. We have pushed AI-assisted writing to full novel length repeatedly, at our own expense, specifically to find where it falls apart. Not a thousand words. Ninety thousand, over and over.
Here is what actually broke, and why it should matter to anyone paying someone else for AI content.
It sags. It does not collapse.
This is the finding that changed how I work, and it is the one nobody warns you about.
Read any thousand words in isolation and they are fine. Clean sentences, correct grammar, reasonable structure. You would sign off on them. Read forty pages and something else becomes obvious: everything sounds the same. The rhythm never varies. Every paragraph is built the same way.
Quality does not fall off a cliff where you would notice it. It slowly flattens. On a novel you catch it eventually because you have to read the whole thing. On a website nobody ever reads the whole thing, which means the sag goes undetected until a search engine or a customer decides your site feels like every other site.
It drifts to the middle of the road
We asked for comedy. Unhinged, specific, uncomfortable comedy. What came back, more than once, read like a documentary about comedy. Technically about the right subject, entirely without the thing that made it worth writing.
Left alone, the model produces the most average version of whatever you asked for. That is what it is built to do. For fiction it is a problem you can fight with editing. For a service page it is fatal, because average is precisely what you are trying not to be. If your plumbing page reads like the median plumbing page, you have paid to sound like everyone you are competing with.
It develops tics you cannot see coming
One 27,000 word draft used the word eleven 108 times. Not eleven of something. Just the word, latched onto and repeated, in a book that had nothing to do with the number.
Nobody wrote a rule that produced that. It emerged. And once we knew to look, we found the same pattern in other drafts with other words. A phrase gets established early and then quietly colonises everything after it.
The practical version for your website: run a hundred generated pages and some phrase you never chose will be on all of them. You will not notice, because you will never read all hundred.
Fixing it in bulk makes it worse
The obvious response to a tic is find and replace. We tried it. A scripted swap silently broke nine sentences, because the word appeared in contexts the pattern did not account for. The output still looked clean. It was not.
Anything worth fixing has to be found by reading, which means the cost of a mistake at volume is that you have to read all of it anyway.
It cannot know what actually makes a page win
This one has nothing to do with quality. The details that make a local service page rank are the ones only the business owner has. Which jobs you turn down. What your competitors get wrong. The neighbourhood where the houses all have the same failing system. What customers say on the phone before they say what they think they should say.
None of that exists in a model. It exists in your head, and it has to be extracted by asking you. There is no prompt that substitutes for that conversation.
Generating stops being the hard part immediately
Within a week of running the pipeline properly, producing text was no longer the bottleneck. Editing was the whole job, and it stayed that way.
That reframes what anyone selling you AI content is actually offering. If they are pricing on how many pages they can produce, they are pricing the part that turned out to be free. The expensive part is judgment about what to cut, and that is the part that does not scale.
What this means if someone is selling you AI content
Three questions worth asking.
- How much have you actually run through this? Ten blog posts is not a test. It is a demo.
- Who reads the output, and how much of it? If the answer is a spot check, the sag will reach your site.
- What did you learn that you had to change because of? Anyone who has run this at volume has a list of things that went wrong. If they do not, they have not.
Where I use it in client work
I do use AI. I am specific about where: research, first drafts I intend to rewrite, structured data, and the tedious middle of a technical audit. Those are tasks where average output is genuinely fine, or where the work is mechanical enough that a model does not need judgment.
I do not use it to fill a client site with pages. Not on principle, and not because of a policy. Because I have watched exactly what that produces at length, on my own money, and I would not sell it to anyone.
I am a digital strategist in Tampa. I lead a team at an ad agency and take on a short list of clients on the side, doing SEO, AEO and GEO, lead generation, website builds and automations. If you want a straight answer about whether AI belongs anywhere in your marketing, send me your site.
Discover more from J. Mike Oliver
Subscribe to get the latest posts sent to your email.