AI vs Human Content Case Study for SEO Performance in 2024

Introduction to SEO Case Study

SEO is something that I have avoided doing my entire marketing career, but last year I took the dive to start writing for Performify.

I now know why I avoided it for so long, it just doesn't make any sense.

The performance metrics inside GSC are confusing, not ranking at all is frustrating, and writing these blogs takes a lot of time & energy.

Personally, I blame the disconnect between the Google Search team's PR staff and the actual developers. 

I think this is what leads to SEO sudo science and all the misconceptions floating around in the industry.

So if an AI can pump out perfectly optimized articles in minutes, why not just use that?

You're about to learn if AI writers outperform human-written articles in SEO. 

Let's save time and start with the results...

Case Study Methodology

So I wrote a custom script using Pytorch and several Python NLP libraries, like spaCy & TextBlob, to generate these results.

The code is available in this Github Repo, totally free for anyone to see or use.

Comparing Human SEO Blog Stats

ChatGPT Humanizer is a hand-written blog we'll be comparing in this case study.

My first blog, written in 15 minutes, remains our best-performing piece. It brought in numerous new users, with nearly 1 in 4 readers signing up for a free Performify account.

This success prompted me to investigate why the content performed so well.

Hand-Written Blog SERP Performance Statistics

• Readability: Scored 27.8% higher on the Flesch Reading Ease scale.

• Semantic Coherence: 28.0% higher average score.

• Sentiment Consistency: More consistent tone changes (16 vs. 15).

• Context Score: Higher at 0.32 vs. 0.25 for AI-generated content.

Comparing AI Written SEO Blog Stats

AI Form Filler is an AI-generated blog we'll be comparing in this case study.

A recent blog written almost entirely with the Claude 2.1 model inside Performify started to overtake the ChatGPT Humanizer blog in clicks.

This blog targeted high-impression, low-competition keywords.

AI Generate Blog SERP Performance Statistics

• Word Count: Higher at 1,477 words vs. 1,349 for the hand-written blog.

• Unique Words: More at 567 vs. 475.

• Polarity & Subjectivity: Slightly higher scores (0.20 and 0.52).



• Mentions of Organizations: 85.7% more frequent (26 vs. 14).

Overall SEO Ranking Differences

The difference between an AI writer and a human writer can be huge.  Both from the reader's perspective and from a marketing performance perspective.

Despite similar engagement metrics, the human-written "ChatGPT Humanizer" blog had a noticeably higher number of returning users.

But which one ranks higher in the search engine results?

Case Study Takeaway and Statistics

• Better Readability, scoring 27.8% higher on the Flesch Reading Ease scale compared to the AI-generated blog.

• A 28.0% higher average semantic coherence score, which indicates better contextual flow and logical structure.

• Sentiment was also more consistent in the hand-written piece, changing tone 16 times vs 15 for the AI blog, which could may have kept readers interested.

• The handwritten blog had a higher score for context - 0.32 vs 0.25. This means ideas flowed together better.

Takeaway: What I think made the Human blog perform better was that It had longer sentences and fewer complex words, so it's easier and more enjoyable to read.

The hand-written blog's score of 60.14 falls into the "Plain English" category, which is easily understood by 13- to 15-year-old students.

In contrast, the AI-generated blog's score of 47.08 falls into the "Difficult to Read" category, which is best understood by college graduates.

AI-Generated Blog Performance

• The AI-generated blog had a higher word count (1477) and more unique words (567) compared to the hand-written blog (1349 words and 475 unique words).

• It also mentioned organizations (ORG) 85.7% more frequently than the hand-written blog (26 vs. 14 mentions).

• The AI-generated blog had slightly higher polarity (0.20) and subjectivity (0.52) scores compared to the hand-written blog (0.18 and 0.47, respectively).

• It used fewer ordinal numbers (ORDINAL), only 2 mentions compared to 6 in the hand-written blog.

Takeaway: The AI-generated blog's strengths lie in its ability to cover a wider range of topics and keywords, potentially improving its keyword coverage and topical authority.

However, it may lack the readability and coherence that human-written content provides, which could impact user engagement and search rankings.

Overall SEO Ranking Differences

Taking a look through the analytics, there aren't many huge differences between blogs as far as engagement metrics go, despite the difference in number of page visitors.

However, the 1 noticeable difference is that the number of returning users is much higher for the human-written blog. ("ChatGPT Humanizer")

SEO Statistic Takeaway: The hand-written blog scores 27.8% higher on the Flesch Reading Ease scale compared to the AI-generated blog (60.14 vs. 47.08).

SEO Statistics Comparing Hand Written Articles to AI Writers

Below are some of the raw stats I documented during the NLP analysis of both blogs.  These factors are what contribute to the final performance scores and stats used in the above section.

Flesch Reading Ease:

Hand-written blog: 60.14

AI-generated blog: 47.08

Gunning Fog Index:

Hand-written blog: 10.39

AI-generated blog: 10.51

Sentiment Analysis:

Hand-written blog: Polarity: 0.18, Subjectivity: 0.47
AI-generated blog: Polarity: 0.20, Subjectivity: 0.52

Word Count:

Hand-written blog: 1349

AI-generated blog: 1477

Unique Words:

Hand-written blog: 475

AI-generated blog: 567

Average Sentence Length:

Hand-written blog: 23.86 words

AI-generated blog: 21.17 words

Sentiment Changes:

Hand-written blog: 16.55
AI-generated blog: 15.56

Pros and Cons of Using an AI Writer

While it is possible to personalize AI content at scale, when it comes to performance marketing, client success, and engagement, I believe that context is king.

Aligning your ideas with your audience's pain points is critical to driving home the message you want to share.

An AI writer like Performify can drastically help with this.

We created a shareable smart AI form that syncs up with over a dozen different AI models to enrich your AI responses.

The result is a more personalized output, the first time around.  And as you can see in the above stats, that can greatly improve SEO performance.

Pros and Cons of Writing Original Content

Writing original content by hand has its advantages. Human writers can infuse their unique insights, experiences, and perspectives into the content, creating a stronger connection with readers.

They can also craft compelling narratives and adapt their writing style to suit different audiences and purposes.

However, creating original content is time-consuming and resource-intensive, which may limit the amount of content that can be produced.

Despite these challenges, the personal touch and creativity that human writers bring to the table cannot be easily replicated by AI.

Original content has the potential to stand out in a sea of generic, AI-generated articles, attracting more engagement from readers.

Takeaway from Case Study Results

Focus on context.

When I first started my AI automation marketing agency, I realized that writing for client's means delivering content that is completely accurate to their company.

Learn More about How Performify Helps You Write Higher Performing Blogs