AI Content Detection – Real World Results From Our Content Team Vs External Contract Writers

A game-changing event happened in November 2023: OpenAI, an American AI (artificial intelligence) research company, launched its ChatGPT service, a general-purpose chatbot that can generate texts.

The rapid development in AI content generation following this event has caused many people and institutions to worry about AI content generators’ adverse impact on various sectors and industries.

For instance, business owners today may be anxious to know whether they are getting their money’s worth on high-quality online content or wasting their hard-earned money on fake content.

Fortunately, AI content detection tools are also innovating at an equal pace with AI text generators’ developments. 

However, if you work in the content creation industry, you would want to know if these AI content detectors actually live up to their hype. 

Moreover, you may wonder if these tools can protect you from unethical content creators who submit AI-generated content as their own (we’ve caught out our fair share).

As a search agency heavily focused on editorial SEO with a view to long-term niche domination, ensuring all our internal and contract writers submit human-researched and written content (thereby operating well within the guidelines of Google’s TOS) is something we rely on heavily.

Our own investigation and results on AI content detectors indicate that, while AI detection technology is relatively new and each AI detection tool varies in accuracy, the detectors were fairly good (when combined) in differentiating between human-written and AI-generated texts, with a low % of false positives (if you follow certain rules).

Using the six tools below, we test the articles that the writers submit from our internal writers (control) and our contract writers:

• AI Classifier 
• Originality.AI
• Copy Leaks
• Content At Scale
• Corrector App
• Crossplag

We used the data from this table to compute the percentages of AI content in the contract external writers’ submitted articles. 

As of February 2023, we’ve gathered the following data and information from our investigation using the articles submitted by the external agency, and here are the real-world results we got:

We constantly update our dataset to give real-time information and improved evaluations. 

AI Text Classifier or AI Classifier is an AI detector tool from the makers of ChatGPT (OpenAI’s chatbot).

Originality.AI is currently the most accurate AI detector based on our data.

• The overall average AI% score (using the highest AI% scores across all tools) is 18.05%.

As of February 2023, Originality.ai has the highest accuracy and optimal features of the six AI detection tools we investigated.

What Is AI Detection?

AI detection is the process of detecting AI-generated content. 

Free and paid AI detection tools can help business owners, journalists, and marketing agencies determine whether a piece of content is human-written (original) or AI-generated. 

How Do AI Detectors Work?

AI content detector tools or AI detectors use various methods to identify machine-generated content. For example, they can do the following:

1. Analyze the text’s features: This procedure involves examining how often some words appear, how fluent the text reads, and whether there are punctuation and sentence structure patterns.
2. Compare the AI text analysis to a dataset of human- and machine-written text: Some AI detectors are large language models (LLM) themselves. 

For example, GLTR (Giant Language Model Test Room) has access to the GPT-2 117 M LLM, one of the largest publicly available models.

Developers can retrain these models on some AI-generated and human-written texts so that they can learn to differentiate between the two.

Other AI detectors can detect AI plagiarism by conducting a train-test-validation split on a dataset. 

The GPT-2 output detector uses this method on open-source data from OpenAI to examine the likelihood that  OpenAI ChatGPT generated the text. 

What Is AI-Generated Text?

AI-generated text is a type of content that is digitally produced using machine learning (ML) and natural language processing (NLP). 

ML is a branch of AI and computer science that enables systems to simulate human intelligence, allowing machines to predict outcomes accurately and improve over time.  

NLP, a subfield of ML, is the process of converting raw text data to computer-understandable (structured) data. This NLP uses this structured data to create human (natural) language.

Can Google Detect AI-Generated Content?

Yes. In a Google SEO office-hours session from November 2022, Duy Nguyen from Google’s search quality team stated that Google has “many algorithms to go after” those who use AI writing tools to plagiarize content. 

Furthermore, the latest Google spam policy states that the search engine will penalize those who publish AI-generated content on their web pages.

How to Tell if an Article Was Written With AI

AI content generation is revolutionizing the content creation industry. We’ve yet to see whether this technology will be a net positive or negative for the digital world.

Meanwhile, business owners, journalists, and digital marketers today are searching for ways to detect AI content.

At Digital Spotlight, we investigated our external and internal writer’s content using six AI content detection tools, including AI Classifier, Originality.ai, Copyleaks, Content at Scale, Corrector App, and Crossplag, to ensure that the articles are 100% human-written.

How to Read AI Detection Scores

Our standard process is to copy and paste the article draft into the AI detection tool. Any output with more than a 25% AI score on any AI content detector is considered failed, meaning that it was likely produced by AI. 

Note that a 25% AI score does not mean that the article has 25% AI content. Instead, a 25% AI score means there’s a 25% probability that the article is AI-generated.

Our current data shows that 3 of the 21 articles reviewed failed in two AI detection tests. Meanwhile, five articles failed in Originality.ai alone.

We then provide the feedback to the writer and request a revision to match our standards. We continue to optimize the articles until we get a less than 25% AI score.

This strict content creation process shows how Digital Spotlight’s values original content – the kind of content that is relatable and engaging.

  The following table shows our latest AI detection report on our internal and external writers’ submitted work, and here are the real-world results we got:

For internal writers

For external writers

This article actually got the following scores:

• Corrector App: 0.18% AI 

• Crossplag: 5% AI

• Copyleaks: 93.80% Human 

• Content at Scale: 96% Human

• Originality.ai: 24% Human

Note: We regularly update these trackers to give accurate results and feedback.

Current Problems with AI Content Detection

Readers must also note that AI content detection technology is relatively new. 

These AI content detection tools require much improvement, especially regarding their accuracy and reliability in predicting AI content.

For example, a 2,438-word article from one of our internal writers initially got a 32% AI content score from Crossplag. However, the writer managed to bring the Crossplag AI percentage score down to 5% simply by switching the placements of two paragraphs.

Another article (with a 1,938-word count) from our internal writers initially got a 72% human content score from Content at Scale. However, the human content score improved after a mere change in line spacing and became 92%.

These real-world examples indicate that users should consider AI content detectors as just one of the many tools used in a holistic evaluation of a writer’s work.

OpenAI Releases Tool to Detect AI-Generated Text, Including From ChatGPT

On January 31, 2023, OpenAI launched AI Classifier, a fine-tuned GPT model trained to differentiate between human-written and AI-generated text.

New AI Classifier for Indicating AI-Written Text

AI Classifier can help identify AI-generated text, including ChatGPT-generated text. The tool aims to counter false claims that a piece of AI-generated content is human-written. 

Limitations

Despite the tool’s potential uses, OpenAI stated that its newly released AI detection tool is not yet fully reliable. 

For example, based on OpenAI’s evaluations, the AI Classifier correctly identified 26% of AI-produced text (true positives) as “likely AI-written.” However, the tool incorrectly marked human-made content as AI-written 9% of the time.

Consequently, they advised users not to use AI Classifier as a primary decision-making tool but only as a complementary method to other tools. 

The company listed some limitations that you should consider when using AI Classifier, including the following:

• The tool becomes more unreliable on texts with less than 1,000 characters.
• The classifier works significantly better for English texts than other texts written in other languages.
• Content with very predictable text cannot be reliably identified as AI-generated.

Impact on Educators and Call for Input

AI text generators have raised concerns among educators about the potential for AI-enabled cheating. 

 Some educators are asking the AI community for input on how to detect AI-produced content in educational settings better.

OpenAI published a preliminary resource regarding ChatGPT, which lists the uses and limitations of the chatbot.

Training the Classifier

While significant limitations exist for the AI Classifier, OpenAI can update and retrain the language model based on successful attacks. 

For example, OpenAI tweaked the confidence threshold for their web app so that the false positive rate becomes low, meaning the app only marks text as “AI-generated” if the classifier is very confident.

Originality.AI (Professional Writing)

We also use Originality.AI, a paid tool, as a plagiarism checker and AI detector. 

The founder of Originality reported that 99.41% of the time, the company’s AI detector correctly identifies a text as GPT-3-, GPT-3.5- (the latest OpenAI model), or ChatGPT-generated.

Acceptable Detection Scores

At Digital Spotlight, it’s a standard policy that if a piece of content constantly shows less than a 25% AI score, we consider it most likely human-written. 

Checking Entire Sites

Originality.AI has a tool that checks entire websites at once. However, AI checks consistently showing low or high detection scores should be your most significant indicator of AI-generated content.  

Copyleaks AI Detector

Another tool that we use to build our dataset of AI% scores is the Copyleaks AI Detector.

The Copyleaks AI detector is a free tool that can help identify texts produced by almost any AI content generator, including ChatGPT. 

Our latest data shows that Copyleaks is also quite good at detecting AI content. However, the tool might be less accurate in AI content detection than Originality’s AI detector.

Like other AI detection tools, Copyleaks AI detector can simulate human learning to improve its AI detection abilities over time.

AI Content Detection With Full Spectrum Protection

Copyleaks uses full spectrum protection to detect a broad range of AI-written texts, from simple text generators to advanced deep-learning models. 

Users can benefit from the tool’s features below.

API (Application Programming Interface) Integration

Users can integrate Copyleaks AI Detector into other software applications using the tool’s API.

  If you are a business owner, you can use this feature to access the detector’s functionality within your company’s workflow and systems.

LMS (Learning Management System) Integration

Educators can access the functionality of Copyleaks AI Detector from their native LMS platform. They can use the tool’s plagiarism and AI content detection to verify the students’ submissions.

Copyleaks Platform

Users can also access AI detection using the Copyleaks single, easy-to-use interface to verify content originality.

Chrome Extension

In addition, Copyleaks AI Detector is available as a Google Chrome extension, enabling users to verify the originality of the web pages they visit. 

Consequently, you can check posts on your favourite shopping sites and news articles on social media. 

Which Languages Does Copyleaks AI Detector Support?

The Copyleaks AI Detector works across various languages, including English, Spanish, German, French, and Portuguese.

Content at Scale AI Detector

We also used Content at Scale AI Detector to collect AI% scores from various articles. Our latest data indicates that this tool has similar accuracy to Copyleaks AI Detector.

Human-Generated Content

Content At Scale AI Detector provides users with a human content score, indicating whether a text is AI-generated or humanly optimized. 

AI-Written Content

Like the AI detectors above, the Content At Scale AI Detector can help identify if the content comes from an AI writer, including ChatGPT, GPT-3, and other AI models.

Corrector App

This free AI detection tool is specifically designed to identify whether the content is partially or entirely GPT-3 algorithm-created.  

GPT-2 Content Detector

Generative Pre-trained Transformer (GPT)- 2 content detectors can detect texts generated by GPT-2 based AI. 

For example, Huggingface, a New York-based data science company, developed the GPT-2 output detector to detect OpenAI-generated content.

GPT-3 Content Detector

GPT-3 is a machine learning neural network trained using internet data to generate text of any type.

Examples of GPT-3 content detectors include Originality.ai, Copyleaks AI Detector, and Writer AI Content Detector.

These tools offer free and paid services. You can click their pricing tab to see what services best apply to you.

GPT-4 Content Detector

As of January 2023, the CEO (chief executive officer) of OpenAI has yet to confirm the release of GPT-4. This update implies that there are no existing GPT-4 content detectors today.

Crossplag

Crossplag is a fine-tuned model based on the OpenAI dataset. This tool can analyze 1,000 words at a time and only works for English texts.

Crossplag is another AI content detector designed to identify AI-generated content across a wide range of applications. 

AI Content Detector

Our data suggest that Crossplag may be less accurate than most existing AI detection apps.

For example, Originality.ai detected 60% AI content in “Article 1”. In contrast, Crossplag gave the same article a 1% AI content score.

Other Online Detection Methods

Other AI detection mechanisms can also help with AI content detection.

For example, another potential AI detection method is “watermarking.” A watermark for chatbots can signal AI-written texts. 

Some reports say that OpenAI, the creator of ChatGPT, is currently developing a watermark to determine output from its GPT text AI.

Technical Signs

Another way to detect AI-generated texts is to examine the content’s technical aspects, including sentence length, repetition of words and phrases, data accuracy, and generalities.

Verify Sources and Author Credibility

Suppose an author uses sources from questionable websites or asserts information without a source. In that case, the author may be using AI to generate content.

AI Detector Use Cases

You can use AI detection tools in various ways. Below are some areas in which AI detection apps can help you. 

SEO

Google’s policy prohibits programmatically generated content or content that does not create value (lacks originality). The Google algorithm will read any AI-generated content as spam, resulting in lost rankings.

Academia

AI content detection can help uphold academic integrity.

If you are a student, you can know if your sources are original or AI-generated. 

If you are an educator, AI content detectors can give you more confidence in the originality of your students’ output.

Publishing

Increasingly, AI-generated text appears in books and articles. AI detectors can prevent plagiarism, helping you guard your reputation.

Reviews

In recent years, text bots have taken over review sites, providing false assessments of products or services. With AI detectors, you can protect yourself against false and potentially harmful reviews from AI copy generators.

AI Content Detection: A Losing Battle Worth Fighting

The pace of development in AI content generation suggests that AI writing will become more capable of producing realistic, human-like content. 

For many, this scenario means the best AI systems will eventually succeed in evading AI content detection mechanisms.

However, while some computer scientists work to make AI writers more human-like, others develop AI detection tools to improve their detection accuracy.

Digital Spotlight’s current data suggests that AI content detectors can detect AI content.

Consequently, we believe high-quality content that provides real value to users remains a great asset to those working in the content creation industry.

Original, creative content by human writers provides fewer generalities and more valuable insights – human-written content is more relatable and engaging. 

The ‘Burstiness’ of Human Prose

On January 2, 2023, Edward Tian, a senior at Princeton University, tweeted his beta version of GPTZero.

GPTZero is a software application that aims to detect whether a piece of text is human-written or ChatGPT-generated. The app uses these two metrics to make its “judgments”: perplexity and burstiness.

Perplexity refers to the randomness of the text’s word choice, while burstiness compares this factor across sentences.

High perplexity score suggests that a sentence is human-made. Burstiness is the perplexity score of the entire content.

Detectors Without Penalties

Our data indicate that existing AI-writing detection tools have different detection accuracies. 

These results suggest that even high AI content percentage scores may only sometimes mean that the text’s author is an AI. 

Suppose you are an educator. In that case, you should use AI content detection software to provide feedback to your students instead of just focusing on penalizing plagiarized submissions.

A Long-Term Challenge

Even though current AI-writing detection tools are imperfect, any writer hoping to pass off an AI writer’s work as their own could be exposed as detection tools become accurate.

Higher Ed Adapts (Again)

The education system can adapt to the post-ChatGPT world by understanding what makes human-written texts different from AI-generated ones. 

For example, the Higher Education system can focus more on oral-based evaluations or optimize exams without using digital technologies.

The 2023 Artificial Intelligence Boom

Business owners, digital marketers, journalists, and other people who work in the content creation industry think that AI content is now part of the competition.

While some content creators see this development as a threat, Google’s current policy against “auto-generated” content shows that 100% AI-generated content is not a long-term solution.

Moreover, our data suggest that relying on 100% AI-generated content is not a long-term solution. The real-world results we have gathered so far indicate that AI detection tools will only improve. 

At Digital Spotlight, we value original and creative content. Our internal writers’ submissions undergo six AI content evaluations by our team of specialists. 

References

1. Catching a Unicorn with GLTR: A tool to detect automatically generated text
   http://gltr.io/
2. English Google SEO office-hours from November 2022
   https://www.youtube.com/watch?v=TepFVYrBVg0&t=559s
3. Spam policies for Google web search
   https://developers.google.com/search/docs/essentials/spam-policies
4. New AI classifier for indicating AI-written text
   https://openai.com/blog/new-ai-classifier-for-indicating-ai-written-text/
5. roberta-base-openai-detector
   https://huggingface.co/roberta-base-openai-detector
6. OpenAI CEO Refuses to Confirm If Chat GPT—4 Will Be Released This Year
   https://iblnews.org/openai-ceo-refuses-to-confirm-if-chat-gpt-4-will-be-released-this-year/
7. Edward Tian ’23 creates GPTZero, software to detect plagiarism from AI bot ChatGPT
   https://www.dailyprincetonian.com/article/2023/01/edward-tian-gptzero-chatgpt-ai-software-princeton-plagiarism