AI and Writing: Detection….the next level.

It’s impossible to scroll more than a few seconds on social media without encountering someone mentioning how artificial intelligence is taking over the world & that everyone will probably be out of a job soon.

As great of a sci-fi movie as that would become, I don’t really think we’re at the point where AI can fully replace employees yet. That’s for another article.

Now can employees leverage AI to optimize their workflow & climb the work ladder faster? Absolutely. So much of a desk job can be sped up using AI (without any special training, too!)

From the release of ChatGPT to more nuanced writing tools like Jasper and Copy, AI is quickly becoming one of the most powerful tools in a digital employee’s arsenal.

Though many including myself raise the question:

Can anything written with AI be detectable? How does AI writing detection work?

The honest answer is not really.

AI detection is based on predicting patterns, and as large-language models continue to evolve, it’s getting harder to separate these robotic pieces from human-written ones.

Why Is AI Text Detection Even Important?

AI is increasingly being used to create content in fields like journalism, digital marketing, academia, and even law.

Yes… law! Some lawyers thought they could get away with using it in the courtroom.

It’s not that AI-produced content is necessarily bad, but shouldn’t you have the option to determine what is and isn’t AI writing?

One of the biggest concerns going on is the importance of academic integrity. While AI tools offer numerous benefits, using AI-generated content in academic or professional settings without proper attribution can lead to severe consequences (and that’s if one gets caught using them).

What’s crazy is that 65.8% of people believe AI content to be equal to or better than human-written content!

Sometimes academic researchers can’t even detect the difference between the two.

The safety implications that could result from a world where ChatGPT-produced thesis papers are published could certainly result in some dangerous things. Even if you can use AI to cite sources, you reach an ethical quandary.

Identifying AI text with a ChatGPT detector can help a person evaluate the efficacy of content up to a certain human standard, but it’s getting harder with how incredible AI writing has gotten in just the last few months.

On the bright side, knowing which pieces of copy have been written with AI can even help certain businesses decide if it’s worthwhile to keep their writing teams.

Certain tasks simply just don’t require much human creativity and can be outsourced. If you run into a good piece of marketing copy written with AI, you may benefit from using the same AI tools to recreate it yourself.

What Does AI Writing Detection Mean?

At its core, AI writing detection relies on reverse engineering language patterns to determine predictive text. This means that the machine breaks down a piece of text & then uses algorithms to detect patterns within those words.

If a pattern is easier to identify – it’s more in tune with what an AI would write, increasing the odds it was written by AI.

The first letter in AI is actually really important – artificial. That’s what separates humans from machines. AI writing detection is further based on noticing the differences in how words are arranged and used.

Machines write text based on the billions of data & patterns it was trained on, while natural human writing utilizes an aspect of creativity that can’t easily be reproduced by one of these bots. More on that later.

AI detection is also about context. As persuasive as it may seem, machines don’t actually understand the meaning of words, but they can identify patterns that are commonly used really well. They can also pick up on repeated phrases & words, which is often a tell-tale sign of automation or copy-pasting.

Is AI Writing Detection Accurate?

No, not really. AI Writing detection tools are really just a prediction giving insight into where writing came from. If you suspect writing as being AI-written and run it through a detector that says 100% AI, it probably was written by an AI tool like ChatGPT.

If you see anything under 80% I wouldn’t be too confident about accusing it as being AI. There’s just too many unknown and unprovable variables to account for. Just look at what’s happening to students across the world.

Take all the results you see with a grain of salt, as they really are just predictions.

Their have even been tools that claim to bypass AI detection. Tools like Undetectable.AI and HideMyAI try to get around detectors, making the problem even worse.

How Do AI Detectors Work

So now that we’ve gone over why detection is important and a little bit about what it means – we’ll explain what goes on behind the scenes.

Tools that “predict” AI content are largely based on analyzing the context to the left of the following word.

Imagine the sentence “The best part of my day is when I wake up for ___.” In this example, work is the most commonly predicted word based on the 117 million data points the GPT-2 language model was trained on.

The AI model will think back to all of its training data, then identify and analyze patterns in the context of the word set. It might know, for example, that the word “day” is often used after the words “best” and “part”. The algorithm will then calculate the likelihood of each word being the next predicted word, based on these contexts.

Based on training data, the word work had a 41% chance of occurring (the highest probability compared to other words), so it predicted it.

Perplexity & Burstiness

Some other words you might see used when working with an AI detector are perplexity and burstiness.

Perplexity measures the unpredictability of a text. In simpler terms, it measures how often a piece of text might confuse or ‘perplex’ a reader. AI models strive for low perplexity, aiming for content that reads smoothly and logically.

Human writing often has higher perplexity, with more creative language choices and occasional errors.

Burstiness, meanwhile, evaluates the variation in sentence structure and length.

Temperature

An important thing to understand when working with AI-generated text is the concept of temperature. Temperature probability is a measure of the randomness of predictions. If the temperature is low, a model will probably output the most correct text, but it will be quite boring as it has a smaller degree of variation.

If temperature probability is high, the generated text will be more diverse – but includes a higher chance of the model producing grammar mistakes or straight nonsense.

Consumer-facing AI text generation tools like Jasper and ChatGPT seem to err on the side of caution. Although ChatGPT responses produce larger variations than what Jasper does, they are still fairly predictable models.

If you’re using an online tool to help write content, you’re working with pre-trained models (which generally set conservative temperature probabilities to reduce mass errors)

So after calculating this for a single sentence, keep going with the rest of your text. If a sampled piece of text consistently selects the most predictable word throughout paragraphs, you’re almost certainly working with artificially generated text.

Think about it from a personal perspective – the best writers often make use of complex language and explain things in unpredictable, creative ways. Artificial writing doesn’t.

As language models become more and more complex, predicting AI based on the context of words will become a lot harder. The more data in a set, the more variability in generations. But for now, you could follow this pattern to analyze large chunks of text.

It’s extremely simple in concept: To what extent can an AI model predictively regenerate a given example of text?

So How Accurate Is It?

It’s crucial to differentiate between AI detectors and plagiarism checkers.

While both tools aim to ensure content originality, they operate waaay differently.

AI detectors analyze text characteristics to determine if it’s AI-generated. Plagiarism checkers compare content against vast databases to identify potential copying.

One of these has a direct source, one doesn’t (AI). You might be able to get a prediction based on running a sample of writing through a detector, but just know you’re not looking at anything that can be proved. OpenAI discontinued its official detector a while back. More recently, they said AI detection in general doesn’t even really work.

Best Tools To Detect AI Writing

Besides using math, there are grammatical and syntactical ways you can help identify if something was written with AI, but you could do that just from reading.

So how do you determine what percentage chance the context of a word has? Well for starters, you could use a few online tools.

We wrote a larger article on how to detect AI content, but depending on what kind of writing you’re checking you could use CopyLeaks (free) or Originality (paid).

If you want to check academic, industry, or professional content (especially in mass) – look into Copyleaks. It’s free and is better than most other tools out there. We wrote a full review testing it out in much depth.

If you want to check for marketing or copywriting material that got forged using ChatGPT, I’d say Originality is a more in-depth and comprehensive choice.

Originality lets you check for AI, plagiarism, and gives you a percentage that it believes a block of text was written with AI. Check out our full review to see everything it can do.

Detecting AI Writing Manually

Beyond automated tools, you could do a little more digging yourself. I won’t lie – it’s very hard. It’s getting extremely difficult to detect since these generators are just able to write more and more like humans.

If you’re looking at poorly-generated AI writing, these texts are often written a monotonous tone, have predictable word choices, and lack bold, original statements.

AI-generated content might lean heavily on hedging phrases or display inconsistencies in voice and style.

The Future of AI Writing & Content Detection

Whether you like it or not, there’s really no guaranteed way of determining if something was written with AI at this point in time. After ChatGPT went viral, tons of questions and concerns were raised about how this will impact the world: education, industry, and even literature.

The realm of AI isn’t limited to text. AI-driven image and video generators are on the rise, making it super important to detect AI-created visuals, especially with the proliferation of ‘deepfakes’.

Although a guest researcher at OpenAI revealed that they’re developing a tool for “statistically watermarking the outputs of an AI text system.” Whenever a system generates text, the tool would stamp an “unnoticeable secret signal” indicating where the text came from.

If other companies follow suit, we might mitigate some of the ethical dilemmas raised by this new technology. I’m just not sure if it’s even mathematically possible to do.

Regardless, an unpredictable storm started and won’t be over for a damn long time! Generative writing tools are only going to get more nuanced, more creative, and eventually more complex.

For now, it’s best to use your intuition combined with detection tools if you’re skeptical. The next few years are going to be very interesting and fun to see what’s in store. How long until artificial intelligence can seamlessly integrate with our society? We honestly might already be there.