Can an AI critique human writing?

My previous post on OpenAI’s latest offering – ChatGPT – demonstrated how far the technology has come in essay writing.

ChatGPT is a large language model developed by OpenAI. It is capable of generating human-like text based on a given prompt or input. ChatGPT uses a technique called “transformer” architecture to process the input text and generate responses. This allows the AI to generate responses that are coherent and consistent with the input, while also incorporating its own ideas and creativity.* In a very short period of time, we have seen public releases of AI writing technology which can rival student-level writing. This includes the structure, accuracy, and formal essay conventions like inline quotes and analysis.

One of the criticisms of the current technology, however, is that it frequently invents content or takes itself down mysterious rabbit-holes which end in nonsense. I experimented with forcing accuracy, including specifying that the AI had to use “real quotes” and to provide “actual, real references” which I could manually check. This actually noticeably improved the accuracy and voracity of the output.

I’ve been playing around with ChatGPT since it launched, and I’ve tried many different uses. Outside of the essay-style writing required by students, one of the most interesting so far has been using the AI to critique my own fiction.

Everyone’s a critic

I write a bit of fiction – generally sf and speculative fiction. Every now and again I’ll go through a period which is more productive than usual, and submit a round of short stories to different publishers and magazines. Sometimes, these stories end up published. Often, they bounce back with polite rejection letters which often include feedback from the reviewers. The feedback is generally very helpful, and on occasion I’ve edited and resubmitted a story based on the feedback which has later been published.

This process – drafting, editing, submitting, receiving feedback – isn’t unique to short story writing. It’s a process that most writers will go through on the road to publication. I wanted to see if OpenAi’s latest technology could successfully augment some of this process.

Earlier this year I submitted a short story to a science fiction magazine. It passed the first and second rounds, but was rejected at the final round. I later submitted the story to a second magazine, where it was again rejected. Along the way, the story received some great feedback from the various reviewers, including the following comments:

The story was okay, but perhaps not excellent in its current form. It was slightly difficult to tell the points of views apart and I believe the author wished to make Cathyߴs nature ambiguous till the end, but there seemed to be something mechanical of her existence from the start (e.g. tipping her whole frame forward) that maybe gives too much away (had that been the intention).

All too possible future, great characters well written, great tension and an emotional ending.

The writing in this story is efficient and endearing. Setting and history are compactly conveyed. The characters are neatly conveyed in few words. I think the story needs a deeper plot to give these strong elements greater gravitas; the protagonist or his ߴintervieweeߴ could take some action, make some decision or sacrifice to achieve an outcome or bring some change. While the ߴreporter finds mythical creaturesߴ outcome is effective, I think there needs to be a little more.

This is a great concept that’s well-written and emotionally charged, but overall I felt that the journalist needed to play a more active role instead of finding the town/Cathy by luck. As it is, this piece seems more like a vignette that focuses on worldbuilding/backstory.

I really like this, but it could be better. The opening isn’t as compelling as the rest of the story – reconsider the start point. Revealing Cathy’s hometown as McAffery is probably unnecessary given that the arrival scene happens so quickly. There doesn’t seem to be any recognition from Cathy or interaction with the narrator, despite the fact that she has clearly recognised someone new is in town.Feedback from five different (human) reviewers

As you can see, the reviews ranged from lukewarm to positive. In terms of positives, the story was rated as being well written, conceptually sound, “emotionally charged”. As for the negatives: the points of view were problematic, the plot is shallow, and the opening a little weak.

The story sat in a (virtual) drawer until yesterday when I decided to see what our machine colleague might think of it. I pasted the story – about 3500 words in total – into ChatGPT and provided a very simple prompt: Write a 700 word critique of this story. List suggestions for improvements at the end.

Here’s what it came up with:

This story is set in a post-apocalyptic world where the regions outside of the city are desolate and inhospitable. The protagonist, a journalist, is on a mission to find a story in these dead regions, despite the warnings of his colleagues. As he drives, he reflects on the lifeless landscape and the challenges of his assignment. Meanwhile, a character named Cathy navigates the desolate region on foot, encountering another survivor named Amir.

One of the main weaknesses of this story is its lack of clarity and direction. The narrative jumps between the journalist’s point of view and Cathy’s without any clear connection or purpose. The reader is left confused and unsure of the significance of the scenes and characters. Additionally, the characters are underdeveloped and lack depth, making it difficult for the reader to connect with them or care about their experiences.

Another issue with the story is its lack of world-building. The post-apocalyptic setting is intriguing, but the story does not provide enough details or descriptions to help the reader visualize and understand the world. We are told that the regions outside the city are dead and inhospitable, but we are not given any information about what led to this state or what the city looks like. This lack of context makes it hard to fully engage with the story and its themes.

To improve the story, I would suggest providing more details and descriptions to help the reader visualize the world and understand the characters’ experiences. This could include descriptions of the landscape, the city, and the characters’ physical appearances and movements. I would also recommend providing more context for the post-apocalyptic setting, such as how it came to be and how the characters are coping with it.

Additionally, I would suggest clarifying the purpose and connection between the journalist’s and Cathy’s stories. Are they related? If so, how? This could be done through dialogue, narration, or other means of showing the connection between the characters.

Finally, I would recommend fleshing out the characters to make them more relatable and interesting. This could include adding details about their backgrounds, motivations, and struggles in this post-apocalyptic world. This would help the reader connect with the characters and care about their experiences.

Overall, this story has potential but needs more clarity, world-building, and character development to truly engage the reader. With these improvements, it could be a compelling tale of survival and resilience in a post-apocalyptic world.Critique via ChatGPT

The first paragraph is a fairly innocuous summary of the plot. As soon as we reach the second paragraph, however, things start to get more interesting. The first point it picks up on is the point-of-view: “One of the main weaknesses of this story is its lack of clarity and direction. The narrative jumps between the journalist’s point of view and Cathy’s without any clear connection or purpose.” This mirrors the first reviewer’s comment that it is “difficult to tell the points of views apart”.

Next, the AI critiques the “lack of world-building” and “lack of context”. Compare this to the human reviewers’ comments that the story “seems more like a vignette” that “needs a deeper plot”. The AI even uses the term “world-building”, as did one of the reviewers. Ultimately, the AI makes recommendations based on those flaws. One of them, “showing the connection between the characters,” echoes the final reviewer’s comment that there “doesn’t seem to be any recognition from Cathy or interaction with the narrator”.Want more of this? Join the mailing list

What does this mean for human writing?

Much of the discussion around these AI language generators has been about “cheating”. My own earlier experiments – writing essays and short stories – centred on how students might use these technologies to perform the kinds of assessment tasks we set in schools.

The implications of this approach are much more interesting. Had I presented my short story to ChatGPT prior to submission, I would have received very similar feedback to that which I ultimately received after several months of the human review process. I could have worked on these aspects, strengthening the relationship between the characters and the overall depth of the plot. In fact, I could have gone through this process multiple times. I re-ran the prompt several times, and each time ChatGPT provided simliar feedback. This included, “the characters themselves are also not well-developed,” and that the story “lacks a clear conflict or plot.” Subsequent runs also provided more specific and nuanced feedback, including:

The descriptions of the desolate regions outside of the city are well-done, but the constant mention of the “orange dust” that covers everything becomes repetitive and dull. The inclusion of the error code and file retrieval failure in Cathy’s perspective also feels out of place and adds nothing to the story.

The potential to iterate through writing with the support of a keen-eyed AI assistant is much more interesting to me than the simple threat of “cheating”.

The more we experiment with these technologies, the more we will find creative ways to work with artificial intelligence in education. If you’re a teacher, writer, parent, or in any way involved in education I’d encourage you to check out what ChatGPT has to offer.