The AI diaries: how we really, really tried to find a use for ChatGPT
A few weeks ago, we attended a marketing conference in Bristol. We made it roughly eight minutes into the initial coffee-and-hello part of the event before someone asked us The Question: “So, how much is AI impacting what you do?”
We get it; everyone wants to talk about ChatGPT and the ramifications for writers these days. Us included – barely a day goes by without at least one conversation about AI, from the ethical questions around its use to how it will make it more difficult to cut through noise and cynicism.
Having seen generative AI at work, we’ve always been pretty confident that it won’t be replacing great human writers any time soon. But while it might not be up to crafting an expert-led article or inventive web copy by itself, we didn’t want to write it off entirely without giving it a good faith go at showing what else it’s capable of.
So for the last few months, I’ve been experimenting with adding AI into the work we do – not having it produce content for clients but asking what other parts of the creative process it could help make more efficient.
ChatGPT doesn’t have a creative bone in its binary code
Getting past the blank page
One of the first experiments I tried with ChatGPT was seeing if it could speed up the process of creating a blog outline. Not write a blog by itself – just to take our notes from a briefing or the transcript of a client call and order the key points into a skeleton we can then write from.
The results were mixed at best but let’s start with the positives. I was surprised at how well ChatGPT could identify the key points of a conversation from only a video call transcript and not only list them out but also provide bullet point summaries of what a client had said around each point.
But when it came to threading those points together into a blog outline, the limitations hit hard. For one, ChatGPT doesn’t have a creative bone in its binary code, so it couldn’t identify any narrative behind the key points it had picked out. It could only create an outline that followed the same course as the conversation, from one point to the next, neatly tucked between a school essay-style intro and conclusion.
It also struggled with the idea that each key point doesn’t need to be a separate section. If given little instruction, it would simply turn each element of the conversation into its own subhead. The result would be a dozen or more sections, plus intro and outro – way too many to squeeze into a +/- 1,000-word article.
If I got more detailed in my prompts, specifying 3-4 subheadings and asking it to group certain key points together to make one blog section, it would get confused. Sometimes it cut down its previous attempt to four subheads by cutting off the bottom eight rather than grouping things together. Other times it just hallucinated and returned any random number of subheads with an apology after for not being able to count to four.
To structure an article, you have to be able to discern – not just what was said in a briefing call but which points are most relevant to the topic at hand, and what narrative thread links them together in the way that’s most interesting to the person reading. And as it stands, that’s just not in ChatGPT’s wheelhouse right now.
The world’s most eager research assistant
Given that one of ChatGPT’s primary goals is to help people find information fast, I was curious how well it would function as a research assistant. Researching can sometimes be the longest part of writing an article. You know the right statistic is out there but finding it on a site that isn’t disreputable, out of date or, worse, a client’s competitor can be a slog. So any way that AI could speed that up would be a huge win.
There are two obvious problems to work around here:
- Hallucinations. The hardest thing ChatGPT can do is say it doesn’t have an answer, which means if it can’t find the stats you’re asking for, there’s a good chance it will make up results instead. You can ask ChatGPT to provide sources for its answers, but that doesn’t stop it from creating false stats – it just makes it quicker to fact check them.
- Recency. ChatGPT’s training data has a cutoff date, meaning anything published online after that isn’t in its knowledge banks. If you’re looking for anything from the last year (at time of writing, the free version powered by GPT-3.5 is only aware of events up to January 2022), ChatGPT simply cannot help.
But there was one other problem I didn’t expect to find when using ChatGPT to help with research. When it comes to research to support blogs, we’re not just looking for stats that back up what we want to write – we’re also looking for details we hadn’t considered before that spark new ideas and enrich the content.
For example, I was recently writing a blog about the importance of translation and localisation in the adoption of new tech products. When ChatGPT couldn’t return any relevant stats showing that localising marketing materials or UI does lead to greater adoption, I went searching myself. And while I didn’t find the stats I was looking for, I did find ones breaking down the proportions of non-English speakers in predominantly English-speaking countries.
It wasn’t the information I was originally looking for but it did lead me to a worthy point about the need for multilingual marketing even if you’re not planning to sell your products outside of one market – for example, Spanish in the US or French in Canada. But because ChatGPT is designed only to reply to the prompts you give it, it would never have been able to join the dots like this and suggest an angle I hadn’t yet considered.
Optimising articles for SEO
So when it comes to outlining blogs and sparking ideas, ChatGPT isn’t the best. But what about when an article is already written and just needs to be garnished with SEO keywords?
I figured this would be a simple experiment. I give ChatGPT the text of the article and a list of the necessary keywords, ask it to pepper them throughout the blog at least once each, and see what it does. SEO optimisation isn’t a particularly time consuming task, but any way AI could speed that up would be helpful.
Or so I thought.
On its first attempt, ChatGPT returned the exact text of the blog I’d pasted into my prompt. No alterations, no keywords, nothing. Just a word-for-word replica, and the proud assertion that it was a revised version.
After I pointed out what had gone wrong the first time, ChatGPT tried again. And to its credit, it did at least include most of the keywords this time – although they were all added on to the end of random paragraphs and with no accompanying sentence or context. Just… keyword.
I wondered if my prompt was the problem. I’d originally asked ChatGPT to add these words with minimal alteration to the blog itself, so for the third attempt I specified that it could make changes to the body text, such as adding new sentences in the middle of paragraphs, in order to insert the keywords. Just to be on the safe side, I also explicitly asked it to insert the keywords into paragraphs and not tack them onto the end as it had done before.
Third time’s the charm, right? Except… not quite. My attempts to cover off any loopholes or confusion in my prompt just led to ChatGPT returning the exact same copy as it had in its second response. The same keywords in the same places as before. I gave up and did it myself.
My theory on why this went so wrong comes back to the fact that ChatGPT can’t discern like humans can. Inserting SEO keywords might look like a simple find-and-replace job, but there’s more to it than that. To weave in terms without them feeling intrusive to the reader, you have to be able to look at the wider context of a paragraph or whole section. You have to understand which phrases are relevant to which key points in the article.
In other words, you have to use judgement and a little creative thinking – and as we’ve already established, neither of those are qualities ChatGPT can boast.
So… was using AI ever any quicker?
As I’ve been experimenting with AI, there’s one question that keeps coming to mind. It’s not “Is ChatGPT good enough to do X, Y or Z for me?” Instead, it’s “Am I just using AI for the sake of using AI?”
Let’s go back to blog outlines for an example. While ChatGPT falls short when it comes to narrative structures, there are some other kinds of content where AI can actually help. When we’re writing longer form pieces with a focus on SEO, it can help to have ChatGPT lay out distinct talking points in a linear fashion. And because it creates its responses from scraping its banks of training data, the subheads it suggests are usually pretty SEO-friendly.
But structuring a pillar page or a longform SEO piece is rarely something that needs speeding up. Unless you’re writing about a topic that has genuinely never been covered before, it’s pretty evident what information readers need to see and how you need to order it.
So does AI actually make this process any faster? Not really, no – or at least not enough for us to change our entire workflow and become part-time prompt engineers.
We’re all asking questions about AI at the moment – is it good enough, is it ethical, is it the thin end of the robot apocalypse wedge? Often those questions feel like they’re way too big for an individual to answer.
But there’s one question we should all be able to answer, AI expert or not – is using AI for this task worth it? Are ChatGPT, Bard, Dall-e and the like helping you to do something faster, to free up time you can better use elsewhere? Or are you just using them because they’re shiny, they’re new and they’re available?
Because if artificial intelligence can’t help us be any more effective or efficient where we need to be, we might be better off sticking with the intelligence we already have.
Back to hompeage
