In order to fully grasp what ChatGPT can and cannot do, it is important to understand (on a very basic level) how it and other so-called Large Language Models (LLMs) work. The information on this page is primarily written within the context of ChatGPT (and more specifically the May 24 version GPT-3.5, which is the free version of ChatGPT), but the information applies to other similar generative AI models as well.

Provided with input (prompts) from users, ChatGPT uses models to produce its output: it generates text. This text is generated in a stepwise manner. At every step of the process, the following question is asked: what would be the most probable next word? For example, when provided with “ladies and”,  the model might predict that “gentlemen” is most likely, and this word is added to its output (see figure 1). Left with “ladies and gentlemen”, it goes through the process again: which word will most likely follow? These steps are repeated, until the best (or, most probable) option is to end the sentence or text.

Figure 1

That leaves us with a question, though: how do the models predict what the most likely next word is? To understand, we first need to learn some basic concepts of machine learning. In machine learning, large datasets (called the training data) are used to train a model. This model can be seen as a mathematical representation of how the AI tool ’understands’ the data and finds patterns in it. The model is then applied to new data (that was not in the training dataset) and can provide new output.

In the case of ChatGPT, the training data consisted of an incredibly vast amount of text (approximately 300 billion words) from a wide range of different sources from up to September 2021: social media, websites, scientific articles, literary sources, and so on (Hughes, 2023). This data was used to train a model consisting of complex mathematical functions (with 175 billion parameters!) that allows it to predict which word comes next in a text. After this step, further methods were (and are still being) used to improve and update the model. Human workers ranked ChatGPT’s output and gave feedback. Besides looking at output quality, they paid attention to controversiality and harmful content. In addition, any user can provide feedback on ChatGPT’s output via the thumbs up and down icons. All of this feedback is then incorporated when updating the model.

The above is important for two reasons. First, it means that ChatGPT is a black box. Even if we were to get insight into (any one of) ChatGPT’s 175 billion parameters, we would not be able to understand them. There is no way for us to understand or trace how ChatGPT produces its output. This is a challenge when using it for academic writing, in which it is very important to acknowledge your sources. The second important point is that its models consist of ’just’ maths. During no step of the process of generating its output does ChatGPT engage with the contents or meaning of any actual data. Its output is based only on continuous mathematical predictions of what the next best word is. This means that the (un)truthfulness and meaning of its output is entirely coincidental.

We cannot deny ChatGPT’s potential nor its usefulness in many different applications  but we should be mindful of its limitations, their implications for education, and the ethical issues it entails.