Errordle: a serious game

In January 2022, the online game Wordle soared in popularity. It’s a word-guessing game where you guess a five-letter word and are told for each guess whether a letter is in the right place in the target word, present in the solution but in a different place, or not present at all in the target word. I fully expect that its popularity will crash soon enough, like most fads, although not before the NYT acquired it.

At the end of January 2022 I (virtually) attended a Dagstuhl seminar on the human factors of error messages, and one of the organisers, Brett Becker, put a call out for fun activities for the opening Sunday night session. So I came up with Errordle: an error message guessing game. I was fortunate that others had already written an open-source clone of Wordle that I could adapt.

Errordle has the same basis as Wordle, but you guess words in a target Java error message, rather than letters in a target word. As a hint, you are shown the single line of Java code on which the error occurs, according to the compiler. (Technically: the line where the error starts, in the case of multi-line errors.) Your job is to then guess the error message by putting together words that can occur in the error message.

Errordle is now publicly available and you can play here: https://bluej.org/errordle/ If you’re interested, read on for details about the data, some privacy and technical considerations, and some reflections on it.

Blackbox Dataset

The source code lines and error messages in Errordle come from the Blackbox Mini subset of our Blackbox dataset. This means that they are all real error messages that occurred in the wild, for some random user of our BlueJ Java IDE who has opted in to data collection. I did handpick the final set for this public version, both for privacy reasons and to provide a varied challenge.

The Blackbox dataset records participant’s full source code with light anonymisation, and as such some of the data can still contain identifying details. For this reason the dataset is not public, and we cannot expose the complete source code that produces the error. So unfortunately you will only see the single line of code that is exposed, and may well believe that the error cannot possibly be produced by that line. More on that at the end.

The errors are not the full range of Java error messages. I restricted the errors to those that are five words or less, and it turns out that this excludes all type-related error messages, and only leaves some syntax error messages, which tend to be terser.

Some Reflections

Participants at the Dagstuhl seminar seemed to enjoy the game. I ran a wrap-up session late in the conference where I provided my own reflections:

• The “keyboard” of the words in the errors that you use to make a guess shows the words learners are confronted with. On the whole, they are a mix of technical terms (class, interface) and negative words (illegal, missing).
• There are many options for the exact wording, even knowing what the error should be and with a limited selection of words. There is not noticeable consistency between the messages.
• Many errors are not complete sentences. (Although this is slightly biased by the restriction of five words or less!)
• The line indicated by the compiler as the error location is often not the cause of the syntax error. It depends on the lines before to fully understand the error. I’ve deliberately made the game frustrating by only showing one line; this is what Java learners have to deal with, if they focus on the red underline showing the error and do not realise that they need to check the lines above.

So, go and play: https://bluej.org/errordle/

Advertisement