ICER 2014 Roundup

In August I was at the International Computing Education Research (ICER) 2014 conference in Glasgow. This post is a round-up of my notes from the conference (for our own paper, see the previous post). Unfortunately, not all papers are publicly available, but if there is one that interests you, I suggest contacting the author directly for a copy.

Simon gave an interesting talk about academic integrity, pointing out that most university ethics policies on plagiarism, etc, are written in the context of essays, and don’t always apply well to computing. You are generally not allowed to take paragraphs you wrote for one essay, and re-use them in a subsequent essay. However, should you similarly be disbarred from taking a procedure you wrote in a previous programming assessment and re-using it another assessment? And what about group work? Some interesting questions, and they gathered opinions from students and educators, but there is no single right answer to many of these issues.

Colleen Lewis pointed out the CS Teaching Tips website that she is involved in running, which attempts to collect small suggestions for teaching computer science. I suggest any computing teachers reading this should take a look and contribute a tip or two — she has one or two from me, which is a clear sign that she needs some higher quality tips.

I enjoyed Michael Hewner’s ICER talk last year and this year, on students’ perceptions of Computer Science, and how they go about choosing modules in their degree. I recommend having a read of the paper — the interview snippets scattered throughout make it a more approachable read than many academic papers (last years’ paper, too). You may nod your head throughout, but it’s one of these pieces of research where almost any outcome will seem obvious in hindsight. But this does not mean it’s all as you would predict. For example: I would have predicted that what their friends took (or had taken) would be an influence, but that’s not the case. They also do not necessarily shy away from difficult courses or those where it is known to be harder to get marks.

I was pleased to learn about Parsons problems from Barbara Ericson and others (see an example here). These are programming problems where students are given some lines of code, and are asked to put the lines into order, including the indentation, to form a program that solves a given task (e.g. drawing a given shape in a turtle language). This seems like quite a nice way to provide a structured introduction to block-based programming.

Speaking of which, there were a couple of posters from PhD students Alistair Stead and Mark Sherman, who plan to look at issues surrounding the transition from block-based programming to text-based programming. It’s clearly going to be a hot topic of research for the next 2–3 years, whether it is researchers investigating the difficulties of the transition or building new tools to try to bridge the gap. I believe John Maloney (designer of Scratch) is working on one such tool with Alan Kay, there is another group doing the same that slips my mind — and our team is also building a new editor for Greenfoot to try to bridge this gap. It will be interesting to see what we all come up with! (Addendum: Shuchi Grover pointed to this recent paper on her work in the area.)

Leave a comment

Filed under Uncategorized

Educator Beliefs vs Student Data

Sitting in a talk a year ago, at ITiCSE 2013, I heard the speaker make a familiar statement: “We all know that [Java] students mix up assignment and equality.” My colleagues and I have heard many such claims about what the most common Java student mistakes are (messing up the semi-colons, getting string comparison wrong, etc). With the launch of our Blackbox project, we had the opportunity to investigate these claims at a large scale. And not just investigate what mistakes students made, but also what mistakes the educators thought the students were making. I’d just been reading the Sadler paper which suggested educators’ knowledge of student mistakes was important, and it seemed interesting to investigate this issue in the context of learning to program in Java.

The results are published at this week’s ICER 2014 conference, in a paper with my colleague Amjad Altadmri. The paper is freely available; this blog post attempts to informally summarise the results. We got 76 complete responses from educators, and we had available data from 100,000 users via Blackbox. We tweaked a pre-existing classification of beginners’ Java mistakes into a set of 18. The plan was fairly straightforward: we asked the educators to assign frequency ratings to the 18 mistake categories (which we then transformed into a set of ranks). Then we looked at how often the students actually made the mistakes in those 18 categories. For our analysis, we compared educators to educators (do they agree with each other?) and educators to students (do their beliefs generalise to all students?).

Student Mistakes

Let’s start with the student data. The top five mistake categories actually committed by students were, in descending order:

  1. Mismatched brackets/parentheses.
  2. Calling a method with the wrong types.
  3. Missing return statement.
  4. Discarding the return type of a method. (Note: this is contentious because it is not always an error.)
  5. Confusing = with == (assignment with equality).

Everything is obvious in hindsight; so how did the educators fare with their predictions of frequency?

Educator Beliefs

Now on to the educator data. Our first surprise was that the educators did not agree with each other very strongly (equivalent Spearman’s correlation: 0.400). There were a few mistakes where the educators agreed “no-one makes that mistake”, but these “gimmes” actually boosted the educators’ agreement and accuracy. At the top end, there was little consensus on which were the most accurate couple of mistake categories.

Given this lack of agreement between educators, it is not a big surprise that the educators did not agree strongly with the student data (if all the educators predict fairly different rankings then, automatically, only a small number can be close to the student data, whatever it turns out to be). The average Spearman’s correlation between each educator and the student data was 0.514. This graph is an attempt to show what that looks like:

Example Accord

The diagonal line represents perfect agreement between frequency rank in Blackbox (X axis) and by the educator (Y axis). You can see that the best educator, shown as empty circles, comes pretty close to this ideal. But the educator with the median agreement, shown as black squares, was not really very close — for example, they assigned the 12th rank (of 18) to the mistake that was actually the most frequent. The red pluses are the worst educator, who had a negative correlation to the actual results.

So, educators in general did not predict the ranks very well. But it could be that the less experienced educators in our sample (e.g. around half had less than 5 years experience teaching introductory Java) are dragging down the scores of more experienced teachers. We checked for an effect of (a) experience in teaching, (b) teaching intro programming, or (c) teaching intro programming in Java specifically. And we found… nothing. Experience (measured as a, b or c) had no significant effect on educators’ ability to predict the Blackbox mistake frequencies. In graphical form, here is an example match between experience teaching introductory programming in Java (X axis) and prediction of Blackbox data (Y axis):

A great example of a lack of correlation! If you want more details on the results or exact methodology we used, please read the paper.

Conclusions

So, what can we take from this study? Well, we could take the results and head off on a jolly crusade: “Teachers know very little about the mistakes their students make, and experience is worthless!” But we have explicitly not pushed this interpretation. There are several more plausible reasons for our results. Firstly, I think ranking student mistakes is a difficult task, and one that does not necessarily align well with teacher efficacy. Sadler’s paper looked at whether teachers could predict the likely wrong answer to a specific question; we ask here about most frequent mistakes across a whole term/semester of learning, which is a different challenge. It seems likely that this task is too alien to educators, and thus they all struggle with it, regardless of experience.

Another factor that came up in comments after the paper presentation (I think by Angelo Kyrilov, but maybe someone else too?) was that it may matter which environment teachers are using to program — IDEs like Eclipse may minimise the incidence of bracket errors when smart bracketing is used, in contrast to BlueJ (which all the students in our data set used). I don’t think this would explain the whole pattern, but in hindsight I wish we had captured this information when we surveyed the educators.

We can draw two fairly definite conclusions from our study. Firstly: beware any educator who tells you what the most common mistakes students make are. You’ll get a different answer from their colleague and they are unlikely to be accurate on a wider scale. (They may, of course, be spot on about their own class — but unless you teach their class, that is not really relevant to you.) Secondly: our results for the frequency of different student mistakes, available in the paper, must be surprising. After all, none of our participants predicted them!

Leave a comment

Filed under Uncategorized

Computing in UK Schools, Chapter Two

In spring of 2013 my co-authors and I published a paper on Computing At School in the UK. Due to the lead time in publications, the majority of the paper was written in summer 2012, and tweaked at the end of 2012. After that paper was published, the change from ICT to Computing in the English National Curriculum was proposed and confirmed, and CAS’s Network of Excellence project started, to try to fill the massive training gap that was created by these developments.

We have now written an updated account of what is happening with Computing in the UK (especially England, where many of the biggest developments occurred in the past year). Entitled “Restart: The Resurgence of Computing in UK Schools”, it has just been published in ACM’s Transactions on Computing Education journal, as part of a timely special issue on computing in schools worldwide. You can download the paper for free.

The special issue includes several other interesting papers on computing in schools in other countries. Alas, by default these papers are not publicly available. (As a starter, Mark Guzdial and Barb Ericcson’s is linked in the comments on his blogpost.) However, if any interest you, contact the authors to ask them for a copy, or even better, to make a copy publicly available.

Leave a comment

Filed under Uncategorized

Simplicity vs Complexity

Yesterday PC Gamer published an article entitled “No coding required: How new designers are using GameMaker to create indie smash hits”. The article discusses how several recent indie this were made in GameMaker, which allows either a drag-and-drop interface to specifying behaviour, or program code. Tom Francis, creator of Gunpoint, explains that the drag-and-drop interface allowed him to ease in to game making [transitioning from critic to creator]:

“The cool thing for me is I wasn’t confident enough to start coding, and so I was able to use these drag-and-drop actions to define how I wanted things to work on a very basic level.”

This idea of having a simpler layer than programming has occurred several times before in educational systems, such as flowchart-based programming systems (like the original Lego Mindstorms interface). The Unreal Engine Blueprint system mentioned in the article is another example. However, I think the most telling quote comes later on, again from Francis:

“Now I work entirely in code, and the drag-and-drop interface is, at best, a way of organizing code. But even for that I’m starting to think it might be better to just write it all in text.”

Computer science is often characterised as the fight against complexity. Structured programming is the best weapon for the fight; professional program-creation tools all use text-based structured programming, and even the most successful systems for younger ages (Scratch and co) use structured programming with a tweaked interface. The idea of easing in via another interface is interesting, and clearly worked for some GameMaker users. These simpler interfaces can work for simple programs, but as complexity increases, nothing has yet beaten structured programming.

Leave a comment

Filed under Uncategorized

Programming: A Series of Interesting Decisions

Imagine you are watching a Java programmer writing some code. Can you predict what they are most likely to do next in each of these situations?

  • The programmer moves the cursor to a new line, and types the three characters “boo”. What will they type next?
  • The programmer types “x = gteLength();” then gets an error that the method name is unknown. There is a method declared in the class named “getLength()”. What will the programmer do now?
  • The programmer types “ArrayList items;”. No class named ArrayList has been imported, but the standard library class java.util.ArrayList exists. What will the programmer do now?
  • The programmer has a variable named “old”. They have just renamed the declaration to “oldValue”. Now all the uses of the variable “old” are showing an error that the variable “old” does not exist. What will the programmer do next?

No prizes for guessing the most likely action in each case (but to be clear: type “lean” to make “boolean”, correct the typo, import the ArrayList class, change all the uses of “old” to use “oldValue”). These are so obvious that most IDEs now have automated support for helping in these situations. In Eclipse, Ctrl-Space (then Enter) will complete “boo” into “boolean”, hovering over the second error will offer a one-click option to correct the method name, pressing Ctrl-Shift-O in the third case will import the java.util.ArrayList class for you, and pressing Alt-Shift-R before renaming in the fourth case will rename all uses of the variable at the same time.

did-you-mean

One measure of a computer interface, such as an IDE, is the extent to which it reduces the predictability of your next sequence of actions. This may initially sound odd, but your next actions should always be hard to predict — if it’s easy to predict the next sequence of actions (e.g. “boo” being followed by “lean”), the interface should roll it into one action (code completion) or, if it’s 99+% clear what the action will be, just do it for you. Contrast the boo-lean case with what the programmer is going to type when they’ve just written “int ” in a method — it will be a new variable name, but it’s very difficult to predict what the name will be.

There are parallels to information entropy here — the letter E was one dot in morse code because it makes sense that the most frequent action has the shortest code. Similarly here, the most likely behaviour should have the shortest sequence of interface actions to trigger it. (You have to be a bit careful because the user might not want to fix the error immediately, especially in the third case. They may want to keep coding, so you don’t want to interrupt them with a dialog or anything too extreme.)

IDEs in Education

In a professional IDE, the goal is to support an expert programmer in writing good code quickly. However, our team makes educational IDEs, which have a different goal: to support novice programmers in learning how to program. Assistance can of course be positive — it can help move novices past the boring, frustrating parts of learning coding (syntax errors, obscure error messages) on to the interesting parts (like algorithms and design). However, assistance can also have downsides for novices:

  • If you present novices with a list of actions, they may not understand them fully, and may choose one just because it’s available. For example, “You are trying to pass a Person to this method but the type required is the Product interface. Do you want to (a) cast Person to Product or (b) let Person implement the Product interface?” The correct fix is to pass a different value as that argument, but that fix is not offered — a novice may think the only possible fixes are those in the list.
  • Doing too much in one action may lead to not understanding what assistance was given. For example, Eclipse has an option to pick a parent method from a class and override it. Will students understand why the “public void setLocation(int x, int y)” method in Crab overrides a corresponding method in Actor (but “public void setLocation(double x, double y)” does not)? At the extreme case, you don’t want Clippy popping up with “You seem to have written a bug in your quicksort implementation. Click here to automatically fix it”.
  • Too much assistance may be overwhelming at first. Some IDEs start offering completion as soon as you start typing a type or method name. If all the novice is trying to do is write their first variable declarion, do you want to present them, unbidden, with a popup list with all the available classes in the Java standard library?

There is no single right or wrong answer to these issues, and the question of how much assistance is the right level to provide. In the past, BlueJ and Greenfoot have offered only minimal assistance, preferring too little rather than too much. However, in our work at the moment on Greenfoot 3, we are moving towards having more assistance: code completion will be more featured (though I think it will still not popup automatically), and there will generally be more help offered by the environment in writing code and correcting errors.


Postscript: The title for this post comes from Sid Meier, who said that games (which, for him, means strategy games) are “A series of interesting decisions”. If there are no decisions to be made then there is no player involvement and there is not much point in them being sat there. If the decisions are obvious, then you are wasting the player’s time. The reason for the player to be involved is that there is a non-obvious decision for them to make. This seems to me to have a clear parallel in designing interfaces.

2 Comments

Filed under Uncategorized

The ugliest beautiful line of code

Some program code is considered beautiful, often because it accomplishes a great deal just by combining a few simple primitives. One such line in C is the loop that copies one string (const char* src) into another (char* dest), using only programming language primitives:

while (*dest++ = *src++);

(For those who don’t know C very well, “*src++” takes the value from the src pointer, then increments the src pointer. “*dest++ =” stores the value in the dest pointer then increments the dest pointer. The result of the expression is the copied value, and as long as this value is non-zero, the loop continues. Because C strings end in a zero, usually called a null terminator, this loop keeps copying from src to dest until the zero has been copied.)

The code does have an elegance to it. But the more I think about it, the more I realise that this one line manages to contain many of the flaws in the C language:

  • The expressions have both a prefix and postfix operator on the same item, which makes precedence very hard to guess. Is that (*dest)++ or *(dest++)? Both are reasonable answers, but you can’t judge it easily from the syntax. (See also “const char * p” — is that “const (char * p)” or “(const char) * p”?)
  • The loop body is a single semi-colon. This can cause problems for beginners, who instinctively put a semi-colon at the end of every line. For example, what does this code do?
    int n = 0;
    while (*dest++);
    {
        n++;
    }
    
  • The loop condition has a side effect, which is why no body is needed. In general, side effects in loop conditions and if conditions are a source of bugs because they make refactoring trickier and you are more likely to miss seeing side effects in conditions than other code. Not to mention that assignments in conditions are easily confused with equality checks. What do these two loops do: “while (*dest++ = 0);” vs “while (*dest++ == 0);” ?
  • The condition uses C’s ability to have any numeric type in an if condition, with non-zero meaning true. This is another source of bugs, as you won’t notice if you forget to add “== 0″ or “< n" on the end of a condition.
  • Perhaps most crucially, this code works because C’s strings are null-terminated. Unless you can guarantee that src has a null terminator within the bounds of the dest array (not src!), this is a buffer overflow that will write beyond the end of the dest buffer and trash your stack until src happens to finish. Oh, and if dest is pointing to a location in src before the end of the src string, it’s an overflowing infinite loop.

So what lesson should be taken from this, apart from the fact that C is a dangerous language? I wonder if we should beware any such terse elegant code. I’ve written some wonderfully brief code in Haskell before, and usually had to rewrite it again so that it is more easily intelligible for next time I look at it. A lot of people mock Java’s verbosity, but I at least find it reasonably rare that I have to deal with Java code that is too clever for its own good.

1 Comment

Filed under Uncategorized

Localisation of Programming Languages

Localisation is a technical term meaning the translation of software into foreign languages (German, Chinese, etc). This typically relates to a software’s interface: the translation of menus and text in dialog boxes into the local language. For example, if you set Greenfoot’s interface language to German, the “Edit” menu becomes “Bearbeiten”, Copy becomes “Kopieren” and so on.

Almost all programming languages have English keywords: if, while, etc. For a non-native English speaker, these keywords may well appear to be magic invocations, just as it would be to me if the keyword for “while” was a Chinese character. (One study suggests that the names for Java keywords are not very obvious for English speakers anyway, but at least they are in the right language!) In a block language like Scratch, there is the opportunity to localise the programming language itself: to translate the text in the blocks to the local language. The “change colour” block could become “Farbe ändern”. [Note: that one is picked from Google Translate, excuse any mistranslation] Is localising a programming language worthwhile?

I think the value of localising a programming language depends greatly on two things: the age group you are pitching your programming language at, and the control you have over the programming language and libraries. If you are aiming at six year olds then it is unlikely they will have had much chance to learn much English, and especially going outside their native character set would be confusing. (I believe that ScratchJr uses pictures for instructions partly to solve the localisation problem, but also to some extent to overcome reading weaknesses at such young ages.) The languages used in systems for this age group tend to be custom-designed, so the designers of the system have full control over the small language and libraries. So for small systems at young age groups, localisation is feasible, and there is a lot of benefit to be gained from it.

However, consider a system aimed at universities. For better or for worse, all major professionally used languages are in English. Java, C#, Python and the rest have English keywords, and (perhaps even more importantly) English APIs. Even if you provide a thin pre-processor to translate the keywords, you can’t feasibly do the same for all the APIs. There is no way you can translate the entire Java ecosystem and its documentation into another language (e.g. French). So even if you created an educational language, X, which can be fully localised into Arabic, is there much advantage to using it at a university when the students’ next language will be Java and they will have to learn English anyway?

Edit: I note that being able to put variable names in your local language is definitely useful for the programmers, but because the APIs are in English, there will still be a lot of English throughout any code; programmers will need to know English even if they don’t use it everywhere, and whether the keywords are localised doesn’t make a whole lot of difference as they are easy to learn over the long-term.

Summary

The older (and thus closer to professional programming) that you aim, the less sense it makes to try to translate the language. In Scratch 1.4, you can only use the blocks provided with the system, but in a language like Java, the core APIs (even List, String) etc are in English, and so are all the add-on libraries (e.g. to talk to MySQL), so you will have to know English to read the documentation and use the methods. Trying to localise programming languages at older ages seems to be a fool’s errand, and only protects students temporarily from the inevitable truth: to be a professional programmer, you need to learn English.

Leave a comment

Filed under Uncategorized