Accessing Computing Education Research

This post is intended to be a guide to how to get into reading published research (especially computing education research), inspired by some discussion of how to judge the quality of a research paper.

What Use Is Research?

First of all: why would you want to read computing education research? It’s probably because you want to find out which education techniques have been found to work (or not work). The way I see it, a lot of empirical research is just the result of this process of finding out what works:

The meta-process of research.
The meta-process of research.

Ben Goldacre has talked a lot about how medicine used to rely too much on personal experience — essentially, doctors too often stopped at the top step in the diagram and didn’t appraise the problems in relying on their own experience. Things like randomised control trials and all manner of other methodological improvements only sprang up once they started applying the other two steps.

As a very recent example, I was at the CAS #include launch this week where, naturally, the issue of gender was discussed. One or two people mentioned that boys and girls learn differently. Do they learn differently? And is it enough of a difference that we should teach them differently? I don’t know, but this seems like an ideal case for turning to research to find out. (I hope to write another post on this in future, if I can find enough studies.)

The best research rigorously evaluates the effectiveness of an idea or reliability of a finding. This is especially important because the findings do not always follow intuition. For example, in science education, it has been found that live demonstrations are harmful to learning (but there are fixes). But research should not be taken as a complete panacea: research is rarely comprehensive, and there are rarely straight “yes” or “no” answers to any question. That’s life!

Finding Research

The first challenge is finding relevant research. This is usually best done by finding a research-specific search engine. For computing (including computing education) I can think of two:

  • Google Scholar — this encompasses all fields, which can be a mixed blessing. It’s not perfect — it will find unpublished papers (I’ll come back to that), and the wide focus means you’ll get more irrelevant results, but it does have Google’s typical comprehensive reach and search power. It also tends to have direct links to the PDF.
  • ACM Digital Library — this has the benefit of being restricted to computing, and only papers that have actually been published. You can also browse by conference/journal — for computing education at ACM, look for SIGCSE, ICER, ITiCSE in the proceedings section, and TOCE in the journals. The main drawback is that to actually access the papers you find, you’ll need to jump through hoops to then access the papers (keep reading to learn how…).

Another place to try is JSTOR, but I believe they only index journals, and a lot of computing-related content is in conference proceedings, not journals.

A Giant Web of Links

Research papers are a giant network linked together by citations. Once you find one paper you are interested in, you can follow its citations, find other interesting papers and follow their citations, and thus you can find a whole lot more once you have a starting point. It’s generally best to look for recent papers, for two reasons:

  1. It’s possible that old research is now discredited. If you read only old papers, you won’t know about the more recent developments. That’s not to say that old papers are wrong, but read the more recent papers that cite them to check that they are still cited as valid.
  2. Following references only ever takes you backwards in time, so you’ll find a better chunk of research if you start now and follow the references, than start in the 80s and head backwards from there.

Getting Hold of Research

Once you have found some papers that sound interesting, the next challenge is to actually get hold of them! Let me say this as an aside to researchers: if your paper is not freely available on the Internet, you are losing. Papers that are freely available online are more likely to be cited (see here, here), for the fairly obvious reason that more people can read them!

Google is your friend for finding research (other search engines are available)
Google is your friend for finding research (other search engines are available)

The challenge for those not at universities or other institutions with journal subscriptions is that a copy of the paper is not always directly available from where you find it (especially if you find it in the ACM digital library). However, many researchers do have a freely available copy on their own webpages (ACM allow this). When you find mention of a paper that you want to read:

  1. Google for the title! It’s an obvious step, but a lot of people don’t think of it (because they think there is unlikely to be a public copy anywhere else). If the paper does have a public PDF somewhere, searching for the title will usually find it in the first page of results.
  2. Google for the author(s). If Google doesn’t locate the paper, often the author will have a copy on their personal website or their staff webpage where they work. Don’t forget, they are quite likely to have moved institution since they wrote the paper — search for their name, not the institution declared on the paper.
  3. Email the author(s). Generally, I believe authors will be happy to email you a copy of the paper if you ask for it. However, they are often very busy and totally deluged by email, so this method is not guaranteed to get you a response.

Evaluating Research

Like any work, research varies greatly in quality — and so you will doubtless want to know how to identify quality work. I don’t mean to put a dampener on things, but the only reliable way to evaluate research is to read the paper and apply your brain. There are a lot of heuristics for trying to shortcut this, such as looking at the impact factor of the publication venue, or citation counts. These are quite fuzzy and I hesitate to recommend them. Journal papers tend to be of a slightly higher quality than conference papers, but computing research is published primarily in conferences, so narrowing your focus to journal papers will miss most of the work.

I will say this: one simple check is whether the paper has actually been published. Google for the paper title and check that you can find a record of it having been published somewhere. Publishing somewhere is not hard — don’t take publication as any particular sign of quality, but if the paper has never been published anywhere, that is a danger sign. And books don’t count: anyone can publish what they want in a book, but at least if it went into a conference or a journal then someone else faintly knowledgeable will have looked at it and decided that it’s not completely bonkers.

I would also stay away from looking at who wrote the research. It’s well known that good researchers can write nonsense — there is a term, “nobel disease”, which refers to nobel prize winners who go on to endorse pseudoscience. As a famous example, Linus Pauling won two Nobel prizes but got obsessed by vitamin C (with a lasting negative impact) and was wrong about quasi-crystals. Good research stands on its own merits, not those of its author.

Evaluate the work, ignoring who wrote it.
Evaluate the work, ignoring who wrote it.

Read a paper like you would check a student’s work: keep your brain engaged and look for holes in the logic. Did they only try an educational intervention with just one class or with only one teacher (especially if this was one of the authors)? Did they evaluate the outcome in an appropriate way? (Feedback questionnaires saying the students liked it is weak evidence by itself.) Most papers use fairly basic statistics — t-tests where you check if a p-value is beneath a threshold (0.05). If they used many t-tests this may be a sign that they went fishing for significant results. And even if the result is significant, does it actually matter? If doubling the time spent on marking produced a 2% average increase in exam mark, do you care?

Empirical research papers are not like mathematical proofs — one mistake or imperfection does not necessarily invalidate the results. There is no such thing as a perfect research study. But problems like small samples or poor evaluation mean that the research finding may be a chance finding, or is less likely to transfer to other settings.

Conceptions of Research

Research is often reported in the media as “Scientists say X” as if each study is a definitive answer decided by a cohesive cabal of scientists. More and more, I have the feeling that the media reporting on individual studies is like writing a news article every time someone scores in a basketball game. An individual research study is only a small part of the whole effort, which may provide evidence in both directions. It is only by looking a larger body of research (effectively, the score at the end of the game) that you can generally extract reliable knowledge.

Like a successful shot, individual research studies are often only a small part of the picture; meta-analyses are like the end-game score and summarise multiple studies.
Like a successful shot, individual research studies are often only a small part of the picture; meta-analyses are like the end-game score and summarise multiple studies. (Original photo by mvongrue, CC BY-NC 2.0.

The best papers for getting a digest-version of research are meta-analyses which look across an entire area and try to determine if there is a reliable effect. If all you want is a reliable take-away from research, just read the discussion and conclusions of meta-analyses. For example, googling for “algorithm visualisation meta analysis” turns up this paper. (It would be nice to see more sites like the Best Evidence Encyclopedia which attempt to distill research findings from meta-analyses.)

I also suggest following the maxim: don’t ask a question to which you are not prepared to hear the answer. If you go looking to find out if your favourite technique X is backed by research, the answer may well be “not really”. It can take courage to really try to find out, but you should be prepared to accept a negative answer, or there’s no point looking. For example: would you dare to consider if education itself is worthwhile? This paper (via Guzdial) suggests that education to instill financial literacy is costly and harmful to the students (it increases confidence without increasing competence!).

Getting Involved in Research

After you start reading some research, you may find yourself interested in getting involved, either as a participant or as a researcher. If so, I would recommend making contact with a university that does research in your area. I do not believe that research can only be performed by university staff, but there should be a few benefits:

  • If they know the research area, they may have a good idea of what has been researched and what has not.
  • They should have expertise in study design and running studies. No amount of analysis can recover from a bad design, so it’s important to get the design right.
  • They may have money to help run a study, or have a good idea of where there are grants available to help with research.
  • They also have expertise and practice in writing up the work and getting it published. Unpublished and undisseminated research is a waste of time — communicating the research to others is just as important as actually doing the research in the first place.


I hope this is a useful guide to those who want to make use of research findings, but don’t know where to start. In essence: search for what you want, use Google to find a PDF copy, prefer meta-analyses, read discussion/conclusions if you can’t stomach the whole paper, and keep your brain fully operational while reading! Here’s one paper to help get you started, entitled “How To Read a Research Paper”. Edit: and, just as I polished the final draft of this post: To know is not enough: research knowledge and its use (about education research in general).

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s