The perfect grading strategy, or: How to benefit from your own research

Grading exams is one of the burdens of being a teacher, especially when it is a large course. My biggest course has approximately 120 students, which means that right now 120 exams consisting of 6 essay questions each are sitting on my desk.
What is the right grading strategy? There seem to be two strategies.
One is to do one exam at a time, and grade all six questions. There are several problems with this approach. The main problems is the risk that your standards start drifting: if students are bad on average you may become more forgiving, or more stringent if they do well. The second problem is the change of mindset necessary after each question, which is mentally taxing and time-consuming.
The second strategy is the one I think most people use, which is to grade all question 1's, then shuffle the whole stack of exams, grade all question 2's, etc. This seems to be the perfect solution that solves the problems of strategy 1: your standards are less likely to drift, and if they do the effect is more randomly distributed. You also don't have to shift mindsets, because you grade the same question over and over again.
So isn't strategy 2 the perfect method? Unfortunately not. When reading and grading answers that are all similar but not the same it becomes increasingly difficult to separate them in your mind. Each new version of the answer needs more mental focus to keep it apart from the previous answers from other exams. That is why I often use this as an example of a case where interruption is beneficial instead of harmful: an interruption helps clearing the mind of previous answers, reducing interference. But regular interruptions do not, of course, speed up the process of grading, and as we all know interruptions often lead to more distraction.
The solution is so simple that I wonder why I haven't thought of it before. Isn't it obvious: grade TWO questions per exam, so first all questions 1 and 2, then all question 3 and 4, etc. Alternating two questions delivers just the amount of memory interference needed to disentangle the current answer from previous answers. Keeping two answer standards in mind is still doable, and the standard drift is also reasonably under control.
I am now halfway through with grading, and although there still a lot of work to do, I can report that it works very well, and I can recommend it to everyone in the same situation.

Seven plus or minus two: hard to eradicate

The website of "de Volkskrant", a Dutch newspaper, had an interesting bit of news: ToDo lists don't work. The article referred to a blog of the Harvard Business Review by Daniel Markovitz. One of the main claims of that article is that our brain cannot handle more than seven choices, and will therefore be overwhelmed when a ToDo list is longer than seven items.

Seven plus or minus two has been a staple of Cognitive Psychology since Miller published a paper on the limitations of short-term memory in the fifties, which led to the naive theory (not endorsed by Miller himself!) that human short-term memory consists of seven (plus or minus two) slots in which items can be stored. But even though cognitive psychologists have abandoned this theory for decades, it still floats around in the applied domain as a serious limitation of human cognition.

To get back to the original topic, I always find ToDo lists quite useful. From a multitasking perspective they perform a very useful service, helping you keeping track of your uncompleted goals. So now this blog claims that these lists do not work, because your brain can only handle seven uncompleted goals.

The cited research for this claim is a set of studies that show that people want choices, but not too many of them. Being able to choose between 3 brands of detergent is nice, but choosing between 20 brands is ridiculous. So, what's the threshold? Seven. Really seven? In many situations I'd rather not choose at all, that is why I like house brands in supermarkets. Maybe in a restaurant I prefer some more choices on the menu than just seven.

But even if this were all true, the generalization from choosing between similar candidates for a purchase and which item to do next on a ToDo list is pretty far-fetched. And the idea that your brain will overheat if the length of your ToDo lists exceeds the threshold is pretty ludicrous.

To conclude: I think ToDo lists are pretty useful as a memory aid. But maybe the lesson is to be really, really suspicious about any type of psychological advice that involves the number Seven. Plus or Minus Two.

Rationality in Multitasking

There is typically a big gap between the tasks studied in psychological experiments and real-life cognition. This is because psychologists want to study the individual psychological functions in great detail, and therefore develop tasks that try to isolate these functions. Although this offers a wealth of information, the danger is that the interaction between those functions is not well understood. For example, in the example of the attentional blink that I discussed earlier, researchers still do not agree on whether the phenomenon is due to a attention, memory or control.

When Dario Salvucci and I were working on our book on multitasking, we also found an example of this gap. Observational studies had shown that the cost of interruption can be very large, in terms of tens of minutes. However, interruption costs in experiments are typically in the order of only a second. To study interruption behavior at a more realistic scale, Dario and his graduate student Peter Bogunovich designed an experiment that was in between real life and a basic experiment. Subjects had to answer emails for which they had to look up information using a web browser. Occasionally though, a chat window blinked in the background. If subjects clicked on this window, a question about movie preferences was asked which they then had to answer.

The important aspect of the task was that subjects had complete freedom in when they wanted to switch between email and chat, something that is typically not done in experiments. They could switch to the chat right when it would come up with a new question, or they could wait until they were done with the current email. The mail task was structured in such a way that information had to be remembered. For example, in the example in the picture, the price of the mp3-player Killor U-32 had to be looked up, which takes three clicks in the web browser. In the real experiment (as opposed to the picture) the windows were always on top of each other. So, when you have just clicked on "mp3-player" in the browser, it is not very clever to switch to the chat, because then you probably have forgotten "Killor U-32" when you get back.

It turned out that the subjects in Dario's experiment were indeed smart about this, and continued on the mail task until they reached a smart switch point, a point during which they did not need to remember any information.

Together with Jelmer Borst and a group of project students (Joost Timmermans, Anita Drenthen en Tom Janssen), we redesigned the experiment in order to try to tempt subjects into switching to the chat window at moments that information from the email had to be remembered. We did this by introducing delays in the web browser (after clicking a link it took a few seconds before the subsequent page appeared) and the email program (it took a second for an email to load). As a result, a substantial number of subjects now switched from mail to chat during a delay in the browser, at a moment when information had to be kept active. Moreover, the average time needed to answer an email with delays was longer than without delays, even if all the delays were subtracted from this time first. In the strongest condition the extra cost was more than 6 seconds (ok, still not minutes, but better than just a second). In other words, subjects would have been better off if they had just waited during delays instead of making a switch. Trying to use the waiting time for something else turned out to decrease efficiency instead of increasing it.

What these experiments show is that people are typically smart about their choices in switching from one task to another. But delays can surely thwart this rationality. Apparently, we'd rather act than wait, and that is probably why I prefer taking the bicycle over waiting for the bus, even though the latter option might get me there faster.

Become a better multitasker through... sport!

Last week I gave a talk about multitasking in a symposium "The Science and Art of Brain Maintenance". Two of the other speakers in the symposium, Chris Visser and Erik Scherder, talked about the benefits of physical exercise. We all know that exercise is good for your physical health in a great number of ways, but Chris and Erik reported on research that shows that exercise can improve your mental prowess. Chris' research demonstrates that children that are good at sports on average also perform better in school on mathematics and language. Erik showed that exercise during your lifetime is correlated with a lower incidence of diseases like Alzheimer's and other forms of cognitive degeneration.

At first this may sound surprising, but if you think about it, sports is not just a matter of physical practice but also of mental practice. Sports requires motor coordination, discipline and skill learning. According to Chris and Erik, the main cognitive function of importance that connects physical and mental prowess is executive control. Executive control, located in the frontal areas of the brain, concerns our goals and the handling of our goals. It keeps us focussed on what we do, help us in determining our actions in the absence of outside information, and plays a role in juggling all the items that we needs to maintain in working memory. Sounds familiar? Those are exactly the functions that are needed for multitasking.

In my own talk, I showed a picture of brain pictures associated with sequential multitasking: an area in the frontal cortex and an area in the parietal cortex (made by Jelmer Borst). To my surprise, both Chris and Erik showed pictures of roughly the same areas.

If we connect the two together: physical exercise improves executive control, and executive control is the main factor of success in multitasking, then we can conclude that physical exercise can make us better multitaskers.

The pitfall of this line of reasoning is that most evidence is correlational. We cannot be sure whether physical exercise actually causes better executive function. Apart from that, it is an interesting thought that this is another reason why body and mind are maybe not as separate as we tend to think.

What makes a Genius?

After last post's New York experience, I will add a new one about one of the other hubs of multitasking frenzy, London. What do you do when an Apple laptop breaks down on a trip abroad? You go to one of their Genius Bars! Or, the story of how we spend half a day in an Apple Store.

Let me try to explain this without too much technical detail. The problem with the laptop was that it didn't boot, and this problem was caused by repeated attempts to power it up on an empty battery. The attempts made by our Tech gave some interesting insights in how these people are trained. What he basically did was attempt a series of solution procedures that he selected on the basis of association with the problem. Our two "symptoms": computer did start up, but got stuck in the process, and something wrong with power. In this case, the power was a red herring: the real problem was the computer not booting. But before acting on either symptom the first procedure the Tech did was to hook up the computer to a standard hardware test. This showed up no problems with power nor hard disk, but did show up a problem with some sensor: the second red herring. This is something that sometimes happens in the medical profession as well: you go to the doctor with a problem, they do a test, and you end up with two problems.

As a result, the Tech went backstage to try and push all the sensors back in place. An hour later he was back from doing this, with no result. No surprise in retrospect, because the sensor couldn't possibly have been the problem. He then started going through a whole bunch of procedures related to power, up to verifying that our battery was past its expected lifetime (which we already knew), and verifying that the power adapter was indeed working fine.

In the end he did not solve the problem, but I walked away with a pretty good idea what was wrong and fixed it when we returned home.

The point here is not to criticize the Tech. He was obviously not a genius in the normal use of the word, but he was trying his best, but probably on the basis of the wrong strategy prompted by the wrong training. His approach, to reiterate, was to throw solution procedures onto the problem based on associations with the symptoms. What he should have done is try to reason out what possible causes there are of a computer not booting. Defective hard disk? No. Will the computer boot from another source? Yes. Then there must be something wrong with the OS: it needs to be fixed or reinstalled.

So why is this story important for multitasking? In order to be a good multitasking, your skills have to be flexible. When I am cooking from a cookbook, I have to pay close attention to the order of the steps and the ingredients I need to gather. I cannot really multitask very well while cooking from a recipe. But when I am not cooking from a recipe, I can happily watch the news, chat with people and do other things. So why is that? When I am not working from a recipe, the representation of what I need to do is flexible, and I can immediately respond to what I see happening in the kitchen with the right action. But when I cook from a recipe, I have to repeatedly consult the recipe and check what step I am in and what to do next. In the recipe case, I have to maintain a mental representation of the state of the food, and can therefore not afford distraction, but if I cook without recipe the world is my mental state.

Training in terms of learning procedures is very common. I carried out some studies with airline pilots who need to program their onboard computer, and are also trained in doing this by memorizing procedures. The airplane computer is a masterpiece of user unfriendliness, so some pilots despair in using the thing, and training on it is sometimes compared to "drinking from a firehose". The main reason why pilots have so much trouble is that they do not understand what they are doing. They are just carrying out procedures (did someone say "Chinese Room"?).

Our lives are full of procedures: look at any electronics manual, medical practice, pilot training, probably Apple Genius training, etc. But opaque procedures turn us into automatons, and blunt our capacity for creative problem solving. And that while it is not so hard to change things. In our pilot example, just adding an explanation of what each step did increased performance on the task tremendously, and also allowed subjects in the experiment to come up with new procedures for problems they were not trained on.

So what makes a true Genius? If I only knew... But improving training and instruction may be a way to make us all a bit smarter. And better multitaskers too.

Reading a book is also multitasking

Reading a book is often considered as the epitome of monotasking. Indeed, it is the activity that is considered by some as the endangered species in the multitasking world. Well, bad news for you nostalgics: reading a book is also multitasking.

Why? This only became apparent to me in the hub of human multitasking, the New York City Subway. People do a lot of reading on the subway, but in New York the majority uses iPads and other eReaders. When Steffi and I were riding the subway, I noticed a men who exhibited some odd behavior while perusing his iPad. His thumb seemed to have a strange twitch, almost a tremor, and hovered over a corner of his iPad going back and forth. Upon closer inspection it became clear to me that the man was not suffering from a strange neurological disorder, but was caught up in a twist of multitasking conflict. He was reading a book on his iPad, and his thumb was above the corner that turns the page.

Now, think about how you read books (at least it is how I read books). If your gaze approaches the end of the page, say the last few lines before the end, you already lift up the corner of the page in order to be able to turn it right at the moment you read the last word. If you wait with picking up the page until you have read the last word you will lose time and might even disturb the flow of whatever you are reading. So, what you are in fact doing is engage in multitasking. Picking up the page before reading the end of the page is not part of the standard reading process. It is not as if you always pick up the page, say, exactly three lines before the end. Instead there is a parallel monitoring process that itches to pick up the page as you get closer to the end. You don't do it with a newspaper, and when you read a text online you do something different (scrolling).

But now enter the iBook reader on the iPad. It looks very much like a real book, including a stack of pages below the current page you are reading. If you touch the corner of a page, it will go to the next. It is almost like a book, but not exactly. If you act exactly as you would act with a real book, you turn the page prematurely (happens to me all the time).

Now, our poor multitasker was probably halfway learning to use the iBook reader, and the tremor in his hand was produced by the normal book-reading process willing him to touch the corner, and the new iBook reader override trying to convince him not to. Because he was so engaged in the reading but maybe also monitoring the subway ride, he probably didn't even notice it himself.

This small example demonstrates that many of our behaviors are composed of several smaller task that we execute in parallel. When we read a text we have a text-processing process, but also a process that makes sure our eyes are fed words at the right pace. And even though text-processing is usually the same, the word-feeding process may differ depending on the circumstances.

The benefits of multitasking

The discussion around multitasking usually revolves around the question whether multitasking is bad, with the most extreme position that people just cannot multitask. There are, however, several studies that show that multitasking can sometimes improve your performance. In other words, your performance on task A is better when you combine it with a task B than when you do task A alone!

Doodling for concentration

One example is a study that shows that doodling during meetings improves your recall of what has been said during the meeting. Apparently, it is easier to keep concentrated on boring stuff if you do something, well, boring on the side. For this reason I tell my students that making notes during class is a good idea, even if they never look at those notes again (and I pray that my lectures are not as boring as the doodle meetings). However, in such practical studies it is hard to investigate in detail what is going on.

Attentional Blink

To drill down to the details we did a more controlled laboratory experiment that shows the benefits of multitasking in some situations. The experimental paradigm is called Attentional Blink. In these experiments, subjects see a rapid stream of character, most of which are digits, but up to two characters are letters. Subjects are asked to identify and report the letters, and ignore the digits. You can try out the task yourself here.

In a normal attentional blink trial there are two letters in the stream. Due to the speed of presentation, accuracy is never at 100%, but often around 90%. However, the interesting result in an Attentional Blink experiment is that if the two letters are between 200 and 500 ms apart, the second letter often missed much more often. So while correctness on the first letter may be 90%, it is typically only 50% for the second. Interestingly enough, if either the letters are much closer (say, 100 ms), or much further apart, the second letter is reported correctly as often as the first letter (again, around 90%).

The Attentional Blink effect is interesting itself, and many researchers have been thinking about explanations and models. Additional experimentation showed that in some cases people may improve their performance on the task if you tell them not to try to hard, if they hear music in the background, if there is a starfield in the background, etc. So why is that?

To get a better grasp on this issue, some colleagues and I did a new experiment in which we gave people a second task next to just watching for letters. So we added another task to a task that was in itself already quite hard! The additional task was to track a gray dot that circled around the characters that were presented in the center of the screen. In some trials this gray dot momentarily turned red, and subjects were asked to report this in addition to the letters they saw.

And indeed, the results show that people show less "blink" if they are given the extra task.

Why is multitasking sometimes better?
The advantage of a laboratory task is that we can study what happens in detail. In order to increase our understanding, we built a computer model that mimics human behavior on the task. This model could explain why the Blink happens in the first place: once you have seen the first letter, you try to consolidate this in your memory, and you temporarily block additional letters from cluttering your memory process. In other words: you are too focused on the first letter and therefore miss the second.
If we now add a secondary task, we are slightly disrupting this focus. As a consequence, you will not try to remember the first letter too hard, and therefore also process and remember the second.

Back to the real world

Are you still with me reader? The bottom line is that sometimes we do better on tasks if we are a little distracted by a secondary task. Sometimes our automated processes do a better job than when we focus too much. This can be an issue in sport, for example. If an athlete doesn't trust his or her training, then too much thinking can ruin the effort. Another example: I have a hard time swallowing pills. My trick is to distract myself a bit while trying to swallow, for example by reading the package. I before I know it the pills are gone!