Tag: Bad Science

For some reason the world of writers is filled with technophobic troglodytes intent on proving that their old-fashioned way of doing things is better. I’ve written previously about how older people’s favourite hobby since the dawn of time has been complaining about kids these days. This is also true of changes in technology, with people intent on justifying not learning to use a computer or e-reader. Because cutting down trees is the future of communication!

Once again I’ve stumbled across another article that misrepresents scientific studies to try and convince people that we need to clear forests, pulp them, flatten them into paper, cover them in ink, and act as snooty as possible. This time they – the nebulous they: my nemesis!! – are trying to pretend that taking notes with a pen is better than using a keyboard.

Ugh.

When will people learn that paper isn’t the medium we should be promoting? We need to be going back to scratching on rocks and cave walls. When was the last time a paper book lasted more than a hundred years out in the rain, snow, and blazing sun? That doesn’t even begin to compete with the longevity of the 50,000 year old cave paintings. Data retention for rock far surpasses the much inferior paper.

This isn’t the first article I’ve seen on The Literacy Site misrepresenting science. Hopefully they will acquire come scientific literacy soon and overcome their biases. If I turn blue and pass-out, try to act concerned. Let’s dive in.

New Research Explains How The Pen Is Mightier Than The Keyboard

It’s great when articles improve on the titles of science papers. I mean, who wants to read the science paper The Pen Is Mightier Than the Keyboard: Advantages of Longhand Over Laptop Note Taking? Pity that both titles misrepresent the actual findings. Also, is 2014 still regarded as new?

In her graduate assistant days, psychological scientist Pam Mueller of Princeton University used to take notes just like everyone else in the modern age: with a computer. One day, Mueller forgot her laptop and had to take notes the old-fashioned way. Rather than being held back by pen and paper, Mueller left class feeling as if she’d retained far more information than usual on that day. She decided to create a case study that could prove her hunch that writing longhand was actually better for comprehension than typing.

This is actually a good little story and illustrates how a lot of hypotheses are formed in science. This is the anecdote or observation that scientists want to turn into a hypothesis to create actual knowledge. But remember, this is an anecdote, which has as much value as used Easter egg wrappers that have been stuffed between the couch cushions. Putting anecdotal stories at the start of an article can set the audience up to not think too hard about the rest of the article, as you have given them the conclusion in a nice little story.

The study she created, published in Psychological Science, indicated that taking notes by hand is a more effective method than typing them on a laptop when it comes to processing information conceptually.

And here we jump straight off the rails, over the side of the bridge, and careen into the waiting river below. Sure, The Literacy Site is just quoting the press release, but that is lazy. The study itself has this line in the abstract that show how this claim is a misrepresentation of the findings:

We show that whereas taking more notes can be beneficial, laptop note takers’ tendency to transcribe lectures verbatim rather than processing information and reframing it in their own words is detrimental to learning.

In other words, the findings were that people spend all their time typing and no time actually listening and comprehending the lectures. Because the pen is an archaic device that is unwieldy and slow compared to the keyboard, students using a pen only write down notes after they have listened, picked out the key points, and conceptualised that information into a note. But don’t take my word for it, the press release on the University of Michigan website has a few recommendations including:

• To interrupt verbatim note-taking on laptops, break up your lectures with short activities that encourage deeper processing of information.
• Have students use laptops or other technologies to process–not just record–information.

Now it is time to discuss the study details a little bit, because someone might be interested in the methods section. I’m sure those people exist. Somewhere. Interested is probably the wrong word.

In the first of a series of studies led by Mueller, 65 college students watched various TED Talks in small groups, and were provided with either pens and paper or laptops for taking notes. When the students were tested afterward, the results were overwhelming. While the groups performed equally on questions that involved recalling facts, those who had taken longhand notes did significantly better when it came to answering conceptual questions.

Sorry, I need to catch my breath. I’m so shocked at the massive sample size. This is definitely enough people to represent the rest of society. Conclude away I say!

Anyway, these overwhelming results are just a tad whelming.

As you can see the performance on retaining facts was the same, with error bars that suggest 65 people is probably not enough to draw conclusions from. Not that anyone would be trying to claim this study is proof of anything, right? The next thing you see is the benefits of using a pen…. as long as you ignore those error bars and just accept the p-value tells us something of value. Given that those error bars overlap for the two groups, I wouldn’t be drawing conclusions from a p-value. Also, I’m not exactly sure why an ANOVA was used when there were only two groups to compare. KISS principle applies to statistics as well.

Now the study realised that 65 people wasn’t enough, so they repeated the study with a few variations twice more. In the second and third tests they had 151 and 109 people take notes. Each test had the typists writing between 250 and 550 words, whilst the pen wielders wrote roughly 150 to 400 words. Interestingly the note takers were writing verbatim 12-14% with their laptop but the pen users only managed 4-9% verbatim. This shows why the conclusions I’ve quoted above were drawn.

Out of interest, here are the results from the other two tests that were more convincing for that conceptual finding.

The second test with 151 people were tested with pen, laptop, and laptop with a lecture from the tester about how they really should pay attention. With 50 people per group you’d hardly jump up and down about the significance of this test, but clearly telling people to pay attention doesn’t… hey look a squirrel.

The third test with 109 people again tested for pen vs keyboard, but this time they allowed revision of notes before being questioned. This makes the groups even smaller, and again I’d question the significance of such a small sample. But the researchers summed up the results with this erudite paragraph:

However, a more nuanced story can be told; the indirect effects differ for conceptual and factual questions. For conceptual questions, there were significant indirect effects on performance via both word count and verbatim overlap. The indirect effect of word count for factual questions was similar, but there was no significant indirect effect of verbatim overlap. Indeed, for factual questions, there was no significant direct effect of overlap on performance. As in Studies 1 and 2, the detriments caused by verbatim overlap occurred primarily for conceptual rather than for factual information, which aligns with previous literature showing that verbatim note taking is more problematic for conceptual items.

In other words, doing lots of writing, particularly just copying what was said verbatim, makes you suck at understanding what the hell is going on. Oh, and study before the test. Apparently it helps too. Made that mistake at university.

So back at The Literacy Site they are skipping the other tests and just heading to the conclusions:

Mueller found that this was the result of laptop users trying too hard to transcribe the lecture rather than listening for the most important information and writing it down by hand. It may be an era where computers have made handwriting seem useless, but Mueller isn’t the only believer in the importance of longhand.

Notice the nuanced difference that seeing all three tests provides? We could be led to believe that there was overwhelming evidence for the pen, but what we see is that note takers need to readdress their methods of taking notes. Or they could just wing it.

An article in TIME discusses Karin James, an Indiana University psychologist, who published a 2012 study indicating writing is particularly important in the cognitive development of pre-literate children five and under. While using a computer for note-taking in some situations makes sense, it’s important not to overlook the longhand method.

It’s great that the article tries to incorporate some extra research. Citing one study with a small sample size is hardly compelling, certainly not worth writing an article about. But again the research is being misrepresented:

…the benefits of writing: increased creativity, better critical thinking, boosted self confidence, and a correlated improvement in reading capability with writing prowess.

But are these benefits real? The short answer: Mostly not. “There’s lot of caveats in handwriting research,” says Karin James, a psychologist at Indiana University

Curse those damn caveats! Why can’t we have a control group of kids we don’t teach to read and write?!

Which brings me to a final point about these old technologies vs new technologies articles: stop jumping the gun! We’re in a transition phase. This isn’t 1970s velvet suits with platforms versus 2010s hipster atrocities. This is typewriter hipster texting on his phone. Technology is changing and we’re still learning how to use it properly. The studies that are cited in many of these articles have very limited scope, test very few people, and are comparing new and established things. Has anyone taught laptop users to take notes effectively for the new medium? Do you actually need to take written notes at all in this modern age? We need to see more science done on the changes taking place, and we need the articles discussing the science to do more than discuss (one study from) one paper, and highlight the limitations. Well, unless you have already made up your mind about a topic and just want some links to throw at people in an argument. Screw being right!

This blog post is being shared online, in print, and carved into a cave wall. Comment below which format you preferred receiving it in.