Many of us got into journalism to make a difference; some of us got into research, not to put too fine a point on it, to see if we’re making a difference. I’m at a school with a long tradition of research in media effects — that’s the the term for looking at both what people remember about what they see in their news or entertainment, and how they feel and what they do about it.
We have lots of different ways to study and measure these effects, and many of them rely on experiments: Show some people some media “content” such as news stories and pictures, and gauge their reactions. This approach aims to isolate the news messages and keep everything else constant.
I’ve always had a visceral dislike for this sort of experiment, because I don’t believe anything can be isolated from its context. We sort of take it on faith that the effects we see in the experiment are happening the same way out in the real world. A lot of them are. But how many variables does the experimenter not account for?
Did your last effects experiment control for the subjects’ native language?
Something we’ve figured out in the past half-century is that people aren’t coldly rational when making decisions. OK, obvious, right? But we’re irrational in predictable ways, most notably that if we are given the same problem expressed in two ways, one emphasizing possible gain and one emphasizing possible loss, we’ll tend to make different decisions in each case.
BUT. It now turns out that this effect may disappear when a subject is solving problems in a non-native language.
The first experiment involved 121 American students who learned Japanese as a second language. Some were presented in English with a hypothetical choice: To fight a disease that would kill 600,000 people, doctors could either develop a medicine that saved 200,000 lives, or a medicine with a 33.3 percent chance of saving 600,000 lives and a 66.6 percent chance of saving no lives at all.
Nearly 80 percent of the students chose the safe option. When the problem was framed in terms of losing rather than saving lives, the safe-option number dropped to 47 percent. When considering the same situation in Japanese, however, the safe-option number hovered around 40 percent, regardless of how choices were framed. The role of instinct appeared reduced.
Two subsequent experiments in which the hypothetical situation involved job loss rather than death, administered to 144 native Korean speakers from Korea’s Chung Nam National University and 103 English speakers studying abroad in Paris, found the same pattern of enhanced deliberation. “Using a foreign language diminishes the framing effect,” wrote Keysar’s team.
It’s not that testing native and non-native speakers together is bad science; until that study was published (paywall), we didn’t know second languages played a role. But that’s the point: The universe of potential confounding variables is so large that you can’t control for them all.
So when I look at an experimental effects study I always wonder what the researchers might have missed.
This post has not been revised since publication.