A pioneering online international study for the Manchester Science Festival that showed video games may boost your cognitive abilities more than doing exercise, while exercise can help improve your mental health, has made headlines around the world.
The BBC, Irish News, The Daily Telegraph and CBC are among the 200 news outlets that have written in the past few days about the collaboration between the Science and Industry Museum and a team at Western University in Canada for the Festival, which is still under way in Manchester.
The basic idea for the experiment, which is described in this scientific preprint, is to ask participants for biographical information and then make them carry out a series of cognitive tests. For the first wave of the study, more than 2,500 people signed up.
A follow up statistical analysis was then used to untangle various effects of lifestyle and so on, from those of sex on cognition (none) to the links between mental health and activity, and I provide a lay summary of our findings in this blog.
At a special event last Saturday with one of the organisers, Professor Adrian Owen of Western, we were asked by festivalgoers how much we can trust the results of this online experiment, which involved thousands of people around the world.
His co-organiser, Dr Conor Wild, explained how the tests have been under development for decades, beginning with work in the 1990s by Professor Owen while at the University of Cambridge.
At that time, if a scientist wanted to see how a drug affected cognition—the mental processes of everyday life, from reasoning to memory and verbal skills—then patients had to carry out various tests in hospital.
To make the tests more convenient, an online version was developed and the technology was commercialised in 2015 by a company called Cambridge Brain Sciences, now Creyos, where Professor Owen is the Chief Scientific Officer.
But how can we be sure that the score from one of these online games, which took around an hour in the latest study (you can still take part, see here), actually reflect processes in the brain, such as the ability to reason, to plan a sequence of movements, or to remember where items are located in space?
‘Scientific validity is an important concept to consider any time you measure something,’ said Dr Wild. ‘Our cognitive tests rest on decades of research studies using these tasks, or variations of them, to study human cognition in healthy and specific patient populations.’
As one example, he said, the regions at the front of the brain, called the frontal lobes, are known to be involved all sorts of functions, such as movement, problem solving, spontaneity, memory, language, initiation, judgement, impulse control, and social and sexual behaviour. ‘We can see that patients with frontal lobe damage are less likely to use strategies on some tasks, like Token Search.’
The Western team can also validate the tests using brain scanners, notably a method called functional MRI, which picks up tiny changes in blood flow in a brain region when it is used more than neighbouring zones. ‘Using neuroimaging technology, like functional MRI, we have been able to examine and map the brain networks that are engaged by our cognitive tasks.’
Although he said it is too simplistic to say that certain cognitive functions reside within specific regions of the brain, one can see consistent patterns: ‘Cognitive tasks that require some amount of short-term memory tend to recruit brain areas/networks that are associated with short-term memory processes in other neuroscientific studies. Similarly, those tasks that involve linguistic stimuli (such as reading words) reliably engage components of what is typically described as the “language” network.’
‘In other words,’ he said, ‘by scanning people’s brains as they play our tasks, we can see that the expected brain networks are working harder as the participants work harder to complete the tasks. Therefore, it’s likely that we are measuring the thing that we think we are measuring.’
Another issue is whether the thousands of people who took part know that they are doing the online tasks properly. ‘We [Creyos] have put a lot of effort into designing the tasks to be easy to understand and easy to play.’
Clearly, if the design were perfect, it would be self-evident what to do but, given we live in an imperfect world, Creyos and Western created interactive tutorials to try to make the rules of the games as clear and understandable as possible. ‘We’re still working on improving these instructions,’ he said.
Insights also come from the participants as they do the tests. ‘For example, are they going much slower or faster than expected? What kinds of errors are being made? There are various other measures, beside the final score, that we can examine to see if performance is within normal ranges,’ said Dr Wild. ‘These extra insights help us identify participants who might be doing the task incorrectly, not at all (such as clicking away while watching TV), or even cheating.’
A key question is whether tests on the internet work as well as those under more controlled conditions in the laboratory. ‘We have run studies to test exactly this and found that (for the most part) people’s scores were about the same in both locations. In other words, we have data strongly suggesting that computerized tests taken unsupervised at home produce results no different than those taken in a laboratory, both in healthy controls and in a patient population.’ That is one reason why, of the first cohort of more than two thousand people, the team could say that reliable data were gathered on around 1,000.
There is some evidence, however, that the range of scores (which scientists call the variance) can be larger in the internet participants, even though they tend to give similar average scores to laboratory experiments. ‘We suspect that is due to a greater chance of people not understanding or doing the tasks properly, or cheating, when completing them at home, compared to in the lab,’ said Dr Wild. ‘This produces more extreme scores at both ends of the spectrum.’
‘However, as we have already discussed, we have developed ways (and are working on new ways) of identifying those cases from other background data collected during a testing session.’
That is also why, when people took part in the Brain and Body study, ‘we have a screen that asks people to be in a quiet, distraction-free zone, with their phones turned off, before starting the brain games.’
Science Museum Group © The Board of Trustees of the Science Museum
At the festival, visitors can also visit State of Mind, a large scale immersive art installation developed by the collective Squidsoup, with the help of Professor Owen, to probe the mysteries of the brain and perception.
