Rock Paper Scissors

The current study evaluated the ability to use MEG to distinguish between hand gestures made by 12 volunteer subjects. The volunteers were equipped with the MEG helmet and randomly instructed to make one of the gestures used in the game Rock Paper Scissors (as in previous studies of this kind). MEG functional information was superimposed on MRI images, which provided structural information on the brain.

To interpret the data generated, Yifeng (“Troy”) Bu, an electrical and computer engineering PhD student in the UC San Diego Jacobs School of Engineering and first author of the paper, wrote a high-performing deep learning model called MEG-RPSnet.

“The special feature of this network is that it combines spatial and temporal features simultaneously,” said Bu. “That’s the main reason it works better than previous models.”

When the results of the study were in, the researchers found that their techniques could be used to distinguish among hand gestures with more than 85% accuracy. These results were comparable to those of previous studies with a much smaller sample size using the invasive ECoG brain-computer interface.

The team also found that MEG measurements from only half of the brain regions sampled could generate results with only a small (2 – 3%) loss of accuracy, indicating that future MEG helmets might require fewer sensors.

Looking ahead, Bu noted, “This work builds a foundation for future MEG-based brain-computer interface development.”

In addition to Huang, Lee and Bu, the article, “Magnetoencephalogram-based brain–computer interface for hand-gesture decoding using deep learning” (https://doi.org/10.1093/cercor/bhad173), was authored by Deborah L. Harrington, Qian Shen and Annemarie Angeles-Quinto of VA San Diego Healthcare System and UC San Diego School of Medicine; Hayden Hansen of VA San Diego Healthcare System; Zhengwei Ji, Jaqueline Hernandez-Lucas, Jared Baumgartner, Tao Song and Sharon Nichols of UC San Diego School of Medicine; Dewleen Baker of VA Center of Excellence for Stress and Mental Health and UC San Diego School of Medicine; Imanuel Lerman of UC San Diego, its School of Medicine and VA Center of Excellence for Stress and Mental Health; and Ramesh Rao (director of Qualcomm Institute), Tuo Lin and Xin Ming Tu of UC San Diego.

The work was supported in part by Merit Review Grants from the US Department of Veterans Affairs, Naval Medical Research Center's Advanced Medical Development program and Congressionally Directed Medical Research Programs/Department of Defense.