A Hybrid BCI speller based on the combination of EMG envelopes and SSVEP
© Lin et al.; licensee Springer. 2015
Received: 31 July 2014
Accepted: 18 November 2014
Published: 10 January 2015
Recently, much attention has been paid to Hybrid Brain-Computer Interfaces (BCI). In this study, we developed a hybrid BCI speller that simultaneously utilized information from both hand Electromyography (EMG) and SSVEP. This cross-modal BCI speller could increase the target number so as to enhance the information transfer rate (ITR). A 60-target hybrid BCI speller was built in this study. A frame-based sampled sinusoidal stimulation method was used to generate the flickering stimulus on the LCD screen. The 60 targets were equally divided into 4 sections, and each section had the same frequency range. EMG signal was used to distinguish different sections. Subjects were required to repeatedly make a fist from 0 to 3 times when the target was shown in section 1 to section 4. Then by extracting the envelope of the EMG signal and calculating the number of peaks of the envelope, we could know which section the target was in. Canonical Correlation Analysis (CCA) method was used to classify the SSVEP signal. The offline results showed that ITR achieved maximum value when the time window was set to be 2 s. The average classification accuracy of a 2 s time window was 80.5% and information transfer rate was 83.2 bit/min using the proposed hybrid BCI system. While the ITR was 32.7 bit/min for EMG only condition and 58.2 bit/min for SSVEP only condition, which revealed that the hybrid system had better performance than the two single-modal modalities.
KeywordsHybrid BCI EMG Envelope SSVEP ITR
Brain computer interface (BCI) provided a new direct communication channel between human brain and a machine (Wolpaw et al. 2002). Major types of BCI approaches included steady-state visual evoked potential (SSVEP) (Hillyard et al. 1997), P300 potential by oddball paradigm (Donchin et al. 2000; Sellers et al. 2012), and motor-imagery (Pfurtscheller et al. 2006). Each kind of BCI had its own advantages and disadvantages.
Hybrid BCIs combine multiple different approaches in an effort to take advantage of the various strengths that each BCI has on its own (Allison et al. 2010; Pfurtscheller et al. 2010a; Leeb et al. 2011; Lalitharatne et al. 2013;Amiri et al. 2013; Xu et al. 2013; Yin et al. 2013). Generally, two kinds of BCIs can be fused to become a hybrid BCI. For example, SSVEP-motor imagery hybrid BCI combined information from SSVEP and motor imagery to enhance classification accuracy (Allison et al. 2010). Moreover, SSVEP-motor imagery hybrid BCI can be used for orthosis control (Pfurtscheller et al. 2010b). Another kind of hybrid BCI is the P300-SSVEP hybrid BCI. After target stimuli, SSVEP were dismissed and replaced by P300 potentials, and this phenomenon was called SSVEP blocking. By using SSVEP blocking, the hybrid speller achieves higher accuracy and ITR than P300 speller on its own (Xu et al. 2013). SSVEP stimuli could also be superimposed onto the P300 stimuli to increase the difference among targets; this kind of hybrid BCI system can also enhance the accuracy and ITR significantly (Yin et al. 2013). A P300-motor imagery hybrid BCI is another possible combination (Rebsamen et al. 2008). P300 was suitable for discrete control applications and motor imagery was often used for continuous control, therefore the combination of these two types of BCI systems could provide more complicated and practical applications.
Another type of hybrid BCI system combined one BCI system with another system based on other physiological signals such as electromyogram (EMG). Although it is debatable if this type of system should be called hybrid BCI, it can be used for disabled people with all their residual functionalities and enhance the performance of the system. Thus, more and more researchers support this kind of hybrid BCI for practical use (Nijholt et al. 2011; Amiri et al. 2013; Lalitharatne et al. 2013). EMG-motor imagery hybrid BCI was developed to achieve better and more stable performance compared to the single conditions (Leeb et al. 2011). Subjects were asked to move their left or right hand. The results showed that the accuracy of the hybrid system with different kind of fusion method was higher than single modalities. EMG-P300 hybrid system is another kind of EMG based hybrid BCI system. In (Holz et al. 2013), researchers used the EMG signal to cancel any spelling errors that occurred when using a P300 based speller. The efficiency of the hybrid BCI-system was evaluated in terms of time for selection, percent of errors, and users frustration. The results illustrated that the hybrid system improved the performance in all three three aspects.
In this study, we designed a hybrid BCI speller using the information combined from hand EMG signal and SSVEP. The main advantages of SSVEP compared to other BCI systems are its high signal-to-noise ratio (SNR), little user-training, and high information transfer rate (ITR) (Gao et al. 2003; Wang et al. 2010; Bin et al. 2009; Chen et al. 2013). However, SSVEP is only capable of showing a good response within a limited frequency range, which limits the number of targets. Researchers have tried several methods to increase the number of targets such as phase-tagging and using intermodulation frequencies (Jia et al. 2011; Pan et al. 2011; Chen et al. 2013). In this study, we used EMG signal to increase target number. This combination was advantageous because both of the modalities can be recognized within a fairly short time and the interaction between them is negligible. Some researchers utilize different gestures to represent different commands (Chen et al. 2007; Zhang et al. 2009). However, these methods required training before using and the extracted features of this method were not stable due to muscle fatigue and gesture strength. So, we used the features from the EMG envelope of different gesture repetition times to represent different commands. All of the targets were equally divided into 4 sections, and each section had the same frequency range. When a target was shown in a particular section, subjects were required to stare at the target and make a fist several times corresponding to the section simultaneously. Then by calculating the numbers of peaks in the EMG envelope, we could deduce which section the target was in. Canonical Correlation Analysis (CCA) method was used to classify the SSVEP signal in order to determine which target in a particular section one was focusing. Offline studies were conducted among 10 subjects to investigate the feasibility of our hybrid method and determine the optimal parameters for the future online studies.
Ten healthy subjects (four males and six females; mean age 25.6 ± 2.55 years) volunteered to participate in the experiment. The number of subjects was sufficient compared to some other relevant study (Holz et al. 2013; Xu et al. 2013; Hwang et al. 2012; Volosyak 2011). All subjects had normal or corrected to normal vision. Each subject signed an informed consent form prior to the experiment and was paid for the participation.
B. Data acquisition
C. Experiment design
Stimuli were presented on a LCD monitor (23.6-inch; screen resolution 1920 × 1080 pixels, refresh rate 60 Hz). Viewing distance was approximately 70 cm. The stimuli presentation was controlled by Matlab and Psychophysics Toolbox Version 3 (PTB-3) (Brainard 1997; Pelli 1997].
Thus the value of stim(i, f) was from 0 to 1.
The offline experiment contained 2 blocks of 60 6-s trials. All targets were flashed in their particular frequency during the 5 s stimulus period. A red triangle was below one target indicating the target that subject needed to stare at. The focus target was selected randomly and each target was chosen for only one time. During the flickering period, subjects were also required to make a fist in a given repetition time as fast as possible. The repetition time from section 1 to section 4 was 0, 1, 2 and 3 respectively. Different repetition time can lead to different EMG envelop, so from the EMG data we can see which section the subject was staring and the SSVEP data was used to classify different targets in a given section. Then the subject could rest for 1 s before the next trial began. During the rest period, the target which subject was required to stare at turned
D. Data analysis
To classify the EEG data, we used Canonical Correlation Analysis (CCA) method (Lin et al. 2006; Bin et al. 2009; Chen et al. 2014). CCA was implemented using the canoncorr function in Matlab. The reference signals were composed of sinusoids and cosinusoids pairs at the same frequency of the stimulus and its second and third harmonics.
whereis the number of targets, is the mean accuracy averaged over all targets and (seconds/target) is the time for a selection. contained two parts, gaze time and rest time, and in this study, rest time was 1 s. To demonstrate the performance of any of the two single-modal modalities, we also calculate the classification accuracy and ITR for each modality. The EEG data were neglected in the EMG accuracy calculation. If the gesture number determined from the EMG signal matched the corresponding section, the trial was determined to be accurate. In the SSVEP accuracy calculation, a trial was a success if the discriminant target had the same frequency as the actual target regardless of which section the discriminant target was in. In the hybrid accuracy calculation, the trial was considered a success only if the discriminant target was exactly the same as the real target in a given trial. N in equation (3) to calculate the ITR was 4 for EMG only modal, 15 for SSVEP only modal, and 60 for the hybrid system. In order to investigate the influence of the length of time window to the system performance, accuracy and ITR were calculated separately with different epoch time varying from 0.5 s to 5 s with an interval of 0.5 s. Then a two-tailed t-test was conducted on the ITR value to verify the better performance of the hybrid system. Lastly, the classification accuracy for each EMG command was calculated and a two-tailed t-test was done for each pair of EMG commands. The same procedure was also done for the SSVEP classification.
Results of the accuracy and ITR with 2 s’ time window length for each subject
Average (mean ± SEM)
80.8 ± 15.6
94.8 ± 4.6
85.6 ± 15.2
83.7 ± 24.0
33.1 ± 5.3
58.0 ± 18.5
In this study, we proposed a novel hybrid BCI speller. From the results of the offline experiment, we demonstrated the feasibility of this hybrid BCI system and obtained the optimal length of the time window for future online experiment.
A. Feasibility of this hybrid BCI system
A hybrid BCI system was proposed to enhance the system performance. Besides combining two types of BCI approaches, such as SSVEP, P300, and motor imagery, cross-modal BCI systems, which combine BCI with another kind of physiological signal such as EMG was also a practical method by which patients can use their remaining muscular function to improve the BCI system (Leeb et al. 2011; Holz et al. 2013). In this study, the hybrid system we proposed achieved significantly higher ITR than its individual single-modal systems. While one might debate that this hybrid BCI may be not applicable for some paralyzed patients who have totally lost control of their hands, some of these patients might still have the ability to control their facial muscle. Therefore, this method could still be applied by converting the movement from making fists to gritting teeth. This hybrid BCI speller is perfect for Parkinson’s patients, because they might have lost the fine ability to use a real keyboard but still can make fists several times easily. Moreover, any person could also use this hybrid BCI system for entertainment or under conditions where a keyboard is not available.
B. Information transform rate
According to formula (3), there are three ways to obtain a high ITR: (1) by increasing the number of targets, (2) by improving the accuracy of target selection, and (3) by decreasing the time needed to recognize each target. Compared to single-modal BCI system, the hybrid BCI system we designed can enlarge the number of targets with a small sacrifice in accuracy, so as to increase ITR. The total number of targets was 60, which was higher than other studies. The actual ITR with a 2 s time window in the offline experiment was 83.7 bit/min, however the theoretical maximum ITR of the hybrid speller with the accuracy of 100% was 118.1bit/min. In the future work, to further increase the ITR, the classification accuracy of SSVEP and EMG should be enhanced. To increase the accuracy of SSVEP, better stimulus frequencies should be chosen carefully and the area of each target might be enlarged. To increase the accuracy of EMG classification, a more robust classification algorithm should be employed. On the other hand, the time needed to recognize each target was not very short in this study. We could see from formula (3) that recognition time played more important roles than the number of targets. Even though the numbers of targets was relatively high in this study, longer recognition time made the ITR not very high in comparison to (Chen et al. 2014). The recognition time for one target included two parts, stimulus time and rest time. Stimulus time of 2 s was proved to be best in the offline result, so only rest time can be reduced in this study. If the rest time was decreased to 0.3 s and the classification accuracy remained the same, the ITR would reach 118.7 bit/min. However, a short rest time might lead to user fatigue more easily and would not be practical in real use, so rest time was still set to be 1 s in this study.
C. Comparison with other hybrid BCI systems and BCI spellers
In (Xu et al. 2013; Yin et al. 2013), researchers presented a hybrid BCI based on P300 and SSVEP. The ITR in their study was 34.2 bit/min and 56.4 bit/min respectively. Compared to these types of hybrid BCI system, our system achieved much higher ITR which was 83.7 bit/min. Another type of hybrid BCI system was SSVEP and motor imagery based BCI. In (Allison et al. 2010), the mean classification accuracy of the hybrid system with two targets was only 81%, while our system has a classification accuracy of 80.8% for 60 targets which was much more than the SSVEP and motor-imagery hybrid BCI system.
In (Leeb et al. 2011), a motor-imagery and EMG hybrid system was illustrated. The result showed that their method could enhance the classification accuracy when compared to the motor-imagery system and the EMG system on their own. However, the mean accuracy for EMG activity alone was 87% and the fusion approach had only a slightly higher classification accuracy (91%), which showed that their hybrid system was mainly based on EMG and EEG did not have much influence. Another EMG based hybrid system was shown in (Holz et al. 2013). In their P300-EMG hybrid speller, EMG was only used to correct spelling errors. The result showed that the performance (expressed as time for selection and number of errors) was enhanced when compared to no-hybrid speller. However, EMG represents only one target in the study so it only could enhance the system performance slightly. Compared to the two EMG based hybrid BCI system above, the method we proposed has a higher ITR. The fusion method we used was effective and could enlarge the number of targets significantly. Moreover, P300 based and motor imagery based hybrid system required training sessions, which takes additional time and may result in users’ fatigue more easily. Furthermore, these systems required significant mental effort, which may also aggravate users’. Other gesture based EMG recognition methods need training sessions before each test. However in this study, both the EMG portion and the SSVEP portion did not require training sessions, and the system can evoke significantly high SSVEP response without much effort.
In previous study, several BCI spellers were introduced using single BCI modality. BCI spellers based on P300 and motor imagery required training before using and great mental effort to achieve the assumed goal. These types of spellers cannot obtain high ITR. SSVEP based BCI speller can get relatively higher ITR, but the number of target was limited by the frequency. In (Hwang et al. 2012), a SSVEP based BCI speller of 30 targets was introduced and a mean accuracy of 87.58% and ITR of 40.72 bits/min for 6 subjects was reported. In (Volosyak 2011), the mean ITR of 7 participants of their SSVEP based BCI speller with 5 targets was 61.7 bit/min. Our system has more targets and higher ITR than these systems.
In this study we designed a hybrid BCI speller based on EMG envelope and SSVEP. All targets were divided into 4 sections, EMG was used to classify which section the target was in, and SSVEP was used to classify the particular target in the section. The offline results obtained from ten healthy volunteers confirmed that the hybrid BCI speller could be classified effectively for a practical BCI system. Specifically, the offline results revealed that an average classification accuracy of 80.5% and information transfer rate of 83.2 bit/min was achieved using our proposed hybrid BCI system. While the ITR was 32.7 bit/min for the EMG only condition and 58.2bit/min for the SSVEP only condition, thus revealing that the hybrid system had better performance than the two single-modal modalities.
This work was supported by Huawei Technologies Co., Ltd., National Basic Research Program (973) of China (No. 2011CB933204), National Natural Science Foundation of China under Grant 90820304, 91120007, Chinese 863 Project: 2012AA011601. The authors declared that they have no competing interests. KL participated in the design of the study, performed in the data collection, performed in the data analysis and drafted the manuscript. XC and XH participated in the modification of the study and revised the manuscript critically. QD participated in the acquisition of funding and the modification of the study. XG participated in the design of the study, revised the manuscript and supervised the research group. All authors read and approved the final manuscript.
- Allison BZ, Brunner C, Kaiser V, Müller Putz GR, Neuper C, Pfurtscheller G (2010) Toward a hybrid brain–computer interface based on imagined movement and visual attention. J Neural Eng 7(2):026007View ArticleGoogle Scholar
- Amiri S, Fazel-Rezai R, Asadpour V (2013) A review of hybrid brain-computer interface systems. Adv Hum-Comput Interact 2013:1View ArticleGoogle Scholar
- Bin G, Gao X, Yan Z, Hong B, Gao S (2009) An online multi-channel SSVEP-based brain-computer interface using a canonical correlation analysis method. J Neural Eng 6:046002View ArticleGoogle Scholar
- Brainard DH (1997) The psychophysics toolbox. Spatial Vis 10:433–436View ArticleGoogle Scholar
- Chen X, Zhang X, Zhao Z-Y, Yang J-H, Lantz V, Wang K-Q (2007) Multiple Hand Gesture Recognition Based on Surface EMG Signal. Bioinform Biomed Eng ᅟ:506–509. 2007, ICBBE 2007, The 1st International Conference onGoogle Scholar
- Chen X, Chen Z, Gao S, Gao X (2013) Brain–computer interface based on intermodulation frequency. J Neural Eng 10:066009View ArticleGoogle Scholar
- Chen X, Chen Z, Gao S, Gao X (2014) A high-ITR SSVEP-based BCI speller. Brain-Comput Interfaces ᅟ:1–11. ahead-of-printMathSciNetGoogle Scholar
- Donchin E, Spencer KM, Wijesinghe R (2000) The mental prosthesis: assessing the speed of a P300-based brain-computer interface IEEE Trans. Rehabil Eng 8:174–179View ArticleGoogle Scholar
- Gao X, Xu D, Cheng M, Gao S (2003) A BCI-based environmental controller for the motion-disabled IEEE Trans.Neural Syst. Rehabil Eng 11:137–140Google Scholar
- Hillyard S, Hinrichs H, Tempelmann C, Morgan S, Hansen J, Scheich H, Heinze HJ (1997) Combining steady-state visual evoked potentials and fMRI to localize brain activity during selective attention Hum. Brain Mapp 5:287–292View ArticleGoogle Scholar
- Holz E, Riccio A, Reichert J, Leotta F, Aricò P, Cincotti F, Mattia D, Kübler A (2013) Hybrid-P300 BCI: Usability Testing by Severely Motor-restricted End-Users, 4th Workshop on Tools for Brain Computer Interaction (TOBI). Sion, Switzerland, pp 111–112Google Scholar
- Hwang HJ, Lim JH, Jung YJ, Choi H, Lee SW, Im CH (2012) Development of an SSVEP-based BCI spelling system adopting a QWERTY-style LED keyboard. J Neurosci Methods 208:59–65View ArticleGoogle Scholar
- Jia C, Gao X, Hong B, Gao S (2011) Frequency and phase mixed coding in SSVEP-based brain–computer interface IEEE Trans. Biomed Eng 58:200–206Google Scholar
- Lalitharatne TD, Teramoto K, Hayashi Y, Kiguchi K (2013) Towards hybrid EEG-EMG-based control approaches to be used in Bio-robotics applications: current status, challenges and future directions. Paladyn J Behav Robotics 4(2):147–154Google Scholar
- Leeb R, Sagha H, Chavarriaga R, Del R, Millán J (2011) A hybrid brain–computer interface based on the fusion of electroencephalographic and electromyographic activities. J Neural Eng 8(2):025011View ArticleGoogle Scholar
- Lin Z, Zhang C, Wu W, Gao X (2006) Frequency recognition based on canonical correlation analysis for SSVEP-based BCIs IEEE Trans. Biomed Eng 53:2610–2614Google Scholar
- Lin K, Huang X, Ding Q, Li L, Gao X (2014) An Human-Computer Interface based on two channel hand EMG envelopes. In: IEEE Second International Conference on Cognitive Systems and Information Processing (ICCSIP) Accept. ᅟ, Beijing,ChinaGoogle Scholar
- Nijholt A, Allison BZ, Jacob RK (2011) Brain-Computer Interaction: Can Multimodality Help? In: 13th Intern. Conf. on Multimodal Interaction. ACM, NY, pp 35–39Google Scholar
- Pan J, Gao X, Duan F, Yan Z, Gao S (2011) Enhancing the classification accuracy of steady-state visual evoked potential-based brain-computer interfaces using phase constrained canonical correlation analysis. J Neural Eng 8:036027View ArticleGoogle Scholar
- Pelli DG (1997) The VideoToolbox software for visual psychophysics: transforming numbers into movies. Spatial Vis 10:437–442View ArticleGoogle Scholar
- Pfurtscheller G, Brunner C, Schlögl A, Lopes da Silva FH (2006) Mu rhythm (de) synchronization and EEG single-trial classification of different motor imagery tasks. Neuroimage 31(1):153–159View ArticleGoogle Scholar
- Pfurtscheller G, Solis-Escalante T, Ortner R, Linortner P, Muller-Putz GR (2010a) Self-paced operation of an SSVEP-Based orthosis with and without an imagery-based “brain switch:” a feasibility study towards a hybrid BCI. IEEE Trans Neural Syst Rehabil Eng 18(4):409–414View ArticleGoogle Scholar
- Pfurtscheller G, Allison BZ, Bauernfeind G, Brunner C, Solis Escalante T, Scherer R, Zander TO, Mueller-Putz G, Neuper C, Birbaumer N (2010b) The hybrid BCI. Front Neurosci ᅟ:ᅟ. doi: 10.1186/1744-9081-6-28Google Scholar
- Rebsamen B, Burdet E, Zeng Q, Zhang H, Ang M, Teo CL, Guan C, Laugier C (2008) Hybrid P300 and Mu-Beta brain computer interface to operate a brain controlled wheelchair. In: Proceedings of the 2nd International Convention on Rehabilitation Engineering and Assistive Technology, pp 51–55
- Sellers E, Arbel Y, Donchin E, Wolpaw J, Wolpaw EW (2012) BCIs that uses P300 event related potentials,” in Brain-Computer Interfaces: Principles and Practice. Oxford University Press, OxfordGoogle Scholar
- Volosyak L (2011) SSVEP-based Bremen–BCI interface—boosting information transfer rates. J Neural Eng 8:036020View ArticleGoogle Scholar
- Wang Y, Wang YT, Jung TP (2010) Visual stimulus design for high-rate SSVEP BCI Electron. Lett 46:1057–1058Google Scholar
- Wolpaw JR, Ramoser H, McFarland DJ, Pfurtscheller G (1998) EEG-Based communication: improved accuracy by response verification IEEE Trans. Rehabil Eng 6:326–333View ArticleGoogle Scholar
- Wolpaw JR, Birbaumer N, McFarland DJ, Pfurtscheller G, Vaughan TM (2002) Brain-computer interfaces for communication and control Clin. Neurophysiol 113:767–791View ArticleGoogle Scholar
- Xu M, Qi H, Wan B, Yin T, Liu Z, Ming D (2013) A hybrid BCI speller paradigm combining P300 potential and the SSVEP blocking feature. J Neural Eng 10(2):026001View ArticleGoogle Scholar
- Yin E, Zhou Z, Jiang J, Chen F, Liu Y, Hu D (2013) A novel hybrid BCI speller based on the incorporation of SSVEP into the P300 paradigm. J Neural Eng 10(2):026012View ArticleGoogle Scholar
- Yuan P, Gao X, Allison B, Wang Y, Bin G, Gao S (2013) A study of the existing problems of estimating the information transfer rate in online brain–computer interfaces. J Neural Eng ᅟ:10 026014Google Scholar
- Zhang X, Chen X, W-h W, J-h Y, Lantz V, K-q W (2009) Hand gesture recognition and virtual game control based on 3D accelerometer and EMG sensors. In: IUI ’09: Proceedings of the 13th international conference on Intelligent user interfaces. ACM, New York, NY, USA, pp 401–406Google Scholar
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.