User menu

Accès à distance ? S'identifier sur le proxy UCLouvain

Extracting neurophysiological signals reflecting users’ emotional and affective responses to BCI use: A systematic literature review

Primary tabs

Liberati, Giulia [UCL] Federici, Stefano Pasqualotto, Emanuele [UCL]

BACKGROUND: Brain–computer interfaces (BCIs) allow persons with impaired mobility to communicate and interact with the environment, supporting goal-directed thinking and cognitive function. Ideally, a BCI should be able to recognize a user’s internal state and adapt to it in real-time, to improve interaction. OBJECTIVE: Our aim was to examine studies investigating the recognition of affective states from neurophysiological signals, evaluating how current achievements can be applied to improve BCIs. METHODS: Following the PRISMA guidelines, we performed a literature search using PubMed and ProQuest databases. We considered peer-reviewed research articles in English, focusing on the recognition of emotions from neurophysiological signals in view of enhancing BCI use. RESULTS: Of the 526 identified records, 30 articles comprising 32 studies were eligible for review. Their analysis shows that the affective BCI field is developing, with a variety of combinations of neuroimaging techniques, selected neurophysiological features, and classification algorithms currently being tested. Nevertheless, there is a gap between laboratory experiments and their translation to everyday situations. CONCLUSIONS: BCI developers should focus on testing emotion classification with patients in ecological settings and in real-time, with more precise definitions of what they are investigating, and communicating results in a standardized way.

Document type Article de périodique (Journal article) – Synthèse de littérature
Access type Accès restreint
Publication date 2015
Language Anglais
Journal information "NeuroRehabilitation : an interdisciplinary journal" - Vol. 37, no.3, p. 341-358 (2015)
Peer reviewed yes
Publisher I O S Press (Amsterdam)
issn 1053-8135
e-issn 1878-6448
Publication status Publié
Affiliation UCL - SSS/IONS/COSY - Systems & cognitive Neuroscience
Links
  1. Altman, The American Statistician, 46, 175 (1992)
  2. Azcarraga Judith, Suarez Merlin Teodosia, Recognizing Student Emotions using Brainwaves and Mouse Behavior Data : , 10.4018/jdet.2013040101
  3. Balconi Michela, Lucchiari Claudio, EEG correlates (event-related desynchronization) of emotional face elaboration: A temporal analysis, 10.1016/j.neulet.2005.09.004
  4. Baucom Laura B., Wedell Douglas H., Wang Jing, Blitzer David N., Shinkareva Svetlana V., Decoding the neural representation of affective states, 10.1016/j.neuroimage.2011.07.037
  5. Biessmann Felix, Plis Sergey, Meinecke Frank C., Eichele Tom, Muller Klaus-Robert, Analysis of Multimodal Neuroimaging Data, 10.1109/rbme.2011.2170675
  6. Birbaumer Niels, Weber Cornelia, Neuper Christa, Buch Ethan, Haapen Klaus, Cohen Leonardo, Physiological regulation of thinking: brain–computer interface (BCI) research, Progress in Brain Research (2006) ISBN:9780444521835 p.369-391, 10.1016/s0079-6123(06)59024-7
  7. Bradley, International affective digitized sounds (IADS): Stimuli, instruction manual and affective ratings (1999)
  8. Brouwer Anne-Marie, Zander Thorsten O., van Erp Jan B. F., Korteling Johannes E., Bronkhorst Adelbert W., Using neurophysiological signals that reflect cognitive or affective state: six recommendations to avoid common pitfalls, 10.3389/fnins.2015.00136
  9. Buck Ross, Nonverbal behavior and the theory of emotion: The facial feedback hypothesis., 10.1037/0022-3514.38.5.811
  10. Cacioppo John T, Feelings and emotions: roles for electrophysiological markers, 10.1016/j.biopsycho.2004.03.009
  11. Chanel Guillaume, Kierkels Joep J.M., Soleymani Mohammad, Pun Thierry, Short-term emotion assessment in a recall paradigm, 10.1016/j.ijhcs.2009.03.005
  12. Chanel Guillaume, Kronegg Julien, Grandjean Didier, Pun Thierry, Emotion Assessment: Arousal Evaluation Using EEG’s and Peripheral Physiological Signals, Multimedia Content Representation, Classification and Security (2006) ISBN:9783540393924 p.530-537, 10.1007/11848035_70
  13. Chanel G., Rebetez C., Bétrancourt M., Pun T., Emotion Assessment From Physiological Signals for Adaptation of Game Difficulty, 10.1109/tsmca.2011.2116000
  14. Chu Yi, Brown Pat, Harniss Mark, Kautz Henry, Johnson Kurt, Cognitive support technologies for people with TBI: current usage and challenges experienced, 10.3109/17483107.2013.823631
  15. Crabbe James B., Smith J.Carson, Dishman Rod K., Emotional & electroencephalographic responses during affective picture viewing after exercise, 10.1016/j.physbeh.2006.10.001
  16. Cutini Simone, Moro Sara, Bisconti Silvia, Review: Functional near infrared optical imaging in cognitive neuroscience: an introductory review, 10.1255/jnirs.969
  17. Davidson Richard J., Anterior cerebral asymmetry and the nature of emotion, 10.1016/0278-2626(92)90065-t
  18. Ekman, Handbook of Cognition and Emotion, 4, 5 (1999)
  19. Ekman Paul, Friesen Wallace V., O'Sullivan Maureen, Chan Anthony, Diacoyanni-Tarlatzis Irene, Heider Karl, Krause Rainer, LeCompte William Ayhan, Pitcairn Tom, Ricci-Bitti Pio E., Scherer Klaus, Tomita Masatoshi, Tzavaras Athanase, Universals and cultural differences in the judgments of facial expressions of emotion., 10.1037/0022-3514.53.4.712
  20. Federici, Assistive Technology, 1 (2014)
  21. Federici, Assistive technology assessment handbook (2012)
  22. Fox Nathan A., If it's not left, it's right: Electroencephalograph asymmetry and the development of emotion., 10.1037/0003-066x.46.8.863
  23. Frantzidis C.A., Bratsas C., Klados M.A., Konstantinidis E., Lithari C.D., Vivas A.B., Papadelis C.L., Kaldoudi E., Pappas C., Bamidis P.D., On the Classification of Emotional Biosignals Evoked While Viewing Affective Pictures: An Integrated Data-Mining-Based Approach for Healthcare Applications, 10.1109/titb.2009.2038481
  24. Frantzidis Christos A, Bratsas Charalampos, Papadelis Christos L, Konstantinidis Evdokimos, Pappas Costas, Bamidis Panagiotis D, Toward Emotion Aware Computing: An Integrated Approach Using Multichannel Neurophysiological Recordings and Affective Visual Stimuli, 10.1109/titb.2010.2041553
  25. Hadjidimitriou, medical Engineering, 59, 3498 (2012)
  26. Hadjidimitriou S. K., Hadjileontiadis L. J., EEG-Based Classification of Music Appraisal Responses Using Time-Frequency Analysis and Familiarity Ratings, 10.1109/t-affc.2013.6
  27. Hadjidimitriou, medical Engineering, 59, 3498 (2012)
  28. Haselager Pim, Did I Do That? Brain–Computer Interfacing and the Sense of Agency, 10.1007/s11023-012-9298-7
  29. Heger Dominic, Herff Christian, Putze Felix, Mutter Reinhard, Schultz Tanja, Continuous affective states recognition using functional near infrared spectroscopy, 10.1080/2326263x.2014.912884
  30. Hidalgo-Muñoz A.R., López M.M., Pereira A.T., Santos I.M., Tomé A.M., Spectral turbulence measuring as feature extraction method from EEG on affective computing, 10.1016/j.bspc.2013.09.006
  31. Holz Elisa Mira, Botrel Loic, Kaufmann Tobias, Kübler Andrea, Long-Term Independent Brain-Computer Interface Home Use Improves Quality of Life of a Patient in the Locked-In State: A Case Study, 10.1016/j.apmr.2014.03.035
  32. Hosseini Seyyed Abed, Naghibi-Sistani Mohammad Bagher, Emotion recognition method using entropy analysis of EEG signals, 10.5815/ijigsp.2011.05.05
  33. Hosseini S.M. Hadi, Mano Yoko, Rostami Maryam, Takahashi Makoto, Sugiura Motoaki, Kawashima Ryuta, Decoding what one likes or dislikes from single-trial fNIRS measurements : , 10.1097/wnr.0b013e3283451f8f
  34. Jenke Robert, Peer Angelika, Buss Martin, Feature Extraction and Selection for Emotion Recognition from EEG, 10.1109/taffc.2014.2339834
  35. Jie, Biomed Mater Eng, 24, 1185 (2014)
  36. Kamienkowski J. E., Ison M. J., Quiroga R. Q., Sigman M., Fixation-related potentials in visual search: A combined EEG and eye tracking study, 10.1167/12.7.4
  37. Kashihara Koji, A brain-computer interface for potential non-verbal facial communication based on EEG signals related to specific emotions, 10.3389/fnins.2014.00244
  38. Kassam Karim S., Markey Amanda R., Cherkassky Vladimir L., Loewenstein George, Just Marcel Adam, Identifying Emotions on the Basis of Neural Activation, 10.1371/journal.pone.0066032
  39. Kim Min-Ki, Kim Miyoung, Oh Eunmi, Kim Sung-Phil, A Review on the Computational Methods for Emotional State Estimation from the Human EEG, 10.1155/2013/573734
  40. Koelstra Sander, Patras Ioannis, Fusion of facial expressions and EEG for implicit affective tagging, 10.1016/j.imavis.2012.10.002
  41. Kübler Andrea, Holz Elisa M., Riccio Angela, Zickler Claudia, Kaufmann Tobias, Kleih Sonja C., Staiger-Sälzer Pit, Desideri Lorenzo, Hoogerwerf Evert-Jan, Mattia Donatella, The User-Centered Design as Novel Perspective for Evaluating the Usability of BCI-Controlled Applications, 10.1371/journal.pone.0112392
  42. Kübler Andrea, Neumann Nicola, Wilhelm Barbara, Hinterberger Thilo, Birbaumer Niels, Predictability of Brain-Computer Communication, 10.1027/0269-8803.18.23.121
  43. Lee You-Yun, Hsieh Shulan, Classifying Different Emotional States by Means of EEG-Based Functional Connectivity Patterns, 10.1371/journal.pone.0095415
  44. Lemm Steven, Blankertz Benjamin, Dickhaus Thorsten, Müller Klaus-Robert, Introduction to machine learning for brain imaging, 10.1016/j.neuroimage.2010.11.004
  45. Liberati Alessandro, The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration, 10.7326/0003-4819-151-4-200908180-00136
  46. Liberati, Affective Computing and Intelligent Interaction, 838 (2013)
  47. Liberati, Journal of Alzheimer’s Disease, 29, 1 (2012)
  48. Liberati Giulia, Pizzimenti Alessia, Simione Luca, Riccio Angela, Schettini Francesca, Inghilleri Maurizio, Mattia Donatella, Cincotti Febo, Developing brain-computer interfaces from a user-centered perspective: Assessing the needs of persons with amyotrophic lateral sclerosis, caregivers, and professionals, 10.1016/j.apergo.2015.03.012
  49. Liberati Giulia, Veit Ralf, Dalboni da Rocha Josué, Kim Sunjung, Lulé Dorothée, von Arnim Christine, Raffone Antonino, Belardinelli Marta Olivetti, Birbaumer Niels, Sitaram Ranganatha, Combining classical conditioning and brain-state classification for the development of a brain-computer interface (BCI) for Alzheimer's patients, 10.1016/j.jalz.2012.05.1397
  50. Lin Yuan-Pin, Yang Yi-Hsuan, Jung Tzyy-Ping, Fusion of electroencephalographic dynamics and musical contents for estimating emotional responses in music listening, 10.3389/fnins.2014.00094
  51. Lotte F, Congedo M, Lécuyer A, Lamarche F, Arnaldi B, A review of classification algorithms for EEG-based brain–computer interfaces, 10.1088/1741-2560/4/2/r01
  52. Mahalanobis, Proceedings of the National Institute of Sciences, 2, 49 (1936)
  53. McLachlan, Discriminant analysis and statistical pattern recognition (2004)
  54. Mikhail Mina, Ayat Khaled El, Coan James A., Allen John J.B., Using minimal number of electrodes for emotion detection using brain signals produced from a new elicitation technique, 10.1504/ijaacs.2013.050696
  55. Moghimi Saba, Kushki Azadeh, Guerguerian Anne Marie, Chau Tom, Characterizing emotional response to music in the prefrontal cortex using near infrared spectroscopy, 10.1016/j.neulet.2012.07.009
  56. Moghimi Saba, Kushki Azadeh, Power Sarah, Guerguerian Anne Marie, Chau Tom, Automatic detection of a prefrontal cortical response to emotionally rated music using multi-channel near-infrared spectroscopy, 10.1088/1741-2560/9/2/026022
  57. Moher David, Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement, 10.7326/0003-4819-151-4-200908180-00135
  58. MORGANE P, GALLER J, MOKLER D, A review of systems and networks of the limbic forebrain/limbic midbrain, 10.1016/j.pneurobio.2005.01.001
  59. Murugappan Murugappan, Ramachandran Nagarajan, Sazali Yaacob, Classification of human emotion from EEG using discrete wavelet transform, 10.4236/jbise.2010.34054
  60. Murugappan, International Journal of Computers and Communications, 1, 21 (2007)
  61. Mühl Christian, Allison Brendan, Nijholt Anton, Chanel Guillaume, Affective brain-computer interfaces: Special Issue editorial, 10.1080/2326263x.2014.913829
  62. Mühl Christian, Allison Brendan, Nijholt Anton, Chanel Guillaume, A survey of affective brain computer interfaces: principles, state-of-the-art, and challenges, 10.1080/2326263x.2014.912881
  63. Mühl, The Oxford Handbook of Affective Computing, 217 (2014)
  64. Müller Matthias M., Keil Andreas, Gruber Thomas, Elbert Thomas, Processing of affective pictures modulates right-hemispheric gamma band EEG activity, 10.1016/s1388-2457(99)00151-0
  65. Nicolas-Alonso Luis Fernando, Gomez-Gil Jaime, Brain Computer Interfaces, a Review, 10.3390/s120201211
  66. Nijboer Femke, Plass-Oude Bos Danny, Blokland Yvonne, van Wijk René, Farquhar Jason, Design requirements and potential target users for brain-computer interfaces – recommendations from rehabilitation professionals, 10.1080/2326263x.2013.877210
  67. Nijholt, Computer Interfaces, 1, 63 (2013)
  68. Petrantonakis Panagiotis C., Hadjileontiadis Leontios J., Emotion Recognition from Brain Signals Using Hybrid Adaptive Filtering and Higher Order Crossings Analysis, 10.1109/t-affc.2010.7
  69. Picard, Affective computing (2000)
  70. Putze Felix, Schultz Tanja, Adaptive cognitive technical systems, 10.1016/j.jneumeth.2014.06.029
  71. Reuderink B., Poel M., Nijholt A., The Impact of Loss of Control on Movement BCIs, 10.1109/tnsre.2011.2166562
  72. Russell James A, Mehrabian Albert, Evidence for a three-factor theory of emotions, 10.1016/0092-6566(77)90037-x
  73. Schettini Francesca, Riccio Angela, Simione Luca, Liberati Giulia, Caruso Mario, Frasca Vittorio, Calabrese Barbara, Mecella Massimo, Pizzimenti Alessia, Inghilleri Maurizio, Mattia Donatella, Cincotti Febo, Assistive Device With Conventional, Alternative, and Brain-Computer Interface Inputs to Enhance Interaction With the Environment for People With Amyotrophic Lateral Sclerosis: A Feasibility and Usability Study, 10.1016/j.apmr.2014.05.027
  74. Schmidt Louis A., Trainor Laurel J., Frontal brain electrical activity (EEG) distinguishes valence and intensity of musical emotions, 10.1080/02699930126048
  75. Simon, Frontiers in Human Neuroscience, 8, 1039 (2014)
  76. Sitaram Ranganatha, Lee Sangkyun, Ruiz Sergio, Rana Mohit, Veit Ralf, Birbaumer Niels, Real-time support vector classification and feedback of multiple emotional brain states, 10.1016/j.neuroimage.2010.08.007
  77. Soleymani M., Pantic M., Pun T., Multimodal Emotion Recognition in Response to Videos, 10.1109/t-affc.2011.37
  78. Stikic Maja, Johnson Robin R., Tan Veasna, Berka Chris, EEG-based classification of positive and negative affective states, 10.1080/2326263x.2014.912883
  79. Strait Megan, Scheutz Matthias, What we can and cannot (yet) do with functional near infrared spectroscopy, 10.3389/fnins.2014.00117
  80. Tai Kelly, Chau Tom, Single-trial classification of NIRS signals during emotional induction tasks: towards a corporeal machine interface, 10.1186/1743-0003-6-39
  81. van Erp Jan B. F., Brouwer Anne-Marie, Zander Thorsten O., Editorial: Using neurophysiological signals that reflect cognitive or affective state, 10.3389/fnins.2015.00193
  82. van Erp Jan, Lotte Fabien, Tangermann Michael, Brain-Computer Interfaces: Beyond Medical Applications, 10.1109/mc.2012.107
  83. Vapnik, Support vector method for function approximation, regression estimation, and signal processing (1997)
  84. Vytal Katherine, Hamann Stephan, Neuroimaging Support for Discrete Neural Correlates of Basic Emotions: A Voxel-based Meta-analysis, 10.1162/jocn.2009.21366
  85. Yoon Hyun Joong, Chung Seong Youb, EEG-based emotion estimation using Bayesian weighted-log-posterior function and perceptron convergence algorithm, 10.1016/j.compbiomed.2013.10.017
  86. Yuvaraj R., Murugappan M., Ibrahim Norlinah Mohamed, Omar Mohd Iqbal, Sundaraj Kenneth, Mohamad Khairiyah, Palaniappan R., Satiyan M., Emotion classification in Parkinson's disease by higher-order spectra and power spectrum features using EEG signals: A comparative study, 10.1142/s021963521450006x
  87. Zander Thorsten Oliver, Lehne Moritz, Ihme Klas, Jatzev Sabine, Correia Joao, Kothe Christian, Picht Bernd, Nijboer Femke, A Dry EEG-System for Scientific Research and Brain–Computer Interfaces, 10.3389/fnins.2011.00053
  88. Zickler Claudia, Halder Sebastian, Kleih Sonja C., Herbert Cornelia, Kübler Andrea, Brain Painting: Usability testing according to the user-centered design in end users with severe motor paralysis, 10.1016/j.artmed.2013.08.003
Bibliographic reference Liberati, Giulia ; Federici, Stefano ; Pasqualotto, Emanuele. Extracting neurophysiological signals reflecting users’ emotional and affective responses to BCI use: A systematic literature review. In: NeuroRehabilitation : an interdisciplinary journal, Vol. 37, no.3, p. 341-358 (2015)
Permanent URL http://hdl.handle.net/2078.1/169999