NSF CRII: CPS: Wearable-Machine Interface Architectures
Acknowledgment: This material is based upon work supported by the National Science Foundation under Grant Number 1755780
Disclaimer: Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
- Project Overview
Wrist-worn wearable devices provide rich sets of pulsatile physiological data under various modalities and circumstances. An unexploited capability is that the pulsatile physiological time series collected by wrist-worn wearable devices can be used for recovering internal brain dynamics. The goal of this project is to present wearable machine-interface (WMI) architectures related to mental stress and their potential applications for tracking fatigue and arousal states. Decoding brain states using wrist-worn wearables will transform how mental-stress-related diseases are diagnosed and treated. This proposal presents two design classes of WMI architectures related to mental stress: (1) A decoder that using skin conductance data recovers undesired stimuli triggered brain activity from wearables and a controller that delivers intermittent stimulation to reverse the adverse effect of stimuli. (2) A decoder that using cortisol data recovers undesired stimuli-triggered inhibition of cortisol secretion from wearables and a controller that delivers intermittent stimulation to generate the desired cortisol profile. The proposed methods will be validated by analyzing electrodermal activity as well as concurrent cortisol and adrenocorticotropic hormone pulsatile data in the context of mental-stress-related arousal and fatigue.
Video Summary of the CRII CPS Project and Results
– E-Books:
[1] McConnell, G.C., Santaniello, S., Gale, J.T., Faghih, R.T., Kemere, C., Han, M., Hilliard, J.D.,“Towards the Next Generation of Deep Brain Stimulation Therapies: Technological Advancements, Computational Methods, and New Targets“, Frontiers in Neuroscience, 2021.
– Journal Articles:
[1] Steel A. G, Parekh S, Fekri Azgomi H, Badri Ahmadi M, Craik A, Pati S, Francis J. T.Contreras-Vidal J.L., and Faghih R.T., “A Mixed Filtering Approach for Real-Time Seizure State Tracking Using Multi-Channel Electroencephalography Data” IEEE Transactions on Neural Systems and Rehabilitation Engineering 2021.(Supplementary Info)
[2] Fekri Azgomi H, Hahn J-O, and Faghih R.T., “Closed-Loop Fuzzy Energy Regulation in Patients With Hypercortisolism via Inhibitory and Excitatory Intermittent Actuation,” Frontiers in Neuroscience,2021. (Supplementary Info)
[3] Fekri Azgomi H, Cajgas I, and Faghih R.T., “Closed-Loop Cognitive Stress Regulation Using Fuzzy Control in Wearable-Machine Interface Architectures,” IEEE Access, 2021.
[4] Casile A, Faghih R.T., Brown , “Robust point-process Granger causality analysis in presence of exogenous temporal modulations and trial-by-trial variability in spike trains,” PLoS computational biology, 2021.
[5] Wickramasuriya D.S., and Faghih R.T., “A Marked Point Process Filtering Approach for Tracking Sympathetic Arousal from Skin Conductance” IEEE Access, 2020.
[6] Amin, M.R. and Faghih, R.T., 2020. “Identification of Sympathetic Nervous System Activation from Skin Conductance: A Sparse Decomposition Approach with Physiological Priors“. IEEE Transactions on Biomedical Engineering, 2020. (Supplementary Info).
[7] Genty J. X., Amin M.R., Shaw N. D., Klerman E. B., and Faghih R.T., “Sparse Deconvolution of Pulsatile Growth Hormone Secretion in Children,” IEEE/ACM Transactions on Computational Biology and Bioinformatics, 2021. (Supplementary Info).
[8] Pednekar D, Amin M.R., Fekri Azgomi H, Aschbacher K, J. Crofford L, and Faghih R.T., “Characterization of Cortisol Dysregulation in Fibromyalgia and Chronic Fatigue Syndromes: A State-Space Approach” IEEE Transactions on Biomedical Engineering, 2020. (supplementary info).
[9] Wickramasuriya, D.S., and Faghih R.T., “A Mixed Filter Algorithm for Sympathetic Arousal Tracking from Skin Conductance and Heart Rate Measurements in Pavlovian Fear Conditioning,” PLOS ONE, 2020.
[10] Amin M.R., and Faghih R.T., “Robust Inference of Autonomic Nervous System Activation using Skin Conductance Measurements: A Multi-Channel Sparse System Identification Approach,” IEEE Access, 2019.
[11] Taghvafard, H., Cao, M., Kawano, Y. and Faghih, RT “Design of Intermittent Control for Cortisol Secretion under Time-Varying Demand and Holding Cost Constraints” IEEE Transactions on Biomedical Engineering, 2019.
[12] Wickramasuriya DS, Amin MR, and Faghih RT (2019). “Skin Conductance as a Viable Alternative for Closing the Deep Brain Stimulation Loop in Neuropsychiatric Disorders,” Frontiers in Neuroscience 13:780. DOI: 10.3389/fnins.2019.00780
[13] Wickramasuriya D.S., and Faghih R.T., “A Bayesian Filtering Approach for Tracking Arousal from Binary and Continuous Skin Conductance Features,” IEEE Transactions on Biomedical Engineering, 2019. (supplementary info).
[14] Amin M.R., and Faghih R.T., “Sparse Deconvolution of Electrodermal Activity via Continuous-Time System Identification,” IEEE Transactions on Biomedical Engineering (2018).
– Conference Proceedings:
[1] Khazaei, S., Amin, M.R, and Faghih, R.T., “Decoding a Neurofeedback-Modulated Cognitive Arousal State to Investigate Performance Regulation by the Yerkes-Dodson Law.“The Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2021.
[2] Yaghmour, A., Amin, M.R, and Faghih, R.T., “Decoding a Music-Modulated Cognitive Arousal State using Electrodermal Activity and Functional Near-infrared Spectroscopy Measurements.“The Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2021.
[3] Amin, M.R, Tahir, M., and Faghih, R.T.,”A State-Space Investigation of Impact of Music on Cognitive Performance during a Working Memory Experiment.“The Annual International Conference of the IEEE Engineering in Medicine and Biology Society,2021.
[4] Seet M. S., Amin M. R., Abbasi N. I., Hamano J., Bezerianos A., Faghih R.T., Thakor N. V., Dragomir A., “Olfactory-induced Positive Affect and Autonomous Response as a Function of Hedonic and Intensity Attributes of Fragrances,” The Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2020.
[5] Wickramasuriya D.S., Tessmer M.K., and Faghih R.T., “Facial Expression-Based Emotion Classification using Electrocardiogram and Respiration Signals,” the IEEE-EMB Special Topics Conference on Healthcare Innovations and Point-of-Care Technologies, Bethesda, MD, 2019.
[6] Ravindran A.S., Nakagome S., Wickramasuriya D.S., Contreras-Vidal J.L., and Faghih R.T., “Emotion Recognition by Point Process Characterization of Heartbeat Dynamics,” the IEEE-NIH Special Topics Conference on Healthcare Innovations and Point-of-Care Technologies, Bethesda, MD, 2019.
[7] Yadav T, Atique Md M, Fekri Azgomi H, Francis J. T., and Faghih R.T., “Emotional Valence Tracking and Classification via State-Space Analysis of Facial Electromyography,” IEEE Asilomar Conference on Signals, Systems and Computers 2019.
[8] Badri Ahmadi M, Craik A, Fekri Azgomi H, Francis J. T.Contreras-Vidal J.L., and Faghih R.T., “Real-Time Seizure State Tracking using Two Channels: A Mixed-Filter Approach,” IEEE Asilomar Conference on Signals, Systems and Computers 2019.
[9] Fekri Azgomi H, and Faghih R.T., “A Wearable Brain Machine Interface Architecture for Regulation of Energy in Hypercortisolism,” 53rd Asilomar Conference on Signals, Systems, and Computers, Pacific Grove, CA, 2019.
[10] Wickramasuriya D.S., and Faghih R.T., “A Cortisol-Based Energy Decoder for Investigation of Fatigue in Hypercortisolism,” The Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Berlin, Germany, 2019.
[11] Pednekar D, Amin M.R., Fekri Azgomi H, Aschbacher K, J. Crofford L, and Faghih R.T., “A System Theoretic Investigation of Cortisol Dysregulation in Fibromyalgia Patients with Chronic Fatigue,” The Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Berlin, Germany, 2019.
[12] Fekri Azgomi H, Wickramasuriya D.S., and Faghih R.T., “State-Space Modeling and Fuzzy Feedback Control of Cognitive Stress,” The Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Berlin, Germany, 2019.
[13] Wickramasuriya D.S., and Faghih R.T., “A Novel Filter for Tracking Real-World Cognitive Stress using Multi-Time-Scale Point Process Observations,” The Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Berlin, Germany, 2019.
[14] Amin M.R., and Faghih R.T., “Tonic and Phasic Decomposition of Skin Conductance Data: A Generalized-Cross-Validation-Based Block Coordinate Descent Approach,” The Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Berlin, Germany, 2019.
[15] Parshi S, Amin M.R., Fekri Azgomi H, and Faghih R.T., “Mental Workload Classification via Hierarchical Latent Dictionary Learning: A Functional Near Infrared Spectroscopy Study,” IEEE-EMBS International Conference on Biomedical and Health Informatics, Chicago, IL, 2019.
[16] Wickramasuriya D.S., Qi C., and Faghih R.T., “A State-Space Approach for Detecting Stress from Electrodermal Activity,” the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Honolulu, HI, 2018. [Video]
[17] Amin M.R., and Faghih R.T., “Inferring Autonomic Nervous System Stimulation from Hand and Foot Skin Conductance Measurements,” 52nd Asilomar Conference on Signals, Systems, and Computers, Pacific Grove, CA, 2018.
Invited One-Page Conference Papers:
[1] Wickramasuriya D.S. and Faghih R.T., “A Mixed-Filter Algorithm for Arousal Tracking from Galvanic Skin Response and Heart Rate Measurement,” IEEE-EMBS International Conference on Biomedical and Health Informatics, Chicago, IL, 2019.
[2] Amin M.R., Wickramasuriya D.S., and Faghih R.T., “Wearable-Machine Interface Architectures for Mental Stress,” The Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Berlin, Germany, 2019.
Brain Stimulation for PTSD Patients, University of Houston Press Release, 2019
Brain stimulation for PTSD patients, Science Daily, 2019
Sweat-Triggered Brain Stimulation Proposed for PTSD, Other Disorders, Psych Central, 2019
Brain stimulation for PTSD patients, EurekaAlert, 2019
Closed-loop brain stimulator based on sweat response for PTSD patients, News Medical Life Sciences, 2019
Brain Stimulation for PTSD Patients, Biospace, 2019
Brain stimulation for PTSD patients, Neuroscience News, 2019
Mood monitor, IEEE Signal Processing Magazine, 2018
Wearable Devices Explored to Detect Emotional States, Science & Enterprise, 2018.
Texas Researcher Granted Funds to Research Wearable Tech Applications in Mental Health, Fyxes.com, Startup and Tech Stories, 2018.
Wearable Technology to Track Brain, Predict Illness, HealthNewsDigest, 2018
Can I check your watch? I want to see what’s on your mind, MobiHealthNews, 2018
Wearable Technology to Track Brain, Predict Illness, University of Houston Press Release, 2018
Wearable Technology to Track Brain, Predict Illness, EurekAlert, Science News, 2018
Decoding The Brain: A Wirst-worn Wearable Designed to Fight Stress In The Age of Anxiety, FashNerd, a Digital Magazine, 2018
- Provisional Patent
R. T. Faghih. Wearable Machine Interface System for Tracking and Controlling Mental Stress Related Brain Dynamics using Wrist-Worn Wearable Devices, 2018. US Patent App. 62/614,070. [Tech Summary]
Intellectual Merit
- 10 journal articles, 14 conference papers, and 2 one-page invited conference papers.
- This research has been integrated into PI’s courses; the course projects have resulted in 3 conference papers and 36 educational videos.
- 5 senior design teams have been designing wearable-machine interface architectures.
- 1 business school team has been working on the technology transfer for this research.
- 8 Undergraduate Research Projects (Honors Thesis, REU, Summer Undergraduate Research Fellows, and Independent Study).
- PI gave several laymen talks about this research to general audience and broader community
- PI served as a panelist at Montgomery Community College to guide the students through their career path
– Girls Engineering the Future Outreach Event (2021) – (Video Link):
– Girls Engineering the Future Outreach Event (2019) – (Video Link)
– Girl’s Engineering the Future Outreach Event (2018) – (Video Link)
- Educational Videos
Modal Analysis and Canonical Forms – Video by Xinyue Zhang (Fall 2018))
Linear Quadratic Gaussian Control – Video by Rajarsi Roy (Fall 2018)
Matrix Algebra and Linear Equations – Video by Prasanna Kumar Reddy Gade (Fall 2018)
Video on Stability by Nitin Bhabsar (Fall 2018)
Gilbert Realization and Pole Zero Cancellation – Video by Divesh Pednekar
Reference Tracking and McMillan Degree – Video by Aruj Khandelwal
– Videos:
Skin Conductance Video, Undergraduate Research Video by Huy Hua and Zanne Solitz (Summer 2018).
– Undergraduate Research Posters:
Zoch S. R., Amin M.R., and Faghih R.T., “Electromyography Analysis in Emotion Detection and Recognition”. NSF REU Poster Session, University of Houston, Summer 2019.
Tadrous M., Wickramasuriya D.S., and Faghih R.T., “Exploration of Electrocardiography in Emotion Recognition”. SF REU Poster Session, University of Houston, Summer 2019.
Tessmer M.K., Wickramasuriya D.S., and Faghih R.T., “Emotional Valence Classification at High Arousal Using Heartrate Variability and Respiration”. NSF REU Poster Session, University of Houston, Summer 2018.
Hua H., Amin M.R., and Faghih R.T., “A State-space Investigation of Cortisol Alterations in Chronic Fatigue Syndrome”. Undergraduate Research D
ay, University of Houston, Fall 2018.
- Undergraduate Senior Design Project (Spring’18 – Fall’18):
– Wearable Device to Recognize Attentiveness in the Workplace:
– Senior Design Project Members:
Nicholas Hermann, Senior Design Project Computer Engineering Undergraduate Student (senior), University of Houston (January-Decmber 2018)
Huy Hua, Senior Design Project Electrical Engineering Undergraduate Student (senior), University of Houston (January-December 2018)
Zanne Soliz, Senior Design Project Electrical Engineering Undergraduate Student (senior), University of Houston (January-Decmber 2018)
Christopher Theriot, Senior Design Project Computer Engineering Undergraduate Student (senior), University of Houston (January-Decmber 2018)
- Undergraduate Senior Design Project (Spring’19 – Fall’19):
– Wearable Cardiac Monitor
Senior Design Project Members:
Antonio Ruiz, Senior Design Project (January-Decmber 2019), Electrical Engineering Undergraduate Student, University of Houston
Aathira Karippurathu Vijayakumar, Senior Design Project (January-Decmber 2019), Electrical Engineering Undergraduate Student, University of Houston
Harshitha Gandra, Senior Design Project (January-Decmber 2019), Electrical Engineering Undergraduate Student, University of Houston
Tahsin As Sami, Senior Design Project (January-Decmber 2019), Electrical Engineering Undergraduate Student, University of Houston
- Undergraduate Senior Design Project (Spring’19 – Fall’19):
– Emotion Recognition And Regulation System
Senior Design Project Members:
Jin Lu, Senior Design Project (January-Decmber 2019), Electrical Engineering Undergraduate Student, University of Houston
Yuehua Hu, Senior Design Project (January-Decmber 2019), Electrical Engineering Undergraduate Student, University of Houston
Yuzhu Zhang, Senior Design Project (January-Decmber 2019), Electrical Engineering Undergraduate Student, University of Houston
Zhaohan Liu, Senior Design Project (January-Decmber 2019), Electrical Engineering Undergraduate Student, University of Houston
WCE International Business Plan Competition Team
Annie Ngo, Intellectual Property Project (May-December 2019), Wolff Center for Entrepreneurship Undergraduate Student, University of Houston
Cory Gaddis, Intellectual Property Project (May-December 2019), Wolff Center for Entrepreneurship Undergraduate Student, University of Houston
Denah Rooks, Intellectual Property Project (May-December 2019), Wolff Center for Entrepreneurship Undergraduate Student, University of Houston
Kyle Li, Intellectual Property Project (May-December 2019), Wolff Center for Entrepreneurship Undergraduate Student, University of Houston
David Lai, Intellectual Property Project (May-December 2019), Wolff Center for Entrepreneurship Undergraduate Student, University of Houston
Milan Nguyen, Intellectual Property Project (May-December 2019), Valenti School of Communication Undergraduate Student, University of Houston
Sam Matte, Intellectual Property Project (May-December 2019), Valenti School of Communication Undergraduate Student, University of Houston
Alexandria Dave, Intellectual Property Project (May-December 2019), Valenti School of Communication Undergraduate Student, University of Houston
Broader Impact
- Physiological signal variations have been examined in:
– Students taking exams for performance assessment,
– Collaborations with the School of Theater and the College of Hotel Management for investigating emotions during acting and when learning the art of professional culinary tasting.
- Each year, 43.8 million adults in the US experience mental illness.
