Multi-Modal Technology for User Interface Analysis including Mental State Detection and Eye Tracking Analysis

FieldValue
dc.contributor.authorHusseini Orabi, Ahmed
dc.date.accessioned2017-07-31T13:25:44Z
dc.date.available2017-07-31T13:25:44Z
dc.date.issued2017
dc.identifier.urihttp://hdl.handle.net/10393/36451
dc.identifier.urihttp://dx.doi.org/10.20381/ruor-20731
dc.description.abstractWe present a set of easy-to-use methods and tools to analyze human attention, behaviour, and physiological responses. A potential application of our work is evaluating user interfaces being used in a natural manner. Our approach is designed to be scalable and to work remotely on regular personal computers using expensive and noninvasive equipment. The data sources our tool processes are nonintrusive, and captured from video; i.e. eye tracking, and facial expressions. For video data retrieval, we use a basic webcam. We investigate combinations of observation modalities to detect and extract affective and mental states. Our tool provides a pipeline-based approach that 1) collects observational, data 2) incorporates and synchronizes the signal modality mentioned above, 3) detects users' affective and mental state, 4) records user interaction with applications and pinpoints the parts of the screen users are looking at, 5) analyzes and visualizes results. We describe the design, implementation, and validation of a novel multimodal signal fusion engine, Deep Temporal Credence Network (DTCN). The engine uses Deep Neural Networks to provide 1) a generative and probabilistic inference model, and 2) to handle multimodal data such that its performance does not degrade due to the absence of some modalities. We report on the recognition accuracy of basic emotions for each modality. Then, we evaluate our engine in terms of effectiveness of recognizing basic six emotions and six mental states, which are agreeing, concentrating, disagreeing, interested, thinking, and unsure. Our principal contributions include the implementation of a 1) multimodal signal fusion engine, 2) real time recognition of affective and primary mental states from nonintrusive and inexpensive modality, 3) novel mental state-based visualization techniques, 3D heatmaps, 3D scanpaths, and widget heatmaps that find parts of the user interface where users are perhaps unsure, annoyed, frustrated, or satisfied.
dc.language.isoen
dc.publisherUniversité d'Ottawa / University of Ottawa
dc.subjectComputer vision
dc.subjectConvolutional Neural Network
dc.subjectMultimodal Deep learning
dc.subjectMachine learning
dc.subjectHCI
dc.subjectUsability
dc.subjectAffective computing
dc.subjectMental states
dc.subjectEye tracking
dc.subjectSoftware engineering
dc.subjectPsychophysiology
dc.subjectPipeline
dc.subjectUmple
dc.subjectComponent-based Development
dc.subjectEmotion
dc.subjectFacial Action Coding System
dc.subjectFacial Expression
dc.titleMulti-Modal Technology for User Interface Analysis including Mental State Detection and Eye Tracking Analysis
dc.typeThesis
dc.contributor.supervisorLethbridge, Timothy
thesis.degree.namePhD
thesis.degree.levelDoctoral
thesis.degree.disciplineGénie / Engineering
uottawa.departmentScience informatique et génie électrique / Electrical Engineering and Computer Science
CollectionThèses, 2011 - // Theses, 2011 -

Files
Husseini_Orabi_Ahmed_2017_thesis.pdfPhD Thesis5.31 MBAdobe PDFOpen