A brain-computer interface for object selection and tracking in videos

This thesis presents a brain-computer interface (BCI) for object selection and trackingin videos. The presented BCI utilizes steady-state visual evoked potentials(SSVEPs), measured via electroencephalography (EEG), and eye tracking. We introducetwo EEG experiment setups for data collection: an invisible grid and avisible grid, overlaid on videos. A flickering effect is obtained for SSVEP stimulationwith the distinct frequencies assigned to the grid regions. Although the invisiblegrid setup is more comfortable for the users, the visible grid has the advantage offacilitating object tracking. We conducted EEG experiments with 12 participants.Each experiment consisted of 16 videos that included 3, 4, or 5 moving objects (humanor vehicle), and each video was repeated 16 times, resulting in a total of 256trials per participant. In each trial, participants were instructed to focus on theindicated object.For object selection, a trial is assumed to be successful if EEG signal processing(i.e. SSVEP decoding) correctly predicts the object the participant focused on.Participants in the invisible grid setup achieved selection accuracies of 48% withoutaveraging EEG data and 79% with averaging EEG data across the repetitions ofeach video, significantly exceeding the average chance level which is around 27%.Canonical Correlation Analysis (CCA) was utilized in the analysis to decode SSVEPsignals effectively. In the visible grid setup, accuracies reached 51% without averagingand 82% with averaging, consistently surpassing the chance level. Analysesbased on individual videos highlighted the advantages of averaging, especially inchallenging scenarios with overlapping or ambiguous objects.For object tracking, we conduct selection via a sliding window in time. Althoughstandalone eye tracking offers high spatial precision, its accuracy is limited by calibrationdrift and gaze tracking loss, with average accuracies of 43.75% for a 1.25-second window and 35.67% for a 2.5-second window. To address these limitations,a fusion-based methodology combining EEG and eye-tracking data was developed,allowing the system to prioritize eye-tracking data when properly calibrated whileusing EEG data as a fallback in calibration issues or data inconsistencies in eyetracking.This fusion method led to an increase in overall accuracy, rising to 51.04%for a window size of 1.25 seconds and 48.96% for a window size of 2.5 seconds,demonstrating the added value of incorporating EEG signals. Our fusion methodalso reduced the Root Mean Square Error (RMSE) compared to standalone modalities,decreasing values from 1.99 and 2.27 (eye tracker only) to 1.09 and 1.15 forwindows of 1.25 and 2.5 seconds, respectively. This indicates that the fusion methodeffectively compensates for the shortcomings of individual modalities. Overall, thesefindings demonstrate that SSVEP-based BCIs are highly effective for object selectionand tracking, providing robust performance in complex scenarios. This thesishighlights the potential of these systems to improve real-time decision making andinteractions in dynamic environments. This advancement paves the way for futureapplications in various fields, ranging from defense to multimedia technologies.