Headset Removal for Gaze Contact in XR Applications
The volumetric representation of human interactions is one of the fundamental domains in the development of immersive media productions and telecommunication applications. Particularly in the context of the rapid advancement of Extended Reality (XR) applications, this volumetric data has proven to be an essential technology for future XR elaboration. In this work, we present a new multimodal database to help advance the development of immersive technologies. Our proposed database provides ethically compliant and diverse volumetric data, in particular 27 participants displaying posed facial expressions and subtle body movements while speaking, plus 11 participants wearing head-mounted displays (HMDs). The recording system consists of a volumetric capture (VoCap) studio, including 31 synchronized modules with 62 RGB cameras and 31 depth cameras. In addition to textured meshes, point clouds, and multi-view RGB-D data, we use one Lytro Illum camera for providing light field (LF) data simultaneously. Finally, we also provide an evaluation of our dataset employment with regard to the tasks of facial expression classification, HMDs removal, and point cloud reconstruction. The dataset can be helpful in the evaluation and performance testing of various XR algorithms, including but not limited to facial expression recognition and reconstruction, facial reenactment, and volumetric video. HEADSET and its all associated raw data and license agreement will be publicly available for research purposes.
Articles
Geometry-aware video inpainting for joint headset occlusion removal and face reconstruction in social extended reality
Fatemeh Ghorbani, Karen Eguiazarian, Sebastian Knorr
SPIE Journal of Electronic Imaging, 2025.
https://doi.org/10.1117/1.JEI.34.5.051003
Towards Realistic Landmark-Guided Facial Video Inpainting Based on GANs Fatemeh Ghorbani, Karen Egiazarian, Sebastian Knorr
In the proceedings of IS&T Electronic Imaging, Image Processing: Algorithms and Systems XXII, 2024.
Expression-aware video inpainting for HMD removal in XR applications Fatemeh Ghorbani, Karen Egiazarian, Sebastian Knorr
In the proceedings of ACM SIGGRAPH European Conference on Visual Media Production, 2023.
https://dl.acm.org/doi/10.1145/3626495.3626497
Headset: Human emotion awareness under partial occlusions multimodal dataset Fatemeh Ghorbani, Davi Rabbouni, Christine Guillemot, Karen Eguiazarian, Sebastian Knorr
IEEE Transactions on Visualization and Computer Graphics, 2023.
https://www.computer.org/csdl/journal/tg/2023/11/10269730/1QWNRVd3I88