Immersive Multimedia Systems

We conduct research to develop and optimize immersive multimedia systems based on the end-user’s perception, behavior and expectations. Our goal is to tailor the systems around the user, increasing the feeling of presence and immersion, and delivering a better experience in remote communication. Understanding and modeling how humans perceive multimedia signals in immersive environments; what are the social and communication cues that impact user experience when a multimedia system is used; how immersiveness affects the user experience; what are users’ expectations when using the system, is key in order to design and develop algorithms and technologies that provide a satisfactory experience to the end-user. Exploiting this knowledge, immersive multimedia systems can be designed and optimized to provide the best performance. For example, user navigation information can be used to optimize the delivery of large volumetric media, thus increasing the users’ satisfaction while adhering to bandwidth constraints.

A key focus of our group is on human-centered distribution, compression and evaluation of immersive multimedia signals, including research on:

• Analysis of user behaviour and navigation patterns when consuming immersive multimedia contents
• User-adaptive distribution strategies for immersive multimedia contents
• Optimization strategies for transmission of immersive contents, including compression and spatio-temporal interpolation
• Prototypes and infrastructures for studying immersive multimedia distribution and user consumption in real-world settings
• Analysis of performance and user experience with real-time immersive multimedia systems

Topics

• Multimedia streaming
• Experience-aware networking and software defined networking
• Adaptive streaming for immersive experiences
• Perceptually-aware compression
• Point cloud compression
• Omnidirectional image and video compression
• Acquisition of immersive multimedia
• Spatio-temporal augmentation of immersive multimedia signals

Funding

• User-centered Optimization for Immersive Multimedia Systems
• VRTogether
• REVERIE
• Personalized Aftermovies and Festival Experiences
• iNEM4U
• SPICE
• BRICKS

Videos

• Point Cloud Dynamic Human Dataset for Social XR: https://youtu.be/1CctNMseQZ0
• Pipeline for Multiparty Volumetric Videoconferencing: https://youtu.be/noL9pc4OzFY
• VRTogether Social VR Platform: https://youtu.be/Ga9nYBH_Mvk
• World’s First Volumetric Video Conference over a Public 5G Network: https://youtu.be/EuoRfy18_Fc
• TA2: Improving Social Communication Between Groups: https://www.youtube.com/watch?v=5XrT0f0Aw78
• Distributed Tempest: https://www.youtube.com/watch?v=GXEcJX1LIbg
• VRTogether: Ground-breaking Social Virtual Experiences: https://www.youtube.com/watch?v=dL5NX74roBg

Software and Datasets

• Pointcloud Social XR Dataset
• 6-DoF VR User Navigation Dataset
• Proxemics and Social Interactions in an Instrumented Virtual Reality Workshop
• MP3DG: Development framework for point cloud compression algorithms
• https://github.com/tofis/human4d_dataset
• https://www.dis.cwi.nl/software-datasets/videolat/
• MDCS

Publications

• J. R. Williamson, J. Li, D. A. Shamma, V. Vinayagamoorthy, P. Cesar Proxemics and social interactions in an instrumented virtual reality workshop. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (ACM CHI), Yokohama, Japan, 2021.
  
  DOI10.1145/3411764.3445729 
• S. Rossi, I. Viola, L. Toni, P. Cesar A New Challenge: Behavioural Analysis Of 6-DOF User When Consuming Immersive Media. In 2021 IEEE International Conference on Image Processing (ICIP), Anchorage, AK, USA, 2021.
  
  DOI10.1109/ICIP42928.2021.9506525 
• N. Numan, F. ter Haar, P. Cesar Generative RGB-D Face Completion for Head-Mounted Display Removal. In 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Lisbon, PT, 2021.
  
  DOI10.1109/VRW52623.2021.00028 
• A. Chatzitofis, L. Saroglou, P. Boutis, P. Drakoulis, N. Zioulis, S. Subramanyam, B. Kevelham, C. Charbonnier, P. Cesar, D. Zarpalas, S. Kollias, P. Daras HUMAN4D: A Human-Centric Multimodal Dataset for Motions and Immersive Media. IEEE Access, : pp. , 2020 .
  
  DOI10.1109/ACCESS.2020.3026276 
• S. Subramanyam, I. Viola, A. Hanjalic, P. Cesar User Centered Adaptive Streaming of Dynamic Point Clouds with Low Complexity Tiling. In MM '20: Proceedings of the 28th ACM International Conference on Multimedia, Seattle, WA, USA, 2020.
  
  DOI10.1145/3394171.3413535 
• Best Demo
  J. Jansen, S. Subramanyam, R. Bouqueau, G. Cernigliaro, M. Martos Cabré, F. Pérez, and P. Cesar A Pipeline for Multiparty Volumetric Video Conferencing: Transmission of Point Clouds over Low Latency DASH. In Proceedings of the the ACM Multimedia Systems Conference (MMSys), Istanbul, Turkey, 2020.
  
  DOI10.1145/3339825.3393578 
• I. Viola, J. Mulder, F. De Simone, and P. Cesar Temporal Interpolation of Dynamic Digital Humans using Convolutional Neural Networks. In Proceedings of the IEEE International Conference on Artificial Intelligence & Virtual Reality, San Diego, CA, USA, 2019.
  
  DOI10.1109/AIVR46125.2019.00022 
• F. De Simone, J. Gutiérrez, P. Le Callet Complexity measurement and characterization of 360-degree content. In Proceedings of the Human Vision and Electronic Imaging Conference 2019, Burlingame, CA, USA, 2019.
  
  DOI10.2352/ISSN.2470-1173.2019.12.HVEI-216 
• R.G. de A. Azevedo, N. Birkbeck, F. De Simone, I. Janatra, B. Adsumilli, P. Frossard Visual Distortions in 360-degree Videos. IEEE Transactions on Circuits and Systems for Video Technology, : pp. , 2019 .
  
  DOI10.13140/RG.2.2.22377.75364 
• S. Schwarz, M. Preda, V. Baroncini, M. Budagavi, P. Cesar, et.al. Emerging MPEG Standards for Point Cloud Compression. IEEE Journal on Emerging and Selected Topics in Circuits and Systems (IEEE JETCAS), 9(1): pp. 133-148, 2019 .
  
  DOI10.1109/JETCAS.2018.2885981 
• Best Paper
  X. Corbillon, F. De Simone, G. Simon, and P. Frossard Dynamic Adaptive Streaming for Multi-viewpoint Omnidirectional Videos. In Proceedings of the 9th ACM Multimedia Systems Conference (MMSys 2018), Amsterdam, NL, 2018. pp. 237-249.
  
  DOI10.1145/3204949.3204968 
• J.W. Kleinrouweler, B. Meixner, J.W. Bosman, J.L. van den Berg, and P. Cesar Improving Mobile Video Quality Through Predictive Channel Quality Based Buffering. In Proceedings of the 28th International Teletraffic Congress, (ITC 30), Vienna, AT, 2018. pp. 236-244.
  
  DOI10.1109/ITC30.2018.00042 
• M. Rizkallah, F. De Simone, T. Maugey, C. Guillemot, P. Frossard Rate distortion optimized graph partitioning for omnidirectional image coding. In Proceedings of the 26th European Signal Processing Conference (EUSIPCO), Rome, IT, 2018. pp. 897-901.
  
  DOI10.23919/EUSIPCO.2018.8553080 
• 2020 IEEE Transactions on Circuits and Systems for Video Technology Best Paper Award
  R. Mekuria, K. Blom, and P. Cesar Design, Implementation and Evaluation of a Point Cloud Codec for Tele-Immersive Video. IEEE Transactions on Circuits and Systems for Video Technology, 27(4): pp. 828-842, 2017 .
  
  DOI10.1109/TCSVT.2016.2543039 
• Best Paper
  J.W. Kleinrouweler, S. Cabrero, and P. Cesar An SDN Architecture for Privacy-friendly Network Assisted DASH. ACM Transactions on Multimedia Computing Communications and Applications (ACM TOMM), 13(3s), Article 44, 22 pages, 2017 .
  
  DOI10.1145/3092838 
• J.W. Kleinrouweler, S. Cabrero, R. van der D. Mei, and P. Cesar A Model for Evaluating Sharing Policies for Network-Assisted HTTP Adaptive Streaming. Elsevier Computer Networks Journal (ComNet), 109(part 2): pp. 234-245, 2016 .
  
  DOI10.1016/j.comnet.2016.03.023 
• R. Mekuria, M. Sanna, E. Izquierdo, D.C.A. Bulterman, and P. Cesar Enabling 3D Tele-Immersion with Live Reconstructed Mesh Geometry with Fast Mesh Compression and Linear Rateless Coding. IEEE Transactions on Multimedia (TMM), 16(7): pp. 1809-1820, 2014 .
  
  DOI10.1109/TMM.2014.2331919 
• I. Vaishnavi, P. Cesar, D.C.A. Bulterman, O. Friedrich, S. Gunkel, and D. Geerts From IPTV to Synchronous Shared Experiences: Challenges in Design: Distributed Media Synchronization. Elsevier Signal Processing: Image Communication (Elsevier IMAGE), 26(7): pp. 370-377, 2011 .
  
  DOI10.1016/j.image.2011.01.006 

Theses

• Jan Willem Kleinrouweler, Modeling and Optimization of Network Assisted Video Streaming, Vrije Universiteit Amsterdam, The Netherlands. PhD Thesis, 2020.
• Rufael Mekuria, Network Streaming and Compression for Mixed Reality Tele-Immersion, Vrije Universiteit Amsterdam, The Netherlands. PhD Thesis, 2017.
• Mario Montagud Climent, Design, Development and Evaluation of an Adaptive RTCP-based IDMS Solution, Universidad Politecnica de Valencia, Spain. PhD Thesis, 2015.
• Ishan Vaishnavi, Coherence in Synchronous Shared Experiences, Vrije Universiteit Amsterdam, The Netherlands. PhD Thesis, 2011.

Standards

• MPEG-4.3D Graphics Compression Model