Quality of Experience (QoE) and User Experience (UX)
Media consumption has become a ubiquitous part of life. Through evaluating the QoE of media experiences, we aim to quantify the users’ perceived quality of a system or service that affects their experience. What kind of compression technology satisfies users’ expectations for viewing quality? How much delay is tolerant in telecommunication for not disrupting the communication? How much bandwidth does a video stream need to ensure a satisfactory viewing experience? The group does not only focus on QoE, but aims understanding and quantifying the overall user experience, by taking into account personal and situational factors.
The Distributed and Interactive Systems group conducts controlled lab experiments and pilots with users to collect subjective scores regarding QoE. The group uses mixed methodologies for better understanding (qualitative) and, quantifying and predicting (quantitative) the experience of users. From in-depth interviews and ethnography to questionnaires and physiological signal analysis, the group is equipped with a varied toolset to truly understand and model the quality of experience of users.
Topics
- Social VR evaluation: Social Virtual Reality (Social VR) systems have the potential to support the richness of interactions resembling the face-to-face ones. In this area, we are developing tools and protocols to evaluate such immersive and interactive experiences in Social VR. In particular, we have developed a new evaluation protocol, a new questionnaire and are investigating what objective data to collect for adequate modeling;
- Point clouds: Point cloud is one of the technologies used for representing 3D scenes and objects in virtual immersive systems. A point cloud is a set of points in a 3D space, that together make up the surface of an object. We perform subjective experiments using both quantitative and qualitative methods to evaluate point cloud compression quality. The quantitative study aims to collect ground truth subjective scores, and observe user reliability when assessing point cloud quality. The qualitative study aims to explore users' perception of point cloud quality, and uncover quality dimensions that need to be considered when building objective metrics for point clouds;
- 360-degree video: Fully omnidirectional cameras, able to instantaneously capture the 360-degree surrounding real world scene, have recently started to appear as commercial products and professional tools. In order to design perceptually-optimized omnidirectional visual communications, the availability of tools to quantify the level of distortion introduced by each processing step, and, ultimately, the overall quality of the processed signal and the 360-degree experience is critical. At DIS, we study the visual distortions characteristics of 360-degree content production and processing and their impact on user’s QoE;
- Multi-party desktop videoconferencing: The objective is to estimate and predict the quality of experience of specific users in a specific situation when using a rich media-based conferencing service. The Personal QoE takes into account signal aspects, but as well the situational context, the user behavior and the user’s individual traits. To be able to achieve this goal, we need to understand how these various non-system factors influence the impact that system factors have on the experience of the user. In addition, user studies, which normally consist of system factors as input factors and user ratings as output factors, need to be extended to also incorporate user and usage factors;
- End-to-End latency: When using a videoconferencing system there will always be a delay from sender to receiver. Such delays affect human communication, and therefore knowing the delay is a major factor in judging the expected QoE of the conferencing system. Additionally, for implementers, tuning the system to reduce delay requires an ability to effectively and easily gather delay metrics on a potentially wide range of settings. In order to support this process, we make available a system called videoLat. VideoLat provides an innovative approach to understand glass-to- glass video delays and speaker-to-microphone audio delays. VideoLat can be used as-is to do audio and video roundtrip delays of black box systems, but by making it available as open source we want to enable people to extend and modify it for different scenarios, such as measuring one-way delays or delay of camera switching.
Funding
Videos
- MediaScape XR: https://youtu.be/zzB7B6EAU9c
- MediaScape XR Short: https://youtu.be/I7kY1cMZyD0
- Celebration of CWI’s 75th anniversary in SocialVR: https://youtu.be/KcRpp0s50RQ
- A Social VR Clinic for Knee Arthritis Patients with Haptics: https://youtu.be/c89E98SQRqk
- VRTogether Pilot 3: https://youtu.be/0l82gbTc1WCQ
- VRTogether: Ground-breaking Social Virtual Experiences: https://youtu.be/dL5NX74roBg
Software and Datasets
- Color-based objective quality metric for point cloud contents
- PCM_RR
- PointPCA
- Subjective assessment of dynamic point clouds
- MP3DG: Development framework for point cloud compression algorithms
- VideoLat
- VMC-TB
Publications
-
Subjective evaluation of visual quality and simulator sickness of short 360° videos: ITU-T Rec. P.919.
IEEE Transactions on Multimedia,
: pp. ,
2021
.
-
Evaluating the User Experience of a Photorealistic Social VR Movie.
In Proceedings of IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2021),
Bari, IT,
2021.
-
CakeVR: A Social Virtual Reality (VR) Tool for Co-designing Cakes.
In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (ACM CHI 2021),
Yokohama, JP,
2021.
-
A reduced reference metric for visual quality evaluation of point cloud contents.
IEEE Signal Processing Letters,
: pp. ,
2020
.
-
A new Objective Quality Metric for Point Cloud Contents Based on Color.
In Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX),
Athlone, Ireland,
2020.
-
Comparing the Quality of Highly Realistic Digital Humans in 3DoF and 6DoF: A Volumetric Video Case Study.
In Proceedings of the IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR),
Atlanta, GA, USA,
2020.
-
Measuring and Understanding Photo Sharing Experiences in Social Virtual Reality.
In Proceedings of the CHI Conference on Human Factors in Computing Systems Proceedings (CHI 2019),
Glasgow, Scotland, UK,
2019.
-
The Contrast Effect: QoE of Mixed Video-Qualities at the same Time.
Springer Quality and User Experience,
3(1): pp. ,
2018
.
-
Towards individual QoE for multi-party video conferencing.
IEEE Transactions on Multimedia (TMM),
20(7): pp. 1781-1795,
2018
.
-
2020 IEEE Transactions on Circuits and Systems for Video Technology Best Paper Award
Implementation and Evaluation of a Point Cloud Codec for Tele-Immersive Video. IEEE Transactions on Circuits and Systems for Video Technology, 27(4): pp. 828-842, 2017 .
-
User-centric video delay measurements.
In Proceedings of the ACM Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV 2013),
Oslo, Norway,
2013.
pp. 37-42.
-
Are We in Sync?: Synchronization Requirements for Watching Online Video Together.
In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI'11),
New York, NY, USA,
2011.
pp. 311-314.
Theses
- Marwim Schmitt, Personal QoE: Accurately Modeling QoE for Multi-Party Desktop Video-Conferencing Based on Systems, Context and User Factors, Vrije Universiteit Amsterdam, The Netherlands. PhD Thesis, 2019.
- Ernestasia Siahaan, Visual Quality of Experience: A Metric Driven Perspective, TU Delft, The Netherlands. PhD Thesis 2018.
Standards
- P.919 (ex. P.360-VR) - Subjective test methodologies for 360º video on head-mounted displays: https://www.itu.int/rec/T-REC-P.919/en
- P.1305 (ex P.DTM) - Effect of delays on the tele meeting quality: http://www.itu.int/itu-t/workprog/wp_item.aspx?isn=8897