Inclusive Web Empirical Studies in Remote and In-Situ Settings: A User Evaluation of the RemoTest Platform

Abstract

Web accessibility evaluation requires tests to be carried out with real users with disabilities performing real tasks or activities. To recruit an appropriate group of users and to observe their performance in the real world is difficult. For this reason we have developed RemoTest, a platform that assists researchers designing experiments, conducting remote and in-situ experimental sessions and analyzing the data gathered while the users are accessing the Web. Although this tool is oriented to experimenters, it is necessary to check whether the evaluation environments created by RemoTest are accessible or not to the users that participate in the tests. To this end, we conducted formal in-situ evaluations with 36 users with diverse characteristics. For this assessment, the participants were asked to install the platform, to fill in some automatically created questionnaires and to carry out several web navigation tasks. From the data gathered we analyzed the ease of the installation process, the accessibility of the automatically generated questionnaires, and user satisfaction. The results revealed the suitability of the platform for conducting inclusive experiments both in remote and in-situ contexts and provided guidelines on how the experiments should be set out.