Desenvolvendo um processo de Teste A/B para avaliação de Usabilidade e Experiência do Usuário

Abstract

This work presents a proposal for a software evaluation process using A/B testing, usability, and user experience. Traditionally, when acquiring software, factors such as functionality, cost, and implementation time are considered for the purchase decision, without taking into account the opinions of the users who will use the software in their daily activities. This study aimed to develop a generic process to evaluate two software options by employing usability and user experience criteria, thus supporting the decision-making process. A literature review was conducted to identify the state-of-the-art in comparing software in terms of usability and user experience, resulting in the identification of six relevant scientific articles. The study mapped their steps, techniques, methods, platforms, and resources used in testing. Subsequently, stages that could be replaced or added to the process were identified, making the new technique flexible for adaptation and customization according to specific needs. Building upon this study, a first version of a new software evaluation technique was created, named ABTUX, which stands for A/B Testing, Usability, and User Experience Evaluation Process. The process underwent adaptations and refinements with the guidance and support of software engineers, resulting in a second version. To validate the ABTUX process, it was applied to the Udemy and Coursera e-learning platforms, as they are competitors with similar functionalities. Responses were collected, and both objective and subjective answers were analyzed to assess the results. Ultimately, Coursera was identified as the software with the highest evaluation in terms of usability and user experience, based on the test’s premises and objectives. Furthermore, a questionnaire was administered to assess the evaluators who conducted the ABTUX test, aiming to qualitatively and quantitatively evaluate the process and its artifacts, identifying opportunities for improvement and guiding future studies.