About this project
The IVEA app is the result of my master’s thesis, a project provided by the UMCG with the aim of better upholding emergency medical service workers’ knowledge of rarely executed procedures over time.
This page describes in more detail what the data collection and analysis process looked like. Find out more about the full process of this project on the IVEA project page.
Roles: Researcher, data analist
Skills: Data analysis, data collection, research design, project management, UX research, (descriptive) statistics
Tools: R studio, FigJam, Firebase
Experiment
To see whether or not the intervention of the app was effective in upholding the necessary (declarative) knowledge of the cricothyroidotomy procedure, an experiment was performed. In total, 7 participants took part in the experiment, all nurses at the Emergency Medical Service. The experiment tested both the effectiveness of the measure and the usability of the app.
The experimental group was asked to use the IVEA app for two weeks, prior to the 'evaluation' session.
During the evaluation session, participants were asked to perform a cricothyroidotomy procedure, while wearing video glasses and while explaining the steps they are taking. The performance of this procedure by both the control group (who did not use IVEA) and the experimental group was compared.
Experiment design
Data collection
App usage
Through the database connected with the application, it could be viewed how the app was being used. This usage included which videos have been viewed and which answers were correct and incorrect. Furthermore, if a participant experienced a bug, they could notify me through a special page in the application. These notifications would also appear in this database. Finally, the demographic answers (age, years of occupation and whether or not they have had to execute a cricothyroidotomy before) could be found in this database.
Demographics questions in app upon signin up
Usability
To investigate the extent to which IVEA was useful and usable, the usability of the product was investigated mainly in a qualitative, but also in a quantitative way.
Qualitative data on the usability of the product was gathered through open questions in the usability questionnaire, as well as through informal unstructured interviews after the procedure was executed. The findings from the informal unstructured interviews were recorded by noting down the most significant comments from participants during and immediately after the interview. Quantitative usability data were gathered through the SUS (System Usability Scale) questions, which are substantiated in their validity through years of research and applied by many UX researchers.
Part 1 of the SUS usability questionnaire
Video footage
Video footage of the experiment was made using Video glasses, so that it is visible what the participant is doing from their own point of view.
iVUE professional video glasses
Data analysis methods
Evaluation by medical experts
The video footage that was recorded with the video glasses was evaluated by three EMS educational experts. The videos were qualitatively evaluated, describing per participant to what extent the appropriate steps were taken in the correct manner during the procedure. All three experts evaluated the video footage separately without discussion of the results. This way, the findings were not influenced by each other.
The results from expert evaluation of the video footage of the experiment showed that generally, the control group participants tended to perform better at the cricothyroidotomy procedure than the experimental group participants. From these results, however, no firm, significant, conclusion can be drawn about the performance due to the most confounding factor of the experiment, the low number of participants and other confounding factors such as the age and experience of participants
Video footage of participant's point of view
Quantitative usability
The SUS questionnaire results are described through descriptive statistics as well as a SUS score, which indicates the overall usability of the product.
The averaged results of the SUS questions generally indicated high usability. The SUS score that resulted from this is 80.83, as calculated through the formula that corresponds to the SUS questionnaire. This score translates to a 'good' to 'excellent' on the adjective rating scale (Bangor et al., 2009). In terms of usability, this indicates that the product is acceptable to use.
SUS results
Personal observation of video footage
The video footage was also observed with a focus on the comparison between comments by the participants from the experimental group and the questions and answers in the application. The specific phrasing of the content of the application (e.g. questions, answers, spoken text in the videos) was compared to the literal comments of participants during the execution of the procedure. Also, the results from the app usage (e.g. which questions were answered correctly and incorrectly) were compared to the performance in the video footage.
Analysis showed that indeed, participants who performed well iterate (sometimes as literally worded in the video exercises) the steps showed in the practice videos.
Qualitative
The analysis of the qualitative usability findings was performed through a thematic content analysis, a method created by Rosemarie Anderson in 2007.
The qualitative findings of the open questionnaire questions and the informal unstructured interview were first combined. Relevant quotes and descriptions were then highlighted. The criteria for 'relevant' descriptions and quotes were that they were only focused on the use of the app, not on the download process (since this was not representative of how the actual download process would be). Furthermore, the focus was on finding indications of usability issues and the underlying opinions and motivations of participants. Furthermore, suggestions and positive comments were considered 'relevant'. This process is referred to as 'coding'.
From this thematic coding, it became apparent that four main themes could be established: the current situation of studying for an evaluation, potential improvements for the application, positive notes of participants and technical difficulties experienced with the application. The suggestions for improvements and comments about the current situation can provide interesting directions for future development and research. Also indications were found of limitations of the application, namely the fact that the videos were considered somewhat inconsistent and incoherent. The different videos used different materials for the procedure.
Overall, the positive notes give indications that the concept of IVEA would be useful in the everyday lives of the nurses.
