Early Usability Assessment of a Conversational Agent for HPV Vaccination

Conversational user interfaces, or conversational agents (CA), such as chatbots are an emerging technology trend. This paper explores the notion that "harnessing CA for health communication could consistently and systematically perform dialogue tasks while alleviating the communication burden of the provider and supplement the delivery of health information, specifically related to the HPV [human papillomavirus] vaccine." The aim is to evaluate the usability and acceptability of the system in order to refine the idea of using an automated CA for HPV counseling at a clinical environment.

The researchers employed the Wizard of OZ protocol, which simulates speech interfaces with a potential user who thinks he or she is interacting with an automated machine or robot. In order to develop the dialogue script for the protocol, they used an HPV vaccine survey called the Carolina HPV Immunization Attitudes and Beliefs Scale (CHAIS), which contains survey items categorised by the Health Belief Model (HBM). This provided an initial baseline of talking points to communicate HPV vaccine information to prospective users. Previous studies have found that when physicians continued to pursue their original vaccination recommendations when speaking with vaccine-hesitant parents (VHPs), significantly more VHPs ultimately accepted the physician's vaccination recommendations. Thus, the researchers incorporated a "pursuit" section - e.g., a provider stating, "If (s)he were my child, I would definitely go ahead with the vaccination", as opposed to just accepting parental resistance (in which no vaccine is given) or delaying the vaccine. In addition, the researchers developed a dialogue sequence to handle questions during the session.

An iPad and Mac platform application system was used to perform the Wizard of OZ experiment. The iPad application served as the user-facing speech interface, and the tablet communicated with a separate laptop application through Bluetooth connectivity. Text is transmitted directly to the tablet, where the tablet speaks, using text to speech (TTS), to the participant. The iPad tablet can capture the participant's speech using offline speech recognition, where it is transcribed and sent to the laptop application.

The study was conducted from February to July of 2018 at the University of Texas in the United States (US). Sixteen adults with at least one child under age of 18 took part. Each participant completed a pre-assessment survey that included basic information about the subject and a Parent Attitudes about Childhood Vaccines (PACV) survey, which measures vaccine hesitancy. Participants then went through the simulated automated counseling system with the operator coordinating the interaction through the guidance of the dialogue script. At the end of the simulated counseling session, the researchers administered a usability survey.

On average, participants relatively agreed that the system was easy to use and had the capabilities they expected. Positive comments mainly regarded: the interactivity, how informative it is, the accessibility, and the clarity of the system. However, there were some negative comments, which mainly concerned: the response time, the repetitiveness, the lack of visuals, the need to humanise the system, and the inability to answer all questions.

Considering how to learn from this experience, the researchers note that, if the system were to be automated and utilised artificial intelligent components, the system would need to be relatively quick in responding to the user's utterance. In addition, they have been experimenting with the use of visualisations of emotions that could be used to augment the interface.

Lastly, the aim of the speech system is to alleviate some of the communication challenges in a clinical environment. Some of the users expressed a desire to speak to a human. The researchers stress that important discussion points, like personal contextual health information as it pertains to vaccines, should be handled by the provider to avoid confusion. Throughout the dialogue, they emphasised that user should confer with the doctor for more nuanced and specific information, especially since the researchers envision this CA to be stationed as a kiosk or tablet in the waiting room of the physician's office.

Currently, work is underway in developing an ontology-driven dialogue engine and lightweight question-answering component to answer natural language questions.