Human-Chatbot Interaction | Odi's Portfolio

Opera Browser x Uppsala University — 2024

Research: Human-Gen AI chatbot Interaction

The goal is to find out desired interactive features or functionalities that in line with users' use cases.

Key skills developed during this project:

This research project is a collaborative effort between the company and a thesis initiative. It is part of the company's research agenda, aimed at enhancing human-chatbot interactions. At Opera, we currently have a conversational chatbot named Aria, similar to ChatGPT. However, we haven't yet conducted any studies on how users generally engage with chatbots or what types of interactions they want beyond typical conversational exchanges.

ⓘ In this portfolio, I will discuss the research from the industry perspective and will not include any academic theories. For the academic final paper, you can check it here.

Background

Names are anonymized using pseudonyms for privacy and confidentiality, as agreed upon by participants' consent.

Participants short profile

Phase 3: Survey

This survey aims to validate the findings from the interview analysis. The survey mainly validated three aspects:

The most common to least common use cases among larger audiences
The most to least frustrating experiences when interacting with chatbots
The most to least desired conceptual interactions as concluded in the analysis above

For the desired interactions, I included simple illustrations to help respondents visualize them. I used Likert scale questions to assess users' opinions on each concept. Some examples of the questions:

Shared materials

That's it! This research provided us with foundational insights into human and conversational chatbot interactions solely from the users' perspective, without technical or legal considerations. I shared this above, but I will share it again here: if you want to dig deeper into this research, you can access:

EventCo Revamp

Next project

Phase 2: Interview analysis - Thematic

I will briefly discuss the key findings, as this phase does not represent the final research findings. If you're interested in learning more, you can find the full analysis here (I’m allowed to share this as part of knowledge sharing 😁).

Firstly, I conducted a thematic analysis of the interviews to group the data based on themes. Based on the interview, the most common use cases for chatbots are to seek information and to complete a task. I also identified some common struggles, such as users feeling they lack control over the chatbot's responses. Similarly, many expressed a desire for more control or the ability to personalize the chatbot based on their preferences.

The Atomic UX method categorizes research findings into facts > insights > recommendations. This method helps narrow down the large set of findings from the thematic analysis, making it easier to inform the survey design.

The analysis resulted in conceptual ideas for user-desired interactions (see the analysis below), which will be validated through a survey.

Interview analysis - Atomic UX

Some important aspects that I didn’t cover in this research:

Didn’t address technical constraints related to feasibility of the solutions/ideas.
Didn’t assess chatbot’s ethical boundaries, potentially resulting in findings that could potentially violate ethical boundaries in gen AI chatbots.

Limitation

This research combines qualitative and quantitative methods (interviews and surveys), and is divided into four phases.

Method

Predominant use cases include:

Getting help with occupational-related tasks, such as work and study (76.3%)
Obtaining occupational-related information (71.1%)
Receiving assistance with writing tasks (65.3%)

💡 Insight

The high level of engagement in utilitarian purposes highlighted the pragmatic attributes of chatbots, which offer practical outcomes that play significant roles in users' lives. Likewise, negative experiences were frequently linked to failures in the practical effectiveness of chatbots.

Low level engagement use cases include:

Social Interaction: 17.4% (this differed from interview results, where there was little difference between social interaction and utilitarian tasks)
Entertainment: 11.6%

💡 Insight

Users don’t primarily rely on chatbots to fulfill their psychological and entertainment needs. Nevertheless, chatbots demonstrated positive potential in the domain of social interaction, as most interview participants expressed positive sentiments regarding their experiences engaging with chatbots on personal matters. While entertainment is not the primary focus, it could serve as a complementary feature that enriches the interplay of pragmatic and hedonic experiences for users.

3 / The desirability of interactive features concepts

Analysis Method: same as frustrating experiences section.

The survey identified the most frustrating experience is when the chatbot failed to provide sources or supplied unreliable or non-existent sources. Qualitative findings from the interviews support this, revealing that users expressed distrust in the sources provided by the chatbot and frequently felt the need to verify the information manually.

💡 Insight

This aligns with findings from an earlier ChatGPT survey, where respondents reported having trust issues, specifically criticizing ChatGPT's tendency to provide inaccurate information, which they described as 'hallucinating.' This corresponds with the use case in which users predominantly utilized chatbots for occupational tasks or tasks that demanded professionalism and accuracy. In these cases, the reliability of the chatbot's responses is essential for users' reliance on them.

The second most bothersome category of experiences was related to disruptions in the conversation flow, which included:

Having to type repetitive prompts
The chatbot going off track during the conversation
Struggling to generate the appropriate prompts.

💡 Insight

Such disruptions were perceived as breakdowns in the chatbot’s pragmatic attributes, hindering users from completing tasks efficiently and effectively. Interestingly, this aligns with findings from an early survey of ChatGPT users, particularly regarding poor prompting experiences. Currently, academics are conducting research on how to enhance prompting experiences. They characterized users as engaging in a trial-and-error process, experimenting with various prompts and adjusting them when the chatbot failed to comprehend.

Third most bothersome experiences category was related to a lack of personalization functionalities.

No control over the chatbot’s responses (unnatural tones, lengthy, etc)
Difficulty in accessing past conversations
Being unable to organize threads or chat rooms

Lastly, the lack of human-like empathy was considered the least bothersome quality of the chatbots. This aligned with the fact that users don’t primarily use chatbots for social interaction, but rather for practical purposes.

The interview aimed to understand these four key aspects:

Users' usage pattern of chatbot usage
Users’ motivation and experience interacting with chatbot
Users' perception of personalization & interactivity in chatbot interaction
Users' continuance intention

Phase 1: Interview planning

Participant sampling

8-10 participants
Age: 20-45 years old (in line with the demographic analysis of the most common users of ChatGPT)
Minimum four participants are active users who have tried at least two different chatbots
No restrictions on domicile or nationality

Recruit participants: Direct approach, social media, and forum/groups
Participant scheduling & screening: Userinterviews.com
Interview platform: Google meet

Tools

Interview questions

Based on previous research, users use chatbots for five categories of use cases: information seeking, task efficiency, social interaction, entertainment, and personalization. The interview questions related to users' motivations will be based on these categories.

In total there were 10 interview participants.
Age range: 21-31 years old
Domicile: European Union (EU), Asia, and the United States (US)
6 tech-related & 4 non-tech occupation

The interviews were recorded with the participants' consent, and each interview lasted for one hour. As a token of appreciation, the participants were given a digital voucher.

Phase 1: User interview

Phase 4: Survey analysis (final research findings)

I analyzed both the survey and interview findings and found correlations between them. In this section, I will explain the findings main points.

1 / Users' use cases

Ranking of use case categories from most to least common:

Task efficiency
Information seeking
Social Interaction
Entertainment

In line with users’ pain points, the most desired features or functionalities categories: Allow them to personalize their sources or any features that would provide them with more trusted and reliable sources.

💡 Insight

Sources = trust. If no sources, made it difficult for the participants to verify the accuracy of the information they receive and hindered the chatbot's usefulness for some purposes, such as research.
If no sources, users eventually will ask chatbot to provide and get unreliable or nonexistent references instead.
Sources personalization would benefit users with certain task, e.g., students who need research paper references.
Not just for verifying facts but also for exploring topics in deeper. In one example, a participant used references provided by the chatbot to communicate with customers in work, boosting their work's professionalism.

2 / Users' frustrating experiences

Analysis Method: The data were analyzed by assigning numerical values to ordinal responses on the Likert scale as follows: Very bothered = 4; Somewhat bothered = 3; Slightly bothered = 2; Not bothered at all = 1. The goal was to rank the experiences from most to least frustrating based on the average score (mean value) of each experience. The mean scores served as the basis for ranking, with higher mean values indicating the most bothersome experiences and so forth.

Second most desired functionalities category was related to minimizing the disruption in conversation flow.

Allowing users to undo and refine their questions when the chatbot's responses began to unravel.
Better prompting experiences to get more accurate answers.
Save and reuse of prompts.

💡 Insight

These features could help bridge the gap between human and machine limitations. When humans and chatbots failed to adapt to each other's communication styles, it could disrupt the experience.

Third category: improve personalization capability. Such as, dividing chatbot capabilities into specialized assistants increased their credibility in performing specific tasks.

💡 Insight

Even though some chatbots already implemented this. As noted by interview participants, these specialized assistants must also be equipped with distinctive features or functionalities that distinguish them from other specializations to truly be deemed experts. For example, an analyst assistant chatbot might support uploading files in multiple formats, while a creative writing assistant might offer unique writing style options for various kinds of writing. Without these distinct differentiators, the assistants might appear as mere general conversational chatbots.

Even though it was not among the top five most desired features, providing critical and relevant resources and support for crisis intervention was still considered valuable, though only a few interviewees recognized its potential to increase chatbot usefulness in such situations.

💡 Insight

This could also boost the perceived moral value of chatbots, even if it’s not relevant from LLM chatbots focus. This was because chatbots are increasingly seen as an alternative new method for consulting on personal issues. Ethical considerations must be addressed in this context.