Understanding users' mental models for AI Explainability

Having an accurate understanding of how a user believes a product works can reduce unexpected behavior, improving user experience. Explainable AI systems can help do this -- by providing information to update a user's beliefs. As part of a team of user researchers and designers working on Explainable AI systems, I helped develop a set of design exercises to improve the understanding of users' mental models. As of December 2022, this project is still under development, with plans for introducing it to the internal design organization soon.

The image on the right has been modified for confidentiality.

Goals

Design research goals

Create resources and exercises for designers to better understand user mental models

Methods

Literature review
User interviews
Participatory design
Usability testing

Crucial insights

UX researchers consider mental models research as a key component for good product design
Mental models of the users and system designers (sometimes referred to as the "conceptual model") are often at odds, and identifying where these differences are offers opportunities for improved design
Not all users desire to have product behavior explained to them -- figuring out where it is necessary is crucial to a good user experience

Research impact

Strategic impact

This work is still in development as of late 2022, and at this time is going through iterative user testing for the purpose of improving Explainable AI design strategy

Stakeholder impact

Self and business stakeholders: The design research team and I received recognition from the governance board, and the work is in planning for further development and publication
Societal stakeholders: Explainable systems improve transparency and accountability. Hopefully, this work will incentivize these user benefits

Product impact

Using design methods that improve the transparency of AI systems and allow more accurate understanding for the user, my hope is this project will go on to reduce misuse of AI systems and improve user experience

We conducted user interviews with various kinds of designers across the organization and prototyped Mural activities for mapping mental models

The final product is intended to serve as a method for increasing the overlap between the user mental model and other models of understanding and function

What I learned

Terminology across disciplines can easily be confused with one another -- using "model" to denote how a user thinks, as well as the statistical properties of the AI system created a communication challenge, so make sure to establish clear definitions when working with new teams
Iterating on a design should also allow for backwards movement if something is not headed in the right direction