User research on generative-AI: insights and approaches from the GOV.UK Chat experiment

In this session we will share our experience of conducting user research to inform the design of a chatbot that uses generative AI technology to provide natural language answers to users’ everyday questions.

We will share our research approach and the lessons we learned through a mix of storytelling and practical activities and discussion. Participants will be exposed to the difficulties we faced in measuring the accuracy and conversational memory of the tool and how we worked towards reconciling a trusted brand with a technology that carries inherent risk. Our session will culminate in the sharing of quantitative and qualitative insights from scaled testing and an an open discussion of how we might research and design AI proactively, progressively and responsibly.

Participant takeaways:

• an appreciation of how to measure the accuracy of outputs generated by a large language model chatbot 
• recommendations on how to triangulate the chat bot user experience from multiple data points 
• our latest insights on the user experience of generative-AI chatbots for government

Themes:

AI, User research