As a critical component of Human-Computer Interaction, conversational user interfaces (CUIs) can potentially revolutionize the way we interact with technology. This course is designed for graduate students who want to gain a deeper understanding of CUIs and their real-world applications. We will focus on the HCI side of conversational AI. Throughout the course, we will explore cutting-edge research and methodologies for designing, implementing, and evaluating CUIs. Various topics of conversational interfaces will be covered, including design principles, evaluation, human agency, etc.

Prerequisites: Introduction to Human-Computer Interaction (EN.601.490/690), at least one graduate-level computer science course in Artificial Intelligence or Machine Learning (including NLP, Computer Vision, etc.) is preferred, or permission of the instructor. Students must be comfortable reading recent research papers and discussing key concepts and ideas.

Acknowledgements Thank you to Prof. Anjalie Field and Prof. Daniel Khashabi for sharing the course website template! And thanks for Alec Jacobson and Colin Raffel for their thoughtful idea on running paper-reading seminars.

Schedule

The current class schedule is below. The schedule is subject to change, particularly the specific readings:

Date Topic Readings Work Due
Mon Jan 22 Course Overview - History and Evolution of Conversational User Interface [Slides] No Required Reading
Wed Jan 24 From Text to Multimodal CUI [Slides]
  1. (Optional) ELIZA—a computer program for the study of natural language communication between man and machine
No Required Reading
Mon Jan 28 Communication Principles [Slides]
  1. Grice's Maxims of Conversation: The Principles of Effective Communication
  2. Resilient Chatbots: Repair Strategy Preferences for Conversational Breakdowns
Reading Responses by 12:00pm Monday
Wed Jan 31 Metaphors in Conversational User Interface [Slides]
  1. Computers are social actors
  2. Conceptual Metaphors Impact Perceptions of Human-AI Collaboration
  3. (Optional, Book) Metaphors We Live By
Reading Responses by 12pm Wednesday
Mon Feb 5 Conversational Recommender Systems: Understanding Humans [Slides]
  1. The Adaptive Place Advisor: A Conversational Recommendation System
  2. The Ethnobot: Gathering Ethnographies in the Age of IoT
Reading Responses by 12pm Monday
Wed Feb 7 Conversational Search: Mixed-Initiative Interactions [Slides]
  1. Tell Me More: Understanding User Interaction of Smart Speaker News Powered by Conversational Search
  2. Conversational Interfaces for Information Search
Reading Responses by 12pm
Mon Feb 12 CUI for Healthcare [Slides]
  1. "I Hear You, I Feel You": Encouraging Deep Self-disclosure through a Chatbot
  2. Voice-Controlled Intelligent Personal Assistants in Health Care: International Delphi Study
Anonymous course feedback and Reading Responses by 12pm
Wed Feb 14 LLMs I: Building LLM-based CUI [Slides]
  1. LLM-Powered Conversational Voice Assistants: Interaction Patterns, Opportunities, Challenges, and Design Guidelines
  2. (No Response Needed)How to Build a Chatbot Using the OpenAI API & Pinecone
  3. (No Response Needed) AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation
Reading Responses by 12pm
Mon Feb 19 LLMs II: Building CUI with LLM [Slides]
  1. ProtoChat: Supporting the Conversation Design Process with Crowd Feedback
  2. Powering an AI Chatbot with Expert Sourcing to Support Credible Health Information Access
Reading Responses by 12pm; Project Brainstorm Results by 11:59 pm
Wed Feb 21 LLMs III: Responsible Use of LLMs in CUI [Slides]
  1. Understanding the Benefits and Challenges of Deploying Conversational AI Leveraging Large Language Models for Public Health Intervention
  2. Evaluating Verifiability in Generative Search Engines
Reading Responses by 12pm
Mon Feb 26 Project Proposal
Proposal Slides by 3pm
Wed Feb 28 Project Proposal
Mon Mar 4 Evaluation I: Measurement Theory [Slides]
  1. Construct Validity in Psychological Tests
  2. Evidence-Centered Design
Reading Responses by 12pm
Wed Mar 6 Evaluation II: Benchmarks [Slides]
  1. Heuristic Evaluation of Conversational Agents
  2. ChatEval: A Tool for Chatbot Evaluation
  3. Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings
  4. (Optional) Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
Reading Responses by 12pm; Project Proposal by Mar 8th, 11:59 pm
Mon Mar 11 Evaluation III: Controlled Experiments [Slides]
  1. Studying the Effects of Cognitive Biases in Evaluation of Conversational Agents
  2. The controlled experiment approach to evaluating user interfaces
Reading Responses by 12pm
Wed Mar 13 Evaluation IV: Conversation Analysis
  1. Conversations Gone Awry: Detecting Early Signs of Conversational Failure
  2. ConvoKit: A Toolkit for the Analysis of Conversations [Video]
  3. Conversation analysis. What is it and when should I use it?
  4. (Optional) How does your Alexa behave?: Evaluating Voice Applications by Design Guidelines Using an Automatic Voice Crawler
  5. (Optional) From Writing Dialogue to Designing Conversation: Considering the potential of Conversation Analysis for Voice User Interfaces
Reading Responses by 12pm
Mon Mar 18 Spring Break
Wed Mar 20 Spring Break
Mon Mar 25 Human Agency in CUI I: Anthropomorphism I - The Good
  1. Human-animal teams as an analog for future human-robot teams: influencing design and fostering trust
  2. On AI Anthropomorphism
Reading Responses by 12pm
Wed Mar 27 Human Agency in CUI II: Anthropomorphism II - The Ugly
  1. Mirages. On Anthropomorphism in Dialogue Systems
  2. Psychology: Measuring Anthropomorphism
  3. (Optional) ANTHROSCORE: A Computational Linguistic Measure of Anthropomorphism
Reading Responses by 12pm
Mon Apr 1 Human Agency in CUI III: Privacy
  1. “I just shared your responses”: Extending Communication Privacy Management Theory to Interactions with Conversational Agents
  2. Take Back Control: User Privacy and Transparency Concerns in Personalized Conversational Agents
Reading Responses by 12pm
Wed Apr 3 Beyond Dyadic CUI
  1. Having an Animated Coffee with a Group of Chatbots from the 19th Century
  2. SearchBots: User Engagement with ChatBots during Collaborative Search
Reading Responses by 12pm
Mon Apr 8 Designing Explainability for CUI
  1. Democratizing Chatbot Debugging: A Computational Framework for Evaluating and Explaining Inappropriate Chatbot Responses
  2. Why or Why Not? The Effect of Justification Styles on Chatbot Recommendations
Reading Responses by 12pm
Wed Apr 10 Natural Language Interface beyond CUI
  1. TextWorld: A Learning Environment for Text-Based Games
  2. Beyond ChatBots: ExploreLLM for Structured Thoughts and Personalized Model Responses
Reading Responses by 12pm
Mon Apr 15 Conversational UI for Social Good
  1. Exploring Effects of Chatbot-based Social Contact on Reducing Mental Illness Stigma
  2. Situated Understanding of Older Adults' Interactions with Voice Assistants: A Month-long In-home Study
  3. Effectiveness of an Empathic Chatbot in Combating Adverse Effects of Social Exclusion on Mood
  4. Talking with machines: Can conversational technologies serve as children's social partners?
  5. Co-Designing AI Agents to Support Social Connectedness Among Online Learners: Functionalities, Social Characteristics, and Ethical Challenges
Reading Responses by 12pm
Wed Apr 17 User Simulation for Evaluation
  1. Information and Attitude Diffusion in Networks
  2. Evaluating Large Language Models in Generating Synthetic HCI Research Data: a Case Study
  3. Social Simulacra: Creating Populated Prototypes for Social Computing Systems
  4. The Role of Embodiment and Simulation in Evaluating HCI: Theory and Framework
  5. What simulation can do for HCI research
Reading Responses by 12pm
Mon Apr 22 Project Final Presentation
Wed Apr 24 Project Final Presentation
Mon Apr 30 No Class - Reading Days
Wed May 2 No Class - Reading Days

Policies

Attendance policy This is a graduate-level course revolving around in-person discussion. Students are expected to attend class and may notify instructors if there are extenuating circumstances.

Course Conduct This is a discussion class focused on cutting-edge research. All students are expected to respect everyone's perspective and input and to contribute towards creating a welcoming and inclusive climate. We the instructors will strive to make this classroom an inclusive space for all students, and we welcome feedback on ways to improve.

Academic Integrity This course will have a zero-tolerance philosophy regarding plagiarism or other forms of cheating, and incidents of academic dishonesty will be reported. A student who has doubts about how the Honor Code applies to this course should obtain specific guidance from the course instructor before submitting the respective assignment.

Discrimination and Harrasment The Johns Hopkins University is committed to equal opportunity for its faculty, staff, and students. To that end, the university does not discriminate on the basis of sex, gender, marital status, pregnancy, race, color, ethnicity, national origin, age, disability, religion, sexual orientation, gender identity or expression, veteran status, military status, immigration status or other legally protected characteristic. The University's Discrimination and Harassment Policy and Procedures provides information on how to report or file a complaint of discrimination or harassment based on any of the protected statuses listed in the earlier sentence, and the University’s prompt and equitable response to such complaints.

Personal Well-being Take care of yourself! Being a student can be challenging and your physical and mental health is important. If you need support, please seek it out. Here are several of the many helpful resources on campus: