Supporting Designers in Expressing Human Experiences to AI and Context-Aware Technologies

How can we build artificial intelligence (AI) and context-aware systems that understand and facilitate human experiences? What might it look like for these systems to identify opportune moments for loved ones living far apart to engage in shared experiences, such as enjoying a warm meal on a cold day together? While humans have a conceptually-rich understanding of situations that support human experiences (e.g., the kinds of places serving food that are good for a cold day), AI systems have a primitive understanding of context (e.g., ‘person is at a restaurant serving soup’); thus, how can we bridge the gap between human concepts and AI representations of context? 

Ryan Louie, a sixth-year PhD student in Northwestern’s Technology and Social Behavior program (TSB), is exploring these complex questions.

Louie recently received a 2022 Google PhD Fellowship in Human Computer Interaction (HCI) to support his work building human-AI expression tools that aid other designers in effectively translating their natural understanding of a human situation into a machine representation using AI context-detectors, thereby enabling the creation of  novel, context-aware computational systems that opportunistically facilitate human experiences and activities.

“AI and context-aware applications have an enormous potential to be helpful and responsive within many situations that arise in people’s lives, such as facilitating shared experiences across contexts. However, designing applications that understand human situations and experiences is neither easy or straightforward; designers need support in expressing their high-level concepts in terms of the AI representations,” Louie said. 

Created to recognize graduate students who are doing exceptional research in computer science and related disciplines, Google PhD Fellowships support PhD candidates who are poised to shape the future of technology. In addition to receiving funding through the completion of his degree — Louie expects to graduate in June 2023 — he is matched with a Google Research Mentor who can help connect Louie to the AI and HCI research network within Google and provide feedback.

TSB is a dual PhD program in computer science and communication through Northwestern Engineering and Northwestern's School of Communication. Louie is advised by Haoqi Zhang, associate professor of computer science and design at the McCormick School of Engineering, and also collaborates extensively with Darren Gergle, professor of communication studies and (by courtesy) computer science.

Supporting designers of context-aware technologies with human-AI expression tools

Louie aims to support application designers in creating context-aware AI technologies that proactively identify moments where it could be helpful to intervene or to encourage an opportunity for either connection or reflection. He is building and studying tools and technologies within two domains of the context-aware AI technologies field.

One project centers around opportunistic collective experiences, or social experiences for spatially distant users powered by computer programs that identify opportune moments when users are in similar situations and conveniently structure shared activities across distance.

Louie developed the computational platform called Cerebro to design, implement, and execute these programs that identify and facilitate opportunistic experience in users’ daily lives. Unlike interactions on social media feeds for posting and passively engaging in other’s individual lives, interactions in opportunistic collective experiences promote active engagement in shared experiences and activities. With this focus on facilitating convenient moments for active engagement, Louie aims to help people build stronger social connection and improve their emotional well-being.

Louie is also collaborating with Kapil Garg, a fifth-year PhD student in the TSB program, on workplace network orchestration technologies that use AI to identify situations in the workplace in which users need support or would benefit from being directed to help-seeking opportunities.

“In larger work communities, mentors don't always have a continuous way to check in to know when people are struggling,” Louie said. “If we instrumented more work practices data, could we start to express our ideas for where support needs could be most useful to machines as well?”

In both domains, Louie is working to develop tools for designers to connect their high-level concepts for using context-aware technologies with the practical capabilities of AI machines.

“If a designer has a particular vision for how these technologies should be coordinating or facilitating our interactions, I want to make sure that the way the technologies actually operate are aligned with those ideas,” Louie said. “I’m trying to help designers bridge between their creative vision with the available AI capabilities.”

Encoding high-level concepts

In collaboration with Gergle and Zhang, Louie developed a visual programming environment called Affinder that helps designers of opportunistic experiences to follow an iterative expression process for fleshing out a high-level concept of a situation and linking that to low-level context-detectors. The team’s paper — “Affinder: Expressing Concepts of Situations that Afford Activities using Context-Detectors” — was published in the Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems.

“I created the core features in Affinder to address challenges I observed designers face when they try to encode a high-level concept about an experience using the low-level detectors for locations,” Louie said. “For example, a designer’s expression can be narrowly defined if they recalled one location detector for a concept, like ‘park’ for the concept ‘grassy fields to toss a frisbee’, but forget other detectors, such as ‘soccer fields’, that also match.”

The tools Louie develops draws on techniques from diverse fields, from machine learning to design and creativity studies. One bridging tool uses an unlimited vocabulary search technique based on text associated with location context-detectors, such as Yelp reviews, to allow designers to query relevant detectors based on their conception. Another bridging tool prompts designers to reflect and expand their own conception by using design-by-analogy methods for re-representing ideas through more abstract schemas.

“To create AI systems that can truly understand and facilitate human goals, we need to create human-AI interfaces that empower designers’ and end-users’ capacities for expressing desired experiences using the AI capabilities provided,” Louie said.

Building capacity for human expression

Focusing on people and their capacities for expression were core themes of research projects Louie conducted in collaboration with Google Magenta and People + AI Research scientists, which aimed to empower novice composers in co-creating music with generative AI models. 

During his first internship project, the team developed steering tools for generative models to overcome challenges novices face when interfacing directly with a deep generative model for music. They demonstrated in their paper published at CHI 2020 that having steering interfaces that allow novice musicians to partition and constrain how an existing ML model generates content significantly improved composers’ creative experience and their partnership with the AI.

In a second internship project, published in their IUI 2022 paper, the team studied how better and more coherent models and more intuitive steering interfaces can help composers express emotions in music as judged by listeners. The results highlighted that interfaces for steering were critical for enabling composers to create music samples that are less likely to be generated by ML models due ML training biases, but more aligned with a user’s expression and musical goals.

Mentoring students

Louie, who earned a bachelor’s degree in engineering in 2017 from the Olin College of Engineering in Needham, Massachusetts, is a graduate student researcher with the Delta Lab, an interdisciplinary research lab and design studio, and is also a special interest group (SIG) head and student co-mentor with Northwestern Engineering’s Design, Technology, and Research (DTR) program. DTR is a fast-paced, multiple-quarter course directed by Zhang that is structured around students learning to self-direct their own research projects within a supportive community.

Louie leads the opportunistic collective experiences SIG, and Garg leads the network orchestration technologies SIG.

“The students I mentor are trying to push this question about how technologies can find opportune moments and anticipate our needs for connection,” Louie said. “We have project openings for students to lead a new research question or lead the development of a new technology, so the students’ interests are quite aligned with our overall vision, and we co-develop it together.”

Two DTR student teams Louie mentored earned top awards in the Undergraduate Category at the Association for Computing Machinery (ACM) Computer-Human Interaction Student Research Competition.

Jennie Werner (’18) and Allison Sun (’18) won second place in 2018 for the project “Cerebro: A Platform for Opportunistic Collective Experiences,” which demonstrated how Cerebro automatically gathers and coordinates participation in experiences by detecting information about users' physical surroundings.

Cindy Hu, an undergraduate student in computer science at Northwestern Engineering and in communication studies at Northwestern’s School of Communication, won second place in 2022. She applied the Cerebro platform to aid users in initiating connections and developing friendships with new acquaintances for her project “Self-Disclosure for Early Relationship Development through Situated Prompts in Opportunistic Collective Experiences.” 

Reflecting and looking ahead

Prior to graduating, Louie plans to continue building on platforms he has developed in previous years, refining the value of these technologies, and prototyping with teams in DTR.

“I would really love for the types of designer tools and techniques I’m developing to better loop into the authentic practice of actually deploying these technologies,” Louie said. “A key element of the process is evaluating the interface between the designer’s vision for an experience and how it actually plays out. And there are opportunities to loop back that feedback so that designers can make changes during a deployment.”

Louie reflected on his experience in TSB and the lessons in perseverance he has confronted in pioneering a new way for technology to support people.

“It took three pilot deployments to really get to the first paper that showed the promise of this approach, and that was a good milestone,” Louie said. “Along the way, trying to communicate or understand this new concept for interaction, it was easy to mistake failures as reasons the idea is not going to work. It was a good lesson for me to try to develop a little resilience for new ideas. They will continuously not be perfect on many angles, but that doesn’t mean the whole idea is not a good one to pursue.”

Louie plans to apply to several research related roles to continue developing human-AI interfaces to support designers of AI and context-aware technologies that enable human connection and creative empowerment. His experience mentoring students in DTR has also inspired him to pursue a faculty role as well.

“I've had just a tremendous opportunity to mentor undergraduates along these research directions,” Louie said. “There's a lot of joy in seeing them push forward on milestones and merge their abilities as both technologists and designers.”