The Promises and Possibilities of Scenarios We Can Create – Communications of the ACM

This is the first blog in a series that will consider positive uses of technology. We start by introducing the idea of a physical/tangible interface that modifies a Generative AI prompt to make a student aware of the way different personas might view a question.

The question “Is technology a force of good?” might be asked from a skeptical, thoughtful, confident or curious position; Generative AI’s answers to these very different positions will also be very different from each other. This blog highlights an approach to teaching students the value of considering different points of view.

We have gotten accustomed to requesting detailed information from our digital world. We have organized data networks so we can access an infinitude of human-curated knowledge with search engines. Now we have organized knowledge in Large Language Models (LLMs) to aggregate what is known in response to our requests.

Until we were given the Generative AI (GenAI) interface, LLMs were only a tool for us geeks. With GenAI, they have become attractive and usable by hundreds of millions of people. These GenAI chatbots represent a new kind of human-computer interaction. And while many experiments have exposed people trying to use them in ways that show off their weaknesses, there are so many ways to use them that show off their strengths.

The conventional route to designing something would likely begin by mapping out desired features, and then mapping out interface elements to expose them productively for users. The new design feature GenAI brought is that of expecting users to use their natural language to explain what they want to know and how they want it presented in language, code, or image. Now anyone, and not just people who are programmers, can access deep AI representations knowledge and communication.

We have been thinking of ways of making users more aware of how their GenAI prompts guide how the systems respond. A powerful prompt approach includes telling the tone (like confident) you want to hear things in. We began thinking about how a user might tell the GenAI a point of view it wants to consider.

As computer scientists, we think of languages, structure, procedures, and interfaces. As people, we consider how to communicate. We have been provoking our students with questions that will ignite their imaginations, lead to some understanding, more questions, and help the student in the world. A big question like “Are we alone in the universe” has complex dependencies and big implications.  

Of course, changing your prompt changes the responses of GenAI. Indeed, the tone of the prompt is a often-overlooked aspect that changes the response. Our emotional cues end up as part of the prompts that can drive the GenAI exploration. If we prompt a GenAI to answer in a skeptical tone, we will get quite a different answer than if we ask it to answer in a confident tone. 

The possibility of supporting many new kinds of questions is here. Imagine we could tell the system the way we want it to look at a problem simply, but also in a way that makes us recognize the way we are positioning our question. We like to start with scenarios; let’s start with a simple one. 

Against all odds, our phones, tablets, and computers all have amazing cameras with enough hardware and software to interpret what they see; let’s show them a face that prompts a GenAI system to answer from a curious, thoughtful, skeptical, or confident way of discussing a question. Imagine providing such affordance with a tangible interface that could come in a box of cereal.

We start with a physical sponge with faces drawn on its sides.  

We asked GenAI to come up with its own taxonomy by writing the following prompt:

“I have a face that can change expression; give me four faces that notice and epitomize different perspectives that I can bring to a Big Question; what faces will showcase different perspectives on the question.”

Copilot:

That’s an interesting idea! Here are four different facial expressions that can help you showcase different perspectives on a big question:

1. Curious Face: Raised eyebrows, slightly open mouth, and wide eyes. This expression shows you’re open to exploring new ideas and eager to learn more.

2. Thoughtful Face: Furrowed brows, slightly pursed lips, and a focused gaze. This expression indicates deep contemplation and careful consideration of the question.

3. Skeptical Face: One eyebrow raised, lips slightly pressed together, and a sideways glance. This expression conveys doubt and critical thinking, questioning the validity of the information.

4. Confident Face: Relaxed brows, a slight smile, and direct eye contact. This expression shows assurance and a strong belief in your perspective or solution. These expressions can help you convey different attitudes and approaches when tackling a big question.

It’s not a stretch to say that the camera can recognize these affective faces and be used to drive results.

A student takes the sponge out of its plastic bag. The picture on one face of the sponge tells them to hold it towards a phone or iPad, which takes them to a webpage to download a game. Pointing its QR code at their phone or laptop camera lets them set up the student’s GenAI workplace and its special interface. The interface encourages the student to choose a question and also to hold the sponge in front of the camera as they do. For the purpose of a teaching unit around universal “big questions,” the screen can show a list of AI prompts on big questions from which the student can choose. After their selection, they hold the sponge up to the camera and squeeze it for system response. The GenAI can be trained specially to work with materials that a teacher wants their students to consider.

They can also see them on the screen. Whichever face of the sponge points to the camera, that’s the “personality” that answers the question. The sponge is designed to engage students in a new way, focusing on the affect you present when asking a question.

There are many implications of this for learning, scenario development, and translation of technology.

The faces are a simple language of perspectives. Answering with a confident tone is very different than with a curious tone; it makes people aware of the impact of projected affect in question answering. We will also explore this from constructive angles in future blogs.

Trying the different faces allows a commonality of perspectives that permits a student, parent, or teacher to share common points about similar questions. Letting everyone ask the same questions with a few different tones gives students a shared experience that lets them discuss the ideas together. We also will talk about other points about computer-assisted learning in future blogs. 

By using a sponge, we are showing that sophisticated and novel user interfaces don’t have to be expensive or difficult to build. We love talking about provocative interface scenarios. We will talk about new interaction scenario opportunities as a matter of course in these posts. We love working to provoke learners with new technology and questions, such as what can be done with GenAI. Our posts will regularly include these perspectives.

There are always new affordances that new scenarios present. Our posts will try to help you imagine new directions for AI interaction scenarios.

The interface proposed in this blog was created to show how as UX engineers, we can guide a user to see tools’ affordances. The way we ask a question biases the way it is answered. We hope our excursion helps you consider the ways our new AIs work, the power in seeing the different parts of the things you make, and the ways you think about things.

Ted Selker is a computer scientist and student of interfaces.

Berry Billingsley is an educator interested in philosophical “big questions.”