Imagine a boombox that tracks your every move and suggests music to match your dance style. That’s the idea behind “Be the Beat,” one of several projects in the MIT 4.043/4.044 (Interactive Intelligence) course taught by Marcelo Coelho of the Faculty of Architecture, which will be presented at the 38th NeurIPS (Neural Information Processing Systems) conference in December 2024. With more than 16,000 participants in Vancouver, NeurIPS is a competitive and prestigious conference dedicated to research and science in the fields of artificial intelligence and machine learning, and a premier venue to showcase cutting-edge developments.
This course explores the emerging field of large-scale linguistic objects and how artificial intelligence can be extended to the physical world. While “Be the Beat” transforms the creative possibilities of dance, other student submissions span the fields of music, storytelling, critical thinking and memory, generating innovative experiences and new forms of human-computer interaction. Taken together, these projects reveal a broader vision of artificial intelligence: one that goes beyond automation to foster creativity, reinvent education and reimagine social interaction.
Become the rhythm
Developed by MIT Mechanical Engineering and Design student Ethan Chan and MIT Mechanical Engineering and Music student Zhixin Chen, “Be the Beat” is an AI-powered boombox that suggests music based on a dancer’s movements. Dance has traditionally been guided by music throughout history and across cultures, yet the concept of dancing to create music is little explored.
“Be the Beat” creates a space for human-AI collaboration in freestyle dance, empowering dancers to rethink the traditional relationship between dance and music. It uses PoseNet to describe movements in a large language model, analyzes dance styles, and queries the API to find music with similar style, energy, and tempo. Dancers who interact with Boombox report feeling more control over their artistic expression and cite Boombox as a new approach to creatively exploring dance genres and choreography.
A mystery to you
“A Mystery for You” is an educational game by Mrinalini Singha SM ’24, a recent graduate of the Arts, Culture and Technology program, and Haoheng Tang, a recent graduate of the Harvard Graduate School of Design, designed to develop critical thinking and fact-checking skills in young learners. The game leverages large-scale language models (LLMs) and visual interfaces to create an immersive investigative experience. Players take on the role of citizen fact-checkers, responding to AI-generated “news alerts” output by the game interface. By inserting a combination of cartridges to prompt the next “news update,” players navigate ambiguous situations, analyze evidence, and weigh conflicting information to make informed decisions.
This human-computer interaction experience challenges our news consumption habits by eliminating touchscreen interfaces and replacing continuous scrolling and skimming with tactile analog devices. By combining the capabilities of slow media and new generative media, the game encourages thoughtful and tangible interactions while providing players with the ability to better understand and challenge today’s polarized media environment, where misinformation and manipulative narratives are rampant.
Memory
Memorscope, by MIT Media Lab researcher Kim Gun-wook, is a device that creates collective memories by combining the highly human experience of face-to-face interaction with advanced AI technology. Inspired by how microscopes and telescopes are used to examine and explore hidden, unseen details, Memorscope allows two users to “peek” into each other’s faces, using this intimate interaction as a portal to create and explore shared memories.
The device leverages AI models from OpenAI and Midjourney to introduce different aesthetic and emotional interpretations to create a dynamic, collective memory space that goes beyond the limitations of traditional shared albums, offering an interactive and fluid environment where memories are not just static snapshots but living, evolving stories shaped by the ongoing relationships between users.
Storyteller
Created by Harvard Graduate School of Design students Shiyin (Alia) Bao and Yubo Zhao, Naratron is an interactive projector that allows users to co-create and co-act children’s stories through shadow puppets using large language patterns. Users can press the shutter button to “shoot” the protagonist they want to feature in the story, and the machine takes the hand shadow (e.g., animal shape) as the protagonist’s input. The system then develops the plot as new shadow characters are introduced. The story is projected on the projector as a background for the shadow puppets, and the speaker tells the story as the user turns the handle to “play” it in real time. By combining visual, auditory, and physical interactions in one system, the project aims to inspire creativity in shadow puppet storytelling and enable multimodal human-AI collaboration.
Perfect syntax
“Perfect Syntax” by Karyn Nakamura ’24 is a video art piece that explores the syntax behind movement and video. The project uses AI to manipulate video footage to explore how the fluidity of movement and time can be simulated and replicated by machines. Drawing inspiration from both philosophical studies and artistic practice, Nakamura’s work explores the relationship between perception, technology, and movement that shapes our experience of the world. By reimagining video through computational processes, Nakamura explores the intricacies of how machines understand and represent the flow of time and movement.
