My Thesis paper “Reinventing Museums Spaces: Site-specific Mixed Reality for Gamified Learning” has been accepted by Duke Graduate School and is in the queue for delivery to ProQuest.

PROJECT OVERVIEW

This master’s thesis consists of a written component and a digital project. The written component includes an introduction to the project, a literature review on analog and digital approaches to enhancing learning and engagement in museums, an overview of design strategies with documentation, and a reflection. The digital component features a mixed reality museum experience, developed using Unity and Meta Quest 3.

CCONCEPT & INSPIRATION

During my master’s study, I witnessed the launch of cutting-edge mixed reality devices like the Meta Quest 3 and Vision Pro that are equipped with advanced spatial computing capabilities. To begin my journey into MR, I chose to explore its potential for reimagining a static space with a predetermined use. Meanwhile, I recall the movie Night at the Museum, where exhibits come to life. This idea resonated with me as I envisioned the possibilities of MR to create similarly immersive and engaging experiences. Therefore, I set out to design a MR museum experience that blends narrative-driven exploration, hybrid reality immersion, and educational escape room.

OBJECTIVES

• Explore how mixed reality can enhance learning and engagement in a traditional art gallery
• Identify what are the challenges and potentials of developing and implementing such an experience within traditional museum contexts
• Lay the foundation for my long-term goal of exploring the intersection of MR and AI.

PROCESS & METHODOLOGY

Storyboard & Wireframe

https://www.figma.com/proto/y03rqK0PltMwSxmLJoySkp/MR%2B?node-id=134-32&t=btI1GQhxFxBBdfMi-1

Character & Clothing Modeling

To accurately represent Mayu as a 13-year-old aclla from the Inca Empire, I sourced images of young Peruvian girls online, identified key facial features including almond-shaped eyes, slightly flat nose, and reddish and textured skin. I chose Character Creator 4 for modeling the character due to its powerful capabilities for realistic character creation and seamless integration with Unity. Using its Headshot Plug-in, I converted an image of a Peruvian girl into a 3D face, then refined facial features such as the eyes, nose, and cheeks to capture representative characteristics of a young Peruvian girl.

For Mayu’s clothing, I referenced Guaman Poma’s drawings of queens and acllas as well as online images of traditional Peruvian clothing. I used Marvelous Designer, an industry-standard fashion design software, to create a 3D model of Mayu’s attire.

AI NPC Interaction

To ensure natural interaction and maximize learning, Mayu needs to be able to listen and respond to players’ questions in real-time. Convai, an AI character tool powered by Large Language Models (LLMs), is used to create human-like conversations. By integrating a detailed background story and a knowledge bank, Mayu can engage in conversations based on her identity and expertise, providing accurate and contextually relevant information to enhance the player’s learning experience.

Leveraging Convai’s narrative design functionality, I enabled the character to initiate conversations, pose questions, and share insights proactively as if she could truly perceive the players’ actions and environment. Additionally,I explored multi-path storytelling, designing the NPC to offer varied responses based on the players’ choices.

Photogrammetry

Key Virtual Asset–Moray Terraces

Unity

Unity is used to develop all the game mechanics and interactions. Meta XR SDK–which contains various building blocks and key packages such as Mixed Reality Utility Kit–is used to load scanned scene into Unity and enable hand and controller interactions. A spatial anchor script is written to position each object in the desired spot in the gallery.

Early Stage Demo

User Testing

Chicha Making Puzzle

Based on small scale, 1 on 1 user testing, I made the following design changes:

• Applied glowing effect (visual hints) to a larger number of key spots to enhance player guidance
• Added interactive buttons to trigger audio, encouraging active learning
• Highlighted the current panel during audio playback for improved focus

USER TESTING @ NASHER

OUTCOMES

REFLECTIONS

What are working?

A few elements that suggest a promising direction:

• Co-presence with NPC who can move around the gallery
• Conversation with AI NPC –> promotes active learning
• Narrative design
  • AI NPC initiating conversations and asking players questions proactively –> potential of NPC to shape players’ actions
  • Potential of multi-paths storytelling

What are not working?

• The balance between maintaining immersion and offering sufficient/right information at the right time to help players navigate and feel grounded in the experience
• The puzzles may not fully achieve their intended educational objectives

Future Directions

• User testing
  • small-scale, in-depth testing to identify 1) which designs enhance learning and engagement and which fall short; 2) issues with game flow
  • larger-scale user testing, conducted in collaboration with the Nasher, to assess whether this MR experience effectively enhances learning and engagement
• Multiplayer
• Spatial Video
• Incorporate tangible interactions with physical artifacts
• MR + VR:
  • To extend the MR experience beyond the gallery, a complementary VR experience could be developed, allowing users to continue their journey with Mayu and explore the Inca Empire off-site. This VR segment could culminate in a narrative point that seamlessly transitions to the next MR experience at a different museum dedicated to Andean cultures. Such an approach would create an interconnected, large-scale mixed-reality experience across multiple museums, transforming individual visits into a cohesive, globally connected exploration.

GALLERY

ACKNOWLEDGEMENT