The project leverages MR technology to create an interactive bedtime storytelling space in children's bedrooms. Children can lie down and immerse themselves in a narrative world enhanced by AI speech recognition and scene generation technology. When specific keywords are mentioned in the story, the headset generates interactive objects and adjusts the overall atmosphere accordingly. For instance, the skybox changes from day to night when those words are mentioned, and a bird will fly in and land on the child's hand when a bird is mentioned. This allows children to experience the story dynamically or create their own imaginative narratives, thereby enhancing their logic and language skills.
The ceiling transforms into a virtual dome that displays various scenes and environments, changing with the plot. The bed surface also becomes part of the story, visually morphing into grass, hillsides, or log cabins. Children can lie down and feel like they are part of the story, watching the wind blow through fields or Lilliputians from "Gulliver's Travels" running around. This immersive experience makes storytelling more engaging and magical, all while remaining in their own bedrooms.
Interactions are further enriched by animations triggered by storytelling keywords, which appear around the child and can be interacted with. Models generated around the player can be grasped or altered with hand movements, adding a tactile element to the storytelling experience. This innovative approach transforms bedtime stories into an engaging and educational adventure.
To realize this project, the core technology is divided into AI speech recognition for scene generation and depth recognition of the real space within the MR device. This involves confirming the locations of the ceiling and the bed. The focus will be on the MR aspect, which includes connecting the MR device with Unity and building basic scenes, recognizing hand movements, determining the depth of the room, and setting up collision bodies. Successfully addressing these components will resolve the major technical challenges of the MR portion of the project. Subsequently, the project will require implementing space depth recognition, analyzing the positions of the ceiling and bed, and generating models within these specific areas.
In addition to advancing the technical aspects of our project, focusing on the user interface (UI) and user experience (UX) is equally crucial for this kid-oriented app. We aim to balance simplicity, intuition, and fun in our design. The interface will feature intuitive visual elements to appeal to children, avoiding complexity and information overload. Smooth scene transitions and interactive designs, such as drag-and-drop and rotating models, will enhance engagement and exercise children's hands-on abilities and spatial imagination. Challenges such as the limitations of current AI tools and compatibility issues in Unity highlight the potential of MR technology. Looking ahead, we are excited about MR's ability to transform everyday spaces into interactive and immersive environments. Transforming bedroom ceilings into dynamic storytelling canvases is just the beginning; we envision MR seamlessly integrating into daily life, providing entertaining, educational, and therapeutic experiences.