A minigrant project, led by principal investigators Dr. Leigh Jin and Weimin Zhang from San Francisco State University, explored the intersections of art, technology, history, modernity, the past and the future. Entitled, “Bring the Ancient Instruments back to Life through an Augmented Reality App,” Dr. Jin, Zhang, and their student assistants utilized augmented reality (AR) to capture 360-degree images of ancient South Asian, East Asian, and indigenous Mexican musical instruments to construct a mobile app to display the virtual collection.
With expertise in app development, Professor of Information Systems Dr. Leigh Jin collaborated with Professor of Cinema Weimin Zhang. Zhang, who previously had integrated AR/VR into her cinema curriculum, eagerly accepted the invitation to partner. “When Leigh told me about this innovation grant, I was very excited. I thought, ‘Wow that matches perfectly for technology integrated with art.’ This was something I was excited to explore further,” Zhang shared.
Zhang also reflected that typical AR/VR demonstrations only featured the technology itself and lacked a storytelling component. Jin weighed in saying, “A lot of these demos… just demo for technology’s sake. There’s no storytelling behind it. It’s not interesting. ” With that sentiment, storytelling and mean-making became a part of the mission guiding this project.
To imbed experiential learning into this endeavor, the project team visited the Music Instrument Museum in Phoenix, Arizona to learn more about the selected ancient instruments and capture the necessary photos. Anh Tran, a key student assistant, obtained the necessary photos of five musical instruments using Polycam, a 3D capture app. Those images were then uploaded using Lidar and photogrammetry to construct the 3D models. Sound recordings were later paired with the images so that users could aurally engage with the instrument.
While photographing these images, the team realized limitations with fully capturing some of the rarer, delicate instruments that could not be moved from their display. For example, Chinese bronze bells were displayed in such a way that it was not possible to get a 360-degree view so it rendered partial images. However free-standing, and round instruments proved to be much simpler to photograph- like the Aztec Tlalpanhuehuetl and the Japanese Taiko drums.
After building the AR models, Tran took the lead in the second phase to construct a prototype of an app fit for Apple iOS to allow user interaction with the instruments. Tran, now a recent SFSU graduate, has a strong passion for Apple technology and brought a valuable skillset to prototyping using various tools such as: SwiftUI, RealityKit, ARKit, and Xcode. Tran developed a fully functional mobile app with the ancient instruments that included the ability to manipulate the size, placement, and proximity with occlusion features. Watch a demo of the interactive features below.
The team had the opportunity to present their app to Apple’s Education Team in late June where they received valuable feedback on ways to enhance and scale their work. With this development, the project team opened the door for experiential learning and cultural appreciation for students and historians alike to experience artifacts that they may not otherwise have encountered.
The team also sees applicability of AR to future art forms, especially in NFTs. Soon after receiving a certification in App Development with Swift from Apple, Tran got to see relevant examples of AR in art and animation at the NFT Art Exhibit in San Francisco. Dr. Jin reflected that, “this was very inspirational and made us realize… what the potential is for augmented reality in art and that this is truly for the future.”
With the first iteration of the AR instruments app completed, the team possesses an even deeper understanding and appreciation of the work they completed. Dr. Jin concluded, “Before, we just stayed in the classroom facing the computer coding like crazy. But this truly brought a lot of inspiration into the project. We are not just coding to showcase but… we understand the history, culture, meaning, and stories behind those ancient musical instruments. We hear their sounds, we hear their stories.”
