Constructing Virtual Reality Exhibitions with Multimodal Interactions

This master thesis aims to help museums to construct virtual exhibitions with multimodal interactions with capacity, connectivity, immersiveness, and flexibility in an efficient manner..








This thesis hypothesizes that constructing VR-based virtual museums is an effective method to expand museums’ virtual identity and shift their paradigm in response to the social and technological changes of contemporary society; and a design system that is modularized and object-oriented could help modern museums construct virtual exhibitions in VR with capacity, connectivity, immersiveness, and flexibility in an efficient manner.


For Museums

Benefits of virtual exhibition in VR

In comparision with a traditional physical exhibition


In terms of methodology, the thesis could be divided into four parts. A background research that covers the timeline of museum’s evolvement in western history, the concept of museum 4.0, the birth of virtual museum and early examples. The second part is a field survey covering current states of the development of virtual museum, including precedents, categories, interviews with professionals, as well as a questionnaire. The third part is two VR exhibitions constructed as part of this thesis. And finally a design guideline is written based on the previous sections.

Background Research

Timeline of Museum in Western World

When considering the timeline of museums in the Western world, which spans more than 2,000 years, one could find that the definition and identity of museums have shifted radically. There are the repetitive shifts of a museum’s role as a public institute and private sector in addition to its longtime evolution, from emphasizing the collection itself to focusing on the buildings that hold the collection and eventually provide other services to the public. The role of the museum architecture also shifted from being a container and backdrop of the collection to the “object” on display, itself.

Museum 4.0

In alignment with the evolution of museum space, Mark Walhimer identifies the four development stages of a museum: museum 1.0 as the cabinet of curiosities format; museum 2.0 as museums such as science centers that incorporate interactive technologies into their exhibitions; museum 3.0 as “open-ended constructivist” with exploratorium, multilayered, and inquiry-based exhibitions; and museum 4.0 as a “museum with walls.” For museum 4.0, the museum experience is divided into three parts: pre-visit, in-person visit, and post-visit, each of which could be fulfilled by physical spaces, digital contents, or both. In addition to the adoption of various technologies, such as radio frequency identification RFID, social media, and Bluetooth mesh technologies, museum 4.0 should be able to provide a personalized experience depending on the visitor’s age group, background, and interests.

Definition and Evolvement of Virtual Museum

Even though the concept of the virtual museum is still young compared with the “brick and mortar” museum, it has evolved for nearly three decades now. As early as 1998, the term "virtual museum" was defined as

“... a logically related collection of digital objects composed in a variety of media which, because of its capacity to provide connectedness and various points of access, lends itself to transcending traditional methods of communicating and interacting with visitors …; it has no real place or space, its objects and the related information can be disseminated all over the world ”

------ Andrews and Schweibenz, 1998

As technology develops, the targeted media of virtual museum has been changed from CD-ROM, World Wide Web, application on a hand-held device, to VR and AR in head-mounted device. During its initial stage, virtual museums could be categorized into two groups: one that considers using digital media as a way to communicate, which is prior to physical exhibitions and paper publications, and one that emphasizes the creation of an immersive space for display, which is tightly connected with the concept of VR.

Field Survey

Dataset of Virtual Museums and exhibitions

As part of the field survey, I started compiling this dataset of projects that loosely fit into the concept of virtual museum and virtual exhibitions

Example of Online Collection

Example of Virtual Tours

Example of Exhibition Experiences in VR

Categories of Virtual Museums and exhibitions

In addition to VR/AR applications that are often installed as part of a physical exhibition, virtual museums that can be accessed offsite could be divided into three major categories: AR/VR application or WebVR contents, virtual tours, and online collections published by museums. From left to right among the three categories, the cost of constructing such an exhibition decreases as does its level of customization.


As part of the research, I also interviewed individuals who work in the field as different roles: curators, conservators, and art preparators. I interviewed them with questions regarding their daily job at museums and galleries, their general perspectives of the identity of museums at the current time, their worries and concerns about museums, and their viewpoints on the concept of virtual museums and virtual exhibitions realized through VR technology.


As part of the field survey, a questionnaire is sent out to collect opinions of common museum-goers. According to the survey, 62.1% participants didn’t have experience with any type of virtual museums and online exhibitions, and 35.1% participants never have experience with any type of VR devices.

In addition, more than 50% of the participants have experienced VR content as part of some museum visiting experience, and for those who did, the most frequent comment is that using VR in public spaces like museums makes people uncomfortable, vulnerable, and feeling isolated. Other concerns that have been mentioned in the response are the long wait time, low-quality content.

Case Studies

I. MFA Case Study

A case study is included based on the Asian Art Collection at the Museum of Fine Arts in Boston. The collection of more than 100,000 objects including paintings, prints, sculpture, ceramics, metalwork, and other art forms from Japan, China, Korea, South, and Southeast Asia, and the Islamic world. The research analyzes the characteristics of different types of objects, and propose different ways of representing them or modeling them in digital format. And I have select 5 objects from the collection and created a digital archive containing their textured models, photos, and metadata. The textured mesh model is created by photogrammetry, while the other information is collected from MFA’s website, Youtube Channel, and printed catalog. When reading about the description of each object, I tried to summarize the information and created five categories. Those categories together could cover the written description of any objects to be displayed in museums. After the categories of information are identified, I come up with five templates designed to showcase each of the information. The design themselves are still under development and need to be tested. The populated template can then be linked together to create paths of navigation, therefore completing the museum experience.



MFA_VR_Museum_Unity_Prototype from Yichen Jia on Vimeo.

II. Art of Memory Case Study

The Art of Memory is an exhibition of a collaborative project between Charisse Foo (Cornell University, B.Arch ‘18) and myself. The project start with a 12-day trip to four Italian towns that have been indelibly changed by natural disasters. It discusses the different nique approach towards destruction and preservation and explores the notion of architectural memory. We have created different types of documentations, and a series of illustrations and diagrams for the project, which was all displayed in the exhibition at MIT Rotch Library. As part of the case study, I compared the different design process of the two physical exhibition, and the virtual exhibition. Similar as the last case study, I wanted to utilize a modularized design system and a object-oriented approach for the design of this virtual exhibition. The diagram below shows that when designing physical exhibitions, artworks are usually arranged based on their sizes and sizes of the exhibition spaces. On the other hand, when designing virtual exhibitions, the list of artworks that will be included in the exhibition are first grouped by types. There is a specific scene designed for each type or a collection of types, and scenes are connected in the end to form the storyline.


User Experiment

Based on this VR exhibition, I conducted user experiments in an attempt to learn how effective the design since. From the diagram above, one could see that Scene E attracts significantly less attention than the other ones, potentially due to the fact that it is hard to find its entrance. Besides it is observed that people get bored after entering scenes with the same template several times, even though the content is different. It is clear that all participants have experienced frequent jumps in between different scenes, and entering the same scene for more than one time. It is potentially due to the lack of indication that whether this scene was entered before. I also looked at information architecture of each scene and examine how it is related to the average total time people spend in that scene. Besides the fact that Scene D and Scene E have significantly less information than the other ones because they are designed to showcase single artworks instead of the collection, participants tend to stay longer in the scenes that are more balanced in terms of number of interactions and contents. Therefore, balancing between the amount of content, interactions, and connections is essential when designing virtual exhibitions. In addition, two type of interactions and locomotions are compared as part of the user experiment. For interaction, participants give mixed feedback in terms of which method they prefer. Generally speaking, participants who have more previous experience with VR are more used to using the laser pointer, while those who didn’t have much previous experience prefer trigger by location. For locomotion, even though 2 participants indicate that they prefer nature walk, all 3 of them mainly used sliding because of various reasons.

Besides the above aspects, the other takeaway of the user experiment is a evaluation metrics subtracted from how participant critic the design of the virtual exhibition. And also reflections in terms of the design, which is then incorporated into the design guidelines and templates.

Major Takeaways In Terms of Design

Design Guideline

Finally, as a final deliverable, I have complied a design guideline that could be used by art museums as reference when they designing such exhibitions in VR, which includes design principles for both VR and exhibition design, as well as design factors that covers a wide range of elements. As part of the design guideline, there are also templates that museum can refer to and swap their own content into the scene.


Back to Home >>>