Transforming Virtual Reality into Social Experiences: A Social-Aware Multimedia Approach
A social-aware multimedia system for interactive cultural and educational experiences
The paper presents a "social-aware" multimedia system that integrates Virtual Reality (VR) installations with social media interaction and real-time video streaming. Deployed at the Foundation of the Hellenic World (FHW), it enables remote experts to participate in live VR tours while visitors interact via a dedicated mobile application connected to Facebook and Twitter.
TL;DR
This research tackles the "loneliness" and "rigidity" of traditional museum visits by introducing a distributed system that connects Virtual Reality (VR) theater visitors with a remote panel of experts via social media. By leveraging Facebook and Twitter as interactive layers, the system provides a dual benefit: guests receive an immersive, expert-led experience, while organizers gain granular, real-time data on visitor engagement.
Background: Breaking the "One-Way" Barrier
Historically, virtual reality installations in cultural centers like the "Tholos" theater at the Foundation of the Hellenic World (FHW) have operated as one-way communication systems. An educator leads a group through a pre-rendered script, and visitors remain silent observers. This paper identifies two major gaps:
- Expert Accessibility: Real experts (archaeologists, historians) are too expensive to keep on-site 24/7.
- Feedback Loop: Organizers rely on boring questionnaires to see if visitors liked the show, which lacks context and spontaneity.
Methodology: The Social-Aware Architecture
The proposed system bridges these gaps through three core technological pillars:
1. The Expert Remote Participation (AVCC)
To bring experts into the room without them being physically present, the Audio Visual Content Component (AVCC) captures the live VR output and educator's audio, encoding it via H.264/AAC and streaming it through a RTMP server. This allows experts to see exactly what the audience sees with minimal latency.
2. Contextual Social Interaction (SocialIntegrator)
The "SocialIntegrator" acts as a bridge between the physical VR theater and Social Networks (SN). A key innovation is the use of representative snapshots. Since typing in the dark is hard and VR navigation is fluid, the system provides a gallery of pre-selected images. A visitor can "Like" a snapshot or post a question under it, which automatically tags the content for the expert to see.
Figure: The overview of the social-aware multimedia orchestration.
3. Real-time Analytics (Experiment Monitoring)
The Experiment Monitoring (EM) component collects every "Like," comment, and question. This turns social noise into structured data, allowing curators to see exactly which "scene" in a VR movie was the most popular.
Experimental Deployment at "Tholos"
The system was deployed in a real-world setting at the Hellenic Cosmos cultural center. In this environment, visitors used Android devices while watching a 3D show on a 13-meter diameter dome.
Figure: A visitor interacting with the mobile application during a live VR session.
Key Insights from the Results:
- Content Optimization: Scenes like the "Agora of Miletus" received significantly higher "Likes," signaling to organizers that more time should be allocated to these sections in future iterations.
- Expert Efficiency: The Expert Application (EA) allows experts to prioritize questions that are "trending" among the audience, ensuring that the most curious guests get their answers first.
- Cross-Network Transparency: By abstracting the complexities of Facebook and Twitter APIs into a single interface, both visitors and experts could communicate regardless of their preferred platform.
Figure: The EM interface providing curators with structured feedback on different movie segments.
Critical Analysis & Conclusion
Takeaway
The genius of this work lies in its Social-Awareness. It recognizes that modern users are already "multi-screening" (using phones while watching TV). Instead of fighting this behavior, the system embraces it to create a collaborative learning environment.
Limitations & Future Work
While successful, the system currently relies on 2D video for experts. The authors suggest that future work could involve 3D reconstruction of the expert, superimposing a holographic representation of the scientist directly into the VR environment. Additionally, moving beyond Facebook/Twitter to include more visual platforms like Instagram or LinkedIn could broaden the demographic reach.
In conclusion, this project serves as a blueprint for the Future Media Internet, where cultural heritage is no longer a static exhibit, but a living, digital conversation.