Context-Aware Multimodal Sharing: Bringing Digital Emotions into the Physical World

TL;DR

Communication in the digital age often feels "cold" because it lacks the rich non-verbal cues of face-to-face interaction. This paper proposes a system that captures a user's emotional state via neuroheadsets and "broadcasts" it into the receiver's physical environment using smart surfaces and a robotic painting named Aphrodite. By being "context-aware," the system smartly chooses whether to show an emotion on a TV, speak it through a speaker, or mime it via a robot depending on the room's conditions.

The Missing Dimension of HCHI

Human-to-Computer-to-Human Interaction (HCHI) has a fundamental flaw: the loss of paralanguage. While emoticons were a step forward in 1982, today’s Social Awareness Streams (SAS) like Twitter are still largely confined to small screens and 2D layouts. The authors argue that because humans naturally share emotional experiences shortly after they occur, our digital tools should support a more immersive, "natural" way to transmit these feelings into our surroundings.

Methodology: The Fusion of Context and Affect

The core of the system lies in its ability to handle Multimodal Fission—the process of taking an abstract piece of information (an emotion) and deciding how to "render" it.

1. The Architecture

The system relies on two key frameworks:

• Inter-Face: Manages heterogeneous interactive surfaces and neuroheadset drivers.
• NAIF: The "brain" that gathers context data and decides which device should speak or display information based on the environment.

2. Sensing and Mapping

Instead of just manual input, the system uses the Emotiv EPOC neuroheadset to sense facial expressions via surface Electromyography (sEMG). These are then mapped to Plutchik’s Wheel of Emotions:

• Smile $\to$ Happiness
• Frown $\to$ Sadness/Anger
• Wink $\to$ Trust
• Laugh $\to$ Ecstasy

3. Ambient Feedback (The Aphrodite Factor)

The most unique output is Aphrodite, a robotic painting of Botticelli’s Venus. It acts as an anthropomorphic agent, miming the sender's facial expression to create an empathetic connection for the receiver.

Intelligent Reasoning: Why Context Matters

A "smart" system shouldn't just be multimodal; it must be opportunistic. The authors implemented reasoning logic based on environmental sensors:

• Luminosity: If the room is too bright for a projector, the system automatically redirects visual feedback to a high-contrast TV.
• Noise Levels: If the environment is noisy, vocal synthesis (TTS) is bypassed in favor of visual emoticons.
• Privacy: Using RFID tags, the system can identify if a specific user is near a device and deliver "private" emotional tweets only to that screen.

Experimental Validation

The system was tested in a "Smart Living Room" setup. During public demonstrations, users moved from skepticism to high engagement. The choice of multiple input modalities was praised, though the physical bulk of the neuroheadset was a noted limitation (Inductive bias: comfort is key to adoption).

Critical Insight & Future Outlook

This work highlights a shift from Personal Computing to Atmospheric Computing. The real value isn't just in the Twitter integration, but in the logic that treats a room as a unified display.

Limitations: The current reliance on sEMG is cumbersome for daily use. However, the authors suggest that as vision-based face recognition and wearable wristbands (like the Affectiva Q-Sensor) mature, the "friction" of sharing emotions will decrease. Future social networks might not be apps we browse, but environments we inhabit, where the "vibe" of our social circle is physically manifested around us.

Summary Table