Inside the Robotic Gut: Emotional Evaluation of Bandit Problems

TL;DR

Researchers from the Technical University of Munich have bridged the gap between abstract machine learning and psychological realism. By applying the Zürich Model of Social Motivation to the Multi-Armed Bandit (MAB) problem, they have enabled artificial agents to experience "security" and "arousal." This allows a robot to translate its learning progress and reward history into human-understandable emotions like joy, fear, and uncertainty, moving beyond clinical metrics like "regret."

The "Believability" Gap in AI

When we evaluate a Reinforcement Learning agent today, we look at the Reward Curve. If the curve goes up, the agent is "succeeding." However, a robot operating in a human environment needs more than a high reward; it needs to be believable.

Existing systems typically focus on Emotion Recognition (detecting how the human feels). The authors argue that agents need their own Internal Emotional State to build trust. Without a "gut feeling" about its own decisions, an agent remains a black box, incapable of expressing the tension of exploration or the satisfaction of a stable strategy.

Methodology: The Zürich Model Meets MAB

The core of this research is the adaptation of the Zürich Model, a systems-theoretic psychological model. The authors focus on two primary subsystems:

1. The Security System (Red): Driven by familiarity and relevancy (calculated through eligibility traces). It models the agent's sense of protection and stability.
2. The Arousal System (Blue): Retriggered by new or uncertain situations, mapping to feelings of curiosity or alarm.

Architectural Integration

The authors treat the MAB arms as objects in a psychological space. Using Eligibility Traces, they determine:

• Familiarity (Fi): How often have I picked this arm?
• Relevancy (Ri): How much reward has this arm given me?

These inputs are processed through feedback loops where they are compared against reference values: Dependency (D) for security and Enterprise (E) for arousal. The discrepancy between the current state and these references generates "Momentum Vectors," effectively a psychological drive.

Experimental Insights

The researchers tested the model using the Gittins Index policy—an optimal strategy for Bernoulli bandits. They introduced environmental disturbances (e.g., swapping which arm provides the best reward) to observe the emotional fallout.

Key Observations:

• Phase 1 (Learning): High arousal leads to "uncertainty."
• Phase 2 (Convergence): As the agent finds the best arm, security rises while arousal falls, manifesting as "joy" and "trust."
• Phase 3 (Environmental Shift): When the reward structure breaks, the agent experiences a spike in arousal and a drop in security, resulting in "aversion" and "fear."

The table below highlights the rule-based logic used to bridge the gap between numerical "feelings" and human labels:

Critical Analysis & Conclusion

Why this matters

This work provides a zero-training framework for AI self-evaluation. It doesn't require a neural network to learn "how to feel"; the feelings are an emergent property of the decision-making mathematics combined with biological axioms.

Limitations

Currently, the emotional state is a passive observer. The agent's "fear" or "joy" does not influence its next elective action—it only reports them. Furthermore, the mapping from (Security, Arousal) to human labels like "Anger" or "Trust" (as seen in Table II) is a heuristic defined by the authors and lacks a rigorous psychological baseline.

Future Outlook

The next logical step is Emotional Feedback Loops. Imagine an agent that chooses to explore because it feels "bored" (low arousal) or retreats to a safe strategy because it feels "fearful" (high arousal/low security). By closing this loop, we could create agents that not only look human-like in their reactions but also possess biologically inspired Inductive Biases for learning in the real world.