Can self-referenced social preferences enable cooperation without observing others' rewards?

Cooperation usually depends on a private signal

Sequential social dilemmas are games where individual incentives conflict with collective welfare. The paper uses three: Escape Room, where some agents must pay a cost to pull a lever so others can leave; Clean Up, where cleaning a river earns nothing directly but keeps apples growing; and Commons Harvest, where agents must leave apples uneaten so patches regenerate. Independent deep reinforcement learners fail to cooperate in these games.

The standard remedy is to give each agent a social preference, an extra term in its objective that depends on how other agents are doing. Inequity aversion penalizes earning more or less than peers, social value orientation weights others' outcomes alongside one's own, and prosocial preferences add others' rewards to one's own. All of these raise cooperation, but they do so by reading the other agents' rewards. The paper argues this is unsatisfying: a reward is an agent's private signal of how good an experience was, and handing it to another agent supplies information that agent could not get by interacting with the world.

The study therefore keeps the social preferences but replaces their input. Each agent receives only the observations and actions of its peers, not their rewards, critics, or parameters. It learns a model of its own reward and applies that model to the peers' observed transitions, producing a self-referenced assessment of how the peer is doing from the agent's own perspective. The paper's question is whether cooperation can be learned from that substitute.

The estimate can enter the reward or the update

The paper studies two integration points. In the first, called SRR, the self-referenced assessment modifies the learning reward, so the agent's own reward is adjusted by its social preference before the critic learns from it. In the second, called SRA, the assessment is used to weight policy updates, shifting credit between actions rather than changing the reward the agent learns from.

A value look-ahead is added to SRR in the configurations the paper highlights. The critic's value distinguishes positions before any reward has been assessed, which matters in Escape Room: an agent that never reaches the door never sees a reward to assess, so a social term built only on assessed rewards stays at zero. The look-ahead gets cooperation started by valuing positions the agent has not yet been rewarded for.

The paper also introduces a standing ledger for repeated interaction. Because the same agents meet in every episode, each agent keeps a running average of every agent's estimated episode return, which discounts the gains of an agent that has been ahead. This is meant to separate an always-me configuration from one where the costly role rotates.

Cooperation appears in all three games

Across all three environments, agents learn cooperative behavior without observing others' rewards, including settings where independent learners fail. In Escape Room, plain PPO agents never learn to pull the lever, but a self-referenced agent can recognize the benefit because it has learned from its own exits that reaching the open door pays ten. SRA-EI and SRR-IA+V reach the optimum of 17, as do agents given true rewards.

In Clean Up, every self-referenced learner produces about as much as its true-reward counterpart, so self-referenced estimates are enough to sustain a public good. Under inequity aversion, SRR does better than the true-reward learner: SRR-IA+V produces more than TR-IA and more than doubles the return of its poorest agent. The paper's explanation is that true rewards are the obstacle here, because an agent that has just eaten an apple looks far ahead of one that is cleaning, even though apples are about to grow for the cleaner.

Commons Harvest is where self-reference falls furthest behind. SRA-EI doubles plain PPO's collective return and its agents eat apples four times later on average, but TR-EI and TR-SVO still produce more. The paper attributes the gap to harder estimation: the reward model is learned through a shared representation with the policy, whose image features were not trained to recognize rewards, and the beam's large penalty dominates what it learns.

Inequity aversion and benevolence need different placements

The effective integration point depends on the social preference. Inequity aversion works only with SRR: SRR-IA+V is the better of the two methods in every game, while SRA-IA never escapes Escape Room. The paper's mechanism is that being ahead costs the agent part of its return when the estimate enters the reward, which it can learn to avoid; in the policy update, being ahead only shifts weight between scores, so nothing pushes the agent to stop being ahead.

A purely benevolent preference such as EI only adds up the others' outcomes, crediting actions with other agents' gains. That makes SRA-EI the better method in Escape Room and Commons Harvest and about as good in Clean Up. SVO also adds outcomes but weighs the agent's own as well, and in the policy update this caps the weight assigned to others at a fraction of the agent's own weight, which the paper finds is not enough to match EI.

The paper's summary rule is that a preference comparing agents belongs in the reward, while a benevolent preference such as EI is safer in the policy update. The look-ahead is what makes SRR work, but under EI it brings no clear gain outside Escape Room: in Clean Up, SRR-EI+V produces about 290 apples less than SRR-EI, and in Commons Harvest it stays at plain PPO's level.

High collective return can hide an unequal split

Collective return alone does not show who pays for cooperation. In Clean Up with five agents, TR-EI produces 1,530 apples, but its poorest agent earns nothing for the whole run: the richest agent eats a third of the apples and cleans almost nothing, while the two poorest do 96% of the cleaning. Nash welfare, a geometric mean that falls toward zero when any agent is left with nothing, is only 70 for TR-EI even though its agents average about 300 apples.

SRR-IA+V is the exception. All five agents' returns climb together, every agent takes about a fifth of the apples and a fifth of the cleaning, and its Nash welfare of 270 equals its average return, at a collective return about 12% below TR-EI's. The paper stresses that this high welfare comes from sharing the work, not from every agent earning equally little.

The paper reports both collective return and Nash welfare, plus the lowest agent return, because neither metric is enough alone. In Commons Harvest, Nash-welfare settings make the SRR learners equal by reducing their production to plain PPO's level, which the paper calls the failure mode of that metric. The standing ledger addresses the repeated-game version of the same problem: in Escape Room with seven agents, Nash welfare selects SRR-IA with standing at a gate strength of 2, reaching 72% of the optimum, over a setting that produces more but leaves the poorest agent at a loss.

Partial views and larger groups test the limits

Under partial observability, with a 15 by 15 window in Clean Up and Commons Harvest, the policy-update approach continues to support cooperation for both feed-forward and GRU agents. Group size exposes a limit of SRA: its coefficient averages over the N minus 1 others, so any one agent's action matters less as the group grows, and SRA-EI falls further behind TR-EI. SRR does not dilute in the same way, and the paper reports that inequity aversion with true rewards declines as group size grows while the self-referenced version with the look-ahead holds.

The paper also tests heterogeneous private valuations, where agents value an apple differently. There, self-reference becomes a counterfactual assessment: the reward the agent itself would receive in the other's situation. No learner in that test acts on the value difference, so the paper states the test cannot show whether self-reference would mislead an agent that otherwise would.

The claim holds when rewards are shared

The central scope condition is the shared reward function. A self-referenced estimate recovers another agent's reward only when every agent is rewarded for the same events in the same way. Agents whose rewards come from different events, or from events their transitions do not show, are beyond a model of one's own reward. The paper suggests keeping one's own model as a prior and correcting it from what others do, combining self-reference with inverse reinforcement learning, as an open direction.

Every learner in the study is independent PPO. The paper argues the reward placement needs only a reward and the policy-update placement only a weighted policy gradient, so both should carry over to other actor-critic methods and the reward placement to value-based learners, but neither has been tested. Settings are also chosen on two or three tuning seeds by mean minus one standard deviation, a rule of thumb rather than a confidence bound; the paper notes visible failures of that rule but states that because every reported number comes from held-out seeds, the failures add noise rather than favoring any learner.

The practical priority that follows is not simply whether self-reference works, but which placement matches which preference. For inequity aversion, put the self-referenced estimate in the reward with a value look-ahead; for a benevolent preference such as EI, weight the policy update instead. And when comparing configurations, read collective return together with Nash welfare and the lowest agent return, because the most productive Clean Up configuration in the paper leaves one agent with nothing while a slightly less productive one shares the work.

Sources used in this answer