An agent that must find goals without rewards
The paper studies an unsupervised goal-conditioned setting. The agent receives no demonstrations and no reward function; it must explore from scratch and learn to maximize the likelihood of reaching commanded goals. Performance is measured as the number of time steps, out of 1000, that the agent stays near the goal, and a single reported score averages the last five epochs of training.
This setup matters for interpreting every number below. A gain here means more time spent near commanded goals in simulation, not a reward-driven score, and the agent had to discover the behavior through its own exploration rather than imitate a demonstrator.
Depth means dense layers, scaled jointly
The paper defines network depth as the total number of Dense layers across all residual blocks. Each residual block repeats four units of a Dense layer, Layer Normalization, and Swish activation, with the residual connection applied after the block's final activation. In all main experiments, depth refers to the actor network and both critic encoder networks, which are scaled together; only a later ablation separates them.
The baseline is a 4-layer MLP, the depth used by the JaxGCRL benchmark and by relevant prior work. The study then tests depths of 8, 16, 32, and 64, and extends to 256 and 1024 on the hardest humanoid mazes. Stabilizing such deep networks required residual connections, layer normalization, and Swish activations; ablations report that removing layer normalization performs significantly worse and that scaling with ReLU hampers scalability.
Gains range from 2x to 50x by task
Compared with 4-layer models, deeper networks produce 2-5x gains on robotic manipulation tasks, over 20x gains on long-horizon maze tasks such as Ant U4-Maze and Ant U5-Maze, and over 50x gains on humanoid-based tasks. The paper's table makes the spread concrete: Arm Push Easy rises from 308 to 762 time-at-goal steps (2.5x), Ant U4-Maze from 11.4 to 286 (25x), Humanoid from 12.6 to 649 (52x), and Humanoid Big Maze from 0.06 to 59 (1051x).
The authors note that the largest benefits appear on tasks with the largest observation dimension. Against external baselines, scaled contrastive RL outperforms SAC, SAC+HER, TD3+HER, GCBC, and GCSL in 8 of 10 environments. The exception is SAC on the humanoid maze environments, where SAC is more sample-efficient early on, though scaled contrastive RL eventually reaches comparable performance.
Performance jumps at environment-specific critical depths
Improvement is not smooth. Instead of gradual gains, performance shows pronounced jumps once a critical depth threshold is reached, and that threshold varies by environment, from 8 layers on Ant Big Maze to 64 layers on Humanoid U-Maze. Further jumps appear at depths of 256 and 1024 on the humanoid mazes.
The authors connect these jumps to qualitatively different learned behaviors. A depth-4 humanoid agent collapses or throws itself toward the goal; at depth 16 it walks upright into the goal; at depth 64 it struggles and falls in Humanoid U-Maze; at depth 256 it folds forward into a leveraged position to propel itself over walls and shifts into a seated posture to move over an obstacle. The paper states this is, to its knowledge, the first goal-conditioned approach to document such behaviors on the humanoid environment.
Depth helps contrastive RL more than width or other algorithms
The paper compares scaling width against scaling depth. Prior work suggested width can enhance performance, but here depth yields more impactful gains; in the Humanoid environment, the reported comparison favors depth over width at matched parameter counts.
Depth scaling is also tied to the algorithm. SAC, SAC+HER, and TD3+HER do not benefit from depths beyond four layers, consistent with prior findings, and GCSL and GCBC fail to achieve meaningful performance on Humanoid and Ant Big Maze. One exception is GCBC on Arm Push Easy, which improves with depth. The authors conclude that the self-supervised contrastive RL algorithm is critical to the effect. A separate test with the quasimetric CMD-1 architecture finds scaling benefits are not limited to a single network parametrization, though its poor performance on Ant U5-Maze suggests more work is needed for consistent scaling.
Actor and critic scaling matter differently by task
An ablation scales the actor and critic separately. In Arm Push Easy, scaling the critic is more effective; in Ant Big Maze, the actor matters more; in Humanoid, scaling both is necessary. The authors read this as actor and critic scaling complementing each other for contrastive RL.
This is a practical distinction for anyone trying to reproduce the gains: the headline recipe of scaling everything jointly is not equally important in every environment, and the paper's own grid shows that a deep actor alone or a deep critic alone can leave substantial performance on the table.
Offline goal-conditioned RL shows little benefit
The positive results come from online training. In preliminary offline experiments on OGBench, the paper found little evidence that increasing contrastive RL depth improves performance. Ablations that scaled critic depth while holding the actor at 4 or 8 layers, and that applied cold initialization to the final critic encoder layers, still often had baseline depth-4 networks achieving the highest success.
A separate offline test on three OGBench environments reports that performance declined drastically in two of three as depth scaled from 4 to 64, with a slight improvement on antmaze-medium-stitch-v0. Offline GCBC did scale with depth on antmaze-medium-stitch, while behavioral cloning and QRL showed negative results. The authors treat adapting the method to offline goal-conditioned RL as future work.
Compute cost sets the priority for adopting depth
The paper's stated primary limitation is that scaling depth costs compute. That reframes the result as a trade-off rather than a free win: the largest reported gains, such as the 52x on Humanoid and the 1051x on Humanoid Big Maze, come from the deepest networks, and the 1024-layer runs required holding the actor at 512 layers because the actor loss exploded at the onset of training.
The most useful reading is therefore conditional on task and setting. If the task is an online, unsupervised goal-conditioned problem with a large observation dimension and the algorithm is contrastive RL, depth is the lever the paper identifies, with critical thresholds to watch for rather than a smooth return. If the setting is offline, or the algorithm is a TD-based or imitation-based baseline, the paper's evidence does not support expecting the same jump.
Sources used in this answer
1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities
Increasing contrastive RL network depth from 4 to 64 layers, and up to 1024 on humanoid mazes, raises goal-reaching performance by 2x to 50x in simulated locomotion and manipulation tasks, with gains appearing as jumps at environment-specific critical depths.