Do generic centrally symmetric convex outer billiards admit unbounded orbits?

What the Moser–Neumann question asks

The outer (or dual) billiard map about a compact strictly convex body K sends a point x outside K to its reflection through the point of tangency p of the supporting line to K through x, with the tangent line chosen by a fixed orientation convention. The map is an area-preserving homeomorphism from the exterior of K to itself. Unlike classical inner billiards, its phase space is unbounded, so it is natural to ask whether every orbit stays bounded.

Moser phrased the problem as "Decide whether every orbit is bounded," attributing it to Neumann; Neumann's own formulation asked whether the map admits invariant curves. The two are closely related, because arbitrarily large invariant curves would confine all orbits and hence imply boundedness. The paper states the question as: is there a convex table for which the outer billiard has an unbounded orbit?

The paper answers this question affirmatively in a specific regularity class: centrally symmetric, strictly convex, compact planar bodies whose boundary is a regular C1 curve. The result is stronger than Schwartz's earlier question, which asked only for a strictly convex domain or a C1 domain with unbounded outer billiard orbits.

Why smooth tables cannot produce this

For sufficiently smooth tables all orbits are bounded. If the boundary of K is of class C6 with positive curvature, Moser's invariant curve theorem, applied to the return map near infinity, produces invariant curves that confine every orbit; the complete proof is due to R. Douady. Gutkin and Katok studied caustics of outer billiards in this regime, and Boyland exploited the twist-map structure of the return map near infinity.

The paper's Main Theorem does not conflict with this. Smooth tables with positive curvature form a meager subset of H, and for them the radius-of-curvature measure is absolutely continuous, so they lie outside the residual set R. The result therefore lives in the gap between very smooth tables, where confinement is forced, and polygonal tables, where the dynamics is a piecewise isometry.

At the polygonal extreme, orbits are bounded for lattice and quasi-rational polygons and for trapezoids, while Schwartz showed that the outer billiard about the Penrose kite has unbounded orbits, and Dolgopyat and Fayad did the same for the semicircle. The paper targets the intermediate case of strictly convex C1 tables with no further regularity.

The radius of curvature measure as the key variable

The paper works with the space H of support functions of centrally symmetric, strictly convex compact bodies whose boundary is a regular C1 curve, endowed with the C1 topology. For a C1 function h on R/πZ, the radius-of-curvature measure is defined as ρh := h dθ + Dh′, where D is the distributional derivative. When h is the support function of a convex body, ρh is a positive measure.

The space H is characterized exactly as the set of h in C1(R/πZ) with h > 0, ρh ≥ 0, and support of ρh equal to all of R/πZ. Each condition has geometric meaning: ρh ≥ 0 makes h the support function of a convex body; π-periodicity encodes central symmetry; h in C1 means the body is strictly convex and is equivalent to ρh having no atoms; and full support of ρh means the boundary has no corners and is a regular C1 curve. The last condition is not a consequence of h in C1, since the support function of a strictly convex body with corners is still C1.

The intuition the paper offers is that purely singular radius-of-curvature measures make a C1 strictly convex domain behave, as far as its outer billiard dynamics is concerned, as similarly to a polygon as possible. For a convex polygon with edges e1 through en, the radius-of-curvature measure is a sum of atoms ℓk δθk, where ℓk is the length of edge ek and θk is the outward normal direction of that edge. Polygons are neither C1 nor strictly convex, so to stay close to polygons while remaining in H, the paper considers purely singular ρh with full support.

The rigidity theorem that drives the proof

The key ingredient is a rigidity theorem. If h is in H and ρh is purely singular, then every continuous invariant tangent graph of the outer billiard map about the associated body consists entirely of periodic points with the same period. The proof rests on the rigid and complementary behavior of the map at edge-like and vertex-like points, which is the mechanism that forces invariant graphs to be periodic.

This rigidity is what makes the rest of the argument tractable. For tables with purely singular radius-of-curvature measure, eliminating invariant graphs reduces to eliminating invariant graphs made entirely of periodic points, a far more classical problem. Periodic configurations are critical points of an explicit action functional, and the paper shows that for each period q and each compact region of the phase space there is an open dense subset of H for which the corresponding outer billiard maps admit no periodic graphs of period q in that compact region.

Combining these open dense sets with the residual set of tables having purely singular radius-of-curvature measure yields the residual set R of the Main Theorem. The paper also treats invariant graphs that touch the boundary circles of the compactified annulus directly, since the rigidity theorem alone does not address them.

What the Main Theorem guarantees

The Main Theorem states that there exists a residual, meaning Gδ dense, subset R of H such that for every h in R three things hold. First, the radius-of-curvature measure ρh is purely singular with respect to Lebesgue measure, carried by a Lebesgue-null set. Second, the outer billiard map T about the body Kh has no continuous invariant curve in the exterior of Kh that is a graph over the contact angle. Third, there exist points x outside Kh with |T^n x| tending to infinity as n tends to infinity.

In particular, there exist h in R arbitrarily C1-close to the support functions of disks or ellipses. Since the outer billiard maps about disks or ellipses are integrable, the paper interprets the third part of the Main Theorem as a manifestation of Arnold diffusion for C1-generic perturbations of those integrable maps.

The three parts of the theorem are not interchangeable. Purely singular radius-of-curvature measure is a property of the table; absence of continuous invariant graphs is a statement about invariant sets of the map; and the existence of escaping orbits is a statement about individual orbits. The proof chains them together, but they answer different questions and should not be substituted for one another.

How escaping orbits are constructed

The passage from "no invariant curves" to "orbits escaping to infinity" uses Mather's mechanism for diffusing in a Birkhoff region of instability. Because the outer billiard map in this setting is only a homeomorphism, the paper uses Le Calvez's topological formulation of that mechanism rather than the smooth variational version.

Two results of Le Calvez are central. The first says that for a non-wandering, V-monotone homeomorphism of the annulus, the frontier of an invariant lower or upper annulus is an essential simple loop transverse to the vertical foliation. The second says that if the two boundary circles are the only invariant essential simple loops transverse to that foliation, then there is a point whose backward orbit approaches one boundary circle and whose forward orbit approaches the other. The paper shows that the compactified return map is V-monotone, with its inverse V-decreasing and the map itself V-increasing.

A technical obstacle is that non-wanderingness is automatic for area-preserving homeomorphisms of a compact annulus by Poincaré recurrence, but not here, because the invariant measure is Euclidean area, which is infinite on the exterior of K and infinite near the outer boundary circle of the compactification. The paper handles this by observing that a wandering open set is itself a source of unbounded orbits: if a non-empty open set W satisfies T^n(W) ∩ W = ∅ for all n ≥ 1, then |T^n x| tends to infinity for Lebesgue-almost every x in W. Consequently, either the compactified map is non-wandering, or the outer billiard already has a positive-measure set of forward escaping orbits.

Where the result applies and where it stops

The result applies to centrally symmetric, strictly convex, compact planar bodies whose boundary is a regular C1 curve, with the origin placed at the center of symmetry. It is a statement about a residual subset of H in the C1 topology, not about every table in H. The residual set includes support functions arbitrarily C1-close to those of disks or ellipses, but closeness to an integrable table does not by itself guarantee escaping orbits for a particular table.

The paper is explicit that the Main Theorem does not conflict with boundedness for C6 tables with positive curvature, because those tables form a meager subset of H and their radius-of-curvature measures are absolutely continuous, placing them outside R. The result also does not claim that every table with purely singular radius-of-curvature measure has unbounded orbits. Appendix B constructs C1, centrally symmetric, strictly convex domains with purely singular radius-of-curvature measure that admit an invariant curve consisting of periodic points of period 4, and these tables can be constructed arbitrarily C1-close to disks or ellipses. The paper notes that this does not by itself prevent the existence of unbounded orbits for those tables.

A further scope condition is that the invariant curves excluded by the theorem are those that are graphs over the contact angle. The paper also shows that an invariant graph touching the outer boundary circle must coincide with that circle, and that an invariant graph touching the inner boundary circle must lie in a compact region uniformly away from the outer circle, so such graphs do not obstruct escaping orbits.

What to take from the result

The paper's decisive move is to separate two questions that are easy to conflate: whether a table has purely singular radius-of-curvature measure, and whether its outer billiard map has unbounded orbits. The first is a regularity condition on the support function; the second is a dynamical conclusion. The Rigidity Theorem bridges them by forcing invariant tangent graphs to be periodic, and the generic absence of periodic invariant graphs then removes the obstruction to escaping orbits.

For readers comparing this result with earlier work, the relevant threshold is regularity. At C6 with positive curvature, invariant curves near infinity confine every orbit. At the polygonal extreme, the dynamics is a piecewise isometry and boundedness depends on the polygon. The paper's contribution is the intermediate C1 strictly convex regime, where it shows that a residual set of centrally symmetric tables has orbits escaping to infinity, and that this residual set reaches arbitrarily C1-close to disks and ellipses.

The most useful thing to carry away is the role of the radius-of-curvature measure as the discriminating variable. Purely singular measures with full support are what make a C1 strictly convex table behave dynamically like a polygon, and it is exactly on the residual set of such tables that the paper proves the absence of continuous invariant tangent graphs and the existence of escaping orbits. The period-4 example in Appendix B is the sharpest reminder that purely singular radius-of-curvature measure alone does not settle the dynamical question for an individual table; genericity in H is doing real work in the theorem.

Sources used in this answer