Why tilting the light changes what comes back
A multilayer reflector is a stack of thin layers that reflects a band of wavelengths through interference. Light can be split into two polarization components, and the stack does not reflect those two components equally; the reflection of each depends on the incidence angle and on the optical properties of the layers [1]. As the angle changes, the relative strength of the two reflected components changes too, so the polarization of the reflected beam is not fixed but angle-dependent.
The clearest general picture comes from a single interface, which the multilayer stack is built from. When a circularly polarized beam hits an interface, the reflected light contains a normal component that keeps its spin and an abnormal, spin-flipped component that picks up a geometric phase and shifts sideways [1]. The strengths of these two modes are set by the incident angle and the interface's optical properties, so the reflected polarization is a direct function of angle [1].
What is special about the Brewster angle
The Brewster angle is the incidence angle at which reflection of one polarization component falls to a minimum, effectively vanishing at a simple interface. Near this angle, the reflection coefficients for the two circular polarization components take opposite signs, so the normal and abnormal modes interfere destructively [1]. The result is a highly deformed reflected beam pattern with anomalously enhanced spin-Hall shifts, meaning the beam's polarization structure and position become unusually sensitive to small angle changes [1].
This is not a small correction. Because the two components cancel in a structured way rather than simply dimming, the reflected light near the Brewster angle can look qualitatively different from what a simple brightness argument would predict [1]. The same kind of Brewster-like behavior is expected to appear in both reflection and transmission geometries, which is why the angle dependence matters beyond a single interface [1].
How this plays out in real multilayer and structured reflectors
In practical multilayer devices, the angle dependence is a design parameter rather than an accident. A Ka-band multilayer polarization-rotating reflector built for polarimetric radar was optimized for 45° oblique incidence, rotating a linearly polarized field by 90° at selected frequencies [2]. That design shows that a multilayer stack can be engineered so that its polarization response is tuned to a chosen oblique angle, though this particular study did not examine Brewster-angle behavior [2].
Other structured reflectors show the same angle sensitivity in different physical systems. Pitch-gradient cholesteric liquid crystal reflectors display polarization-selective reflection that varies with incidence angle across four input polarizations, with spectra measured at typical angles such as 20° and 40° [4]. In biological structural color, multilayer reflectors are also associated with polarized beam splitting accompanied by Brewster-angle reflection, linking the same angle-dependent polarization physics to natural optical systems [5]. Older work on periodic optical waveguides treats multilayer reflectors and notes that results carry over to the orthogonal polarization and that behavior at the Brewster angle affects how power is reflected [3].
What a reader should expect at different angles
At angles well away from Brewster, the reflected polarization changes gradually: the two components are reflected with different but stable strengths, so the output polarization rotates or becomes more elliptical in a smooth way as the angle tilts [1]. As the angle approaches Brewster, one component is suppressed and the sign relationship between components can reverse, making the polarization state swing sharply and the beam profile distort [1]. Beyond Brewster, the balance shifts again, so the same reflector can produce quite different polarization outputs on either side of the angle [1].
The practical lesson is that a multilayer reflector's polarization performance should be specified at the angle it will actually be used. A stack optimized for normal incidence or for one oblique angle, such as the 45° radar design, is not guaranteed to behave the same way near Brewster [2]. Where Brewster-angle operation matters, the angle-dependent interference between polarization components is the effect to design around, not a detail to ignore [1][3].
Sources used in this answer
Revisiting the anomalous spin-Hall effect of light near the Brewster angle
Near the Brewster angle, reflection coefficients for the two circular polarization components have opposite signs, causing destructive interference between normal and abnormal modes and producing highly deformed reflected beams with enhanced spin-Hall shifts.
A Ka-Band Polarization Rotating Multilayer Reflector for Polarimetric Radars
A Ka-band multilayer polarization-rotating reflector was designed and measured to rotate linear polarization by 90° at selected frequencies when optimized for 45° oblique incidence.
Analysis and applications of periodic optical waveguides
Periodic optical waveguide analysis treats multilayer reflectors, notes that results carry over to the orthogonal polarization, and discusses power behavior at the Brewster angle.
Anomalous Polarization-Selective Reflection in Pitch-Gradient Cholesteric Liquid Crystals
Pitch-gradient cholesteric liquid crystal reflectors show polarization-selective reflection that varies with incidence angle across four input polarizations.
Structural color
Structural color systems can be treated as multilayer reflectors in which interference is accompanied by Brewster-angle reflection and polarized beam splitting.