How does MaRK condition state space models with Markov-adapted recurrent kernels?

Conditioning moves inside the recurrence operator

Most ways of steering a pre-trained state space model for iterative generation act outside its temporal dynamics. Input-stream injection concatenates the conditioning vector into the sequence, and AdaLN scales layer activations. The paper argues that both expose the model to context while leaving the underlying recurrence fixed, so the SSM's temporal behavior does not itself depend on the diffusion timestep.

MaRK instead treats the context vector as an input to the operator. It projects the timestep context into bounded shifts of the frozen backbone's recurrence (A), read-in (B), read-out (C), skip (D) and discretization (Δ) parameters. Because these matrices define how past inputs persist and decay, changing them changes the model's memory profile rather than only its activations. The backbone stays frozen; only the adapter that produces the shifts is trained.

Markov parameters make the conditioning measurable

The paper frames the SSM through its Markov parameter sequence, H(t) = {C(t)A^k(t)B(t)} for lags k ≥ 0. This sequence is the model's input-output memory kernel: it describes how much of an input survives at each temporal lag, and it determines the transfer function and impulse response. The authors note it is invariant under similarity transformations of the state-space coordinates, so it characterizes the dynamics rather than a particular parameterization.

That framing supplies the paper's diagnostic. If MaRK only re-projected features, the Markov sequence should stay close to the frozen backbone's. If it genuinely conditions the operator, the sequence should separate by timestep while still decaying. The experiments measure the head-averaged Markov norm over lags up to 4096 at timesteps t ∈ {0, 0.25, 0.5, 0.75, 1.0} and compare each adapter against unmodulated Hydra.

Bounded shifts keep the modulated recurrence stable

Modulating a recurrence risks pushing it into unstable dynamics, so MaRK constrains the shifts. In Hydra's diagonal SSM, the recurrence eigenvalue is built as A = −exp(A′_log) < 0, the step size as Δ = softplus(Δ_input + Δ′_bias) > 0, and the discretized eigenvalue as Ā_i = exp(A_i · Δ_i), which lies in (0, 1). Hyperbolic-tangent saturation and softplus constraints keep the modulations inside a finite range.

From this construction the paper derives an Affine Quadratic Stability certificate: the identity matrix P = I is a common Lyapunov function satisfying Ā(c_t)ᵀ P Ā(c_t) − P ⪯ −εI for all admissible contexts, with margin ε = 1 − max_i Ā_i² > 0. In plain terms, every modulated recurrence contracts, so the conditioned model cannot blow up. This is an analytic guarantee for the stated diagonal parameterization, not an empirical observation about trained adapters.

Three adapter geometries trade smoothness for flexibility

MaRK is instantiated with three ways of mapping context to parameter shifts, all on the same frozen 111M-parameter, 23-layer Hydra-BERT backbone. The Hypernet uses direct linear projections to produce low-rank scale and shift matrices and is the smallest at about 6.3M trainable parameters. The Chebyshev adapter expands the modulation in orthogonal polynomials up to degree K, at about 11M parameters. The DCT adapter builds a length-L signal from frequency-damped spectral weights over a fixed Type-II cosine basis, at about 7.1M parameters.

The design difference matters because the diffusion timestep is a continuous scalar and adjacent timesteps define nearly identical denoising tasks. A smooth modulation map is therefore appropriate. The Hypernet consumes each timestep as an independent input and must learn smoothness from data, whereas the polynomial and cosine bases encode it structurally. The paper presents this as a regularizing prior on the functional space rather than a difference in raw capacity.

Chebyshev leads, while DCT is the cheaper trade

The head-to-head ablation reports CART-weighted validation loss (lower is better) across 10 seeds with 95% confidence intervals. Chebyshev averaged 2.55, DCT 2.59 and Hypernet 3.77. Both structured bases beat the direct projection on every completed validation dataset. Chebyshev led DCT on four of five datasets and tied on WikiText; the average gap was 0.04 nats.

Parameter counts show the gap is not simply capacity. Hypernet and DCT carry 6.3M and 7.1M adapter parameters yet differ by 1.18 nats, while the extra 3.9M parameters separating DCT from Chebyshev buy only 0.04 nats. The authors conclude Hypernet does not overfit but lacks the smoothness prior. They also note DCT is cheaper at inference, making Chebyshev the accuracy ceiling and DCT the better parameter and latency trade. These are internal comparisons among MaRK variants, not a comparison against separately trained diffusion language models.

Modulating A matters more than modulating Δ

A leave-one-out ablation on WikiText isolates each of the five modulated parameters. Freezing A raises validation loss by 0.18 to 0.25 points across settings, and a variant modulating only B, C and Δ degrades by 0.34 to 0.51 points. The paper reads this as evidence that MaRK cannot be reduced to Mamba-style input selection: B and C control instantaneous throughput, while A governs how long memory persists.

The discretization parameter Δ shows the reverse pattern. Modulating it alone yields marginal gains, and the all-except-Δ configuration is second best for every kernel. The authors attribute this to redundancy: dynamics can be shifted directly through A or indirectly through Δ, since Ā = exp(A_cont · Δ). A-only outperformed Δ-only by up to 0.75 points. Still, every single-parameter configuration trailed the full five-parameter baseline by up to 0.50 points, so joint modulation remained best.

Timestep-conditioned kernels stay dissipative

The Markov diagnostics show all three adapters preserving a stable decay envelope while producing distinct timestep-conditioned sequences. The curves stay approximately geometric over thousands of lags, consistent with the stability construction, but they do not collapse onto the frozen backbone. Hypernet produces the broadest upward shift, especially at medium and long lags, suggesting greater memory persistence. Chebyshev yields a tightly ordered family of smooth decay curves. DCT redistributes energy most selectively: high-noise timesteps amplify early-lag response, while lower timesteps attenuate faster and cross the backbone at intermediate lags.

The authors interpret this separation as evidence that MaRK changes the impulse response, not merely a linear re-projection of features. That reading is supported by the Markov sequence's role as a coordinate-invariant description of the dynamics, but it remains the paper's interpretation of the measured norm curves rather than a direct causal test on downstream generation quality.

Latency overhead amortizes at long sequences

Inference measurements on an NVIDIA A100-SXM4-80GB at batch size 8 show MaRK adding a roughly constant per-layer cost of about 37 to 42 ms, independent of sequence length. At sequence length 4096, that amounts to 14% to 16% additional latency with memory overhead below 1% relative to frozen Hydra. At length 512, the same fixed cost dominates and produces 65% to 104% overhead.

The paper states that MaRK scales O(1) with sequence length and O(N) with diffusion sampling steps, aligning its asymptotic runtime with AdaLN Zero and input-injection baselines. The practical reading is conditional: the operator-level conditioning is cheapest where sequences are long, and relatively expensive for short-sequence inference. The authors also flag that systematic comparison against established conditioning frameworks on downstream benchmarks, and behavior under scaling laws, remain open.

Sources used in this answer