CITRAS: Mastering External Factors in Time Series with Shifting Attention

TL;DR

In the real world, time series never exist in a vacuum; they are influenced by holidays, weather, and price schedules—external factors known as covariates. CITRAS is a new decoder-only Transformer that elegantly solves the "length mismatch" problem of future known covariates using a simple KV Shift and stabilizes noisy inter-variable relationships via Attention Score Smoothing. It doesn't just predict the next value; it understands the contextual influence of the future.

Problem & Motivation: The Heterogeneity Gap

Most time series models are either Channel Independent (CI) (univariate, ignoring relations) or Channel Dependent (CD) (multivariate, often drowning in noise). When we introduce covariates, we face a critical split:

1. Observed Covariates: Historical data only (e.g., yesterday's humidity).
2. Known Covariates: Data available in the future (e.g., next Monday is a holiday).

Current Transformers struggle here. If you use a standard attention mechanism, how do you let a target variable at "Time T" look at a holiday indicator at "Time T+1"? Most models resort to "late fusion" (shoving it in at the end) or complex RNN-hybrid architectures that break the efficient, autoregressive nature of modern Transformers.

Methodology: The Core Mechanisms

CITRAS preserves the pure Transformer decoder structure but reinvents how Cross-Variate Attention works.

1. KV Shift: Solving the Future Alignment

To exploit future known information without introducing temporal leakage, CITRAS uses KV Shift.

• The Logic: At a specific patch step, the Query comes from the target variable. The Key comes from the known covariate at the current step. However, the Value is shifted—it comes from the known covariate at the next step.
• The Intuition: The model learns: "If the holiday indicator today (Key) matches the pattern I'm looking for (Query), then incorporate the information from tomorrow's schedule (Value)."

2. Attention Score Smoothing (ASS)

Patch-level attention is great for fine-grained details but terrible for global trends because local temporal disturbances (noise) can distract the attention mechanism. CITRAS addresses this by applying an Exponential Moving Average (EMA) to the attention scores across time steps. This "denoises" the inter-variable relationship, allowing the model to focus on stable correlations while still being able to adapt to gradual shifts.

Experiments & Results

CITRAS was tested against a massive battery of 14 baselines, including TiDE, iTransformer, and PatchTST.

Performance in Covariate-Informed Forecasting

On the Electricity Price Forecasting (EPF) and Bike Sharing (BS) datasets, CITRAS consistently outperformed MLP-based models (TiDE) and other Transformers (TimeXer). As shown in the visualization below, CITRAS accurately captures the "holiday effect" on electricity demand, shifting the forecast lower when the known covariate signal is detected.

Efficiency and Scalability

Unlike iTransformer, which has a complexity of $O(C^2)$ based on the number of variables $C$, CITRAS manages a cross-variate complexity of $O(CN)$. This makes it significantly more scalable for datasets with hundreds of variables, such as Traffic or ECL.

Critical Analysis & Conclusion

Takeaway

The success of CITRAS lies in its simplicity. By modifying the internal mechanics of the attention block (KV Shift and ASS) rather than adding complex external fusion layers, the authors maintained the expressive power of the decoder-only Transformer while giving it the "eyes" to see future exogenous context.

Limitations & Future Work

• Categorical Encoding: Currently, the model treats categorical events (like "Sunday") and continuous ones (like "Temperature") with the same embedding logic. Future iterations could benefit from specialized categorical encoders.
• Supervised Nature: Like most current SOTA forecasters, it requires dataset-specific training. The next frontier for CITRAS is likely Large-scale Pre-training to create a Foundation Model that understands covariates across domains.

CITRAS is a robust step forward for industrial forecasting where "knowing the schedule" is as important as "knowing the past."