Can a deterministic pipeline caption wearable health time-series data at scale?

Why captions must be regenerable, not just plentiful

Time-series language models learn to describe and reason over sensor signals, but they need paired examples: a stretch of signal plus text that says what is happening in it. The paper argues that the field is blocked less by model architecture than by the supply of such pairs. The largest paired corpus for multimodal consumer wearables covers 59.7 million hours from 103,731 participants but is closed; the largest open alternative is roughly four orders of magnitude smaller in recorded hours and limited to smartphone- and smartwatch-derived modalities, with 5,648 question–answer pairs over 60 subjects.

Using a large language model to write captions could close that scale gap, but the paper identifies a different cost. Language-model outputs cannot be regenerated exactly, so nobody else can re-derive, extend, or audit the resulting corpus. The authors also note a subtler problem: if the same kind of model is later evaluated on captions produced by a language model, part of the evaluation becomes distillation of the captioning model rather than a test of reading the signal.

TimeCap's answer is to remove the language model from the annotation loop entirely. Captions are produced by fixed rules and thresholds applied to the signal, so anyone with access to the signals can regenerate the same captions. That is what the word deterministic means here: the same input and the same channel definitions yield the same text, which makes the corpus traceable back to a rule that can be inspected and changed.

Three extractor tiers plus a cross-channel layer

The pipeline is built from composable extractor tiers that each look at the signal in a different way. Statistical extractors summarize level and variability within an interval; structural extractors detect events and segments; semantic extractors attach meaning drawn from the dataset's own channel definitions and labels. A cross-channel synthesis layer then combines evidence across channels, so a caption can relate, for example, heart rate to electrodermal activity in the same interval rather than describing each channel in isolation.

Extensibility comes from a channel-schema abstraction: adapting TimeCap to a new wearable dataset requires writing channel definitions, not redesigning the annotation pipeline. The paper demonstrates this by applying the same tiers to a second, very different dataset. WESADCap covers 20 chest- and wrist-worn physiological channels, including accelerometry, electrodermal activity, temperature, respiration, blood-volume pulse, electromyography, and heart rate, with binary labels for baseline, stress, and amusement conditions and the meditation phases, instantiated at the session level.

The trade-off is stated plainly in the paper's limitations. Captions come from fixed template banks with 20 variants per caption type, which limits their linguistic diversity. The authors also note that they did not test two adjacent baselines: language-model-generated captions of the same signals, and frontier vision-language models prompted on rendered signal-plot images. Those comparisons would quantify the determinism-versus-lexical-diversity trade-off the design deliberately makes, but they remain untested here.

What MHCap actually contains

Applying TimeCap to OpenMHC, the MyHeart Counts wearable dataset, produced MHCap, described as the largest open caption layer for multimodal wearable time-series data. It spans 19 Apple HealthKit channels covering iPhone activity, Apple Watch physiology, sleep stages, and ten workout categories, and totals 7,005,763 captions over 908,944 person-days from 2,081 participants.

Those units matter for reading later results. A person-day is a day of data from one participant, not a participant; the caption count is a count of generated text units, not of independent clinical observations. The corpus inherits OpenMHC's data-quality filters: a period counts as non-wear when all seven continuous channels are zero or missing for more than 30 consecutive minutes, and a day is retained only if it has at least 12 hours of estimated wear time and passes per-channel variance thresholds that remove near-constant sensor artifacts.

The cohort behind these numbers is skewed toward white, male participants from the US with a median age of 39 years. The authors state that models pretrained on the corpus may not generalize uniformly to underrepresented populations, which is a constraint on the corpus itself rather than on the captioning method.

Captions beat labels on most binary tests

To test whether the corpus is useful, the authors trained OpenTSLM, a time-series language model, on MHCap and benchmarked health-outcome prediction. The comparison is between OpenTSLM-Cap, trained on captions plus labels, and an architecturally identical label-only baseline. On per-user AUROC and AUPRC, caption-augmented training outperformed the baseline on 29 of 32 binary metric-scale tasks across both the 1B and 3B parameter models.

The headline count needs its denominators kept in view. The 32 evaluations are per-user binary metric-scale tasks: eight tasks, each scored with two metrics, at two model scales. The metrics are computed by aggregating each user's weekly binary predictions to a mean and then ranking across users, so they describe how well users are ordered relative to one another, not whether an individual prediction is correct.

The paper's own summary of the pattern is that captions add a signal that persists through fine-tuning and helps on every binary task, whether or not the model saw that task during training. That last clause is the more interesting claim, and it is where the per-week results become relevant.

The largest gains are on unseen clinical tasks

Per-week macro-F1 shows where the advantage concentrates. At the 3B scale, held-out clinical conditions show clear caption gains: Cardiovascular Disease (+3.4), Diabetes (+6.1), Coronary Artery Disease (+2.6), and Atrial Fibrillation (+2.6). These are tasks marked as held out, meaning the model did not see them during fine-tuning. In-distribution ordinal tasks also improve at 3B, with QWK up by +4.8 on BMI and +5.0 on Happiness.

The ablation adds a second layer. Training the 1B OpenTSLM-Cap on each caption family alone raised mean per-user AUROC over the label-only baseline by 1.9 to 3.2 points, more than the 1.3 points separating the families from each other. No single family won everywhere, but only the full mix beat the baseline on all eight binary tasks, and cross-channel captions, the tier TimeCap adds, matched structural ones on average.

The same ablation also marks the limit. Labels alone stayed ahead on the ordinal targets, and per-week differences were small and mixed. Caption supervision is not a uniform upgrade; its benefit is concentrated in binary classification, especially on clinical labels the model was not tuned for.

Where the caption advantage stops

The paper reports additional zero-shot experiments, not included in its main results section, in which ordinal transfer to held-out blood-pressure and sleep-duration categories failed across all configurations. Parse rates collapsed and predictions defaulted to the majority class. The authors read this as evidence that caption supervision alone is not yet sufficient for clinical labels with strong class imbalance.

A second boundary is the level of prediction. Per-user AUROC between 59.4 and 70.5 on the clinical tasks (Hypertension, Sleep Diagnosis, and the four held-out conditions) indicates population-level patterns rather than reliable individual predictions. The authors state that any individual-level use would require independent prospective validation and safeguards against misuse, for example by insurers or employers.

A third is measurement reliability on ordinal tasks. The paper notes that Spearman and QWK reliability scales with the spread of both predicted and ground-truth scores, so heavily concentrated tasks have lower effective signal. Happiness, for instance, has 75% of its 328 users in the High or Very High levels, while BMI's 557 users are spread across five levels. Low ordinal scores on concentrated tasks partly reflect that distribution, not only model quality.

What to take from a reproducible corpus

The paper's distinctive claim is not that captions help a model. It is that a corpus can be built so that its captions are regenerable, auditable, and independent of the models evaluated on it, and that such a corpus is large enough to train on. The 29-of-32 result is the demonstration; the deterministic pipeline is the contribution the authors expect others to reuse.

For a reader deciding whether this matters, the useful split is by task type and by prediction level. The caption advantage is strongest for binary clinical outcomes the model never saw during fine-tuning at the 3B scale, present but smaller for in-distribution binary tasks, and absent or negative for ordinal targets where labels alone stayed ahead. The per-user AUROC range on clinical tasks places the results at the population level, not the individual level.

The authors also flag a concrete next step that follows from the corpus design: a per-channel data-availability gate before structural detection, which would prevent missing or zero-filled periods from being captioned as events. That is a reminder that deterministic captions are only as good as the rules that define what counts as an event, and those rules are exactly what the pipeline makes visible and changeable.

Sources used in this answer