Can training dynamics detect noisy keypoint annotations in infant pose estimation?

Why noisy keypoints matter more in infant motion analysis

Markerless pose estimation is increasingly used to derive clinically relevant motion biomarkers from video of preterm infants, whose spontaneous movement patterns are treated as early markers of neurodevelopmental trajectories. Training those models requires large sets of manually annotated keypoints, and annotation is error-prone: occlusions, poor lighting, and motion blur create ambiguous cases.

The paper argues the clinical setting changes the stakes. In generic pose estimation, annotation errors typically cause only modest reductions in aggregate localization metrics such as Average Precision or Percentage of Correct Keypoints. In infant movement assessment, a mislocalized keypoint introduces an artificial displacement into the reconstructed trajectory of the corresponding joint, so the error propagates as an artifact into the motion descriptors used for assessment.

Reformulating noisy-keypoint detection as anatomical classification

The framework has two stages. In the first, a hybrid convolutional-attention keypoint classifier takes a single-infant RGB image and its M keypoints (12 retained from the COCO 17-keypoint format: left/right shoulder, elbow, wrist, hip, knee, ankle) and predicts, for each keypoint, its anatomical class among K = 6 joint categories and its lateral position (left/right).

Each keypoint is represented by combining an appearance encoding, sampled from a pre-trained EfficientNet feature map via bilinear interpolation at the keypoint location, with a sinusoidal positional encoding of its coordinates projected through a two-layer MLP. A cross-attention module then lets each keypoint encoding gather information from the whole feature map, and two separate linear heads predict anatomical class and side, trained with a weighted cross-entropy loss.

The key intuition is that an incorrect keypoint sits at an erroneous spatial location, which yields misleading local visual features, creating a discrepancy between the input and its label that shows up in the loss.

Loss trajectories separate clean from noisy keypoints

In the second stage, the joint loss and side loss for every keypoint in every image are recorded at each epoch, forming loss-trajectory vectors that are smoothed with a moving average filter. Two features are extracted per keypoint: a normalized peak-to-trough loss variation, and a temporal feature measuring when the minimum loss is reached relative to the maximum. Both are computed from the joint loss and the side loss, giving a four-dimensional descriptor per keypoint.

These descriptors are partitioned with vanilla K-means clustering. Because feature values empirically decrease as annotation noise increases, the cluster with the lower centroid is identified as containing noisy keypoints.

The paper introduces a Separability Metric (SeM) to quantify how well clean and noisy loss trajectories separate during training: it models each group's loss distribution per epoch as a Gaussian, computes the overlap area between the two distributions, and averages across epochs. Lower values mean better separation.

Cross-entropy dynamics beat MSE regression dynamics

The comparison baseline follows the same sample-selection paradigm but derives training dynamics from the MSE regression loss of an HRNet model rather than a classification loss, with the same K-means clustering applied to its descriptors. This isolates the effect of loss type from the clustering stage.

On NeoPose, the CE-based approach substantially and consistently outperforms the MSE-based baseline across all configurations of both the random-noise and swapping-noise scenarios, achieving an F1-score of up to 91.9% in noisy-keypoint detection. The paper attributes this to the logarithmic penalty of misclassification under cross-entropy versus the quadratic penalty under MSE: MSE may suffer gradient attenuation for small perturbations around the true keypoint location, producing less discriminative training dynamics.

The SeM values support this. Under random noise at 10%, 20%, 30%, and 40%, the MSE-based trajectories score 0.4524, 0.5385, 0.5843, and 0.6199, while the CE-based trajectories score 0.2413, 0.2475, 0.2615, and 0.2770 — consistently lower, meaning better separated.

Random perturbation and left-right swapping are not equally hard

The two noise scenarios behave differently. Under random positional perturbation, performance stays comparatively high and stable even at elevated corruption levels. The paper interprets this as a structural property of NeoPose: frames are sampled from continuous video of infants lying in a constrained supine position, so the dataset has limited pose variability and strong spatial regularities among keypoints. Small localized coordinate perturbations tend to remain within the same semantic neighborhood as the true keypoint, only marginally disrupting the visual-spatial consistency the classifier exploits.

That property does not extend to left-right swapping. Exchanging left and right labels alters the semantic structure of the supervision signal rather than introducing a local geometric perturbation. As the proportion of swapped keypoints grows, the model faces increasingly conflicting supervision for symmetric body parts, and the problem becomes progressively more degenerate as left/right symmetry is approached — explaining the steeper performance decay for swapping noise at higher corruption levels.

Generalization to COCO comes with a counterintuitive trend

Repeating the random-noise evaluation on a 14K-image subset of COCO, absolute detection performance drops relative to NeoPose for both methods, which the paper attributes to COCO's substantially higher visual and postural heterogeneity — multiple subjects, varied viewpoints, clothing, occlusion patterns, and backgrounds — compared with the constrained infant-in-crib scenario.

Three results still support robustness. First, the CE-based pipeline consistently and substantially outperforms the MSE-based baseline at all tested noise levels: F1 rises from 0.387 to 0.510 at 10% noise, 0.560 to 0.670 at 20%, 0.628 to 0.733 at 30%, and 0.661 to 0.784 at 40%.

Second, detection performance improves as the noise level increases rather than degrading. The paper hypothesizes a class-balance effect intrinsic to unsupervised K-means: the algorithm receives no prior about the relative size of the clean and noisy partitions and is implicitly biased toward producing clusters of comparable size. At low noise, clean keypoints vastly outnumber noisy ones and this bias works against correct partitioning; as the classes become more balanced, clustering more closely matches the true partition.

Third, a per-keypoint breakdown reveals statistically significant heterogeneity in detection difficulty across anatomical landmarks (Friedman test, p<0.05). Post-hoc pairwise comparisons with FDR correction identify the left and right ankle as significantly harder to classify as clean or noisy than all other keypoints, consistent with reports that distal joints are disproportionately affected by localization ambiguity, motion blur, and occlusion.

Filtering detected noisy keypoints helps only above 10% noise

Detection quality is not the same question as downstream benefit, so the paper trains HRNet on each of four COCO random-noise configurations, both on the full corrupted training set and on the set after removing keypoints flagged as noisy by the CE-based pipeline, evaluating both on a held-out clean COCO test set with COCO AP.

The effect is noise-level dependent. At 10% noise, training on the full unfiltered set yields higher COCO AP (0.633) than training on the CE-filtered subset (0.601). At 20%, 30%, and 40% noise, filtering consistently improves downstream performance, raising COCO AP from 0.544 to 0.618, from 0.542 to 0.599, and from 0.522 to 0.573 respectively.

The paper links this to the same class-imbalance bias: at 10% noise the detector reaches its lowest F1 among the four configurations, so more correctly annotated keypoints are erroneously discarded as noisy, and the loss of legitimate clean training signal outweighs the benefit of removing the few truly noisy keypoints. As noise rises to 20–40%, detection F1 improves, clustering more closely approximates the true partition, and the benefit of removing a larger, more correctly identified corrupted set dominates.

Where the framework stops and what to watch

The K-means clustering stage carries a class-balance bias that penalizes detection precision at low noise levels, and the noise scenarios considered, while clinically motivated, do not cover all sources of annotation error encountered in practice, such as those arising from occlusions or ambiguous poses. The paper states that future work will explore clustering strategies explicitly robust to class imbalance, extend the noise taxonomy to additional real-world annotation error types, and investigate integrating the detection framework as an online data-curation step within the pose-estimation training loop.

For readers deciding whether to apply this, the paper's own evidence points to a threshold rather than a blanket rule: on COCO, filtering CE-detected noisy keypoints paid off at 20%, 30%, and 40% random noise but hurt at 10%, where the detector's F1 was lowest. The most decision-relevant comparison is therefore not CE versus MSE detection alone, but whether the estimated noise level is high enough that the clustering partition is trustworthy — and the paper's own explanation is that this depends on how balanced the clean and noisy classes are, not on the detector's architecture.

Sources used in this answer