How can objective performance indicators score robotic inguinal hernia repair skills?

A 0–100 score built from console motion

The score is not a rating given by an expert watching a video. It is computed from objective performance indicators (OPIs): mathematically derived metrics captured by the Intuitive Data Recorder during surgery, such as instrument path length, movement velocity and workspace utilization. The study used all 19 console-specific OPIs supplied for robotic transabdominal preperitoneal (rTAPP) inguinal hernia repair, without adding new engineered features.

Those 19 OPIs were grouped into eight domains, including time efficiency, economy of motion, movement speed, workspace utilization, camera control, hand repositioning, console attention and energy application. Principal component analysis condensed each domain's OPIs into one signal, retaining on average 79.0% (range 56.3–100.0%) of the original information across the 32 task-domain combinations. Ridge logistic regression then combined the domains into a task-specific 0–100 composite, calibrated so that attending surgeons averaged about 60.

The practical meaning is that the output is a probability-like discrimination score, not a pass/fail certificate. A composite of 50 corresponds to a predicted probability of 0.5 that the operator is an attending, and the paper reports accuracy at that prespecified boundary.

Four tasks, four separate scoring models

Each rTAPP repair was divided into four standardized tasks performed sequentially, and on both sides in bilateral cases: peritoneal incision (PI), flap exploration (FE), mesh placement (MP) and closure of peritoneum (CP). Two optional tasks, hemostasis and adhesiolysis with bowel sweep, were excluded because data were limited.

A separate model was fitted for each task rather than one global score for the whole operation. This matters because the domains did not contribute equally. Movement speed carried the largest weight in FE (54.9%), CP (42.2%) and MP (40.1%), while PI drew more evenly on time efficiency (21.4%), movement speed (15.9%), energy application (13.8%) and camera control (13.7%).

Before modelling, cases with too little active console movement were removed using task-specific engagement thresholds: 50% console time for PI, 20% for FE and MP, and 40% for CP. Attending cases with task duration above the 90th percentile of the attending distribution were also excluded so atypical cases would not distort the attending reference.

Moderate discrimination that generalizes across surgeons

In the development data, 4523 task-level observations came from 47 surgeons (6 attendings and 41 trainees); after preprocessing, 2588 observations (57.2%) from 392 unique cases remained, contributed by 6 attendings and 29 trainees, 12 of whom appeared at more than one training level.

Resubstitution AUC ranged from 0.71 for CP to 0.78 for FE, with large standardized differences between attending and trainee composite scores (Cohen d 0.79–1.33). The more honest test is cross-validation, where surgeons in the evaluation folds were not seen in training: those AUCs fell to 0.62 for CP and 0.74 for PI, with an overfitting gap of 0.019 to 0.092. The composite also beat the best single-domain AUC for every task, which is the paper's evidence that combining domains adds discrimination beyond any one kinematic dimension.

Read the two AUC families separately. The resubstitution values describe fit to the surgeons the model learned from; the cross-validated values describe performance on unseen surgeons and are the appropriate basis for judging whether the score would work elsewhere.

Scores rise from PGY-2 to attending

Composite scores increased significantly across training levels for all four tasks, with PGY-1 excluded for insufficient sample size. The rank correlation was 1.000 for FE, MP and CP (P < 0.01) and 0.943 for PI (P = 0.005).

The size of the step differed by task. FE showed the steepest slope at 10.3 points per postgraduate year. The fellow-to-attending gap was small for FE, MP and CP (2.5–4.8 points) but remained substantial for PI at 15.8 points, suggesting PI keeps separating surgeons later in training than the other three tasks.

This progression is known-groups validity evidence: scores behave as expected if they track experience. It is not proof that the score measures operative judgment, and the authors treat the pattern as support for an ordinal skill construct rather than a finished competency verdict.

External AUC of 0.84 hides uneven task performance

The external test used 21 rTAPP repairs recorded at Washington University in St. Louis with a dual-console configuration, yielding 133 task-level observations; after the training engagement filter, 84 observations from 21 cases were analysed. All model parameters were frozen, with no refitting or recalibration.

The pooled overall AUC was 0.84 (95% CI, 0.69–0.93), with confidence intervals from cluster bootstrap resampling at the surgeon level. That headline number is encouraging, but the task breakdown is the more decision-relevant result: discrimination was strong for PI and FE, with FE reaching an external AUC of 0.91 (95% CI, 0.78–1.00), and near chance for MP and CP.

The authors offer two explanations: the model may be sensitive to institutional differences in technique, or those tasks may need a larger validation sample. Either way, a reader should not treat the pooled 0.84 as evidence that all four task scores transfer equally.

Movement speed drives much of the score

Movement speed was the dominant discriminating domain, contributing 40.1–54.9% of composite weight in three of four tasks and showing significant progression across all training levels (ρ = 0.94–1.00, all P ≤ 0.005). LASSO regression independently selected it as the sole retained predictor in FE and CP.

The authors interpret this as attendings committing to purposeful instrument trajectories while trainees move tentatively, and they note the pattern matches prior work in other robotic procedures. Camera control and hand repositioning separated the groups weakly (Cohen d ≤ 0.33), which the authors read as basic console skills that trainees acquire early.

That interpretation is the authors', not a measured causal finding. The paper itself flags the consequence: because one construct carries so much weight, the composite is disproportionately influenced by movement speed, and the other domains contribute less to the overall number.

A binary model inside a continuous skill gradient

The ridge model was trained on a binary attending-versus-trainee label. That collapses a continuous expertise gradient into two classes and pools attendings with 4 to 18 years of experience into one reference group, so the framework cannot distinguish an early-career attending from a senior attending.

Other limits sit close to the results they qualify. Only six attendings were available, so PCA and attending-referenced calibration were performed on the full dataset rather than within each cross-validation fold. The model assumes domains combine linearly and does not account for institutional differences in technique or case complexity; defect size, BMI and complications were not incorporated.

The comparison with GEARS is also a boundary rather than a validation. Across 139 rated tasks, the composite and GEARS showed no association, with a weak non-significant correlation (ρ = 0.136, p = 0.111). The authors suggest the two instruments measure different constructs and may be complementary, but the sample was underpowered for domain-level GEARS comparisons.

Flap exploration is the score's strongest use case

If the question is which task score to trust most, the paper's own ordering is clear. FE combined the strongest within-cohort discrimination (cross-validated AUC 0.74, resubstitution 0.78), the steepest training-level slope (10.3 points per PGY), the largest domain weight on movement speed (54.9%) and the best external discrimination (AUC 0.91, 95% CI 0.78–1.00). PI followed with cross-validated AUC 0.74 and a persistent 15.8-point fellow-to-attending gap.

MP and CP are the weaker links. Their cross-validated AUCs were 0.70 and 0.62, and their external discrimination was near chance, so a low MP or CP score should not yet be read as a reliable statement about a surgeon's skill.

The most defensible use of this framework is therefore targeted rather than global: domain-level feedback on movement speed and economy of motion during demanding dissection, with FE and PI scores carrying the most weight, while MP and CP await larger external validation before they inform competency decisions.

An automated signal, not a replacement for expert judgment

The paper positions the score against the assessment burden in competency-based surgical education: entrustable professional activity assessments have shown low completion rates, and reliable ratings reportedly require 7–10 micro-assessments per EPA. Kinematic scoring needs no additional hardware or observer time, which is the practical advantage the authors emphasize.

The domain structure is what makes the output usable for feedback rather than just ranking. A trainee consistently scoring below peers on movement speed across multiple cases gives a specific target, whereas a single composite number does not. The authors describe robotic skill as developing along two paths: foundational console handling that reaches competence early, and operative judgment reflected in movement speed and economy of motion that keeps refining with clinical experience.

That framing is the paper's interpretation of its domain weights, and it should be read as such. What the evidence supports is narrower and more useful: an objective, automatically computed score that discriminates attending from trainee surgeons well in FE and PI, moderately overall, and poorly in MP and CP.

Sources used in this answer