Why mesh segmentation struggles on real scans
Mesh-based methods recover articulated objects by partitioning a static mesh into functional parts and estimating the joint axes and connectivity that govern their motion. The paper identifies two deployment problems with this approach. Incomplete surfaces and fused components can undermine part segmentation, and partitioning alone cannot recover missing geometry or functional openings. Separately, reliance on task-specific training makes generalization to out-of-distribution objects difficult when their categories or mechanisms are poorly represented in the training data.
Agentic methods avoid the complete-mesh requirement by building objects from scratch as executable programs, using LLM knowledge of everyday objects to cover diverse categories. Without measurements of the real object's geometry, however, generated dimensions and part placement may not match the target instance, and such geometric differences can limit transfer of learned manipulation policies to the real object. USDCraft's stated challenge is to let reliable source geometry guide both program construction and revision while retaining flexibility to rebuild missing geometry and separate fused parts.
Turning a mesh into a metric description an LLM can read
Pretrained LLMs reason well over text but cannot read a mesh directly. Rendered views discard metric scale and hide interior structure, and raw vertex or point lists exceed a practical context budget. Source geometry analysis addresses this by converting the source mesh into a compact metric description the agent can read and query. It combines a global axial geometry encoding with targeted mesh inspection.
The axial encoding samples the surface support of the source mesh on a regular grid in a shared metric frame. The set of grid cells containing sampled source surface is serialized as an ordered sequence of axial slice maps along the vertical axis, together with local metric summaries of bounds, protrusions, openings, and observed interior fittings. Every cell outside that set is marked as unknown instead of empty or solid. This distinction matters for incomplete meshes: a hole in a scan does not imply free space, and the inside of a closed shell does not imply solid material, so the agent remains free to complete such regions from the image and the object's function. Automatically detected features serve as measurement anchors, and their functional interpretation is left to the agent.
Mesh inspection complements this limited-resolution overview on demand. Guided by the reference image and its current part plan, the agent renders the source from chosen viewpoints and queries local bounds, point distributions, and surface orientations within selected regions. Together the two forms of evidence let an LLM reason about dimensions, part placement, and interior structure without training on 3D mesh understanding.
Why the asset is a program rather than a mesh
USDCraft represents each asset as an executable program because a program exposes the quantities that later feedback needs to change. Parts, joints, and physical properties are defined by explicit parameters, so a discrepancy found during inspection can be addressed by a local edit instead of regenerating the asset. A program can also introduce parts and connections that the source never observed, which segmenting the given surface cannot do.
The authoring toolkit combines parametric solid modeling with CadQuery for dimension-controlled mechanical parts and signed distance field modeling with analytic primitives for organic shapes and smooth transitions. Appearance is authored in the same program with physically based materials, textures, and decals. Physical properties are authored directly as PhysX parameters — the quantities used by NVIDIA PhysX, the physics engine of Isaac Sim. The program sets masses, contact friction, joint limits, joint friction, and drives in the engine's own terms, which removes the conversion step needed when physical properties are predicted in a generic form. These values follow authoring guidance distilled from Isaac Sim test outcomes, which also keeps hand-operated mechanisms passive and checks articulated clearances along normal-use sequences. Properties that cannot be measured from the evidence are treated as modeling assumptions. Compiling the program derives collision geometry and exports a USD asset that loads into Isaac Sim with plausible physical behavior and without manual adjustment.
Separating missed source surface from open space
Source geometry analysis informs the initial construction, but errors introduced while writing the program must also be found and corrected. Iterative geometric rechecking expresses this feedback in the same representation as the source evidence. At each revision step, the current candidate is re-encoded in its reference configuration on the source grid. Comparing the source cell set and the candidate cell set returns a coverage ratio plus two discrepancy sets: cells containing observed source surface that the candidate misses, and cells where the candidate has surface but the source is unknown.
The paper treats these two sets differently. Missed source surface is direct evidence of an error and serves as a correction signal. Candidate surface in unknown regions may be an error, but it may also be a necessary completion such as inner walls, image-supported parts, or connections absent from an incomplete scan, so the agent resolves it with the image and the object's function instead of penalizing both uniformly.
Coverage alone cannot tell whether a part is misplaced or wrongly shaped, and the two cases call for different edits. For finer comparison, first-surface maps from six axis-aligned directions measure face offsets, silhouette differences, and the placement of openings and protrusions. The agent can also select a source region and a candidate part and compare their first-surface depths along a chosen viewing direction, without ground-truth part segmentation. The median offset measures the front–back shift along the viewing direction, and the 90th percentile of the residual after removing that offset measures shape difference. For the handle in the overview, the median offset is only 0.64 mm while the profile difference is 3.41 mm, so its profile rather than its placement needs revision; a large offset with a small profile difference would instead indicate a misplaced part. After each revision, the agent recompiles the program and rechecks the affected geometry.
Visual feedback catches what surface comparison cannot
Geometric diagnostics compare surfaces and cannot judge whether an asset looks and works like the intended object. Visual feedback fills this gap by rendering the candidate with its materials and with per-part colors from viewpoints selected by the agent, at the default or specified joint configurations. The agent compares these images with the reference image or requested description to find missing components, implausible assemblies, incorrect appearance, and clearance problems that appear only when parts move, and revises the program accordingly. In generation, where no source mesh is available, visual feedback provides the main check on the candidate.
The same program representation supports generation from text or images, and every compiled asset carries collision geometry, masses, contact parameters, and joint limits for direct use in Isaac Sim. The paper reports leading articulation recovery on two benchmarks and validates USDCraft's effectiveness for real-to-sim-to-real robot manipulation.
Backbone choice matters less than geometric grounding
The paper's discussion states that grounding improves geometry and articulation over segmentation-based and image-only programmatic methods, and that this grounding matters more than the choice of backbone. The reported comparison supports the second claim with a wide spread in cost and quality across backbones. On the generation comparison, USDCraft with Sol records a quality score of 0.9704 at 16.5 minutes, while USDCraft with Claude records 4.5115 at 101.1 minutes and USDCraft with Gemini records 0.8643 at 80.5 minutes. The Mini Workflow baseline with Sol scores 0.6878 at 18.9 minutes, and with Astra 0.4619 at 5.7 minutes.
These are generation-comparison scores averaged over anonymous evaluations by Sol/high, on a quality scale out of 5, with time in minutes; they are not the articulation-recovery benchmark numbers. The practical reading is that the same grounding procedure produces usable assets across backbones at very different cost, while the ungrounded Mini Workflow stays well below USDCraft on quality even when it runs faster.
Where transferred policies hold and where they fail
For the manipulation study, the robot is a UR7e manipulator with a Robotiq 2F-85 gripper and a single external camera. The policy observes RGB images and outputs absolute end-effector poses, and the robot, table, and camera are calibrated to align simulated and real workspaces. For each method–task setting that supports the required interaction, cuMotion automatically collects 200 demonstrations entirely in simulation, and a separate Diffusion Policy is trained once per setting for 80 epochs with batch size 128 and deployed directly on the real robot, without real-world training data or fine-tuning. Each trained policy is evaluated in 20 simulation trials and 20 real-world trials with randomized object positions and initial robot configurations. Success requires opening the drawer by at least 5 cm, pressing the toaster lever down by at least 4 cm, or rotating the toaster switch by more than 5 degrees; rollouts stopped because of dangerous contact count as failures.
Baseline preparation differs by method and limits what the comparison means. USDCraft and Mini Workflow receive image and mesh evidence, Particulate receives the mesh, and Articraft receives an image with its generated asset scaled using the real-scan dimensions. Particulate recovers the toaster-lever joint but cannot support drawer opening or toaster switching, so no demonstrations are collected for those two settings and their entries are recorded as zero success due to asset reconstruction failure. Particulate's reconstructed bodies cannot stand stably, so they are fixed in Isaac Sim, and the reported 50% simulation success for lever pressing uses this fixed-base condition. The executed Mini Workflow–Astra drawer rollouts are reported as 0% real-world success under the dangerous-contact rule.
Dimension preservation matters beyond the manipulated object. The toaster is supported by the drawer for lever pressing and by the box for switching, so support height also determines the toaster's lever and switch positions relative to the robot. The paper also reports limits on what the representation covers: fine lattice structures such as racket strings and shopping-cart wire frames remain difficult to reconstruct faithfully, with a shopping cart retaining its overall form while wire spacing and local connections differ from the source. USDCraft can annotate flexible members and construct their rest geometry, but this does not establish how they bend, stretch, or respond to contact; for the umbrella, rigid joints on the ribs and stretchers do not model canopy tension or rigid–flexible coupling, which the authors state they have not validated in Isaac Sim.
What decides whether a reconstructed asset transfers
The paper's own priority is that measured source geometry, not the choice of LLM backbone, is what makes a reconstructed asset usable for real-to-sim transfer. That priority follows from two comparisons in the evidence: USDCraft's quality scores stay high across Sol, Astra, Claude, and Gemini backbones despite large differences in time, while the ungrounded Mini Workflow falls well below on the same scale; and the manipulation study trains policies only in simulation and deploys them on the real robot without real-world data or fine-tuning, so any transfer depends on the asset matching the physical object's dimensions and articulation.
The scenario split that matters most is therefore between objects whose geometry the source evidence can constrain and objects it cannot. When the mesh and image pin down dimensions, part placement, and openings, the rechecking loop can convert discrepancies into local program edits and the compiled asset can carry the physical properties Isaac Sim needs. When the object is a fine lattice or depends on flexible deformation, the same loop has no measured basis for the missing behavior, and the paper reports those cases as unresolved rather than validated.
Sources used in this answer
USDCraft: Geometrically Grounded Programmatic Modeling of Articulated 3D Assets for Simulation
A pretrained LLM writes and revises executable asset programs grounded in a metric mesh description and iterative geometric rechecking, producing articulated USD assets that load into Isaac Sim without manual adjustment and support simulation-trained policies transferred to real objects.