ACTS · test-set results · nano_action_joint_v8
ACTS: Learning Visuo-Tactile World Models from Human Interaction
Motivation
- The problem. Action-conditioned world models need diverse interaction data, but for contact-rich manipulation, collecting it with robots is costly: teleoperation limits throughput, and a fixed robot or gripper restricts which contact geometries and force-modulation behaviors can be explored.
- The question. Can these dynamics be learned directly from human physical interaction, while remaining usable for robot control?
- ACTS. A visuo-tactile world model trained entirely from human interaction.
- Data collection. Human operators manipulate handheld tactile sensors while we record synchronized third-person and wrist video, bilateral GelSight observations, 6-DoF sensor motion, and tactile-derived force signals — yielding React, a dataset of human interactions spanning four contact-rich tasks.
- Bridging human data and robot execution. Actions are represented at the tactile sensing interface: visual prediction is conditioned on sensor geometry in the visual frame, while tactile prediction uses force-augmented sensor motion in the local sensor frame. ACTS jointly predicts future visual and tactile observations under candidate actions.
- The result. Despite being trained without robot interaction data, ACTS can be used directly for real-robot planning without robot-specific world-model fine-tuning — human physical interaction can provide scalable action-conditioned supervision for visuo-tactile world models that transfer to robotic manipulation.
This page: a Cosmos3-Nano world model that, given the recorded context and the executed sensor actions, predicts what the main camera, both wrist cameras and both GelSight gels see next — evaluated on the held-out test split of mixed_mb_wrist_holdout.
checkpointepoch_004
test episodes5
test windows16,598
short horizon8 frames · 0.53 s
long horizon32 rollouts · 8 s
trainable45.3 M / 15.2 B
Dataset
Two React recordings per task — what the sensors see
Each card plays the five streams the model sees, at the model's own 15 fps, with the recorded normal force for the whole recording underneath. Both hands carry a GelSight sensor, so tactile and wrist views come in left/right pairs. The model on this page is trained on motherboard only. Videos are muted and paused by default — press play on any row.
motherboard2026-09-11 · episode_004_seg00
main view
wrist L
wrist R
tactile L
tactile R
motherboard2026-09-12 · episode_002
main view
wrist L
wrist R
tactile L
tactile R
pushT2026-09-11 · episode_002_seg00
main view
wrist L
wrist R
tactile L
tactile R
pushT2026-09-12 · episode_006_seg00
main view
wrist L
wrist R
tactile L
tactile R
rope2026-09-14 · episode_005_seg00
main view
wrist L
wrist R
tactile L
tactile R
rope2026-09-11 · episode_000_seg00
main view
wrist L
wrist R
tactile L
tactile R
Method
Nine streams, one frozen Nano generator, LoRA
Nine streams — five images, four actions — are projected into one sequence and denoised jointly by a frozen Cosmos3-Nano generator through LoRA. Actions are always clean conditions, future included; only future image latents are supervised, so the model never predicts an action.
window16 @ 15 Hz
tokens / example1,342
LoRAr16 · 252 linears
training5 ep · 4×H100
datav8 · 09-17
Evaluation setup
Test split, contact classes, how to read the strips
Test split5 held-out episodes, 16,598 windows. Three of them are the 09-09 session (different hardware day), two are in-distribution 09-11/09-12 segments.
Contact classesFrom the recorded force, max(FL, FR) > 0.5 N over the 16 frames. none full onset release
pools: none 1,102, full 8,361, onset 1,407, release 1,333, mixed 4,395 (not evaluated). 25 windows per class, fixed seed; 32 also run the 8 s rollout (8 shown as video below).
pools: none 1,102, full 8,361, onset 1,407, release 1,333, mixed 4,395 (not evaluated). 25 windows per class, fixed seed; 32 also run the 8 s rollout (8 shown as video below).
Reading a clipFive streams side by side, GT on top and the prediction below, playing through the 8 history frames and the 8 predicted ones (~1 s). The badge marks history vs. predicted; underneath, a moving playhead tracks the recorded force and the commanded press along the gel normal.
- none
- no contact in any of the 16 frames
- full
- in contact for at least 80% of the 16 frames
- onset
- no contact at the end of the context, contact at the end of the prediction
- release
- contact at the start of the context, no contact at the end of the prediction
Short horizon
8 predicted frames, teacher-forced
Teacher-forced: the 8 history frames are the recorded ones, so this measures one step of prediction rather than accumulated drift. Δ is model minus the copy-last floor.
PSNR (dB)
| class | model | n | view | wrist L | wrist R | tactile L | tactile R | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| model | floor | Δ | model | floor | Δ | model | floor | Δ | model | floor | Δ | model | floor | Δ | |||
| none | action_full epoch_004 | 25 | 28.96 | 26.73 | +2.24 | 24.18 | 22.28 | +1.90 | 25.91 | 23.71 | +2.20 | 49.54 | 43.55 | +5.99 | 49.79 | 43.55 | +6.24 |
| full | action_full epoch_004 | 25 | 26.22 | 22.99 | +3.23 | 17.01 | 15.23 | +1.78 | 17.93 | 16.59 | +1.34 | 38.37 | 36.42 | +1.95 | 44.14 | 39.65 | +4.49 |
| onset | action_full epoch_004 | 25 | 26.45 | 25.00 | +1.45 | 18.43 | 17.01 | +1.43 | 17.00 | 15.55 | +1.45 | 43.62 | 39.99 | +3.63 | 41.02 | 38.03 | +3.00 |
| release | action_full epoch_004 | 25 | 24.30 | 22.40 | +1.90 | 16.14 | 14.19 | +1.94 | 15.41 | 13.96 | +1.44 | 44.96 | 39.16 | +5.80 | 43.19 | 37.48 | +5.71 |
Positive Δ: the model beats repeating the last context frame.
SSIM
| class | model | n | view | wrist L | wrist R | tactile L | tactile R | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| model | floor | Δ | model | floor | Δ | model | floor | Δ | model | floor | Δ | model | floor | Δ | |||
| none | action_full epoch_004 | 25 | 0.907 | 0.846 | +0.061 | 0.717 | 0.698 | +0.019 | 0.737 | 0.712 | +0.025 | 0.994 | 0.989 | +0.005 | 0.994 | 0.989 | +0.004 |
| full | action_full epoch_004 | 25 | 0.848 | 0.746 | +0.103 | 0.524 | 0.459 | +0.065 | 0.604 | 0.552 | +0.052 | 0.967 | 0.962 | +0.005 | 0.981 | 0.976 | +0.004 |
| onset | action_full epoch_004 | 25 | 0.869 | 0.812 | +0.057 | 0.600 | 0.555 | +0.045 | 0.549 | 0.514 | +0.035 | 0.982 | 0.978 | +0.004 | 0.976 | 0.974 | +0.001 |
| release | action_full epoch_004 | 25 | 0.810 | 0.742 | +0.069 | 0.550 | 0.486 | +0.064 | 0.505 | 0.498 | +0.007 | 0.986 | 0.972 | +0.014 | 0.983 | 0.969 | +0.013 |
LPIPS
| class | model | n | view | wrist L | wrist R | tactile L | tactile R | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| model | floor | Δ | model | floor | Δ | model | floor | Δ | model | floor | Δ | model | floor | Δ | |||
| none | action_full epoch_004 | 25 | 0.038 | 0.075 | -0.037 | 0.187 | 0.224 | -0.037 | 0.162 | 0.197 | -0.036 | 0.005 | 0.012 | -0.007 | 0.009 | 0.015 | -0.006 |
| full | action_full epoch_004 | 25 | 0.058 | 0.142 | -0.084 | 0.213 | 0.276 | -0.064 | 0.195 | 0.262 | -0.066 | 0.064 | 0.078 | -0.015 | 0.040 | 0.048 | -0.009 |
| onset | action_full epoch_004 | 25 | 0.047 | 0.082 | -0.036 | 0.217 | 0.280 | -0.063 | 0.255 | 0.321 | -0.066 | 0.050 | 0.055 | -0.005 | 0.080 | 0.081 | -0.001 |
| release | action_full epoch_004 | 25 | 0.080 | 0.141 | -0.062 | 0.288 | 0.352 | -0.064 | 0.310 | 0.343 | -0.033 | 0.027 | 0.057 | -0.030 | 0.040 | 0.084 | -0.044 |
Lower is better, so Δ is negative when the model wins.
noneno contact in any of the 16 frames↑ contents
motherboard_0909_episode_000 · t=14PSNR model (floor): view 35.3 (31.1) · wrist L 42.0 (37.4) · wrist R 40.9 (36.6) · tactile L 47.3 (44.0) · tactile R 49.6 (44.2)
motherboard_0909_episode_000 · t=1266PSNR model (floor): view 26.9 (24.2) · wrist L 25.6 (20.6) · wrist R 12.2 (11.4) · tactile L 49.2 (43.9) · tactile R 47.9 (44.2)
motherboard_0909_episode_000 · t=1302PSNR model (floor): view 26.5 (27.5) · wrist L 22.6 (25.3) · wrist R 17.6 (17.5) · tactile L 49.6 (43.8) · tactile R 48.2 (43.6)
motherboard_0909_episode_000 · t=1316PSNR model (floor): view 25.1 (24.4) · wrist L 15.7 (11.7) · wrist R 19.2 (16.0) · tactile L 50.4 (43.8) · tactile R 39.8 (40.6)
motherboard_0909_episode_000 · t=1819PSNR model (floor): view 24.1 (21.8) · wrist L 18.5 (15.1) · wrist R 10.2 (10.0) · tactile L 50.8 (44.0) · tactile R 50.5 (43.9)
motherboard_0909_episode_000 · t=2033PSNR model (floor): view 25.8 (23.9) · wrist L 12.3 (11.0) · wrist R 17.3 (15.8) · tactile L 50.3 (43.5) · tactile R 51.6 (43.9)
motherboard_0909_episode_000 · t=2037PSNR model (floor): view 26.4 (26.6) · wrist L 13.5 (12.0) · wrist R 17.3 (16.9) · tactile L 49.3 (43.8) · tactile R 51.5 (43.9)
motherboard_0909_episode_001 · t=14PSNR model (floor): view 35.3 (31.3) · wrist L 35.0 (35.9) · wrist R 41.2 (38.1) · tactile L 48.4 (44.0) · tactile R 49.0 (44.1)
motherboard_0909_episode_001 · t=20PSNR model (floor): view 35.3 (31.3) · wrist L 36.3 (35.4) · wrist R 41.0 (38.1) · tactile L 49.5 (44.0) · tactile R 48.9 (44.1)
motherboard_0909_episode_001 · t=36PSNR model (floor): view 35.2 (31.3) · wrist L 39.7 (36.6) · wrist R 41.1 (38.3) · tactile L 48.5 (44.0) · tactile R 49.4 (44.0)
motherboard_0909_episode_001 · t=85PSNR model (floor): view 35.2 (31.3) · wrist L 39.3 (36.7) · wrist R 41.1 (38.2) · tactile L 48.6 (43.8) · tactile R 49.3 (44.1)
motherboard_0909_episode_001 · t=89PSNR model (floor): view 35.2 (31.3) · wrist L 40.2 (36.8) · wrist R 40.8 (38.2) · tactile L 48.7 (43.6) · tactile R 48.8 (43.5)
motherboard_0909_episode_001 · t=104PSNR model (floor): view 35.3 (31.3) · wrist L 40.9 (36.6) · wrist R 40.9 (37.8) · tactile L 48.8 (43.9) · tactile R 49.6 (44.1)
motherboard_0909_episode_001 · t=141PSNR model (floor): view 25.4 (24.1) · wrist L 26.2 (24.5) · wrist R 17.2 (17.4) · tactile L 46.9 (43.6) · tactile R 49.1 (44.1)
motherboard_0909_episode_001 · t=151PSNR model (floor): view 23.2 (20.8) · wrist L 11.3 (11.7) · wrist R 14.9 (11.7) · tactile L 49.5 (43.6) · tactile R 51.5 (43.8)
motherboard_0909_episode_001 · t=1836PSNR model (floor): view 25.9 (24.1) · wrist L 10.5 (10.7) · wrist R 20.7 (17.4) · tactile L 50.3 (43.4) · tactile R 50.7 (43.8)
motherboard_0909_episode_001 · t=3242PSNR model (floor): view 23.0 (21.0) · wrist L 12.9 (12.3) · wrist R 14.9 (12.2) · tactile L 49.4 (44.0) · tactile R 47.7 (43.8)
motherboard_0909_episode_002 · t=2601PSNR model (floor): view 27.1 (26.5) · wrist L 12.0 (11.4) · wrist R 26.8 (25.9) · tactile L 44.4 (41.9) · tactile R 50.7 (43.9)
motherboard_0909_episode_002 · t=2613PSNR model (floor): view 24.4 (24.0) · wrist L 9.2 (8.0) · wrist R 27.3 (25.9) · tactile L 47.3 (42.6) · tactile R 49.1 (43.9)
motherboard_0911_episode_008_seg00 · t=915PSNR model (floor): view 27.9 (25.3) · wrist L 21.4 (18.0) · wrist R 15.4 (13.1) · tactile L 52.1 (43.3) · tactile R 50.8 (43.0)
motherboard_0912_episode_002 · t=16PSNR model (floor): view 31.2 (30.1) · wrist L 28.7 (29.2) · wrist R 35.3 (30.9) · tactile L 52.4 (43.5) · tactile R 52.3 (42.9)
motherboard_0912_episode_002 · t=33PSNR model (floor): view 27.7 (27.0) · wrist L 22.4 (24.4) · wrist R 23.7 (24.0) · tactile L 51.5 (43.3) · tactile R 51.2 (42.8)
motherboard_0912_episode_002 · t=73PSNR model (floor): view 27.3 (23.8) · wrist L 13.5 (12.7) · wrist R 20.9 (16.7) · tactile L 51.7 (43.1) · tactile R 52.7 (43.0)
motherboard_0912_episode_002 · t=726PSNR model (floor): view 25.3 (23.4) · wrist L 20.3 (12.0) · wrist R 16.5 (14.2) · tactile L 51.6 (43.1) · tactile R 52.3 (42.9)
motherboard_0912_episode_002 · t=762PSNR model (floor): view 34.0 (30.7) · wrist L 34.5 (30.8) · wrist R 33.1 (30.4) · tactile L 52.0 (43.2) · tactile R 52.4 (42.8)
fullin contact for at least 80% of the 16 frames↑ contents
motherboard_0909_episode_000 · t=1468PSNR model (floor): view 29.7 (27.7) · wrist L 23.4 (24.0) · wrist R 17.2 (17.7) · tactile L 50.3 (43.8) · tactile R 36.8 (34.6)
motherboard_0909_episode_000 · t=2585PSNR model (floor): view 23.5 (20.4) · wrist L 16.1 (13.8) · wrist R 15.5 (16.1) · tactile L 39.4 (38.1) · tactile R 37.6 (36.1)
motherboard_0909_episode_001 · t=402PSNR model (floor): view 33.3 (31.7) · wrist L 15.7 (13.9) · wrist R 20.0 (21.8) · tactile L 36.2 (34.7) · tactile R 42.7 (40.7)
motherboard_0909_episode_001 · t=783PSNR model (floor): view 22.5 (20.8) · wrist L 13.3 (12.2) · wrist R 16.1 (13.3) · tactile L 31.2 (29.4) · tactile R 51.6 (44.0)
motherboard_0909_episode_001 · t=1448PSNR model (floor): view 22.0 (22.9) · wrist L 18.7 (17.3) · wrist R 16.5 (16.8) · tactile L 50.6 (43.9) · tactile R 34.7 (34.0)
motherboard_0909_episode_001 · t=1800PSNR model (floor): view 23.6 (21.0) · wrist L 23.4 (19.7) · wrist R 20.7 (19.9) · tactile L 49.4 (43.9) · tactile R 32.9 (31.4)
motherboard_0909_episode_001 · t=1977PSNR model (floor): view 20.7 (18.1) · wrist L 20.2 (17.9) · wrist R 14.5 (14.8) · tactile L 37.5 (36.9) · tactile R 51.1 (43.8)
motherboard_0909_episode_001 · t=2537PSNR model (floor): view 25.1 (20.5) · wrist L 10.6 (10.8) · wrist R 20.7 (17.9) · tactile L 33.4 (31.6) · tactile R 49.9 (43.9)
motherboard_0909_episode_002 · t=628PSNR model (floor): view 22.5 (21.2) · wrist L 13.6 (13.3) · wrist R 14.3 (11.3) · tactile L 35.7 (35.2) · tactile R 49.9 (43.7)
motherboard_0909_episode_002 · t=1432PSNR model (floor): view 23.6 (22.4) · wrist L 13.4 (10.3) · wrist R 16.9 (15.1) · tactile L 41.0 (30.1) · tactile R 28.8 (30.3)
motherboard_0909_episode_002 · t=1797PSNR model (floor): view 30.3 (26.6) · wrist L 14.8 (13.3) · wrist R 19.1 (15.9) · tactile L 31.1 (30.9) · tactile R 37.7 (35.9)
motherboard_0909_episode_002 · t=1937PSNR model (floor): view 24.9 (22.9) · wrist L 16.3 (15.1) · wrist R 18.1 (17.5) · tactile L 35.3 (34.0) · tactile R 49.9 (43.7)
motherboard_0911_episode_008_seg00 · t=317PSNR model (floor): view 26.3 (28.9) · wrist L 19.4 (15.8) · wrist R 20.4 (20.5) · tactile L 30.2 (29.2) · tactile R 52.8 (42.8)
motherboard_0911_episode_008_seg00 · t=382PSNR model (floor): view 30.7 (26.1) · wrist L 16.8 (13.7) · wrist R 20.4 (18.2) · tactile L 31.0 (32.3) · tactile R 52.8 (42.9)
motherboard_0911_episode_008_seg00 · t=652PSNR model (floor): view 31.0 (27.7) · wrist L 16.0 (14.8) · wrist R 19.6 (18.9) · tactile L 32.3 (33.9) · tactile R 53.1 (42.9)
motherboard_0912_episode_002 · t=95PSNR model (floor): view 26.6 (21.7) · wrist L 14.8 (12.4) · wrist R 20.7 (18.3) · tactile L 38.9 (39.3) · tactile R 45.5 (40.6)
motherboard_0912_episode_002 · t=2267PSNR model (floor): view 27.8 (19.1) · wrist L 19.2 (17.9) · wrist R 16.7 (15.2) · tactile L 36.2 (41.0) · tactile R 48.5 (42.2)
motherboard_0912_episode_002 · t=2698PSNR model (floor): view 26.8 (19.0) · wrist L 19.3 (17.4) · wrist R 18.3 (17.7) · tactile L 42.0 (38.9) · tactile R 36.1 (35.7)
motherboard_0912_episode_002 · t=2724PSNR model (floor): view 25.0 (19.0) · wrist L 17.2 (16.3) · wrist R 17.3 (16.7) · tactile L 42.0 (37.0) · tactile R 37.5 (36.4)
motherboard_0912_episode_002 · t=2837PSNR model (floor): view 26.4 (22.1) · wrist L 16.5 (13.8) · wrist R 18.8 (16.8) · tactile L 51.6 (43.3) · tactile R 41.3 (38.2)
motherboard_0912_episode_002 · t=3099PSNR model (floor): view 27.7 (23.8) · wrist L 16.9 (15.3) · wrist R 15.6 (13.9) · tactile L 44.0 (40.5) · tactile R 46.8 (41.7)
motherboard_0912_episode_002 · t=4679PSNR model (floor): view 26.7 (24.2) · wrist L 17.3 (15.7) · wrist R 18.6 (16.7) · tactile L 36.8 (36.2) · tactile R 52.4 (42.9)
motherboard_0912_episode_002 · t=5609PSNR model (floor): view 26.1 (21.9) · wrist L 18.9 (16.3) · wrist R 18.7 (16.2) · tactile L 39.5 (39.5) · tactile R 41.7 (40.1)
motherboard_0912_episode_002 · t=5644PSNR model (floor): view 26.5 (22.4) · wrist L 16.7 (15.4) · wrist R 16.4 (13.7) · tactile L 30.6 (34.3) · tactile R 45.8 (41.6)
motherboard_0912_episode_002 · t=5645PSNR model (floor): view 26.3 (22.6) · wrist L 16.4 (14.3) · wrist R 17.0 (13.9) · tactile L 33.1 (32.9) · tactile R 45.8 (41.2)
onsetno contact at the end of the context, contact at the end of the prediction↑ contents
motherboard_0909_episode_000 · t=284PSNR model (floor): view 27.7 (26.1) · wrist L 27.4 (27.5) · wrist R 15.1 (11.6) · tactile L 49.4 (43.9) · tactile R 34.4 (34.2)
motherboard_0909_episode_000 · t=858PSNR model (floor): view 26.3 (26.8) · wrist L 13.4 (13.4) · wrist R 18.0 (15.8) · tactile L 40.4 (39.2) · tactile R 43.0 (40.7)
motherboard_0909_episode_000 · t=1517PSNR model (floor): view 29.0 (27.8) · wrist L 27.5 (22.9) · wrist R 15.7 (14.4) · tactile L 50.5 (43.8) · tactile R 32.7 (32.5)
motherboard_0909_episode_000 · t=1678PSNR model (floor): view 23.8 (23.6) · wrist L 15.0 (15.0) · wrist R 14.5 (13.1) · tactile L 38.8 (38.4) · tactile R 33.0 (33.3)
motherboard_0909_episode_000 · t=1774PSNR model (floor): view 23.5 (23.8) · wrist L 14.7 (11.4) · wrist R 14.3 (14.4) · tactile L 50.9 (44.0) · tactile R 27.5 (31.9)
motherboard_0909_episode_000 · t=1898PSNR model (floor): view 28.4 (25.7) · wrist L 22.2 (22.4) · wrist R 12.6 (11.2) · tactile L 50.9 (43.8) · tactile R 34.8 (35.9)
motherboard_0909_episode_000 · t=3279PSNR model (floor): view 22.2 (22.5) · wrist L 16.6 (18.3) · wrist R 14.3 (14.1) · tactile L 50.4 (43.9) · tactile R 33.9 (34.3)
motherboard_0909_episode_001 · t=1292PSNR model (floor): view 28.7 (26.5) · wrist L 22.8 (18.3) · wrist R 18.2 (14.8) · tactile L 50.6 (43.8) · tactile R 32.7 (32.5)
motherboard_0909_episode_001 · t=1847PSNR model (floor): view 25.7 (25.1) · wrist L 13.7 (18.2) · wrist R 19.9 (18.1) · tactile L 34.4 (34.5) · tactile R 50.4 (43.8)
motherboard_0909_episode_001 · t=2618PSNR model (floor): view 27.9 (24.7) · wrist L 26.2 (24.8) · wrist R 13.8 (13.3) · tactile L 49.1 (43.9) · tactile R 33.5 (33.2)
motherboard_0909_episode_001 · t=2711PSNR model (floor): view 25.6 (25.2) · wrist L 22.5 (22.4) · wrist R 17.5 (16.8) · tactile L 49.2 (43.2) · tactile R 35.3 (34.8)
motherboard_0909_episode_001 · t=3014PSNR model (floor): view 27.2 (26.4) · wrist L 12.3 (10.5) · wrist R 23.9 (23.7) · tactile L 37.0 (37.5) · tactile R 51.1 (44.0)
motherboard_0909_episode_001 · t=3040PSNR model (floor): view 30.8 (28.8) · wrist L 10.6 (10.9) · wrist R 25.1 (24.0) · tactile L 36.5 (36.0) · tactile R 50.0 (44.0)
motherboard_0909_episode_002 · t=260PSNR model (floor): view 23.7 (23.0) · wrist L 29.2 (29.3) · wrist R 14.3 (14.2) · tactile L 48.4 (43.8) · tactile R 33.7 (33.3)
motherboard_0909_episode_002 · t=452PSNR model (floor): view 28.4 (28.3) · wrist L 21.0 (22.7) · wrist R 13.0 (13.0) · tactile L 47.9 (42.7) · tactile R 32.2 (31.9)
motherboard_0909_episode_002 · t=542PSNR model (floor): view 23.4 (21.5) · wrist L 18.9 (17.7) · wrist R 11.6 (12.2) · tactile L 50.5 (43.8) · tactile R 32.0 (35.0)
motherboard_0909_episode_002 · t=726PSNR model (floor): view 25.1 (25.6) · wrist L 12.0 (12.7) · wrist R 19.6 (19.3) · tactile L 38.8 (37.7) · tactile R 51.5 (43.8)
motherboard_0909_episode_002 · t=2790PSNR model (floor): view 22.8 (21.7) · wrist L 15.3 (15.2) · wrist R 14.1 (13.0) · tactile L 32.4 (32.0) · tactile R 44.9 (41.7)
motherboard_0911_episode_008_seg00 · t=278PSNR model (floor): view 27.3 (25.5) · wrist L 19.0 (14.7) · wrist R 20.6 (19.4) · tactile L 33.7 (32.9) · tactile R 51.6 (43.0)
motherboard_0911_episode_008_seg00 · t=798PSNR model (floor): view 31.1 (27.9) · wrist L 14.3 (12.2) · wrist R 23.7 (23.4) · tactile L 30.1 (30.4) · tactile R 50.9 (42.2)
motherboard_0912_episode_002 · t=1145PSNR model (floor): view 26.0 (22.2) · wrist L 17.7 (12.4) · wrist R 17.3 (11.7) · tactile L 51.7 (43.2) · tactile R 43.1 (40.1)
motherboard_0912_episode_002 · t=1940PSNR model (floor): view 26.7 (23.6) · wrist L 15.4 (12.8) · wrist R 15.6 (14.1) · tactile L 36.5 (37.5) · tactile R 51.3 (42.5)
motherboard_0912_episode_002 · t=2751PSNR model (floor): view 25.5 (22.6) · wrist L 17.8 (13.6) · wrist R 16.8 (12.3) · tactile L 52.5 (43.3) · tactile R 38.3 (37.1)
motherboard_0912_episode_002 · t=4246PSNR model (floor): view 28.3 (26.0) · wrist L 19.0 (14.6) · wrist R 20.8 (19.1) · tactile L 33.1 (34.3) · tactile R 53.1 (42.9)
motherboard_0912_episode_002 · t=5284PSNR model (floor): view 26.1 (24.1) · wrist L 16.3 (11.3) · wrist R 14.8 (11.7) · tactile L 46.8 (42.2) · tactile R 50.6 (42.4)
releasecontact at the start of the context, no contact at the end of the prediction↑ contents
motherboard_0909_episode_000 · t=837PSNR model (floor): view 23.3 (22.2) · wrist L 11.1 (8.0) · wrist R 11.7 (10.9) · tactile L 41.8 (29.8) · tactile R 41.6 (35.5)
motherboard_0909_episode_000 · t=2007PSNR model (floor): view 22.3 (21.1) · wrist L 22.3 (19.4) · wrist R 12.2 (12.0) · tactile L 50.1 (43.9) · tactile R 34.4 (31.1)
motherboard_0909_episode_000 · t=2370PSNR model (floor): view 23.0 (19.8) · wrist L 15.6 (15.0) · wrist R 14.1 (12.6) · tactile L 50.5 (43.8) · tactile R 38.0 (36.2)
motherboard_0909_episode_000 · t=2397PSNR model (floor): view 22.5 (20.9) · wrist L 12.2 (10.4) · wrist R 13.8 (13.0) · tactile L 42.0 (39.4) · tactile R 40.2 (37.4)
motherboard_0909_episode_000 · t=2426PSNR model (floor): view 22.7 (22.2) · wrist L 12.8 (11.6) · wrist R 12.0 (12.0) · tactile L 43.2 (40.9) · tactile R 42.8 (38.6)
motherboard_0909_episode_001 · t=640PSNR model (floor): view 23.8 (24.4) · wrist L 11.8 (11.2) · wrist R 14.9 (14.3) · tactile L 41.5 (38.5) · tactile R 37.2 (33.7)
motherboard_0909_episode_001 · t=892PSNR model (floor): view 22.1 (20.1) · wrist L 14.1 (13.3) · wrist R 13.8 (12.5) · tactile L 50.7 (43.8) · tactile R 35.8 (30.0)
motherboard_0909_episode_001 · t=1282PSNR model (floor): view 29.1 (26.5) · wrist L 19.5 (17.5) · wrist R 15.3 (13.9) · tactile L 50.4 (43.7) · tactile R 39.9 (35.3)
motherboard_0909_episode_001 · t=1923PSNR model (floor): view 23.3 (21.8) · wrist L 11.9 (11.6) · wrist R 21.7 (18.6) · tactile L 36.1 (29.7) · tactile R 50.4 (43.9)
motherboard_0909_episode_001 · t=2890PSNR model (floor): view 25.0 (22.8) · wrist L 9.7 (9.5) · wrist R 18.0 (18.7) · tactile L 38.1 (32.5) · tactile R 50.9 (44.0)
motherboard_0909_episode_001 · t=2927PSNR model (floor): view 23.2 (21.2) · wrist L 10.8 (10.0) · wrist R 20.2 (18.1) · tactile L 39.4 (35.0) · tactile R 50.8 (43.9)
motherboard_0909_episode_002 · t=279PSNR model (floor): view 24.8 (22.9) · wrist L 25.8 (27.0) · wrist R 14.1 (12.5) · tactile L 46.8 (43.8) · tactile R 42.7 (35.9)
motherboard_0909_episode_002 · t=436PSNR model (floor): view 25.5 (22.0) · wrist L 23.4 (21.9) · wrist R 11.1 (10.0) · tactile L 49.2 (43.7) · tactile R 44.3 (32.1)
motherboard_0909_episode_002 · t=807PSNR model (floor): view 22.9 (20.4) · wrist L 13.7 (11.5) · wrist R 16.4 (11.8) · tactile L 38.2 (36.1) · tactile R 51.1 (43.7)
motherboard_0909_episode_002 · t=823PSNR model (floor): view 21.0 (19.0) · wrist L 19.2 (16.5) · wrist R 10.9 (11.1) · tactile L 50.5 (43.9) · tactile R 39.3 (37.2)
motherboard_0909_episode_002 · t=1480PSNR model (floor): view 22.7 (21.7) · wrist L 16.6 (14.6) · wrist R 12.2 (11.0) · tactile L 49.2 (43.6) · tactile R 33.9 (28.4)
motherboard_0909_episode_002 · t=1740PSNR model (floor): view 32.5 (31.7) · wrist L 24.9 (24.5) · wrist R 15.7 (16.6) · tactile L 50.2 (43.9) · tactile R 43.2 (35.9)
motherboard_0909_episode_002 · t=2528PSNR model (floor): view 26.5 (24.3) · wrist L 10.7 (9.6) · wrist R 23.1 (23.9) · tactile L 37.7 (30.8) · tactile R 50.4 (44.0)
motherboard_0909_episode_002 · t=2696PSNR model (floor): view 21.3 (19.9) · wrist L 15.4 (13.0) · wrist R 10.7 (11.0) · tactile L 48.9 (43.2) · tactile R 41.8 (35.7)
motherboard_0909_episode_002 · t=2769PSNR model (floor): view 20.6 (19.7) · wrist L 14.2 (13.0) · wrist R 14.5 (14.0) · tactile L 40.4 (34.8) · tactile R 34.4 (28.3)
motherboard_0912_episode_002 · t=1300PSNR model (floor): view 23.9 (20.8) · wrist L 21.6 (17.3) · wrist R 16.9 (15.1) · tactile L 46.6 (41.8) · tactile R 49.6 (41.9)
motherboard_0912_episode_002 · t=2206PSNR model (floor): view 26.7 (22.4) · wrist L 18.4 (13.2) · wrist R 17.0 (15.3) · tactile L 48.1 (38.3) · tactile R 42.9 (40.6)
motherboard_0912_episode_002 · t=3878PSNR model (floor): view 26.5 (23.5) · wrist L 16.1 (12.1) · wrist R 16.6 (13.1) · tactile L 48.7 (41.9) · tactile R 49.5 (42.4)
motherboard_0912_episode_002 · t=5795PSNR model (floor): view 28.2 (26.9) · wrist L 16.2 (14.1) · wrist R 19.2 (15.3) · tactile L 38.4 (35.1) · tactile R 45.1 (40.1)
motherboard_0912_episode_002 · t=6102PSNR model (floor): view 24.3 (22.2) · wrist L 15.2 (8.8) · wrist R 19.4 (11.6) · tactile L 47.4 (37.0) · tactile R 49.3 (41.1)
Long-horizon rollout
8 s free-running, one continuous timeline each
How it runsThe first window is predicted from the recorded context. After that the model's own decoded frames become the next history (decode → re-encode → predict), so the images never see ground truth again. Each window is conditioned on the actions actually executed then — given this action sequence, what do the cameras and gels see? All 32 rollouts ran the full 15 windows; 8 representative ones (2 per class) are shown here as video.
Reading a videoEach stream plays GT on top, the model's prediction below, at 15 fps (~8.5 s total). The badge marks context (recorded, identical in both rows) versus predicted — from there the PRED row never sees ground truth again. Recorded force runs underneath with a moving playhead.
noneno contact in any of the 16 frames at the start↑ contents
motherboard_0909_episode_000 · t=14 · 15 chained windows, 8 s predictedmotherboard_0909_episode_000 · t=1266 · 15 chained windows, 8 s predictedfullin contact for at least 80% of the 16 frames at the start↑ contents
motherboard_0909_episode_000 · t=1468 · 15 chained windows, 8 s predictedmotherboard_0909_episode_000 · t=2585 · 15 chained windows, 8 s predictedonsetno contact at the end of the context, contact at the end of the prediction at the start↑ contents
motherboard_0909_episode_000 · t=284 · 15 chained windows, 8 s predictedmotherboard_0909_episode_000 · t=858 · 15 chained windows, 8 s predictedreleasecontact at the start of the context, no contact at the end of the prediction at the start↑ contents
motherboard_0909_episode_000 · t=837 · 15 chained windows, 8 s predictedmotherboard_0909_episode_000 · t=2007 · 15 chained windows, 8 s predicted