# Method: action-conditioned visuotactile prediction

This section describes the completed `nano_action_joint_v8` run. We adapt the pretrained Cosmos3-Nano generator to predict synchronized future RGB and tactile observations, conditioned on observation history and known actions throughout the prediction horizon. The run uses `task_mode: action_full`: all action tokens are clean conditioning inputs, and the model is supervised only on future image latents. “Joint” refers to prediction of the visual and tactile streams in a shared transformer.

## Modality representations

**Visual and tactile observations.** Each example contains a 16-frame window sampled at 15 Hz, comprising eight history frames and eight future frames. The five observation streams are the main RGB view, left and right wrist RGB views, and left and right tactile images. We train on precomputed Wan2.2 VAE latents rather than pixels. Each stream is stored as a tensor of shape $48\times4\times16\times16$ (channels, time, height, width), with the cache's anchor latent omitted. The first two latent frames are clean history; the last two are prediction targets. Thus, the prediction horizon is eight frames, approximately 0.53 s. The repository uses the base Wan2.2 VAE for RGB and a tactile-adapted VAE with the `refflow` LoRA for tactile decoding; VAE training is outside this run.

Each latent frame is divided into nonoverlapping $2\times2$ spatial patches. Flattening each patch produces a 192-dimensional vector ($48\times2\times2$), giving 64 tokens per latent frame and 256 tokens per observation stream. The three RGB streams share the pretrained RGB input and output projections. The two tactile streams share a separate pair of trainable input and output projections, initialized from the RGB projections. Both project through the generator's 4,096-dimensional hidden space; output projections produce 192-dimensional velocity patches, which are unpatchified into latent tensors.

**Sensor actions.** Each hand contributes 15 nine-dimensional transition vectors, one for each consecutive pair of frames in the window. For sensor position $p_k$, sensor orientation $R_k$, and normal force $f_k$, we use

$$
\begin{aligned}
\Delta\widetilde p_k &= R_k^\top(p_{k+1}-p_k)
  + \kappa(f_{k+1}-f_k)n,\\
\Delta R_k &= R_k^\top R_{k+1},\\
a_k &= [\Delta\widetilde p_k;\operatorname{Rot6D}(\Delta R_k)],
\end{aligned}
$$

where $n=(0,-1,0)^\top$ is the sensor-normal axis and $\kappa=0.0005$ m/N, corresponding to a stiffness of 2 N/mm. Translations are in metres. The rotation representation concatenates the first two columns of the relative rotation matrix, without subtracting the identity. Force change is therefore folded into the translation component rather than added as a tenth coordinate. The left and right sensor streams share a trainable $9\rightarrow4096$ projection initialized from the pretrained Nano UMI action input head (domain ID 6).

**View actions.** Each hand also contributes a separate sequence of 16 nine-dimensional vectors, one per frame. These are calibrated main-view projections of three hand/sensor points: position, normal point, and up point. Each point is represented by $(u/W,v/H,z)$, where image coordinates are normalized by image width and height and depth is converted from millimetres to metres. Concatenating the three points gives nine coordinates. These are per-frame geometric conditions, without temporal differencing. A separate, newly initialized trainable $9\rightarrow4096$ projection is shared between the two view-action streams. Sensor and view actions are aligned to the same extracted frame IDs. Windows require valid sensor transitions and finite view coordinates with positive depths; points outside the image may remain valid geometric conditions.

**Action normalization.** Sensor and view actions use separate channelwise quantile normalizers, each shared across both hands. For each coordinate $d$,

$$
\widetilde a_d=2\frac{a_d-q_{0.01,d}}{q_{0.99,d}-q_{0.01,d}}-1.
$$

Quantile ranges have a numerical floor of $10^{-6}$; normalized values are not clipped. Statistics are fitted only on training data, using every eighth retained training window and pooling both hands. The saved normalizers were fitted on 267,120 sensor-action rows and 284,928 view-action rows. These are sampled rows from overlapping windows, not counts of unique transitions.

## Token packing and attention

For each example, tokens are concatenated in a fixed stream order:

$$
[V, W_L, W_R, T_L, T_R, A_L, A_R, U_L, U_R],
$$

where $V$ is the main view, $W$ denotes wrist views, $T$ denotes tactile images, $A$ denotes sensor actions, and $U$ denotes view actions. Image tokens are ordered by latent time, then spatial row and column; action tokens follow chronological order. Each projected token receives a learned stream embedding, distinguishing the nine streams even when their projections are shared. Sensor-action stream embeddings additionally receive the pretrained UMI modality embedding at initialization. The pretrained noise-level embedding is added only to future image tokens.

| Streams | Tokens per stream | Clean conditioning tokens | Prediction tokens |
|---|---:|---:|---:|
| Main RGB, left wrist RGB, right wrist RGB | 256 each | 128 each | 128 each |
| Left tactile, right tactile | 256 each | 128 each | 128 each |
| Left sensor action, right sensor action | 15 each | 15 each | 0 |
| Left view action, right view action | 16 each | 16 each | 0 |
| **Total per example** | **1,342** | **702** | **640** |

Examples are flattened into a packed generator-only sequence, with no text or understanding-tower tokens. The attention mask prevents communication between different examples; backend alignment may add padding beyond the real tokens. At a local batch size of eight, the unpadded sequence contains 10,736 tokens.

The transformer uses three-dimensional rotary positional coordinates. Image tokens carry the end-frame ticks $(3,7,11,15)$ and their spatial patch coordinates; sensor-action tokens carry ticks $1,\ldots,15$, and view-action tokens carry ticks $0,\ldots,15$, with zero spatial coordinates for actions. A frame tick $k$ maps to rotary time $6k/15$, preserving physical time at 15 Hz while retaining the pretrained temporal scale of six latent steps per second.

Attention follows a fixed modality topology. The main RGB view attends to all five image streams. Left wrist and left tactile tokens attend to the main view and the two left image streams; right wrist and right tactile tokens have the corresponding right-side connections. The main view connects to both view-action streams, each wrist view connects to its corresponding view-action stream, and each tactile stream connects to its corresponding sensor-action stream. Each image–action connection is bidirectional; each action stream also attends to itself. Other direct action connections are masked. These restrictions describe direct attention edges; information can propagate through the main view across layers.

We use `chunk_bidirectional` temporal attention: every permitted stream pair can attend across the entire 16-frame window, including known future actions and noisy future observation tokens. All future observations are denoised jointly. Optional wrist streams can be omitted from packing entirely, preserving the IDs and coordinates of remaining streams. However, all retained training episodes in this run have wrist cameras and wrist dropout is zero, so training uses the full nine-stream layout.

## Training objective

We train with future-only rectified-flow matching. For each example, a noise level $\sigma\in[0,1]$ is sampled using the upstream `waver` training-time distribution with shift 3. Equivalently, if $t$ is the unshifted sample, $\sigma=3t/(1+2t)$. The same $\sigma$ is shared across all observation streams in an example, while Gaussian noise is sampled independently for each stream. For clean future latent $z^m$ and noise $\epsilon^m\sim\mathcal N(0,I)$,

$$
x_\sigma^m=(1-\sigma)z^m+\sigma\epsilon^m,
\qquad v_*^m=\epsilon^m-z^m.
$$

History latents and all sensor/view actions remain clean. The shared generator predicts $v_\theta^m$ for future observation latents. Let $\Omega_{im}$ denote valid future latent elements for example $i$ and stream $m$, and let $B_m$ be examples carrying valid targets for that stream. The objective is

$$
\mathcal L=\sum_{m\in\{V,W_L,W_R,T_L,T_R\}}
\frac{1}{|B_m|}\sum_{i\in B_m}
\frac{1}{|\Omega_{im}|}
\sum_{j\in\Omega_{im}}
\left(v_{\theta,imj}-v_{*,imj}\right)^2.
$$

Each full stream has $48\times2\times16\times16=24,576$ target elements per example. All five stream weights are 1. The optimized loss is the **sum** of per-stream mean squared errors; the logged `image_mean` is their arithmetic mean. The loss contains no action reconstruction term, history reconstruction term, pixel-space term, or additional noise-level weighting. Action projections learn through their effect on future observation predictions. In distributed training, stream losses are normalized by the global number of valid examples, with rank scaling that compensates for DDP's gradient averaging. Missing streams contribute no loss.

## Optimization and implementation details

We initialize from the local Cosmos3-Nano checkpoint and freeze the pretrained base weights. LoRA adapters of rank 16 and scaling $\alpha=32$ are trained in 252 generator-tower linear layers: the attention query, key, value, and output projections and the feed-forward gate, up, and down projections. We also train all nine stream embeddings, the tactile input/output projections, and both sensor- and view-action input projections. RGB projections and the timestep embedder remain frozen; action output heads are unused and frozen. The run reports 45,342,912 trainable parameters, including 43,646,976 LoRA parameters, out of 15,201,601,929 parameters instantiated in the model (approximately 0.30%). The total includes pretrained modules beyond the actively used generator path.

| Setting | Value for this run |
|---|---|
| Dataset | Motherboard recordings selected by `mixed_mb_wrist_holdout.json` |
| Training episodes / segments | 21 |
| Candidate training windows | 71,485 |
| Removed invalid training windows | 258 |
| Retained training examples | **71,227 overlapping windows** |
| Validation episodes / segments | 2 |
| Validation sampling | Every 50th cached window per episode; 109 selected windows from 5,411 candidates |
| Input clock / window | 15 Hz; 16 frames, split into 8 history + 8 future |
| Training window stride | Every cached window (stride 1) |
| Hardware | **4 NVIDIA H100 GPUs on one node**, four DDP ranks; Slurm job 535527 |
| Batch size | 8 examples per GPU; **32 globally**, one optimizer update per batch |
| Training duration | **5 epochs; 2,225 updates per epoch; 11,125 updates total** |
| Optimizer | AdamW, $\beta_1=0.9$, $\beta_2=0.95$ |
| Learning rate | **$10^{-4}$**, with linear warmup for 500 updates, then constant |
| Weight decay | 0.01 |
| Gradient clipping | Global parameter gradient norm capped at 1.0 |
| Precision | bfloat16 model / mixed precision; float32 loss arithmetic |
| Attention implementation | FlexAttention, `flex_triton` preference |
| Memory optimization | Activation checkpointing |
| Data loading | 4 workers per rank; pinned memory; incomplete training batches dropped |
| Random seed | 0 |
| Wrist dropout | 0.0; all training episodes have both wrist streams |
| Checkpointing | Every epoch; best checkpoint selected by natural-validation total loss |
| Logged elapsed time | 21,645 s, approximately 6.01 h |
| Logged peak training memory | 34.8 GiB on rank 0, after resetting the construction-time peak |

The split is episode based: overlapping windows from the same listed episode/segment remain in the same partition. The selected dataset uses the September 17 repaired pose and v8 force release; older no-wrist recordings retained on disk are excluded by the split. Distributed sampling and dropped incomplete local batches yield 71,200 example presentations per epoch, or 356,000 over five epochs, distinct from the 71,227 available training windows.

Validation runs after each epoch with natural wrist availability and a second arm that removes both wrist streams. The latter measures prediction of the main RGB and tactile streams without wrist inputs; its packed layout contains 830 tokens per example. The distributed validation sampler pads 109 selected windows to 112 presentations across four ranks, so logged validation counts include three repeated examples. The best natural-validation total loss was 0.62394 at epoch index 4 (the fifth epoch), also the final checkpoint.

## Local evidence and scope

Run-specific numbers above come from [config.yaml](config.yaml), [train.log](train.log), [metrics.jsonl](metrics.jsonl), [trainable_params.json](trainable_params.json), and [action_normalizer.json](action_normalizer.json). Hardware allocation is documented by [the launch log](../../logs/slurm-535527.log) and [the training launcher](../../scripts/train.sh). Representation, packing, routing, and loss details were checked against [the stream registry](../../vtm_cosmos/streams.py), [view-action reader](../../vtm_cosmos/data/view_actions.py), [action normalization](../../vtm_cosmos/data/actions.py), [layout](../../vtm_cosmos/model/layout.py), [attention mask](../../vtm_cosmos/model/mask.py), [generator adapter](../../vtm_cosmos/model/generator_adapter.py), [noising](../../vtm_cosmos/train/noising.py), [loss](../../vtm_cosmos/train/loss.py), and [training loop](../../vtm_cosmos/train/train.py). The force coefficient and delta-force semantics were also verified in a training episode's action-file metadata. The saved artifacts establish the run settings and outcomes; implementation descriptions reflect the local source inspected on September 17, 2026, because the run directory does not contain an immutable source snapshot.