Add apple tree collision-aware build script and camera capture rig - #8
Open
yogesh2850 wants to merge 42 commits into
Open
Add apple tree collision-aware build script and camera capture rig#8yogesh2850 wants to merge 42 commits into
yogesh2850 wants to merge 42 commits into
Conversation
apple_tree.py builds apple trees with fruit-aware soft collision avoidance enabled and prints each tree's bounding-box extent. apple_tree_cameras.py reuses build_apple_tree() to render a same-distance, fixed-elevation-arc camera rig (above/level/below) per tree, intended as input views for Gaussian splatting. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bnbailey-psl
force-pushed
the
master
branch
3 times, most recently
from
July 28, 2026 19:43
b29db54 to
e30ab6b
Compare
…l strategy Renders each tree from its own frontal-plane camera grid instead of an orchard-wide orbit, trains apple-only (fruit-class) Gaussians via gsplat, and runs both DefaultStrategy and MCMCStrategy on the same seed points so the two can be compared side by side. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CAPTURE_CONFIGS now varies image count (grid density) and number of locations (viewing planes per tree) independently from a default baseline, training/exporting a separate fruit-only .ply per (capture config, splat strategy) pair. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ruth export, visibility/AVUB, kinematics, map+planners Background-agent runs per helios_setup_tasks.md, each isolated to yogesh_dev/: - Phase 0: radiation camera migration, pose validation, honest Tier B/C benchmark - Phase 1: fruit ground truth, semantic/instance/depth export, transforms.json, RGB-D noise model - Phase 2: per-primitive visibility (vis_i), AVUB/AVUB_inf, achievability classes - Phase 3: 5-DOF FK/IK, execution-time model, roadmap, arm geometry - Phase 4: occupancy map, semantic layer, fruit tracker, explore/exploit planners, 3-arm coordination, gimbal refinement T0.6 (CollisionDetection binding) remains a documented, unimplemented blocker propagated through every phase. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Summarizes each phase's real deliverables with links to their LOG/STATUS files, and flags T0.6 (CollisionDetection binding) as the standing blocker every later phase worked around. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… progress so far Adds Phase 6 (metrics harness), Phase 7 (foundation model diagnostics), Phase 8 (scale-up) scope, the upstream helios-core patches worth contributing, and a suggested order of attack from the current state. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…harness - Phase 5: T5.1-T5.9 baselines/oracles (fixed cam, static rig, raster, random, nearest-frontier, greedy oracle, ILP set-cover ceiling, all-views fusion, perfect-perception ablation) against a real 83-fruit scene. T5.7's ILP ceiling caught and fixed two real solver-correctness bugs along the way. - Phase 6: T6.1-T6.11 metrics harness. Reproduces and fixes three real bugs in apple_tree_gaussian_splatting.py (train/test split, masked PSNR, occlusion-aware supervision) without touching that file directly; adds occlusion-conditioned recall, class-stratified F-score, occupancy confusion matrix, discovery curves, oracle-normalized planning score, IG calibration (found anti-correlated, ~-0.26 Spearman - flagged for follow-up), deadline-enforced closed loop, per-module latency table. Both phases blocked only on git push credentials in their own runs, same as every phase before them - nothing else outstanding. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…(D1-D6) T7.1-T7.7 run for real against real Helios data. helios env has none of MapAnything/DA3/pi3/VGGT installed (checked, not assumed) - D1/D2/D3's reconstruction sub-metrics use a documented classical multi-view-geometry proxy instead of faking foundation-model results, flagged in every relevant field. D2's angle sweep found the "no collapse when posed" theory does NOT hold for this proxy - reported honestly rather than smoothed over. D4 (thin-structure recall) beats the <55% benchmark (65-76% here). D6 (metric-scale integrity) mostly within 4-24% of ground truth; canopy volume gap explained as a real definitional mismatch, not silent error. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…s Phases 0-7 Consolidates real bugs found and fixed (8 total, spanning helios-core upstream issues, undocumented API ordering requirements, and pre-existing bugs in the gsplat pipeline), key quantitative findings (render cost scaling, pose accuracy, AVUB numbers, the trace-vs-log-det submodularity failure mode, the IG anti-correlation finding), and what's honestly incomplete or blocked, with pointers back to each phase's log. Phase 8 still running, will be appended once done. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…-we-found per phase Companion to FINDINGS_SUMMARY.md (cross-cutting synthesis): this one goes phase by phase (0-7), each with a "What we did" and "What we found" breakdown covering every subtask's concrete work and real result, pulled from each phase's own log/status files. Phase 8 to be appended once done. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…lete) Phase 8 (scale-up): seeded canopy factory with disjoint dev/test seeds, fractional-factorial screening design run at a documented reduced scale (24 canopies vs the full 320+ spec) with a real ~6.8hr full-scale cost estimate extrapolated from measured timing, apple vs apple_fruitingwall interpenetration comparison (29% vs 4% at 1.5m spacing), full hand-rolled statistics suite (bootstrap CI resampling canopies not views, paired Wilcoxon, Cliff's delta, Holm-Bonferroni), pre-registered metrics with a passing Tatarchenko degenerate-baseline check, and a LiDAR digital-twin path (99.88% leaf-area fidelity, exact rank preservation across 4 policies). Two new real bugs found and fixed along the way. Updates FINDINGS_SUMMARY.md and PHASE_BY_PHASE_FINDINGS.md with Phase 8's results, and COMPLETE_SETUP.md to reflect all 9 phases (0-8) as done. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
W0 (orchard.py, run_w0/w0b/w0c.py), all measured on this machine: - Seeded 2x10 apple orchard via buildPlantCanopyFromLibrary, reusing Phase 8's seeding discipline, Phase 1's ground truth + semantic_class_id derivation. - Determinism verified: same seed -> 830,392 prims / 519 fruit twice; a different seed -> 846,331 / 549. - Build cost 13.5 s @540 d, ~38 s @720 d (752k-830k primitives). - advanceTime at 20-tree scale MEASURED (the plan flagged it unknown): 0.9-2.8 s in the frozen window, 1.3-5.1 s in the real growth window, against ~7-38 s for a rebuild. - Age->fruit curve: the apple model is piecewise constant in age, NOT smooth. Fruit exists only in ~550-730 d; ALL geometry is frozen from 580-730 d and gone by 740 d. The only window where anything actually grows is 540-580 d. This gates the whole growth channel and is reported, not worked around. - advanceTime path == direct build to the same age (identical organ counts, areas agreeing to 1.3e-7 relative), so growth stages cost ~2.5 s not ~7 s. W1 (render.py, run_w1.py): - One solve serves every registered camera; solve time is FLAT at ~4.9 s from N=1 to N=256, i.e. 4.98 s/image at N=1 and 0.019 s/image at N=256. Readback of all four modalities costs 0.009 s/image. - Pixel-array orientation determined empirically (fliplr, unanimous 8/8 views). The obvious test -- lit-vs-hit IoU -- FAILS because diffuse sky flux lights background pixels too; replaced with a leaf/fruit green-red ratio test that scores 6.23 for the right orientation and ~1.0 for the wrong ones. - One fixed global exposure scale for the whole dataset, computed once. - Added a textured ground plane: without it 61% of every frame was empty background. With it, sky is 18% and the lower half of the frame carries real parallax texture. - Depth/semantic sky masks agree pixel-for-pixel (1.000000). W2 (actions.py, run_w2.py): - 4-D view action (dx,dy,dz,dyaw) + scalar growth action; three trajectory families. Acceptance passed: all trajectories in bounds, action replay reproduces poses to 4.7e-7. - Orbits pass through canopy 41.8% of the time on average -- reported rather than hidden, since Helios cameras have no collision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…jection check Three real bugs found and fixed while verifying the render rig, all documented in WORLD_MODEL_LOG.md: 1. Orientation was aggregated by MEAN score. One calibration view with almost no fruit pixels produced a green/red ratio of ~1e7 and outvoted 13 correct views. Now a majority vote, with >=200 leaf and >=200 fruit pixels required per view. 2. Orientation was calibrated at growth stage 0 (540 d), which W0b already measured has ZERO fruit -- so the test found no usable view and silently fell back to the default, which would have mirrored the whole dataset. Calibration now runs at the last stage, pools exposure over first+last stage, and RAISES instead of falling back. 3. The calibrated flip was applied to ALL FOUR modalities. That keeps them mutually consistent and the render still looks like a perfectly good orchard -- which is why it is dangerous -- but depth and the label maps already come out of the Phase 0-validated pixel-labelling pass, so flipping them mirrored every modality relative to the recorded pose. Only raw RGB needs the flip. Added V7, a fruit-centroid reprojection test that exists specifically to catch (3): project each fruit's known world centroid through the recorded pose and compare against its instance-mask centroid. Result: n=135, mean 0.80 px, median 0.71 px, p90 1.09 px at 128x128 -- matching Phase 0 T0.3's 0.71 px. Also added a ground plane with seeded per-patch reflectance jitter. Without it 61% of every frame was empty background (PlantArchitecture canopies have no ground); with it sky is 18.4% and the lower half of the frame carries real optical-flow texture for the action-conditioned model to use. Plus: rssm.py (DreamerV3-style RSSM), data.py (loader + prefetch), train.py, evaluate.py, generate.py, run_w3_verify.py, WORLD_MODEL_LOG.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- rssm.py: DreamerV3-family RSSM. GRU deterministic carry + 32x32 discrete stochastic latent with straight-through gradients and 1% unimix, KL balancing (0.5 dynamics / 0.1 representation) with 1.0-nat free bits, symlog targets for depth and fruit visibility. Four decoder heads: RGB, depth, semantic (7-class), fruit-visibility scalar. Losses are masked by pad_mask -- growth episodes are only 8 frames, so a fixed-length window pads them with repeated frames and zero actions, which is semantically valid but trivial to predict and would inflate the growth channel's apparent accuracy. - data.py: seed-split-aware loader with a bounded LRU episode cache, explicit view/growth channel balancing (the two channels differ by ~24x in frame count), and a single-thread prefetcher (more than one would race the RNG and break reproducibility). - train.py / run_w4.py: training with checkpointing + resume, --overfit-one for the W4 acceptance, --zero-actions for the W6 no-action ablation model. - evaluate.py: horizon-resolved open-loop rollout metrics (PSNR, SSIM, depth MAE in metres over non-sky pixels, semantic mIoU excluding absent classes, fruit-visibility MAE) against copy-last-frame, eval-time zeroed actions, shuffled actions from another episode, and optionally a separately-trained no-action model. LPIPS is NOT reported -- `lpips` is not installed and there is no network access; SSIM substitutes and the substitution is stated. - gsplat_baseline.py: self-contained 3DGS view-synthesis reference in two information settings (matched: same 5 context frames as the world model; generous: every frame but the targets). Does not modify or import the repo-root gsplat script, which is out of write scope and needs pyhelios and gsplat in the same env. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…und and fixed Overfit-one-trajectory acceptance on a real recorded episode (train/view_s10000_g0_row_traversal.npz, 32 frames, 64x64, 12k steps): teacher-forced reconstruction PSNR 30.67 dB SSIM 0.956 depth MAE 0.047 m mIoU 1.000 open-loop imagination (ctx=5) t+1 31.00 dB t+5 30.89 t+10 31.07 t+25 30.14 dB Holding ~30 dB out to a 25-step open-loop rollout on the memorised trajectory means the RSSM dynamics are working, not just the autoencoder. REAL BUG FOUND: the first attempt at this run DIVERGED in its final 500 steps. Reconstruction had reached rgb MSE 7.7e-4 / depth 2.3e-4 at step 11500, then at step 12000 jumped to 1.35e-2 / 3.77e-2 -- a 17x regression -- and the saved final checkpoint scored only 20.05 dB. Cause: Adam eps=1e-8. Once gradients are that small, eps stops regularising the denominator and the effective step size explodes. Fixed by defaulting to eps=1e-5 (the DreamerV2/V3 value) and by checkpointing on best loss rather than trusting the last step. Also: generation now writes VAL and TEST orchards FIRST. At ~5 min per orchard a run that has to be cut short would otherwise leave a dataset with no validation or test split; a smaller train split is merely smaller. On resume it also reuses the manifest's stored exposure/orientation calibration instead of recomputing, so the dataset's single global exposure cannot become a function of when the run was restarted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…TE_SETUP patch note Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… noise-floor finding
Dataset: 800 episodes / 64,000 frames / 1.61 GB. 12 train + 4 val + 4 test orchard
seeds, provably disjoint (checked as declared ranges AND as actually used).
8 growth stages x 3 view families x 128 steps + 16 growth probes per orchard.
Generated at ~5.3 min/orchard. This is 20 train seeds short of the 20/4/4 target;
the run was stopped at 12 to leave time for training and evaluation, which is a
deliberate, reported reduction.
W3 acceptance A1 ("regenerating one shard with the same seed is byte-identical")
FAILS -- 0/40 episodes byte-identical -- and the per-array diagnostic explains
exactly why:
depth, semantic, pose, state, a_view, fruit_vis : bit-exact
instance : fruit mask IoU 1.0, identical 796-ID set, 0.0042% of pixels differ
rgb : 41.1% of pixels differ, mean |d| 6.2 levels, p99 134, max 255
Cause: the OptiX Monte-Carlo radiative solve samples with an RNG PyHelios does
not expose. Geometry is deterministic; radiance is not.
The consequence matters more than the failure: re-rendering the SAME frame from
the SAME pose agrees only to 20.82 dB PSNR. That is a noise floor on the
simulator, and therefore an approximate ceiling on any model's held-out RGB PSNR
-- the stored target carries a noise realisation that is not a function of scene
or pose and so is unpredictable in principle. Every W6 RGB number must be read
against 20.8 dB.
Also ran the gsplat view-synthesis reference (6 test episodes, 3000 iters, 64px):
matched (same 5 context views) t+1 20.98 dB -> t+25 16.67 dB
generous (28 posed views) t+1 21.67 dB -> t+25 20.36 dB
The generous setting landing at ~21 dB, right at the simulator's own re-render
noise floor, is an independent consistency check on that ceiling.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… negative result Trained two models identically except for --zero-actions (the real no-action ablation), 30k steps, batch 24, seq 32, 64x64, 12 train orchards, 2674 s each run concurrently. Evaluated on 4 held-out test orchard seeds. WHAT WORKS - The full model beats copy-last-frame on RGB PSNR at every horizon (18.14/17.98/17.79/17.45 dB vs 16.71/15.63/15.38/15.14 at t+1/5/10/25). - Ordering full > zeroed > shuffled > (trained no-action) holds at every horizon. WHAT DOES NOT - The margins over the ablations are 0.03-0.31 dB. On depth the model LOSES to copy-last at short horizons (1.04 m vs 0.66 m at t+1) and on mIoU too (0.267 vs 0.333). The qualitative rollouts show why: the model has learned the global layout (sky band, textured ground, right average colours) and almost no canopy structure. Its PSNR advantage comes from being a good blurry predictor, not from tracking the scene. PSNR that barely degrades from t+1 to t+25 is the tell. - The GROWTH channel does not work: the full model is worse than the same model with the growth action zeroed at every horizon, and far worse than copy-last on depth (1.47 m vs 0.26 m). Growth frames are 4.0% of the dataset and ~6% of the loss after pad-masking -- the imbalance the plan's risk table anticipated. METHODOLOGICAL FINDING Added `action_sensitivity` to evaluate.py because the plan's PSNR-based no-action ablation turned out to be a weak instrument here. Rolling out from the same context with true, zeroed and negated actions and comparing the PREDICTIONS to each other shows the model displaces its prediction by 21.9% (t+1) rising to 39.4% (t+25) of the magnitude of real inter-frame motion when the action is zeroed, and 24.5% -> 42.8% when it is negated. So the model IS substantially action-conditioned -- the PSNR ablation understated it by an order of magnitude because moving a blurry prediction in the right direction buys almost no PSNR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… and W7 reports
Retrained both models under a corrected best-checkpoint rule (validation
RECONSTRUCTION, not total loss -- the KL term rises as the model uses more latent
capacity, so total loss went up while the model got better, freezing "best" at
step 5000 when reconstruction actually improved to 14000). Both models select
step 14000: main val-recon 0.7407, no-action 0.7605 -- a 2.7% gap in the
action-conditioned model's favour, which is a far cleaner action signal than the
0.04-0.25 dB PSNR margins.
Final test-split evaluation (4 held-out orchard seeds):
view RGB PSNR full 18.13/17.96/17.78/17.45 dB at t+1/5/10/25
copy-last 16.71/15.63/15.38/15.14
trained no-action 18.09/17.87/17.58/17.20
action sensitivity zeroing the action moves the prediction by 20.3% (t+1)
rising to 35.3% (t+25) of real inter-frame motion magnitude
growth channel full is WORSE than zeroed-action and than copy-last up to t+3;
only overtakes at t+5 (25 simulated days)
depth the model LOSES to copy-last at every horizon (1.06 m vs 0.66 m
at t+1); mIoU flat at ~0.26
Also found: training ran at 2.75 it/s until --cache-size was raised from 64 to
512 (npz decompression was 78% of every batch); 11.3 it/s after, a 4x speedup.
And continuing to 42k steps made validation WORSE while training loss fell --
the model is limited by having only 12 distinct orchards, not by step count.
output/w5/curves.png shows the divergence.
W7: WORLD_MODEL_LOG.md, WORLD_MODEL_STATUS.md and FINDINGS.md are complete.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The webhook config (~/.config/claude-notify/slack_webhook_url) does not exist on this machine, so notify_slack() no-ops by design; and notify_slack.py is no longer at the repo root, though it was present at session start. Neither was touched by this work -- all six commits are confined to yogesh_dev/world_model/. Flagged in WORLD_MODEL_STATUS.md rather than papered over or worked around by writing outside the task's write scope. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ffolding
R2-A (run_r2_growth_signal.py) measures, from the stored dataset only, whether
the growth channel is learnable at all. Two decisive results on the test split:
* The growth ACTION is constant. All 320 growth episodes carry the identical
a_grow sequence (5,5,5,5,5,5,10,0) -- one distinct sequence in the whole
dataset. It therefore carries no information beyond the frame index, and
W6's "zero the growth action" ablation was an out-of-distribution query
rather than a counterfactual.
* The RGB growth signal sits at the simulator noise floor. Consecutive stages
differ by 20.60 dB PSNR (23.8 RMS levels) against a measured 20.82 dB
re-render floor (23.2 RMS levels) -- a ratio of 1.03. Depth and semantics do
carry real signal (MAE 0.20 -> 0.06 m per step, self-mIoU 0.48 -> 0.87), but
only 4.5% of the RGB step is explained by a scene-independent mean delta, so
95% of it requires solving canopy structure.
Round 2 code:
* data.py: optional growth stage-subsampling. Growth windows are built from a
random increasing subsequence of the stored stages, so a_grow takes 5/10/15/20
day values instead of one constant. Zero extra rendering -- same real renders,
different step between them. Measured dt histogram is reported in stats().
* train.py: --growth-subsample, --growth-max-stride, --weight-decay (AdamW).
* run_r2_growth_eval.py: a growth evaluation that is a real counterfactual --
from the 545 d frame, predict with a_grow in {5,10,15,20,25,35} d, each with a
true stored target. Reports dt response, dt identification accuracy, and
accuracy vs copy-last. On the Round 1 model identification is 16.7% = exactly
chance, with every prediction landing nearest the 550 d frame regardless of
the dt asked for.
* run_r2_noise_floor.py: re-renders the growth probe rig twice per stage to pin
the noise floor and the stage step at the same poses, same orchard.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
R2-C (run_r2_recon_floor.py) measures the teacher-forced POSTERIOR
reconstruction -- encode a real frame, decode it straight back, no dynamics --
which is an upper bound on every rollout number the model can produce. On the
Round 1 model, held-out test orchards:
PSNR depth MAE mIoU
posterior recon 18.10 dB 1.012 m 0.266
open-loop t+1 18.03 dB 1.040 m 0.264
copy-last t+1 16.57 dB 0.650 m 0.328
97% of the depth error at t+1 is already in the reconstruction; one step of
dynamics adds only +0.028 m. The same measurement on TRAIN orchards gives
0.997 m / 0.294 mIoU -- the model reconstructs frames it was trained on barely
better than frames it has never seen. So the reason it loses to copy-last on
depth is that it cannot render a sharp orchard at all, not that its dynamics or
its orchard diversity are the binding constraint. Round 1's "data-limited"
diagnosis is right about the loss curve and wrong about the ceiling.
Pipelines:
run_r2.sh split re-check -> noise floor -> growth signal -> three matched
40k-step runs on 44 orchards (r2_main, r2_noaction, r2_growth)
-> W6 eval -> growth counterfactual -> curves.
run_r2b.sh the capacity experiment the attribution implies: identical to
r2_main but base 32->64, deter 512->1024, stoch 32x32->48x48,
run alone; then the reconstruction floor for every model on both
splits, W6 eval, and combined curves.
run_r2_check_split.py re-verifies split disjointness against the manifest as
written after the --resume extension, plus directory/file/dup checks.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Per-class breakdown of the Round 1 model's teacher-forced reconstruction on
held-out orchards (run_r2_recon_floor.py, pooled over the split):
ground IoU 0.857 gt 42.37% predicted 46.80%
fruit IoU 0.000 gt 2.04% predicted 0.00%
leaf IoU 0.374 gt 19.62% predicted 18.80%
shoot IoU 0.013 gt 6.33% predicted 0.19%
petiole IoU 0.000 gt 0.22% predicted 0.00%
peduncle IoU 0.000 gt 0.03% predicted 0.00%
sky IoU 0.618 gt 29.39% predicted 34.20%
Those seven numbers average to exactly 0.266 -- the "flat mIoU ~0.26" of Round 1.
The model emits ground, leaf and sky and literally never emits fruit, petiole or
peduncle; copy-last-frame scores 0.328 only because copying reproduces every
class for free. So the mIoU result is class collapse under an unweighted
cross-entropy on a distribution whose rarest class is 0.03% of pixels.
* rssm.py / train.py: optional per-class weights on the semantic cross-entropy.
--sem-class-weights auto measures the class histogram from the training
sampler itself and uses sqrt(median_freq / freq) clamped to [0.25, 4]
(measured: 0.40, 1.58, 0.58, 1.00, 4.00, 4.00, 0.49). Median-referenced, not
mean-referenced, so the 0.03% class cannot hijack the gradient.
* evaluate.py rebuilds the weight buffer from the checkpoint args.
* run_r2c.sh trains r2_sem (identical to r2_main except for the weights) and
evaluates it. Its validation-reconstruction number is NOT comparable with
r2_main's because the semantic term is on a different scale; the comparison
that counts is held-out unweighted mIoU.
Also corrects WORLD_MODEL_STATUS.md: Round 1 flagged the missing Slack
notification as an unexplained anomaly. The cause is known -- notify_slack.py and
the webhook file were deleted deliberately by the user on 2026-07-30, mid-run.
Retracted and documented; the guarded call sites no-op and are left alone.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
run_r2_recon_floor.py now also reports, for the teacher-forced reconstruction:
* RGB sharpness -- mean |spatial gradient| relative to the ground-truth frame.
Round 1 called the model's output "low-frequency smears" from looking at the
rollout strips; measured, the ratio is 0.113. The reconstruction carries 11%
of the real frame's gradient energy (copy-last, being a real frame, scores
0.995). Since the decoder is trained on MSE, its optimum IS the conditional
mean, and with a 32x32 categorical latent (160 bits/frame) the conditional
entropy of an orchard frame is large -- the blur is the architecture working
as specified, not a bug.
* depth MAE split by ground-truth distance:
0-2 m (35.2% of pixels) 0.810 m vs copy-last 0.684 m
2-4 m (40.1%) 0.634 m vs copy-last 0.457 m
4-8 m (18.3%) 1.145 m vs copy-last 0.643 m
8+ m ( 6.3%) 4.396 m vs copy-last 1.805 m
Roughly half the total error comes from the 24.6% of pixels beyond 4 m. The
model trains on MSE in SYMLOG depth but is scored on MAE in METRES, and
symlog charges about a fifth as much for a metre of error at 16 m as at 1 m,
so the far field is systematically under-penalised during training. That
mismatch is part of the copy-last gap -- but not all of it: the model is
worse than copy-last even in the 0-2 m band.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
--depth-loss {mse,l1}. Round 1 trains depth with MSE in symlog space, whose
optimum is the conditional MEAN -- a blur (R2-C measured 0.113 gradient energy
relative to ground truth) -- and which is not the loss the model is scored on
(MAE in metres). L1's optimum is the conditional median, and R2-C's
error-by-distance table shows the far field, where symlog compresses hardest, is
where the model loses most heavily to copy-last.
run_r2d.sh trains r2_best = r2_main + class weights + L1 depth, concurrently with
phases B (capacity) and C (class weights alone), so that r2_sem vs r2_main
isolates the class weights and r2_best vs r2_sem isolates the depth loss.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The RSSM samples its categorical latent, so two evaluations of the same checkpoint differed by ~0.015 m depth MAE and ~0.03 dB. Seeding torch per checkpoint makes the model-to-model comparison exact; verified by running the same checkpoint twice and getting identical numbers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rowth eval
Stage subsampling with stride <= 3 produces a_grow in {5,10,15,20} days, but the
counterfactual eval also queries 25 d and 35 d, which no model here has ever
seen. The eval now keeps the full per-episode error matrix and reports
identification accuracy twice: over all six candidates, and restricted to the
four the augmentation can actually produce, each against its own chance level.
On the Round 1 model both are exactly chance (16.7% of 16.7%, 25.0% of 25.0%)
with every prediction landing nearest the 550 d frame whatever dt is asked for.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ness allows
run_r2_blur_baseline.py scores the PERFECT prediction -- the ground-truth frame
itself -- after blurring it, using exactly evaluate.py's metrics. Test split,
64x64:
sigma sharpness depth MAE PSNR mIoU
0.0 1.000 0.000 m 120.00 1.000
1.0 0.271 0.622 m 21.27 0.476
2.0 0.139 0.841 m 19.84 0.357
3.0 0.100 0.972 m 19.35 0.324
6.0 0.060 1.225 m 18.62 0.288
copy-last 1.000 0.657 m 16.71 0.333
The Round 1 model's measured sharpness is 0.113, i.e. between sigma 2 and 3. At
that sharpness a perfectly correct depth map still scores ~0.94 m -- worse than
copy-last-frame's 0.657 m. So the depth criterion is unreachable at this output
sharpness no matter how accurate the prediction is; beating copy-last on depth
requires sharpness above ~0.26 (sigma ~1), more than double what the model
produces. The model's own 1.03 m is only ~0.09 m worse than that bound, so
roughly 90% of its depth deficit is blur and not prediction error.
mIoU behaves differently: the blurred oracle keeps ~0.33 at the model's
sharpness while the model scores 0.264, so unlike depth there is real headroom
there -- which is consistent with R2-C's finding that the gap is class collapse
rather than blur.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds the Round 2 header and the four settled diagnostic sections:
12 Round 1's diagnosis is half right. Teacher-forced posterior reconstruction
on held-out orchards is 1.035 m depth / 0.268 mIoU / 18.25 dB; open-loop
t+1 is 1.067 m, so 97% of the t+1 depth error is the autoencoder and 3% is
the dynamics. On TRAIN orchards the same model reconstructs at 1.020 m --
a 1.5% generalisation gap. Also quantifies the blur (0.115 of the ground
truth's gradient energy) and locates the depth error by distance.
13 mIoU 0.266 is class collapse, and the seven per-class IoUs average to
exactly the reported number. Four of seven classes are never predicted.
14 A blurred-ground-truth control: the PERFECT prediction, blurred to the
model's measured sharpness, still scores ~0.94 m depth MAE against
copy-last's 0.657 m. Depth is a sharpness problem; mIoU is not.
15 The growth channel. One distinct a_grow sequence across all 320 growth
episodes; RGB growth signal at 1.03x the simulator noise floor; 4.5% of
the change explained by the action alone; and a real counterfactual
evaluation on which the Round 1 model identifies the requested dt at
exactly chance while always predicting one stage ahead.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Round 1's training logs show the KL pinned near its free-bits floor for the whole
run: train kl_dyn 1.371 at both step 17.5k and 18k, val 1.58 -> 1.64. The latent
is 32 categorical variables of 32 classes -- 160 bits per frame of capacity --
and the posterior is diverging from the prior by 1.37 nats, i.e. 2.0 bits. The
encoder is very nearly bypassed, so almost nothing about the specific canopy
reaches the decoder and what it renders is the pose-conditioned average orchard.
That is the same thing as the measured blur (0.115 of the ground truth's gradient
energy) and, per R2-C's blurred-ground-truth control, the reason depth cannot
beat copy-last.
* train.py: --kl-dyn / --kl-rep (the WorldModel already accepted them; they
were hard-coded at DreamerV3's 0.5 / 0.1). evaluate.py restores them from the
checkpoint args.
* run_r2e.sh trains r2_kl with free-bits 1.0 -> 6.0 and KL weights
0.5/0.1 -> 0.2/0.04, everything else identical to r2_main. Both knobs push
the same way on one hypothesis, so this is a single intervention rather than
a controlled pair, and it is labelled that way. Chained on phase C so at most
three training jobs share the GPU at once.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Records the dataset extension (241-323 s/orchard, --resume reusing the stored calibration so exposure and orientation are byte-identical to Round 1), the four diagnostics that reframed the round, the seven-run experiment matrix with what each run isolates, the comparability caveat for the runs that change the loss scale, and Round 2's own dead ends. Also retracts Round 1's Slack anomaly note. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…urement The earlier version compared the growth step (20.60 dB) against FINDINGS section 9's 20.82 dB re-render floor and concluded the RGB growth signal was at the noise floor. That comparison was across different orchards and different poses, and the noise level depends on scene content. run_r2_noise_floor.py measures both at the SAME growth-probe poses on the SAME held-out orchard by rendering each stage twice: re-render identical scene 22.75 dB (18.57 RMS) depth 0.000000 m sem 1.000000 advance one growth stage 20.41 dB (24.32 RMS) depth 0.172 m sem 0.949 The floor at these poses is 22.75 dB, not 20.82. Removing the noise in quadrature leaves a real growth signal of 15.70 RMS levels -- SNR 0.85. So the honest claim is weaker than 'no signal': the RGB growth signal exists and is smaller than the render noise it sits in. Depth and semantics are the opposite extreme -- bit-exact across re-renders at all 8 stages (0.000000 m, 1.000000 agreement) while moving 0.11-0.31 m and ~5% of pixels per stage, so their growth SNR is unbounded. Also updates the action-degeneracy count: still exactly ONE distinct a_grow sequence, now across all 832 growth episodes of the extended dataset. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…owth fix Section 16. Dataset extended to 44 train orchards (166,400 frames, 4.14 GB, splits re-verified disjoint against the manifest as written). r2_main is hyperparameter-identical to Round 1's main2, so the only change is 3.7x the orchards. Loss: best val reconstruction 0.7398 @ 14k -> 0.6959 @ 26k. Better optimum, later overfitting onset. Round 1's claim about the LOSS is confirmed. Held-out metrics: depth MAE 1.061 -> 1.038 m at t+1 (-2.2%), mIoU 0.265 -> 0.268, RGB PSNR 18.13 -> 18.23 dB. Copy-last still wins on depth at every horizon but t+25. Representation ceiling 1.035 -> 1.022 m and sharpness 0.115 -> 0.108 (slightly worse). 99% of the t+1 depth error is still the autoencoder. Stated plainly: Round 1's diagnosis is wrong as an explanation of the held-out result. The failure modes are a sharpness limit, a class-collapse artefact, and a dataset bug -- not orchard diversity. One clean win from the data: the action ablations are now correctly ordered at every horizon (Round 1 had shuffled beating true at t+1). Section 17. Stage subsampling makes a_grow real at zero render cost. dt identification goes 16.7% (chance) for Round 1, 16.1% (chance) for 44 orchards with the unchanged sampler, and 25.0% vs 16.7% chance -- 34.0% vs 25.0% on in-distribution dt -- with subsampling, and the confusion matrix goes from degenerate to diagonal. So the growth channel was a dataset bug, not an unlearnable signal. It is still far from useful, and the measured RGB growth SNR of 0.85 plus the 4.5% action-explained fraction say why. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The blurred-ground-truth control was indexed by RGB sharpness, which gave a bound
of ~0.94 m at the Round 1 model's 0.115 and the claim that ~90% of its depth
deficit was blur. r2_best disproved it: an L1 depth head yields a SHARP depth map
behind a blurred RGB decode, so the two sharpnesses are not locked together.
Measured separately, depth sharpness is about twice RGB sharpness for every model
(r1_main 0.228 vs 0.115, r2_main 0.220 vs 0.108, r2_best 0.281 vs 0.105).
Both scripts now report depth sharpness alongside RGB sharpness, and the bound is
read off the depth column. Corrected conclusions:
* a uniformly blurred perfect predictor needs depth sharpness >= ~0.31 to match
copy-last's 0.657 m; every model here is below it (0.220-0.281), so the
depth criterion is still unreachable at these sharpnesses -- but r2_best is
close to the threshold, not far from it;
* blur accounts for ~35% of Round 1's depth deficit and ~28% of r2_best's, not
90%. The earlier claim is retracted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…umbers Six single-factor runs plus their combination, all on the 44-orchard dataset, all scored on the held-out test split with Round 1's protocol. Ranked by effect on t+1 depth MAE vs r2_main: KL relaxation + class weights + L1 (r2_final) -22% free class weights + L1 depth (r2_best) -13% free KL relaxation alone (r2_kl) -11% free class weights alone (r2_sem) +2% free (worse) 4x capacity (r2_big) -0.5% 4x params 3.7x orchards (r2_main vs Round 1) -2% 2h45m of ray tracing The two levers Round 1 and the Round 2 brief pointed at -- data and capacity -- are the bottom two rows and bought 2.5% between them. Three free changes bought 22%. Against the criteria: r2_final beats copy-last on depth at t+5/t+10/t+25 (0.891 vs 0.968, 0.975 vs 1.096, 1.068 vs 1.205) and still loses at t+1 (0.810 vs 0.657), where section 14's sharpness bound says 0.657 is not reachable at its depth sharpness of 0.293 (threshold 0.31). mIoU 0.265 -> 0.294 on rollout and 0.268 -> 0.304 on reconstruction. RGB reaches 88.8% of the 20.8 dB noise floor, up from 87.2%. r2_big is the proof that the bottleneck is the KL penalty and not capacity: quadrupling the latent to 268 bits left the KL at 1.373 nats, unchanged, and the metrics unchanged to three decimals. Relaxing the KL takes it to 6.2-6.3 nats and is the only intervention that improves RGB sharpness. r2_final was deliberately trained WITHOUT growth subsampling and sits at exactly chance on dt identification, confirming that only the sampler fix touches the growth channel. Also records two methodological notes: val reconstruction has ~7% sampling noise with 8 val batches and all runs share the val stream (so every run's best lands at step 26000), and val recon is not comparable across runs that change the loss scale. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…criteria Depth MAE: PARTIALLY MET -- r2_final beats copy-last at t+5/t+10/t+25 and loses at t+1, where the measured sharpness bound says copy-last is not reachable. mIoU: MET MODESTLY -- 0.265 -> 0.294 rollout, 0.268 -> 0.304 reconstruction. RGB: 87.2% -> 88.8% of the 20.8 dB noise floor, not chased. Growth: the channel was a degenerate action variable, fixed by the sampler at zero render cost (chance -> 1.5x chance); nothing else touches it. Records the correction this round owes Round 1 and its own brief: data bought -2% of depth MAE for 2h45m of ray tracing, capacity bought -0.5%, and three free loss/KL changes bought -22%. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…g point, mistakes, conclusion Records that nothing failed, the throughput actually achieved (7.6 it/s for three concurrent 40k runs with a 1200-episode cache), the moment r2_big's KL came back at 1.373 nats with a 268-bit latent and made clear the bottleneck was the penalty rather than the capacity, why r2_final was added mid-round, and Round 2's own four mistakes including the retracted blur bound and the misattributed growth SNR. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Same pattern as Round 1's committed *_stdout.txt. generate_r2_stdout.txt is the noisy capture (the C++ advanceTime progress bar cannot be suppressed with setProgressCallback -- FINDINGS section 8); the clean per-orchard record is output/dataset/generate_log.txt. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
apple_tree.py builds apple trees with fruit-aware soft collision avoidance enabled and prints each tree's bounding-box extent. apple_tree_cameras.py reuses build_apple_tree() to render a same-distance, fixed-elevation-arc camera rig (above/level/below) per tree, intended as input views for Gaussian splatting.