Teaching the cube-finder to read the real, messy camera view
Our self-taught cube-finder worked beautifully on tidy simulated pictures — but the real camera looks down at an angle, sees several cubes at once, and a cluttered workbench all around them. So we rebuilt the practice world to be harder than real life: a camera that roves and tilts, up to several cubes per picture, and random clutter objects thrown in to ignore. We retrained the model overnight on our own PC. It now reads the real camera's full picture — nothing cropped out — finds every cube, and steps over almost all the clutter.
Open the photo largeThe retrained finder on pictures it had never seen — now with several cubes per picture and clutter around them. The crosses are the model's guesses, the circles the true spots; they still land on top of each other.
Photo: Claude (MuJoCo simulation)
What we did
- Rebuilt the simulator's practice scene to look like the real, awkward view: the camera now tilts and circles instead of staring straight down, so the model learns the cube from every angle.
- Put several cubes in each picture at once — a mix of green and orange on the board, and the occasional stray one off to the side — instead of a single tidy cube.
- Scattered random clutter through every scene — objects of other shapes, sizes and colours — so the finder learns what is *not* a cube as firmly as what is.
- Taught the finder to light up every cube it sees, not just the single best one, so one photo can turn into a whole list of cubes for the arm to choose from.
- Retrained the model overnight on our own PC. On unseen practice pictures it now gets about 96 out of 100 cubes right, with almost no false alarms, landing within about 3 pixels of each true spot.
- Pointed it at the real camera's own pictures — just looking, nothing moving — and it reads the full frame with no cropping, finds the cubes, and ignores nearly all the workbench clutter. The earlier finder needed the picture trimmed down by hand first.
What we learned
- The sim-to-real gap
- A model trained only in a simulator can fail on a real camera whenever reality looks different from the practice pictures. Our first finder only ever saw a tidy, straight-down view with one cube — so the real camera's tilt, clutter and several cubes threw it. The cure is to make the simulator's pictures vary more widely than reality ever will, so the real view is just one more case it has already seen.
- Harder than reality, on purpose
- Rather than try to copy the real desk exactly, we made the practice world messier than the real one: more extreme camera angles, more cubes, more clutter, more colour changes. If the model copes with the exaggerated version, the calmer real thing is easy.
- Finding many things at once
- The first finder read off a single brightest point — fine for one cube. The new one reads every bright peak in its heat-map, so a single picture yields all the cubes at once, each with its own position. That is what lets the arm later pick which cube to go for.
- Teaching what is NOT the target
- A finder that has only ever seen cubes can mistake anything vaguely cube-like for one. By filling the practice pictures with clutter — distractor objects that are deliberately not cubes — we teach it to hold its fire on everything that isn't the real thing.
- Confidence, not just a guess
- Each cube the finder reports comes with a confidence score. That lets the arm act only on the cubes it is sure about and quietly skip the shaky guesses — a simple, safe way to turn a fuzzy picture into a firm decision.
Numbers
| Practice pictures this round (train / check) | 1,500 (1,320 / 180) |
|---|---|
| Cubes found correctly, on unseen pictures | about 96% (with ~95% of all cubes found) |
| How close each guess lands (640x480 picture) | about 3 px typically |
| Cubes handled in a single picture | several per colour, at once |
| Cropping the real picture needs now | none — it reads the full frame |
| Where it runs | our own PC, no cloud |
Problems and fixes
The first finder only knew a tidy, straight-down view with a single cube, so it stumbled on the real camera's angled, cluttered, many-cube picture.
Fix: Rebuilt the practice world to be harder than reality — a tilting, roving camera, several cubes per scene, and random clutter — and retrained. It now reads the real frame uncropped.
A model that has only seen cubes tends to spot 'cubes' in ordinary clutter.
Fix: Filled every practice picture with non-cube distractor objects, so the finder learns to ignore things that merely look busy. A couple of stubborn cases remain, noted for the next round.
Next
- Tighten the finder further so it ignores every stray object on a busy desk, and so it recognises cubes that sit on a coloured base rather than being one solid colour.
- Then let the real arm pick a cube the finder spotted on its own — carefully, with a small camera on the gripper for the final centring.
Photos and videos
Open the photo largeA new, harder set of practice pictures: the camera now looks from an angle, several cubes sit out at once, and random clutter is scattered around to ignore. Every picture is changed at random and labelled by the simulator itself.
Photo: Claude (MuJoCo simulation)
