Occlusions

From PRIMO.ai
Jump to navigation Jump to search

YouTube search... ...Google search

An occlusion happens whenever something the system cares about is blocked from the sensor: a pedestrian steps behind a parked van, a hand covers part of a cup, a ball rolls behind a wall, or a person walks into the next room. Occlusion is one of the oldest and most stubborn problems in computer vision because a camera records only the first surface each light ray hits. Everything behind that surface is missing from the pixels.

Occlusion comes in several forms:

  • Inter-object occlusion: one object hides part or all of another (a car behind a truck).
  • Self-occlusion: an object hides part of itself (the far side of a mug, a person's arm behind their back).
  • Truncation: the object extends past the edge of the frame.
  • Temporal occlusion: an object is fully hidden for a stretch of time and must be remembered until it reappears.
  • Sensor occlusion: the medium itself blocks the signal (walls for visible light, smoke and fog for cameras, heavy rain for lidar).

Humans handle all of these without effort. We see a cat behind a fence as one cat, not as stripes of fur. Psychologists call this amodal completion: perceiving the whole of an object even though only part of it is visible. Albert Michotte named the phenomenon in the 1950s, and Gaetano Kanizsa's illusory-contour figures made it famous. Developmental psychologists such as Renée Baillargeon later showed that infants only a few months old already act surprised when a hidden object seems to vanish, a capacity called object permanence.

Early deep learning systems were the opposite. A standard detector or segmenter is modal: it labels only the visible pixels, and it forgets an object the instant it disappears. That design works on benchmarks built from visible-pixel labels, but it fails exactly where safety matters: the child about to step out from behind the van.

Modern work attacks occlusion from several directions at once:

  • Robust recognition: classify an object correctly from a fragment.
  • Amodal perception: predict the full shape, extent, and even appearance of hidden parts, increasingly with generative models such as diffusion.
  • Persistence through time: keep tracking an object while it is hidden, using motion models and learned memory (SAM 2, SAM 3, CoTracker3).
  • World models and intuitive physics: learn the expectation that objects keep existing when unseen (V-JEPA, IntPhys 2, Genie 3).
  • Sensing past the occluder: use radio waves that pass through walls (RF-Pose, DensePose from WiFi), multi-view geometry, or the faint light that bounces around corners.
  • Occlusion as a probe: hide parts of an input on purpose to discover which parts a model relies on (Explainable AI).

The shift is from "report what is visible" to "maintain a best estimate of the world, including the parts that are hidden right now, and know how uncertain that estimate is." That is the same predictive stance described on the Predictive Analytics and World Models pages, applied to space and time.


A vision system that reports only what it can see loses the world behind the nearest wall. Handling occlusion means predicting what is hidden, carrying that prediction forward in memory, and tracking how uncertain it has become.


Mechanisms

Occlusion handling is not one algorithm. It is a set of mechanisms that work at different time scales and with different kinds of evidence. Most production systems combine several of them.

1. Recognizing objects from fragments

The cheapest defense is to make recognition itself robust. Training-time augmentations such as Cutout and Random Erasing (both 2017) paste blank or random patches over training images so the network cannot depend on any single region. Part-based and compositional models go further: Adam Kortylewski and colleagues' CompositionalNets (CVPR 2020) represent an object as a set of parts with spatial relationships, and explicitly model an "occluder" alternative for each part, so missing parts lower confidence gracefully instead of breaking the prediction. The limitation is that these methods still output only the visible object. They tell you a dog is present, not where its hidden legs are.

2. Amodal completion: predicting the hidden shape and appearance

Amodal methods predict the full extent of an object, including occluded regions.

  • Amodal masks and boxes. Ke Li and Jitendra Malik's amodal instance segmentation (ECCV 2016), the COCOA dataset from Zhu, Tian, Metaxas and Dollár (CVPR 2017), and the KINS driving dataset (CVPR 2019) established the task: label every object with both its visible mask and its complete mask.
  • Generative completion. Recent systems treat completion as conditional generation. pix2gestalt (Columbia University and Toyota Research Institute, CVPR 2024 Highlight) fine-tunes a latent diffusion model to synthesize the whole object given an occluded image and a region of interest. Because the output is a complete RGB object, any downstream model (a classifier, a 3D reconstructor) can then work on the completed object.
  • Video completion. TACO (ICCV 2025) repurposes a pre-trained video diffusion model so the completed object stays consistent from frame to frame instead of flickering between different guesses.

The catch: completion is always a prior-driven guess. A diffusion model produces a plausible hidden part, not a measured one, so amodal outputs should carry uncertainty and should never be presented as observed fact.

3. Persistence: tracking through occlusion

To follow an object while it is hidden, a tracker needs a model of how it moves and a memory of what it looks like.

  1. Predict. A motion model (classically a Kalman filter with constant velocity) carries the last known position and velocity forward every frame, and grows the uncertainty while no measurement arrives.
  2. Gate. When a new detection appears, the tracker only matches it to tracks whose predicted position is close enough, with the allowed distance widening as uncertainty grows.
  3. Re-identify. An appearance memory (an embedding of what the object looked like) confirms that the thing that reappeared is the same object rather than a look-alike.
  4. Update or retire. A match corrects the estimate and shrinks the uncertainty; too long without a match retires the track.

Learned trackers fold these steps into neural networks. SAM 2 (Meta, 2024) adds a memory bank of recent and user-prompted frames, a memory attention module that conditions each new frame on those memories, and an occlusion head that predicts whether the target is visible at all in the current frame. That last piece matters: a tracker that can say "the object is present but hidden" is very different from one that silently paints a wrong mask. CoTracker (Meta and Oxford) tracks many points jointly, so visible points on an object constrain the positions of its occluded points. DeepMind's AlignNet (2020, linked above) attacked the same problem at the level of object-centric slots, learning to keep each object assigned to a stable slot across time.

4. World models and intuitive physics

Object permanence can also be learned as an expectation. A video model trained to predict missing parts of a scene implicitly learns that objects keep existing behind occluders, because predictions that delete hidden objects are wrong when the objects come back. Researchers test this with the violation-of-expectation paradigm borrowed from infant studies: show the model a physically possible video and an impossible one (an object vanishes behind a screen), and measure which one "surprises" the model more, meaning which has higher prediction error.

Generative world models such as Genie 3 face the same problem from the other side. To generate a believable interactive world, the model must remember what is behind the player when they turn around. Persistence of off-screen content is therefore both a measure of understanding and a requirement for playability (see AI-Based Gaming and Occlusion).

5. Sensing past the occluder

When prediction is not enough, systems change the sensor or the viewpoint:

  • Multiple views and depth. Several cameras, lidar, or depth sensors see around each other's blind spots. Autonomous-driving occupancy networks fuse many views into a 3D voxel grid and mark which voxels were actually observed.
  • Radio frequency (RF). Radio waves in the WiFi range pass through drywall and wood and reflect off the human body. The original page's featured work, Through-Wall Human Pose Estimation Using Radio Signals (RF-Pose, MIT CSAIL, CVPR 2018), described it this way: occlusion is a fundamental problem in human pose estimation and many other vision tasks, and instead of hallucinating missing body parts based on visible ones, it used radio signals to accurately track the 2D human pose through walls and obstructions. Through-Wall Human Pose Estimation Using Radio Signals | Mingmin Zhao, Tianhong Li, Mohammad Abu, Alsheikh Yonglong, Tian Hang Zhao, Antonio Torralba, Dina Katabi - MIT CSAIL The key trick was cross-modal supervision: a camera-based pose model labeled synchronized radio data during training, so the radio model learned to output poses without needing a camera at test time, even through walls the camera could not see past.
  • Non-line-of-sight (NLOS) imaging. A pulsed laser bounces light off a visible wall, around a corner, and back. Precise photon timing lets an algorithm reconstruct the hidden scene. The Conditional Adversarial Architecture (CAA) page covers related "AI sees through walls" work.

6. Occlusion as an explanation tool

Occlusion can also be applied deliberately. Matthew Zeiler and Rob Fergus (ECCV 2014) slid a gray square across an image and recorded how the classifier's confidence in the correct class changed. Regions where covering the image caused a large drop are the regions the model relies on. This occlusion sensitivity map is one of the simplest and most widely used techniques in Explainable AI. It is model-agnostic and easy to interpret, but it costs one forward pass per patch, and the result depends on the patch size and on what value fills the occluded patch.

Approach Question it answers Typical methods Main weakness
Robust recognition "What is this, even though part is missing?" Cutout, Random Erasing, CompositionalNets Says nothing about the hidden part
Amodal completion "What does the hidden part look like?" COCOA, KINS, pix2gestalt, TACO Output is a guess shaped by priors
Persistent tracking "Where is it now, while I can't see it?" Kalman filters, SAM 2/SAM 3 memory, CoTracker3, TAO-Amodal Uncertainty grows the longer the object stays hidden
World models / intuitive physics "Should this hidden object still exist?" V-JEPA, Genie 3 Benchmarks show large gaps from human performance
Alternative sensing "Can I measure it some other way?" RF-Pose, DensePose from WiFi, lidar, NLOS imaging Coarse resolution, cost, and privacy concerns
Occlusion as a probe "Which parts of the input does the model use?" Occlusion sensitivity, RISE Slow; sensitive to patch size and fill value

The Physics

Line of sight and visibility

A camera pixel records light along one ray. If a surface intersects that ray closer to the camera than the object of interest, the object contributes nothing to that pixel. Graphics engines formalize this with a depth buffer (z-buffer): for each pixel, keep only the closest surface. Vision systems face the inverse problem. They receive the z-buffered result and must infer what the depth buffer threw away. The lost information is not merely weak; it is absent. Any reconstruction of fully hidden content must come from somewhere else: other viewpoints, other times, other sensors, or prior knowledge.

Completion as Bayesian inference

Amodal completion is best understood as inference over hidden variables: 

p(hidden | visible)  is proportional to  p(visible | hidden) x p(hidden)

The likelihood term says which hidden configurations are consistent with the visible pixels (the visible edge of a cup constrains its full outline). The prior says which configurations are common (cups are usually rotationally symmetric). Diffusion models such as pix2gestalt learn a very rich prior and sample from this posterior. Because the posterior is often broad, a single completion hides real ambiguity. A responsible system either reports several samples or attaches a confidence to each completed region.

Uncertainty growth while hidden

For a target that moves with roughly constant velocity and random accelerations of strength q, a Kalman filter's predicted position variance grows approximately as 

sigma^2(t)  ~  sigma0^2  +  sigma_v^2 * t^2  +  q * t^3 / 3

where sigma0 is the position uncertainty at the moment the object disappeared, sigma_v is the velocity uncertainty, and t is time spent hidden. The cubic term is why short occlusions are easy and long ones are hard: doubling the hiding time can increase the positional uncertainty by far more than double. Learned trackers do not write this equation down, but the same geometry limits them, which is why both SAM 2 and TAO-Amodal report their worst results on long, heavy occlusions.

Why radio sees through walls and light does not

Visible light has wavelengths of about 400 to 700 nanometers, and building materials absorb or scatter it within a fraction of a millimeter. WiFi signals at 2.4 GHz and 5 GHz have wavelengths of about 12.5 cm and 6 cm. At those wavelengths, drywall, wood, and glass are partly transparent, while the human body (mostly water) is a strong reflector. The tradeoff is resolution. Detail finer than roughly half a wavelength is hard to resolve, so a single radio reflection gives a blurry, specular picture of the body. Systems like RF-Pose overcome this by combining reflections over time and space and by learning from camera supervision. For frequency-modulated continuous-wave (FMCW) radios, range resolution is c / 2B, where B is the sweep bandwidth: 1.5 GHz of bandwidth gives about 10 cm. Concrete and metal still block most WiFi-band energy, so "through walls" usually means through interior walls.

Light around corners

NLOS imaging relies on three diffuse bounces: laser to visible wall, wall to hidden object, object back to wall, wall to detector. Each diffuse bounce spreads the light over a hemisphere, so the returning signal falls off roughly with the fourth power of distance. Confocal NLOS imaging (O'Toole, Lindell and Wetzstein, Nature 2018) showed that scanning the laser and detector together turns reconstruction into a fast deconvolution, but the signal is so weak that capture still needs single-photon detectors and controlled lighting.

Observed versus unobserved space

3D occupancy benchmarks make the physics explicit. Occ3D (NeurIPS 2023) casts rays from each sensor and labels every voxel as observed (a ray hit it or passed through it) or unobserved. Its official evaluation scores only observed voxels. That choice keeps the benchmark honest about what sensors measured, but it also means the benchmark does not reward reasoning about what is hidden. Semantic scene completion, a related task, does try to infer occluded regions. The difference matters for safety: a planner needs to know not only what is there, but which regions it simply could not see.

Why This Matters

  • Safety in driving and robotics. Many of the most dangerous moments for an autonomous vehicle involve something hidden: a pedestrian behind a bus, a cyclist in a blind spot. A system that represents "unknown space" explicitly can slow down near it. Robots grasping objects in cluttered bins face constant partial occlusion of the very object they need to pick up (see Robotics and Vehicles).
  • Video understanding and editing. Segmentation that survives occlusion makes rotoscoping, sports analytics, wildlife monitoring, and medical video annotation practical. Meta built SAM 2's dataset with an annotation tool powered by the model itself, and the team reported that annotation time fell sharply as the model improved.
  • Augmented reality. Virtual objects must disappear behind real ones to look real. Phone AR frameworks now estimate depth so a virtual character can walk behind a real couch.
  • Explainability and debugging. Occlusion-sensitivity maps reveal shortcut learning: a "wolf" classifier that relies on snow in the background, or a medical model that keys on a hospital's text marker instead of the anatomy.
  • Cognitive science. Whether object permanence must be built in or can be learned from video is a live question about both infants and machines. V-JEPA and IntPhys 2 give the debate concrete, testable numbers.
  • Generative worlds and games. A generated world in which things vanish when you look away is unplayable. Persistence under occlusion is now one of the main metrics for interactive world models.
  • Privacy and security. Through-wall sensing works with commodity WiFi hardware and is being standardized. That creates real benefits (fall detection for older adults without cameras) and real risks (sensing people who never consented). See Privacy.

What remains uncertain: there is still no general method for knowing when an amodal completion is trustworthy, current video models remain far from human performance on hard object-permanence tests, and long occlusions in crowded scenes (many look-alike objects) defeat even the best trackers.

Recent Developments

Segmentation and tracking that remember

SAM 2 (Meta FAIR, July 2024; ICLR 2025 oral). SAM 2 extended the Segment Anything approach from images to video by treating an image as a one-frame video. Its streaming architecture stores memories of recent frames and of frames where a user clicked, cross-attends to those memories at every new frame, and uses an occlusion head to output whether the target is visible. Meta trained it on SA-V, a dataset of 35.5 million masks across about 50,900 videos built with a model-in-the-loop annotation engine. The paper reports better video segmentation accuracy with 3 times fewer user interactions than prior approaches, and image segmentation that is more accurate and 6 times faster than the original SAM. In the team's public demo, SAM 2 follows the ball in a three-cup shell game through full occlusion. What it does not establish: the authors acknowledge that crowded scenes with several similar-looking objects and long occlusions still cause identity errors, and their remedy is a corrective user click rather than a fully automatic fix.

SAM 3 (Meta, November 19, 2025). SAM 3 adds promptable concept segmentation: give it a short noun phrase such as "yellow school bus" or an example image, and it detects, segments, and tracks every matching instance. Architecturally, an image-level detector and a memory-based video tracker share one backbone, and the tracker inherits SAM 2's memory encoder, memory attention, and occlusion head. Meta reports roughly double the accuracy of existing systems on concept segmentation in both images and video, trained on a data engine that produced about 4 million unique concept labels, and released the model with the SA-Co benchmark. Limitation: recall now depends on how precisely a concept is phrased, and ambiguous concepts still need human review.

CoTracker3 (Meta and Oxford, October 2024; ICCV 2025). Most point trackers are trained on synthetic video because real videos lack ground-truth tracks. CoTracker3 uses existing trackers as "teachers" to pseudo-label real, unlabeled videos and trains a simpler "student" model on them. The authors report better results than other real-data training recipes while using about 1,000 times less data, and the model runs 27% faster than LocoTrack. Its offline mode, which sees the whole clip, is especially good at occluded points because it can interpolate between the frames before and after the occlusion. The authors note that some competing trackers benefit from more data but still cannot track occluded points well, which makes joint tracking of many points the key design choice.

TAO-Amodal (Carnegie Mellon University and Toyota Research Institute, 2023 to 2024). This benchmark adds amodal and modal bounding boxes to the large-vocabulary TAO tracking dataset, covering 833 object categories in thousands of video sequences, including fully occluded and out-of-frame objects. The headline result is sobering: state-of-the-art trackers, even when adapted for amodal tracking, struggle with heavily occluded objects. Simple fine-tuning schemes improved occluded-object detection and tracking metrics by only 3.3% and 2.1%. This is one of the clearest measurements of how far persistent, amodal tracking still has to go.

Generative amodal completion

pix2gestalt (CVPR 2024 Highlight). Researchers from Columbia University and Toyota Research Institute trained a conditional latent diffusion model on synthetic pairs of occluded objects and their complete counterparts. The model completes objects zero-shot on natural photos (Amodal COCO, Amodal BSDS) and even on paintings that break physical priors, outperforms supervised baselines on established benchmarks, and improves downstream object recognition and 3D reconstruction under occlusion. What it does not establish: the completions are samples from a learned prior. They can be convincing and wrong, which matters for any use beyond visualization.

TACO (ICCV 2025). TACO tackles video amodal completion: given a video and a visual prompt for one object, it generates that object's complete appearance consistently across frames. The authors fine-tune a pre-trained video diffusion model on a large synthetic dataset made by imposing occlusions on unoccluded videos, using a curriculum from easy to hard occlusions. They report better results than previous video amodal completion methods and zero-shot performance on par with state-of-the-art video amodal segmentation, with demonstrations on driving, robot manipulation, and Internet video.

Object permanence: learned or built in?

V-JEPA learns intuitive physics from natural video (Meta FAIR, Université Gustave Eiffel and EHESS, February 2025). Quentin Garrido and colleagues tested V-JEPA, a model trained only to predict masked regions of natural videos in a learned representation space, using violation-of-expectation tests from the IntPhys, GRASP and InfLevel benchmarks. V-JEPA was reliably more "surprised" by impossible videos, reaching 98% zero-shot accuracy on IntPhys pairs and performing especially well on object permanence, continuity, and shape constancy. The authors report that pixel-space video prediction models and multimodal large language models performed much closer to chance. Their interpretation: core knowledge such as object permanence may not need to be hardwired, as long as the model predicts in an abstract representation space. Caution: the original IntPhys benchmark was relatively simple, and the next result shows what happens when it gets harder.

IntPhys 2 (Meta FAIR, June 2025). Florian Bordes and colleagues built a harder benchmark in complex, photorealistic virtual environments testing four principles: permanence, immutability, spatio-temporal continuity, and solidity. Most state-of-the-art models performed at chance (about 50%), while human participants scored near ceiling. A follow-up position paper from New York University (NeurIPS 2025 workshop) trained V-JEPA on SAYCam, head-camera video recorded from young children's point of view, and found that every model tested, regardless of data or architecture, scored only 50 to 54% on IntPhys 2. The authors argue that passive visual data alone may not be enough, and that interaction with the world may be needed. This remains an open and active debate.

Interactive world models and memory

Genie 3 (Google DeepMind, August 5, 2025). Genie 3 generates navigable worlds from a text prompt in real time, at roughly 20 to 24 frames per second and up to 720p. DeepMind reports that environments stay largely consistent for several minutes, with visual memory reaching back about one minute: trees that leave the frame are still there when the camera returns. Crucially, this consistency is emergent. Genie 3 builds each frame auto-regressively from previous frames and the user's actions, without an explicit 3D representation such as NeRFs or Gaussian splats. Its predecessor Genie 2 (late 2024) produced worlds playable for up to a minute but forgot off-screen regions much sooner. DeepMind also used Genie 3 worlds to train and test its SIMA agent.

Project Genie (Google, January 29, 2026). Google released Genie 3 to Google AI Ultra subscribers in the US (18+) through Project Genie, a Google Labs prototype that combines Genie 3 with the Nano Banana Pro image model and Gemini. Users can sketch, explore, and remix worlds, with sessions limited to about 60 seconds because of compute cost. In May 2026 Google extended access to Ultra subscribers globally and added Street View integration. Google lists known weaknesses, including physics that does not always follow real-world laws and imperfect character control.

Microsoft Muse / WHAM (Nature, February 2025). Microsoft Research and Ninja Theory trained a World and Human Action Model on more than one billion image and controller-action pairs from the game Bleeding Edge, roughly seven years of gameplay. The team evaluated three properties: consistency with game rules, diversity of possible continuations, and persistency, meaning that a user's edit (for example, adding a character) stays in the generated world. Output resolution was 300 by 180 pixels, and Microsoft released the weights and a demonstrator tool.

Through-wall sensing becomes a standard

IEEE 802.11bf-2025 (approved May 28, 2025; published September 26, 2025). This amendment to the WiFi standard adds formal procedures for WLAN sensing in license-exempt bands between 1 and 7.125 GHz and above 45 GHz. Target uses include presence detection, monitoring in smart buildings, and remote wellness monitoring. Standardization means that sensing through occlusions, including walls, can become a routine feature of ordinary access points rather than a research prototype.

DensePose From WiFi (Carnegie Mellon University, arXiv, January 2023). Jiaqi Geng, Dong Huang and Fernando De la Torre mapped the amplitude and phase of WiFi signals from three transmitters and three receivers (about US $30 of hardware) to dense body-surface coordinates in 24 regions. The paper was a preprint, not peer reviewed when it drew wide press attention, and the authors reported weaker results in room layouts not seen during training. It nonetheless shows how close commodity hardware has come to camera-like body sensing without line of sight.

Occlusion in driving perception

Occ3D (Tsinghua University and collaborators, NeurIPS 2023 Datasets and Benchmarks). Occ3D built 3D occupancy benchmarks from the Waymo Open Dataset and nuScenes with a label pipeline that includes explicit occlusion reasoning: ray casting decides which voxels each sensor actually observed, and both lidar and camera visibility masks ship with the data. Evaluation counts only observed voxels. Occupancy prediction has since become a standard intermediate representation for vision-centric driving stacks, and the observed/unobserved split is a reminder that "free space" and "unknown space" are not the same thing.

Open questions

  • How should amodal predictions express uncertainty so that downstream planners do not treat a guess as a measurement?
  • Can video pretraining alone close the IntPhys 2 gap, or is interaction (robots, games, embodied agents) required?
  • How long can a purely learned world model keep off-screen content consistent before explicit memory or 3D structure becomes necessary?
  • What consent and disclosure rules should apply to through-wall WiFi sensing now that it is standardized?

Predictions

This section covers two kinds of prediction: how AI systems predict what is hidden, and where the field is likely heading. The second kind is editorial forecasting based on the trends above, not established fact.

How AI predicts the hidden

Every occlusion method is a prediction method in disguise. A tracker predicts where the object will reappear; an amodal segmenter predicts the hidden outline; a world model predicts what the player will see after turning around; a violation-of-expectation test checks whether a model's prediction error spikes when a hidden object fails to reappear. The quality of these predictions depends on the same three ingredients discussed on the Predictive Analytics and Predict image pages: a good model of how things change, a memory of recent observations, and calibrated uncertainty.

Where the field is heading (forecasts)

Forecast Evidence today What would confirm it What would count against it
World-model memory will stretch from about a minute to tens of minutes, mostly by adding explicit memory or 3D caches alongside learned generation Genie 2 to Genie 3 jumped from seconds of reliable recall to about one minute; Muse made persistency an explicit evaluation target A world model that returns a player to a location after 10 or more minutes with the scene intact Persistence plateaus without explicit 3D state, pushing games back toward hybrid engines
Visibility outputs ("present but hidden") will become standard in tracking products SAM 2 and SAM 3 ship occlusion heads; TAO-Amodal benchmarks amodal tracking Major annotation and video tools expose per-frame visibility and amodal masks by default Users ignore visibility signals because they are too unreliable to act on
Hard object-permanence benchmarks will stay unsolved by passive video pretraining alone IntPhys 2 near-chance results; NYU's 50 to 54% finding across models A model trained mainly on interaction data (robotics or games) clearly beats video-only models on IntPhys 2 A larger video-only model reaches human-level IntPhys 2 scores
WiFi sensing will ship broadly and trigger privacy regulation IEEE 802.11bf-2025 published; commodity-hardware research like DensePose From WiFi Router and smart-home vendors advertise sensing features, and regulators issue sensing-specific rules Sensing stays a niche enterprise feature
Occlusion-based explanation will move from single images to video and agents Occlusion sensitivity is standard for images; video trackers now expose memory and visibility states Common tools that mask frames or memory slots to explain tracker and agent decisions Gradient- or attention-based methods replace occlusion probes for video

Interactive Demos and Tools Online

Sources

Additional Viewing

Load video
YouTube

Segment Anything 2: Memory + Vision = Object Permanence (Latent Space podcast with Nikhila Ravi of Meta FAIR and Joseph Nelson of Roboflow, August 2024, about 63 minutes). SAM 2's lead author demonstrates the model following a ball through a three-cup shell game, then explains the memory bank, memory attention, and the visibility timeline that shows when the model thinks an object is hidden. She is candid about limits, especially crowded scenes with look-alike objects. Watch on YouTube

Load video
YouTube

Genie 3: A New Frontier for World Models (The TWIML AI Podcast, episode 743, with Jack Parker-Holder and Shlomi Fruchter of Google DeepMind, August 19, 2025, about one hour). Genie 3's co-leads explain how the model generates playable worlds in real time and walk through its key technical challenges, including the visual memory that keeps off-screen content consistent. They also discuss promptable world events and using generated worlds to train embodied agents. Watch on YouTube

Video Gallery

Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube

Retrieved from "https://primo.ai/w/index.php?title=Occlusions&oldid=46999"