Skip to content
Sandeep Kumar ChaudharySandeep
Back to BlogAR / VR / Spatial

What Are Volumetric Videos and How Are They Captured?

By Sandeep Kumar ChaudharyJul 11, 20266 min read
What Are Volumetric Videos and How Are They Captured — AR / VR / Spatial guide by Sandeep Kumar Chaudhary, full stack developer

TL;DR

Here is a clear, practical guide to volumetric videos: the fundamentals, the best practices that actually move the needle, common mistakes to avoid, concrete data points, and a short FAQ. Everything is structured so you can apply it to real projects today.

Key takeaways

  • Build against OpenXR (native) or WebXR (web) rather than a single vendor SDK so your app survives hardware churn across Quest, Vision Pro, and PC headsets.
  • Vision Pro's primary input model is eyes plus pinch, so make targets large, well-spaced, and glanceable rather than porting a mouse-and-keyboard UI.
  • Treat 90 Hz and low motion-to-photon latency as hard requirements, not nice-to-haves, because dropped frames directly cause nausea and users quit.
  • Budget aggressively for performance: standalone headsets render two eye buffers per frame on mobile-class chips, so draw calls, overdraw, and texture memory matter far more than on desktop.
  • Prototype immersive ideas in WebXR first because iteration is faster, distribution is a URL, and you avoid app-store review cycles.

This is a practical, up-to-date guide to Volumetric Videos — what it is, why it matters in 2026, and how to apply it in real projects. It is written for developers and founders who want clear answers and proven best practices, not filler.

Whether you're just starting out or leveling up, treat this as a working reference you can return to. Every section is built to be skimmed, applied, and shared.

WebXR and the immersive web

WebXR is the W3C Device API that lets a web page request an immersive session and render stereo 3D directly to a headset, typically via WebGL or WebGPU and higher-level libraries like Three.js, Babylon.js, or the declarative A-Frame framework. It succeeded the deprecated WebVR API and covers both VR and AR sessions, including hit-testing against real surfaces, anchors, and hand input on supported devices. The huge advantage is distribution: an XR experience is just a URL, with no app-store submission, and it degrades gracefully to a normal 3D view on phones and desktops. Support is strongest in Chromium browsers and the Quest Browser, and Apple added WebXR to Safari on visionOS, though coverage across all Apple platforms has historically been uneven.

Hand tracking and natural input

Camera-based hand tracking estimates the 3D position of finger joints many times per second, letting users pinch, grab, and point without holding anything. It is now standard on Quest and is the primary input on Vision Pro, usually combined with eye tracking so you look at a target and pinch to click. The trade-offs are real: bare-hand tracking has higher latency and no haptic feedback, and it fails when hands leave the camera view or occlude each other, which is why controllers still win for fast games and precise manipulation. Good XR apps therefore treat hands and controllers as interchangeable input sources and design gestures that are forgiving of tracking noise.

AR, VR, and MR on the reality-virtuality continuum

These terms sit on Milgram and Kishino's reality-virtuality continuum, which runs from a fully real environment to a fully synthetic one. Virtual reality replaces your view entirely with a rendered world, so a Quest in immersive mode or a PC headset playing a game blocks out the room. Augmented reality overlays graphics on the real world, as with phone-based AR through ARKit and ARCore or Snapchat lenses. Mixed reality is the middle ground where virtual objects are aware of and occluded by real geometry, which is exactly what color passthrough on Quest 3 and Vision Pro enables when a virtual screen hides behind your real couch. The lines blur in practice, which is why the neutral catch-all XR (extended reality) is often preferred.

The performance and comfort challenge

Comfort is an engineering problem before it is a design one. Users get motion sick when the visual world lags behind their head movement, so systems aim for high refresh rates (commonly 90 Hz or more) and motion-to-photon latency under roughly 20 milliseconds, backed by reprojection to hide the occasional dropped frame. Because standalone headsets render a separate high-resolution image for each eye on a mobile-class GPU, the frame budget is brutal and techniques like foveated rendering, fixed and dynamic resolution scaling, and aggressive draw-call reduction are routine. Locomotion is the other comfort minefield: smooth artificial movement nauseates many people, so teleport locomotion, snap turning, and peripheral vignetting are standard mitigations to offer alongside it.

What spatial computing actually means

Spatial computing is an umbrella term for systems that blend digital content with the three-dimensional space around a user, tracking the position of the head, hands, and surroundings so that virtual objects behave as if they occupy real space. It subsumes augmented reality, virtual reality, and mixed reality rather than being a separate technology. Apple leaned on the phrase to frame Vision Pro as a general-purpose computer you operate with your eyes, hands, and voice, but the concept predates that marketing. The defining shift from flat 2D computing is that input and output are registered to a coordinate system in the physical world, which is what makes a window feel pinned to your wall or a model feel like it sits on your desk.

Inside Apple Vision Pro and visionOS

Vision Pro is Apple's high-end spatial computer running visionOS, built on the same frameworks as its other platforms with SwiftUI, RealityKit, and ARKit at the center. Its signature interaction model is eye tracking to target and a subtle finger pinch to select, so users rarely reach out or hold controllers. Developers build volumetric content and full 3D scenes with RealityKit and the Reality Composer Pro tool, and can create fully immersive spaces with Metal or bring existing iPad and iPhone apps forward with minimal changes. Apple's persistent passthrough and its 'shared space' windowing make it feel more like a heads-up multitasking desktop than a games console, which shapes what kinds of apps land well on it.

Volumetric Videos: Key Facts and Data

According to recent industry research and the official documentation linked below:

  • Camera-based hand tracking is now built into Quest and Vision Pro, letting users interact with pinch and grab gestures without controllers, though most precision gaming still relies on tracked controllers for haptics and low latency.
  • WebXR replaced the older WebVR API and is supported in Chromium-based browsers (Chrome, Edge, and the Quest Browser) and Samsung Internet; Apple added WebXR support in Safari on visionOS, though desktop Safari and iOS coverage has historically lagged.
  • Meta's Quest line has been the dominant consumer VR platform for years, and industry trackers such as IDC and Counterpoint have consistently reported Meta holding a large majority of standalone headset shipments through 2024 and into 2025.

Quick-Reference Summary

A map of what this guide covers:

TopicWhat you'll learn
WebXR and the immersive webWebXR is the W3C Device API that lets a web page request an immersive session and render stereo 3D directly to a headset
Hand tracking and natural inputCamera-based hand tracking estimates the 3D position of finger joints many times per second
AR, VR, and MR on the reality-virtuality continuumThese terms sit on Milgram and Kishino's reality-virtuality continuum
The performance and comfort challengeComfort is an engineering problem before it is a design one.
What spatial computing actually meansSpatial computing is an umbrella term for systems that blend digital content with the three-dimensional space around a user
Inside Apple Vision Pro and visionOSVision Pro is Apple's high-end spatial computer running visionOS

How to Get Started with Volumetric Videos

A simple path that works:

  1. Learn the fundamentals of Volumetric Videos from primary sources, not just tutorials.
  2. Build one small, real project end to end.
  3. Get feedback, refactor, and add tests.
  4. Ship it publicly and document what you learned.
  5. Repeat with a slightly harder project each time.

Build It with a World-Class Full Stack Developer

Sandeep Kumar Chaudhary is a full stack world-class developer. If you want to turn this into a real, production-ready product, get in touch — message directly on WhatsApp at +9779802348957 for a fast, no-pressure consult.

You can also explore the projects already shipped to thousands of users, or start a conversation here.

Final Thoughts

Build against OpenXR (native) or WebXR (web) rather than a single vendor SDK so your app survives hardware churn across Quest, Vision Pro, and PC headsets. The developers and teams who win in 2026 pair strong fundamentals with consistent shipping. Start small, stay curious, build in public, and revisit this guide as your skills grow.

Sources and Further Reading

#spatial computing#webxr#apple vision pro#meta quest

Frequently Asked Questions

What Are Volumetric Videos and How Are They Captured?

Camera-based hand tracking estimates the 3D position of finger joints many times per second, letting users pinch, grab, and point without holding anything. It is now standard on Quest and is the primary input on Vision Pro, usually combined with eye tracking so you look at a target and pinch to click. This guide covers volumetric videos end to end — core concepts, best practices, concrete data, and a step-by-step approach you can apply right away.

What is the difference between AR, VR, MR, and XR?

VR fully replaces your view with a rendered world, while AR overlays graphics on top of the real world you can still see. MR is the middle ground where virtual objects are aware of and occluded by real geometry, such as a virtual screen hidden behind your real couch. XR (extended reality) is the umbrella term that covers all three, used when the exact point on the spectrum does not matter.

How is Apple Vision Pro different from a Meta Quest?

Vision Pro is positioned as a high-end spatial computer running visionOS, with eye tracking plus pinch as its main input and a focus on productivity, media, and multitasking windows. Quest is a more affordable standalone platform running Horizon OS, with a large games and fitness library and physical controllers as a first-class input. They also differ sharply on price and target audience, though both use inside-out tracking and support passthrough mixed reality.

Is WebXR ready for production use?

Yes for many use cases, especially on Chromium-based browsers and the Meta Quest Browser, where WebXR reliably drives immersive VR and AR sessions. The main caveat is uneven support across Apple platforms, so you should feature-detect the WebXR session types you need and provide a graceful 2D fallback. It is particularly strong for product configurators, training, and prototypes where a URL beats an app-store download.

What is 6DoF and why does it matter?

Six degrees of freedom means the system tracks both rotation (looking around) and translation (physically moving through space), as opposed to 3DoF which only tracks rotation. 6DoF is what lets you lean in, walk around a virtual object, and dodge in a game, so it is essential for presence and comfort. All current standalone headsets like Quest 3 and Vision Pro provide 6DoF tracking for both the head and the hands or controllers.

Sandeep Kumar Chaudhary

Sandeep Kumar Chaudhary

Full Stack Software Developer· Nepal's SEO, AEO, GEO & AIO expert and share-market educator. More about me