PoolXR: Revolutionizing Pool Gaming with Mixed Reality

5 minutes

Introducing PoolXR

PoolXR is a mixed reality-based project that uses custom-trained object detection model (for cue and game balls) along with the new Meta Passthrough Camera API. It brings the classic game of pool to life by infusing it with virtual information and guidance, combining real-world interaction with virtual elements.

Behind the Cue: How Shot Prediction Works

  • Detecting balls: I used a custom-trained YOLO11 model, deployed as a Sentis model in Unity3D, to detect both the cue ball and the target game ball in 2D space on the pool table.
  • Ball locked: Once we acquire the 2D coordinates of the balls, we convert them into world space positions and anchor them in place. From this point forward, we no longer need to track their movement.
  • Choose pocket: After the ball positions are anchored in world space, the player is prompted to choose a pocket. This selection is used in the upcoming shot prediction phase.
  • Evaluating shot: With all three key positions (cue ball, target ball, and selected pocket) defined, we use a combination of pool physics calculations and Unity's physics engine to simulate the optimal cut shot. Predicted paths are then visualized with guiding lines.
  • Shot Prediction ready: Once the simulation completes, we display the precise hit point on the target game ball where the cue ball should make contact. This helps the player aim more effectively and increase the chances of sinking the ball.
Step 1: Perform the gesture to initiate ball tracking. Using camera access, every 4th–5th frame is passed through the model to track 2D coordinates of the balls in real time.
Step 2: Once ball tracking is stable, perform the gesture to localize and anchor the real pool table in world space.
Step 3: Select a pocket to start the shot prediction process, guided by visual line overlays on the real table.


Inside the Build: Behind the Scenes of PoolXR

Creating PoolXR was an exciting and rewarding journey — a mix of fun experimentation and deep learning. From the initial idea to training custom models and building the final experience, it took weeks of iteration, problem-solving, and hands-on development. This section offers a behind-the-scenes look at how it all came together — from early concepts to real-world testing.

Early proof of concept showcasing a custom-trained model tracking a table tennis ball in a basic real-world setup.
Custom shader experiment that creates a ripple effect on the table surface based on the cue ball’s world space position, anchored accurately to the physical table.
Manually labeled 450+ images to train a custom object detection model for recognizing cue and game balls in various conditions.
The initial idea was to detect balls using OpenCV — here's the proof of concept. It worked to some extent, but didn’t feel reliable enough for consistent gameplay.

Future Development

I'm continuously exploring new ways to push the boundaries of what's possible in mixed reality gaming. Here's a glimpse of what's next on the roadmap for PoolXR:

  1. Cue Stick Tracking: Adding support for real-time cue stick detection to improve aim tracking and shot direction prediction. Once the cue stick angle is detected accurately, we can also estimate shot success percentage based on alignment and positioning.
  2. Ball Selection Flow: Adding support for selecting the target ball before choosing a pocket, making the prediction flow more intuitive and user-controlled.
  3. Enhanced Shot Evaluation: Improving the shot evaluation system to be more robust and accurate, including a shot accuracy percentage to help players better understand their chances of success.
  4. Lower Detection Latency: Optimizing the performance of the object detection model running on the headset to reduce latency and improve real-time responsiveness.
  5. Multiplayer Integration: Introducing multiplayer capabilities to make the experience more immersive and enjoyable with friends or other players.
in Check LinkedIn post

Aman Bohra

Hi there! I'm Aman Bohra, the developer behind PoolXR. I'm passionate about creating innovative mixed reality experiences that push the boundaries of what's possible in interactive technology.