PoolXR: Revolutionizing Pool Gaming with Mixed Reality

Introducing PoolXR

PoolXR is a mixed reality-based project that uses custom-trained object detection model (for cue and game balls) along with the new Meta Passthrough Camera API. It brings the classic game of pool to life by infusing it with virtual information and guidance, combining real-world interaction with virtual elements.

Behind the Cue: How Shot Prediction Works

• Detecting balls: I used a custom-trained YOLO11 model, deployed as a Sentis model in Unity3D, to detect both the cue ball and the target game ball in 2D space on the pool table.
• Ball locked: Once we acquire the 2D coordinates of the balls, we convert them into world space positions and anchor them in place. From this point forward, we no longer need to track their movement.
• Choose pocket: After the ball positions are anchored in world space, the player is prompted to choose a pocket. This selection is used in the upcoming shot prediction phase.
• Evaluating shot: With all three key positions (cue ball, target ball, and selected pocket) defined, we use a combination of pool physics calculations and Unity's physics engine to simulate the optimal cut shot. Predicted paths are then visualized with guiding lines.
• Shot Prediction ready: Once the simulation completes, we display the precise hit point on the target game ball where the cue ball should make contact. This helps the player aim more effectively and increase the chances of sinking the ball.

Inside the Build: Behind the Scenes of PoolXR

Creating PoolXR was an exciting and rewarding journey — a mix of fun experimentation and deep learning. From the initial idea to training custom models and building the final experience, it took weeks of iteration, problem-solving, and hands-on development. This section offers a behind-the-scenes look at how it all came together — from early concepts to real-world testing.

Future Development

I'm continuously exploring new ways to push the boundaries of what's possible in mixed reality gaming. Here's a glimpse of what's next on the roadmap for PoolXR:

1. Cue Stick Tracking: Adding support for real-time cue stick detection to improve aim tracking and shot direction prediction. Once the cue stick angle is detected accurately, we can also estimate shot success percentage based on alignment and positioning.
2. Ball Selection Flow: Adding support for selecting the target ball before choosing a pocket, making the prediction flow more intuitive and user-controlled.
3. Enhanced Shot Evaluation: Improving the shot evaluation system to be more robust and accurate, including a shot accuracy percentage to help players better understand their chances of success.
4. Lower Detection Latency: Optimizing the performance of the object detection model running on the headset to reduce latency and improve real-time responsiveness.
5. Multiplayer Integration: Introducing multiplayer capabilities to make the experience more immersive and enjoyable with friends or other players.

in Check LinkedIn post