How to unproject a 2D point in an captured image back to its accurate location in the physical world
Hi all, I'm creating HoloLens2 software that detects corners in an image and then shows their actual location in the world. However, there may be slight deviations between the projected points and their actual location, which increase with distance from the user. As following figure shows. Here's my code snippet for projecting the detected corners: I convert the corners from image to screen coordinates, then to NDC for unprojection. Next, I use the inverse of the projection matrix to convert the points from NDC to Camera space and then from Camera space to World space using the camera-to-world matrix. Finally, I instantiate the detected points using a Ray and RaycastHit. For getting the cameraToWorldMatrix and projectionMatrix, i referred to https://github.com/cviss-lab/XRIV/blob/main/XRIV/Assets/LocatableCamera/Scripts/LocatableCamera.cs From my side, i guess it may be the reason that the depth information is not estimated properly or the value of cameraToWorldMatrix and projectionMatrix are not correct. However, i have made efforts on this two reasons, but failed. May dear friends could give me some insights about how to mitigate this projecting devation, i show my great appreciation here in advance!
HoloLens 2 Game Dev Project: Terrace
Super cool that there's finally a forum for MR developers and enthusiasts! I'm an indie game developer working on a boardgame called Terrace for HoloLens, PC, and Xbox. The PC and Xbox versions will have a minimal mode (45° isometric view of the board in a black void) and a "full" 3D mode set in a swanky sci-fi / modern lounge of sorts on a space station. On HoloLens, minimal mode is just the board... which you can place on any surface or leave floating in the air. The menu system is a Near Interactions menu panel which will be dynamically placed at roughly eye level on the closest wall. Instead of a "full" mode, there will be a "mixed" mode on HoloLens, where set pieces from the aforementioned sci-fi scene are dynamically placed in your space (a viewport into space on the wall, the table pinned to the floor with the board on top of it, etc.) Controls will use hand tracking (as shown in the GIF), although I'd love to use gaze and voice at some point for accessibility. Ultimate stretch goal is to eventually add cross-platform shared space multiplayer. If I created an iOS and Android version of the game, those clients could join the same game as someone wearing HoloLens... with up to four players in the same physical space. The host would position the gameboard and the other players would find it (either via adhoc sharing of the world anchor or by the host game creating a very short lived Azure Spatial Anchor). The game has the official blessing of the original IP creator/owner and (if I ever finish it) will be available for free, no advertisements or in-app purchases, and the source code will be published on GitHub. Toryn Farr https://www.linkedin.com/in/torynfarr/
