How to unproject a 2D point in an captured image back to its accurate location in the physical world
Hi all, I'm creating HoloLens2 software that detects corners in an image and then shows their actual location in the world. However, there may be slight deviations between the projected points and their actual location, which increase with distance from the user. As following figure shows. Here's my code snippet for projecting the detected corners: I convert the corners from image to screen coordinates, then to NDC for unprojection. Next, I use the inverse of the projection matrix to convert the points from NDC to Camera space and then from Camera space to World space using the camera-to-world matrix. Finally, I instantiate the detected points using a Ray and RaycastHit. For getting the cameraToWorldMatrix and projectionMatrix, i referred to XRIV work From my side, i guess it may be the reason that the depth information is not estimated properly or the value of cameraToWorldMatrix and projectionMatrix are not correct. However, i have made efforts on this two reasons, but failed. May dear friends could give me some insights about how to mitigate this projecting devation, i show my great appreciation here in advance!
HoloLens 2 App for Timber Frame Assembly
I am very proud about our latest project: We've developed a HoloLens 2 app for timber frame assembly: https://www.youtube.com/watch?v=oSSQ_GUpYsQ The planning of the building projects is done completely digitally using the cadwork software at schaerholzbau in Altbüron Switzerland. Based on these construction plans, machines and employees saw and mill individual wooden beams into the desired shape. These are assembled by hand in the production hall to form transportable elements. The elements are usually different in shape and vary in complexity. For this reason, employees often have to consult the printed construction plan when assembling the elements on the workbench. Since the planning of the buildings and the components is already done digitally and in 3D, these plans can also be displayed spatially in Mixed Reality. This idea gave rise to the ElementBau app for HoloLens 2. Employees see the construction plan as a hologram on the assembly table, are independent of the printed construction plan and have their hands free to do the work. With the use of the HL2 app, a significant acceleration of the assembly process was measured.We just released our first Azure Spatial Anchors app.
Our digital art gallery app Complex States AR is finally here: https://www.youtube.com/watch?v=aNiQXJRQwIw Create your own digital gallery anywhere you are and let others explore it through the miracle of #AugmentedReality and #AzureSpatialAnchors. Get it for iOS: https://apps.apple.com/us/app/complex-states-ar/id1538774460 Get it for Android: https://play.google.com/store/apps/details?id=com.afca.ch.ComplexStatesAR More info about the project: https://afca.ch/en/output/complex-states-ar
HoloLens 2 Game Dev Project: Terrace
Super cool that there's finally a forum for MR developers and enthusiasts! I'm an indie game developer working on a boardgame called Terrace for HoloLens, PC, and Xbox. The PC and Xbox versions will have a minimal mode (45° isometric view of the board in a black void) and a "full" 3D mode set in a swanky sci-fi / modern lounge of sorts on a space station. On HoloLens, minimal mode is just the board... which you can place on any surface or leave floating in the air. The menu system is a Near Interactions menu panel which will be dynamically placed at roughly eye level on the closest wall. Instead of a "full" mode, there will be a "mixed" mode on HoloLens, where set pieces from the aforementioned sci-fi scene are dynamically placed in your space (a viewport into space on the wall, the table pinned to the floor with the board on top of it, etc.) Controls will use hand tracking (as shown in the GIF), although I'd love to use gaze and voice at some point for accessibility. Ultimate stretch goal is to eventually add cross-platform shared space multiplayer. If I created an iOS and Android version of the game, those clients could join the same game as someone wearing HoloLens... with up to four players in the same physical space. The host would position the gameboard and the other players would find it (either via adhoc sharing of the world anchor or by the host game creating a very short lived Azure Spatial Anchor). The game has the official blessing of the original IP creator/owner and (if I ever finish it) will be available for free, no advertisements or in-app purchases, and the source code will be published on GitHub. Toryn Farr https://www.linkedin.com/in/torynfarr/
