The basic idea behind the two Mixed Reality apps described here is to connect the 3Dconstruction plan (CAD) and the need to equip front line workers with access to digital information. So that they can understand complex relationships faster, see invisible actualities and achieve more in the same time.
Both apps were developed in short iterations, followed by tests with employees. After each round of testing, we tried to learn and adapt the solution to the needs of the end users. Both apps use the broad range of capabilities of HoloLens 2 to recognize the user's intentions and take into account the conditions in the current environment. The intuitive experience is enabled by the MRTKv2 and Azure Cognitive Services for speech recognition. The fully automated data exchange with the CAD system is powered by a specialized plugin and an Azure PaaS solution.
Our partner Schaerholzbau ag is a modern timber construction company that builds houses according to ecological, economical, and social principles. The planning of these building projects is done digitally. Based on the digital construction plans, machines saw and mill individual wooden beams. These are assembled in the production hall to form transportable elements. During the construction of the building, these elements are installed as wall, floor, or roof components.
The elements are usually different in shape and vary in complexity. For this reason, employees often need to consult the printed construction plan during assembly. Since the planning of the buildings and the components is already done digitally and in 3D, these plans can also be displayed spatially in Mixed Reality. The “ElementBau” app for HoloLens 2 enables a digital workflow from CAD planning to the assembly on the work bench.
The app projects the construction plan three-dimensionally and in life-size on the workbench and supports the employees during assembly:
- The employee has his/her hands free for work while at the same time always having the most important information in view.
- The component selected by gesture or voice command is highlighted in the virtual model.
- A mini model of the element can be freely rotated and zoomed into as needed.
- The virtually displayed parts can be moved to examine complex sub-areas.
- Information is only displayed where it is useful and does not restrict the field of view while working.
With the use of this tool, a 7% increase in efficiency was measured. Equally pleasing is the fact that all employees have responded positively to the app and love to use it daily. The app has been in use since the beginning of 2021 and will show a positive ROI in the coming weeks due to the efficiency gains. The experience and the success with the ElementBau app immediately motivated us and our partner company to explore other possibilities. In a second stage, a companion app for the work on the construction site was created, which covers two crucial sub-steps in the construction phase: “OnSiteXR” assists in fastening the prefabricated building elements and capture the work done for construction documentation.
The construction plan is projected onto the real environment. At each position, where the building elements need to be fixed with angles, visual cues are superimposed. As the employee approaches and takes a closer look at the location, helpful information about the parts will pop up. This app is completely designed to be operated with voice commands and eye-tracking. So, the hands are always free to perform the required work.
In a subsequent step, the work must be thoroughly reviewed: Since the installed angles are no longer visible in the finished building, the building code stipulates that the work must be thoroughly documented. This step is also aided by the OnSiteXR app: The supervisor takes photos with the HoloLens 2 camera and comments in natural language on the work done. The spoken audio is converted into text using Cognitive Services and is automatically stored in the digital documentation catalog. The contents recorded are immediately downloadable as a PDF.
In the development of these Mixed Reality systems, we focused strongly on providing first-line staff with helpful computing solutions that free up their hands for the actual work. The two apps rely heavily on an Azure PaaS infrastructure in the back end and enable a fully digitized construction process for the first time: From the drawing board to handing over the keys to the new homeowner.
The Mixed Reality solutions for the wood construction industry have caused a stir in timber construction interest groups. For this reason, the solutions have been extended to make them capable to be used by other tenants and a plan to commercialize the system is in the works.
About afca.
Our mission at afca. ag is to bridge the gap between the digital and the real world. Since 2016, we have been working with immersive technologies. Our enthusiasm and our quality standards allowed us to join the MRPP in 2018.
Innovators across a diverse set of industries (science / AEC / production / culture) trust us on their journey into the Metaverse. This has enabled us to deploy over two dozen HoloLens solutions to our customers through the years so far.