HELP: Hololens2 apply camera capture texture for Spatial Mesh

[senthilvelGS]

I successfully saved and displayed the spatial mesh after scene understanding room scanning process. Now I am struggling about how to apply hololens camera live texture to mesh. Any Solution?

[#2]

Just FYI, this is a pretty tricky problem to solve! I've seen it done, but it may require a fair bit of work.

If you're interested, there was a brief discussion on this topic a while ago that does go into a few details:
https://techcommunity.microsoft.com/t5/mixed-reality-feedback-and-help/texturing-spatial-mesh/m-p/21...

It's also quite helpful for us to know what you're hoping to achieve with this as well. Can you tell us anything about your use-case?

[#3]

Thank you for your reply. My use case is display the current Room with real color texture at runtime in front of the hololens camera. It is like a 3D minimap. I am using unity3d(2019.4.23 version) with c#. I have imported the scene understanding plugin for Room scanning. Room scanning and displayed the room mesh functionalities are completed. Now I want apply the real world room texture to the displayed mesh.

I have attached my sample output.

 

[#4]

Hi I am interested in the same thing :), I also just managed to get the scene on .obj and would like to have the texture from the camera over the object, even if in post processing it wouldn't be a problem for me. Did you manage to advance over this problem?

By the way, I guess the code used on the paper mentioned on the other post can be found at https://github.com/samueldong-us/Realtime-Hololens-Retexturing.

[#5]

I'm looking into the same topic. Is there a way to work together on this subject?
As far as I know by now, it's seams to be a very tricky subject. Some have solved it but I cannot find out how they did it

[#6]

Hi,

I searched the holo texture functions in github. Someone developer has successfully completed.

check the link:
https://github.com/tarukosu/HoloCameraTextureMapping

It is a unity 5.6.7 version project and you can take the build without any error. It is properly working in hololens 1. But currently I am working with Hololens2 device. So I tried this project open with latest version(2019.4.4). But lot of errors coming, I could not solve the errors.

Can you help me for the solution?

[#7]

@senthilvelGS 

A person that worked with me diged up on that code and managed to "convert" it to a newer version, supposedly compatible with Hololens 2. You can find a version over https://github.com/danielcamara/Holo2CameraTextureMapping.

Unfortunately it does not work, it gives no compilation errors, but does not work. It can be a better starting point for someone that want to adapt it to Hololens 2.  My skils are shalow, I am not able to fix it... but maybe someone else is 🙂

[#8]

Definitely it doesn't work. Because it is a Unity 5.6.1(very older version) project. So we cannot fix the errors.

[#9]

I am sorry, my misunderstanding. I was under the impression the person had updated it.... but as I knew it wasn't working I never tried to be honest. I was serching for a simpler solution. As I told you, I don't need it to be just in time, for me a post processing, that I can do in another tool, would be OK for me.

What I am doing now is to try to get images during the scan, and try to see if I manage to use the data from the camera to world and projection matrix to have the information I need to project the parts of the collected images over the mesh. But as I never did work with this before,  and I am doing this in my "free time" it is taking a while 🙂

[#10]

Ok. Let me know if you completed. I am also trying projection matrix and shader method. Once I completed I will share to you.

[#11]

I have also been looking at that repo to try and get this type of functionality working for an application framework I am working on. Any progress you make would be greatly appreciated. I haven’t been able to figure out why it will not run correctly yet.

[#12]

@daniel_camara Have you made any progress in this area? I have been trying to get this working for a while now and have not been able to succeed.  

[#13]

@astaikos  Hi I am sorry for the delay in answering!

I got involved with other projects, and I kind of needed to put this on hold! So, nop, I didn't advance... not because it is not possible, too hard, or something like that, just that it kind of decreaed in priority. I will return to it, I don't really know when, for sure not in the next month... but I will and if I manage to do it, I sware to post the code here 🙂