Capturing photos in a quick way (Unity, HoloLens 2)

[WolfoDev]

My objective is to analyze the HoloLens' camera stream. I've been trying to capture what the HoloLens is seeing for image processing but the only properly documented way I've found is through the library UnityEngine.Windows.WebCam. The problem with this library is that taking a photo, no matter how small, always takes a heavy toll on the device's performance.

I've looked up different ways to capture the camera stream from the HoloLens but the documentations are always lacking something. So I've tried implementing the Windows.Media library in Unity to at least take a quick photo, but it doesn't seem to work. Here is how I've implemented it:

using Windows.Media.Capture.Core;
using Windows.Media.Devices.Core;

private MediaCapture _mediaCapture;

private async void StartScanTest()
{
	_mediaCapture = new MediaCapture();

	await _mediaCapture.InitializeAsync();

	StartCoroutine(ScanTestUWP());
}

private IEnumerator ScanTestUWP()
{
	while (true)
	{
		CheckPatternUWP();
		yield return new WaitForSeconds(0.5f);
	}
}

private async void CheckPatternUWP()
{
	var lowLagCapture = await mediaCapture.PrepareLowLagPhotoCaptureAsync(ImageEncodingProperties.CreateUncompressed(MediaPixelFormat.Bgra8));
	
	var capturedPhoto = await lowLagCapture.CaptureAsync();

	var softwareBitmap = capturedPhoto.Frame.SoftwareBitmap;
	byte[] imageBytes = new byte[4 * softwareBitmap.PixelWidth * softwareBitmap.PixelHeight];
	softwareBitmap.CopyToBuffer(imageBytes.AsBuffer());
	Texture2D photoTexture = new Texture2D(1,1);
	photoTexture.LoadImage(imageBytes);

	await lowLagCapture.FinishAsync();

	if (photoTexture)
	{
		CheckFromTexture(photoTexture);
	}
}

This is all under the #if UNITY_UWP preprocessor directive.

Is there a way to get the actual camera stream from the HoloLens? If not, how can I take pictures without stuttering?

[#2]

It has been a long while since I did camera capture stuff, but the MR Lighting Tools I built a while back takes intermittent camera grabs in a manner that sounds similar to what you're describing!

You can find some relevant code over in the repository here that you might find somewhat helpful? I believe it's a mix of PhotoCapture, and some trickery to work with the underlying Windows Media code beneath that. Some parts may be overkill for you though :)

One of the things that I found was that often, a lot of the cost came from copying image data around and the processing associated with that. I'd set the camera resolution to the lowest possible value, and then I'd try my hardest to rely on Unity's methods for texture copy (like UploadImageDataToTexture), since there's a good chance they've got some lower level optimizations in place.

Hope that's helpful!

[#3]

@koujaku Thank you for your answer! Sadly, I have tried using that library but taking a photo is too slow to be usable even once per second, empty app with just photo capturing staggers like it's about to crash. Maybe I'm calling the wrong methods? I initialized CameraCaptureUWP and called RequestTextureAsync, is there some other way?

In the meantime, I've found a way to get a byte stream out of a PhotoCaptureFrame instance: CopyRawImageDataIntoBuffer. Once I get a byte stream I can dilute the amount of work needed to create a Texture2D over a few frames. Problem is, it seems that CopyRawImageDataIntoBuffer has some memory leak problems. Every time I call that method the available memory decreases, until the HoloLens crashes.

Is there another way to get the array of bytes out of the PhotoCaptureFrame?

[#4]

Hello, I also encounter this problem. Do you have any solution?