All our Surface Hubs stopped communicating with Azure Log Analytics

[MarcVDH]

Hello,

All our Surface Hubs have stopped communicating with the Log Analytics workspace and we don't find why.

We have Surface Hubs, Surface Hubs 2S, same problem.

All are configured through Intune,

It has worked for months but suddenly in november (I know it is already a long time ago), the same day, all the SH stopped sending data to the workspace.

We have a graphic showing this :

Since then we can't find the reason for this. No configuration change have been done.

I have installed a machine, a simple windows 10, installed the monitoring agent and configured it with this workspace and it works !

Whatever 'regular' windows 10 I try to connect to the workspace it works but none of the Surface Hub.

We have tried to connect a Surface Hub with a 4G connection, just to see if it could be a problem with our network but it doesn't work either.

In intune I can see that the SH are well communicating, I can reboot them, sync them etc...

The big problem is that with the covid situation we are all working remotely and with a Surface Hub, you can't do anything remotely (you cannot access the logs, take a remote control session...).

The devices are a nightmare to manage remotely.

Anyone have an idea what I can do or where I can look to find a solution ?

Thanks for your help

Marc

[#2]

@MarcVDH  open up a case with Surface Hub support.  We have one open already it appears to be the agent on the surface hub is not starting based on the logs, assuming you are experiencing the same issue but best to open a case, get your logs and submit them.

[#3]

@ToddMethven thanks for your answer.

I had opened a case on 25/01. The answer I got is 'it's a known issue, we are working on it. No ETA, sorry'

[#4]

Funny.
We had exactly the same problem. I passed 1h to try understand how works logs in analytics...

And I have the same idea with firewall problems but not test.

I thought I was going to look on the community to see if anyone had this kind of problem and this is the first post.

I just see this because I didn't need to go on logs before this last days since we have some problems with Miracast.

Monday I had install the new OS for surface hub on a surface hub 2 but I don't see it on logs so the new OS don't seems to resolve this problem.

I open a case to surface hub.

[#5]

@MichaelOliv 

Good luck because for us it is still the same. Support from MS is non-existent, the only communication we receive is 'nothing new'.

And now you have the new Surface Hub Windows 10 Team 2020 Update, still nowhere to be found in WSUS (for both v1 or v2).
The current OS is out of support in 12 days.
We have enrolled a machine in WUfB to see if there we would be able to update the machine...nothing... more than one day later... the status of the machine is 'not scanned yet' and we have no possibility to force a scan for updates.
So what is the solution ? send technicians all over the country with a USB key to do the install ?

These devices are driving us nuts.

[#6]

Hello @MarcVDH ,

Yes, this is a known issue with Log Analytics and unfortunately we can't share anything more with you other than we're looking into it. We also can't share an ETA for a fix as it would be a guess at this point, which others would interpret it as a promise. I can assure you that we are working with the Azure Log Analytics team to resolve this as fast as possible. We're on it, it just takes time so thank you for your patience!

As to WSUS, yes, the 20H2 update is only available through Windows Update, Windows Update for Business and BMR. This was announced here: Surface Hub Windows 10 Team 2020 Update available October 27 - Microsoft Tech Community.  

Best regards,

Cezar

[#7]

Hello Cezar,
Thank you for your answer.
We have a meeting about that problem with Microsoft on monday.

When we started the implementation of the SH in our company (a long time ago) we were told by MS that we could use WSUS for everything because we wanted more control about the deployment of the updates so we didn't looked at the other options.
Now in your article, i agree, it says Windows Update and not WSUS, but WSUS is only a tool using Windows Update so it wasn"t very clear that the update wouldn't come to WSUS. That's why I asked at the end of the article.
In the known issue of the update, now we can see 'If using WSUS, migrate to Windows Update for Business.'.
So ok, we will migrate to WUfb. We are trying with one machine at the moment and the result are not good. The machine is connected for more than one day (I can see the last check-in time changing from time to time) but the last scan time stays desperately on 'not scanned yet'.
I have seen on the forum that the machine should check every 22 hours and that there is no way to remotely force a scan.

I think Microsoft should at least change the date for the end of support of the 1703 version.

Regards
Marc

[#8]

Hello @MarcVDH 

WSUS still works with Surface Hub, except this update and probably a few more to come. My understanding is that WSUS doesn't work with manifests which is required by the 20H2 update.

If you're trying to update a Surface Hub V1, please make sure you enable Full Telemetry first. This is required for the update to be detected.

If you still have issues, please open a case so we can look into this

Thank you,

Cezar

[#9]

Hello Cezar,
We have deployed a configuration profile for configuring the telemetry like indicated in the doc but when deployed, the device status for the CP indicates that the status is 'Not Applicable' for all of our machines.
I am waiting till Monday, then we will wee how it goes.

Thanks
Marc

[#10]

@MarcVDH 

Do you know if this issue has been resolved?
I have an issue where a Surface Hub that is not sending data to Log Analytics.

Thomas

[#11]

@Thomas Forsmark Sørensen 

Hello,

Unfortunately no, they are still investigating the problem.

Marc

[#12]

Hi Again,
OMG it will soon be 3 months since you first reported the problem 😞
Hope they will find a solution soon.
Thomas

[#13]

I would appreciate if you would update this post when you/they have it working again 😄

[#14]

I also hope they will find a solution. The 1M€ question is when....
I will post an update as soon as we have something new.

[#15]

We tried to connect our Surface Hubs since October 2020 and I already opened a case at Microsoft in November.
As we never had it working before, we were not sure if there could be a mistake on our side.
We tried different networks (company, private, 4G) to make sure this is no proxy/firewall issue.
On a regular Windows 10 the monitoring agent ist working fine, but on none of our surface hubs.
BTW: Nobody at Microsoft told us that this is a known issue - we just periodically get some updates that they are still investigating and that there is nothing new.

When I opened the case I attached some findings from the event log but they didn't really seem to care.
The case has been sent to the OMS-Team and back and for what I see they have absolutely no clue, what is going wrong.

@MarcVDH 
Maybe you could check, if you see the same entries in the logs of your Surface Hubs?
If you collect the support logs from a Surface Hub you get a zip file which contains a folder "WindowsEventLog" with a file "Operations Manager.evtx".

Open this file on Windows 10 and you may see some warnings, each followed by an error.

Example (partially translated from German):

Failed to create process due to error '0x800711c7 : Your Organization used Device Guard to block this app - contact your support person for more info.
', this workflow will be unloaded.

Command executed: "C:\Windows\system32\cscript.exe" /nologo "MonitorKnowledgeDiscovery.vbs"
Working Directory: C:\Program Files\Microsoft Monitoring Agent\Agent\Health Service State\Monitoring Host Temporary Files 1\16\

One or more workflows were affected by this.

Workflow name: Microsoft.SystemCenter.Advisor.AgentHealth.KnowledgeDiscoveryMonitor

We only see this error if OMS is enabled on the Surface Hub.

[#16]

hello @baeumeto 

Welcome to the club, if I may say so 😕

We dit everything like you (wired, wireless network, different internet provider etc... etc...)

We had something from MS telling us that the 2020 Update wos solving the problem, and it turns it is not true.

It is even worse now because the 2020 update has apparently introduced new problems (you are not able to make phone call from the SH, it is ringing on the SH but on the other side nothing and after a few seconds you get 'sorry we couldn't connect you'). That was working perfectly before the update.

As for the eventlog, I have checked on one machine, we don't have the same error as you.

We have this one :

AOI-a52dc355-8d13-43ed-b036-70060aaef750
Microsoft.SystemCenter.Advisor.AgentHealth.KnowledgeDiscoveryMonitor
{BFB936BE-6E53-12FA-5ECE-7D059F073B36}
12:45:52 AM
"C:\Windows\system32\cscript.exe" /nologo "MonitorKnowledgeDiscovery.vbs"
C:\Program Files\Microsoft Monitoring Agent\Agent\Health Service State\Monitoring Host Temporary Files 1794\365\
System.PropertyBagData
60
0x80070241 : Windows cannot verify the digital signature for this file. A recent hardware or software change might have installed a file that is signed incorrectly or damaged, or that might be malicious software from an unknown source.

10
5

The message resource is present but the message was not found in the message table

but this error was already present before the Log Analytics problem started.

The error we got when the problem started appeared every hour for approximately a day and then stopped :

Event ID 4007

xxxxxxxx-wwww-yyyy-zzzz-pppppppp.oms.opinsights.azure.com
https://xxxxxxxx-wwww-yyyy-zzzz-pppppppp.oms.opinsights.azure.com/ConfigurationService.Svc/GetCurren...
3448b17f-9a23-4970-c163-bdd85a2e8cda

The message resource is present but the message was not found in the message table

At the moment we are still deploying the config of the Log Analytics workspace on the SH.

Marc

[#17]

We also have noticed another problem with WUfB 

None of the SH enrolled in WUfB are reporting a Last Scan Time and also, the status is always 'Up to date' when it is not true. As you can see on the screenshot, 2 SH, with the 20H2 update, one reporting quality update verion 928 and another one reporting 804 are, according to MS, up to date.
How it that possible ?

Does anyone else also have this problem ?

[#18]

We had the same eventlog message (about the digital signature) in october.

I didn't see it again after we updated to 20H2, but I will have a closer look.

We don't use WUfB and also have different software versions, altough I visited all of the surface hubs shown below last friday and started the search for update process manually.

We see another strange behaviour if I deploy a configuration profile to a Surface Hub.

On devices with manually entered OMS config we get an error for deploying any kind of configuration profile.

If I assign multiple configuration profiles, one gets the error above and the other ones are in failed state.

Every single configuration sets the configurated parameters successfully but failes on the Workspace Key.

We don't have the problem on devices where OMS is not configured.

I don't know if those problems are related, but at least it is all about a "Workspace Key" and only happens if OMS is enabled.

As Intune is already collecting some information about the devices, I thought it would be worth a try, to enable this feature: https://docs.microsoft.com/en-us/mem/intune/fundamentals/review-logs-using-azure-monitor

I enabled all logs including "Devices", but that only led to a new configuration file that failed on all devices.

[#19]

@Thomas Forsmark Sørensen 

I have juste received a mail from MS.

They are closing the case, the issue is now documented on their known issues page for update 2020.

They are still working to identify the cause.

Not happy at all....

1. closing a case that is still ongoing, why ?    must be bad for their solved cases/time stats.

2. they are documenting the problem as a result of the 2020 update which is totally wrong, the problem occured before that.

[#20]

@Marc Vanderhaegen
We created a Microsoft support case on the problem and they have now reported back that they have identified and fixed the problem. So according to Microsoft it should be working again now.
Can you confirm that?