Incoming Webhooks stopped working?

Copper Contributor

We just noticed that the incoming webhooks feature broke sometime around 07/23 12:50am UTC. The webhook returns with HTTP 200 OK, and the only error message is a .NET exception message without the stack trace. Is there a schema change recently that is being rolled out?

 

Oddly, this only affected half of our configured webhooks. Newly created webhooks function normally.

 

$ httpie --json -v POST $url 'text=This is a test.'
POST <redacted> HTTP/1.1
Accept: application/json, */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 111
Content-Type: application/json
Host: outlook.office.com
User-Agent: HTTPie/1.0.2

{
"text": "This is a test."
}

HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Encoding: gzip
Content-Type: text/plain; charset=utf-8
Date: Tue, 23 Jul 2019 05:54:04 GMT
Expires: -1
Pragma: no-cache
Transfer-Encoding: chunked
Vary: Accept-Encoding
X-AspNet-Version: 4.0.30319
X-BEServer: CY4PR17MB0949
X-BackEndHttpStatus: 200
X-BackEndHttpStatus: 200
X-CafeServer: CY4PR13CA0042.NAMPRD13.PROD.OUTLOOK.COM
X-CalculatedBETarget: CY4PR17MB0949.namprd17.prod.outlook.com
X-CalculatedFETarget: CY4PR13CU002.internal.outlook.com
X-FEProxyInfo: CY4PR13CA0042.NAMPRD13.PROD.OUTLOOK.COM
X-FEServer: CY4PR13CA0042
X-FEServer: HK2PR02CA0141
X-MSEdge-Ref: Ref A: 46BA11E2F05D4765AAE6D7854B45323B Ref B: TPE30EDGE0217 Ref C: 2019-07-23T05:54:05Z
X-Powered-By: ASP.NET
X-RUM-Validated: 1
request-id: 19c4e43f-6d27-4e2d-a5c1-75cc15f6b6ba

System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information

 

14 Replies

@jchong_netbase I'm having the same issue.  Just stopped working on 8/1 for no reason.  No changes on our side or in our Office 365 account

Hi @jchong_netbase 

we had a seemingly similar issue, maybe what helped us might help you, too :)

https://techcommunity.microsoft.com/t5/Microsoft-Teams/Webhook-notifications-stopped-working-around-...

(We didn't make any manual changes to our O365, tenant or anything beforehand either)

@Ines_Trenz_HIRSCHTEC After digging around it looks like Incoming Webhook (and all apps) are added to the account level O365 blacklist.  I logged into the Office admin -> Admin Centers -> Teams -> Teams Apps -> Permission Policies -> Org-wide app settings -> Blocked apps, and click X on the Incoming Webhook.  After this I needed to wait ~2-4 hours.  It seemed that some channel webhook's were quicker to come back than others.

I have checked the Org-wide app settings and there are no blocked apps. 

Webhooks were working properly until 10/04/2019 and just noticed that they are not working today 10/07/2019.

The POST request returns 200 OK, but with the following message: System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.

 

Thanks!

Hello
Did u solve this problem?
I have same issue like u.
The most interest thing is that other channel's connector doesn't have any problem.
Specify channel connector has that issue.

@kains4  Even though no one at our company had made any changes to our O365 Teams settings, I had to go into the O365 Admin, go to Teams Admin Center, click Teams Apps, Permission Policies, Org-Wide App Settings, and unblock the  Incoming Webhook.  It seems like MS might have rolled out a massive change.

 

One thing I wish is that I could get an email or subscribe to a feed of any changes both in the future and when they are rolling out.  We're building a number of internal apps around Teams and it would be nice to know if MS is making changes that will break our apps.  In this case we stopped getting internal notifications and thought everything was fine.

@kains4  I just checked this morning and they connectors are working again.

Still broken for me :( @tmorell 

I have de same problem from 10/04/2019 but only happen with one webhook
I have the same problem from 07/10/2019 just with one team/channel

I have same problem.

If I sent same HTTP requests to the same Incoming Webhook, some sent successfully and others failed with exceptions.

Teams Connector seems unstable these days. Connector application list from channel menu is not displayed sometimes and Connectors Developer Dashboard is sometimes unavailable.

@rgcejudo @EduDom @thonda @jchong_netbase 

 

Message I got from MS

 
 

Thank you for providing the details to us.

 

We have analyzed the details and isolated that it is the tenant specific issue.

 

MS created an incident

 

Service Incident TM192739

 

Title: Webhook notification issue 

@tmorell I also found that the connector had a problem was working.

But I didn't do any action for fixing this issue.

I think that this issue just depend on ms service.