Home

Incoming Webhooks stopped working?

%3CLINGO-SUB%20id%3D%22lingo-sub-769012%22%20slang%3D%22en-US%22%3EIncoming%20Webhooks%20stopped%20working%3F%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-769012%22%20slang%3D%22en-US%22%3E%3CP%3EWe%20just%20noticed%20that%20the%20incoming%20webhooks%20feature%20broke%20sometime%20around%26nbsp%3B07%2F23%2012%3A50am%20UTC.%20The%20webhook%20returns%20with%20HTTP%20200%20OK%2C%20and%20the%20only%20error%20message%20is%20a%20.NET%20exception%20message%20without%20the%20stack%20trace.%20Is%20there%20a%20schema%20change%20recently%20that%20is%20being%20rolled%20out%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EOddly%2C%20this%20only%20affected%20half%20of%20our%20configured%20webhooks.%20Newly%20created%20webhooks%20function%20normally.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CPRE%3E%24%20httpie%20--json%20-v%20POST%20%24url%20'text%3DThis%20is%20a%20test.'%3CBR%20%2F%3EPOST%20%3CREDACTED%3E%20HTTP%2F1.1%3CBR%20%2F%3EAccept%3A%20application%2Fjson%2C%20*%2F*%3CBR%20%2F%3EAccept-Encoding%3A%20gzip%2C%20deflate%3CBR%20%2F%3EConnection%3A%20keep-alive%3CBR%20%2F%3EContent-Length%3A%20111%3CBR%20%2F%3EContent-Type%3A%20application%2Fjson%3CBR%20%2F%3EHost%3A%20outlook.office.com%3CBR%20%2F%3EUser-Agent%3A%20HTTPie%2F1.0.2%3CBR%20%2F%3E%3CBR%20%2F%3E%7B%3CBR%20%2F%3E%22text%22%3A%20%22This%20is%20a%20test.%22%3CBR%20%2F%3E%7D%3CBR%20%2F%3E%3CBR%20%2F%3EHTTP%2F1.1%20200%20OK%3CBR%20%2F%3ECache-Control%3A%20no-cache%3CBR%20%2F%3EContent-Encoding%3A%20gzip%3CBR%20%2F%3EContent-Type%3A%20text%2Fplain%3B%20charset%3Dutf-8%3CBR%20%2F%3EDate%3A%20Tue%2C%2023%20Jul%202019%2005%3A54%3A04%20GMT%3CBR%20%2F%3EExpires%3A%20-1%3CBR%20%2F%3EPragma%3A%20no-cache%3CBR%20%2F%3ETransfer-Encoding%3A%20chunked%3CBR%20%2F%3EVary%3A%20Accept-Encoding%3CBR%20%2F%3EX-AspNet-Version%3A%204.0.30319%3CBR%20%2F%3EX-BEServer%3A%20CY4PR17MB0949%3CBR%20%2F%3EX-BackEndHttpStatus%3A%20200%3CBR%20%2F%3EX-BackEndHttpStatus%3A%20200%3CBR%20%2F%3EX-CafeServer%3A%20CY4PR13CA0042.NAMPRD13.PROD.OUTLOOK.COM%3CBR%20%2F%3EX-CalculatedBETarget%3A%20CY4PR17MB0949.namprd17.prod.outlook.com%3CBR%20%2F%3EX-CalculatedFETarget%3A%20CY4PR13CU002.internal.outlook.com%3CBR%20%2F%3EX-FEProxyInfo%3A%20CY4PR13CA0042.NAMPRD13.PROD.OUTLOOK.COM%3CBR%20%2F%3EX-FEServer%3A%20CY4PR13CA0042%3CBR%20%2F%3EX-FEServer%3A%20HK2PR02CA0141%3CBR%20%2F%3EX-MSEdge-Ref%3A%20Ref%20A%3A%2046BA11E2F05D4765AAE6D7854B45323B%20Ref%20B%3A%20TPE30EDGE0217%20Ref%20C%3A%202019-07-23T05%3A54%3A05Z%3CBR%20%2F%3EX-Powered-By%3A%20ASP.NET%3CBR%20%2F%3EX-RUM-Validated%3A%201%3CBR%20%2F%3Erequest-id%3A%2019c4e43f-6d27-4e2d-a5c1-75cc15f6b6ba%3CBR%20%2F%3E%3CBR%20%2F%3ESystem.Reflection.ReflectionTypeLoadException%3A%20Unable%20to%20load%20one%20or%20more%20of%20the%20requested%20types.%20Retrieve%20the%20LoaderExceptions%20property%20for%20more%20information%3C%2FREDACTED%3E%3C%2FPRE%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-769012%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EMicrosoft%20Teams%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-797553%22%20slang%3D%22en-US%22%3ERe%3A%20Incoming%20Webhooks%20stopped%20working%3F%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-797553%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F380525%22%20target%3D%22_blank%22%3E%40jchong_netbase%3C%2FA%3E%26nbsp%3BI'm%20having%20the%20same%20issue.%26nbsp%3B%20Just%20stopped%20working%20on%208%2F1%20for%20no%20reason.%26nbsp%3B%20No%20changes%20on%20our%20side%20or%20in%20our%20Office%20365%20account%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-799454%22%20slang%3D%22en-US%22%3ERe%3A%20Incoming%20Webhooks%20stopped%20working%3F%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-799454%22%20slang%3D%22en-US%22%3E%3CP%3EHi%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F380525%22%20target%3D%22_blank%22%3E%40jchong_netbase%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3Ewe%20had%20a%20seemingly%20similar%20issue%2C%20maybe%20what%20helped%20us%20might%20help%20you%2C%20too%20%3A)%3C%2Fimg%3E%3C%2FP%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2FMicrosoft-Teams%2FWebhook-notifications-stopped-working-around-August-1st%2Fm-p%2F794477%2Fhighlight%2Ftrue%23M36387%22%20target%3D%22_blank%22%3Ehttps%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2FMicrosoft-Teams%2FWebhook-notifications-stopped-working-around-August-1st%2Fm-p%2F794477%2Fhighlight%2Ftrue%23M36387%3C%2FA%3E%3C%2FP%3E%3CP%3E(We%20didn't%20make%20any%20manual%20changes%20to%20our%20O365%2C%20tenant%20or%20anything%20beforehand%20either)%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-799675%22%20slang%3D%22en-US%22%3ERe%3A%20Incoming%20Webhooks%20stopped%20working%3F%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-799675%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F387841%22%20target%3D%22_blank%22%3E%40Ines_Trenz_HIRSCHTEC%3C%2FA%3E%26nbsp%3BAfter%20digging%20around%20it%20looks%20like%20Incoming%20Webhook%20(and%20all%20apps)%20are%20added%20to%20the%20account%20level%20O365%20blacklist.%26nbsp%3B%20I%20logged%20into%20the%20Office%20admin%20-%26gt%3B%20Admin%20Centers%20-%26gt%3B%20Teams%20-%26gt%3B%20Teams%20Apps%20-%26gt%3B%20Permission%20Policies%20-%26gt%3B%20Org-wide%20app%20settings%20-%26gt%3B%20Blocked%20apps%2C%20and%20click%20X%20on%20the%20Incoming%20Webhook.%26nbsp%3B%20After%20this%20I%20needed%20to%20wait%20~2-4%20hours.%26nbsp%3B%20It%20seemed%20that%20some%20channel%20webhook's%20were%20quicker%20to%20come%20back%20than%20others.%3C%2FP%3E%3C%2FLINGO-BODY%3E
jchong_netbase
Occasional Visitor

We just noticed that the incoming webhooks feature broke sometime around 07/23 12:50am UTC. The webhook returns with HTTP 200 OK, and the only error message is a .NET exception message without the stack trace. Is there a schema change recently that is being rolled out?

 

Oddly, this only affected half of our configured webhooks. Newly created webhooks function normally.

 

$ httpie --json -v POST $url 'text=This is a test.'
POST <redacted> HTTP/1.1
Accept: application/json, */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 111
Content-Type: application/json
Host: outlook.office.com
User-Agent: HTTPie/1.0.2

{
"text": "This is a test."
}

HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Encoding: gzip
Content-Type: text/plain; charset=utf-8
Date: Tue, 23 Jul 2019 05:54:04 GMT
Expires: -1
Pragma: no-cache
Transfer-Encoding: chunked
Vary: Accept-Encoding
X-AspNet-Version: 4.0.30319
X-BEServer: CY4PR17MB0949
X-BackEndHttpStatus: 200
X-BackEndHttpStatus: 200
X-CafeServer: CY4PR13CA0042.NAMPRD13.PROD.OUTLOOK.COM
X-CalculatedBETarget: CY4PR17MB0949.namprd17.prod.outlook.com
X-CalculatedFETarget: CY4PR13CU002.internal.outlook.com
X-FEProxyInfo: CY4PR13CA0042.NAMPRD13.PROD.OUTLOOK.COM
X-FEServer: CY4PR13CA0042
X-FEServer: HK2PR02CA0141
X-MSEdge-Ref: Ref A: 46BA11E2F05D4765AAE6D7854B45323B Ref B: TPE30EDGE0217 Ref C: 2019-07-23T05:54:05Z
X-Powered-By: ASP.NET
X-RUM-Validated: 1
request-id: 19c4e43f-6d27-4e2d-a5c1-75cc15f6b6ba

System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information

 

3 Replies

@jchong_netbase I'm having the same issue.  Just stopped working on 8/1 for no reason.  No changes on our side or in our Office 365 account

Hi @jchong_netbase 

we had a seemingly similar issue, maybe what helped us might help you, too :)

https://techcommunity.microsoft.com/t5/Microsoft-Teams/Webhook-notifications-stopped-working-around-...

(We didn't make any manual changes to our O365, tenant or anything beforehand either)

@Ines_Trenz_HIRSCHTEC After digging around it looks like Incoming Webhook (and all apps) are added to the account level O365 blacklist.  I logged into the Office admin -> Admin Centers -> Teams -> Teams Apps -> Permission Policies -> Org-wide app settings -> Blocked apps, and click X on the Incoming Webhook.  After this I needed to wait ~2-4 hours.  It seemed that some channel webhook's were quicker to come back than others.