Forum Discussion
Welcome everyone to the Skype for Business Server 2019 Preview Forum!
Let me provide you with our setup:
Like with Skype for Business Server 2015 we deployed pools in two sites (Denmark (pri) and U.S. (sec))
Denmark pool:
3 Front Ends colocated with Mediation (prisiteFE01.domain.com, prisiteFE02..., prisiteFE03...)
1 SQL Back-End (with CMS DB, pool DBs and Monitoring DB) (prisiteSQLBackEnd.domain.com)
U.S. pool:
3 Front Ends colocated with Mediation (secsiteFE01.domain.com, secsiteFE02..., secsiteFE03...)
1 SQL Back-End (with pool DBs only) (secsiteSQLBackEnd.domain.com)
so US pool sends monitoring data to Denmark pool.
Like mentioned this is the same setup like we had in Skype 2015, we opened same ports, it's configured same and of course part of same topology.
I've been digging into problem nr 2 for few hours yesterday and it seems this is not ocurring constantly, only once in a while when a call completes in US it tries sending QoE metrics and times out and (as I read "by design") does not retry this ever again, jut goes to the next job. Sometimes it times out sometimes it does not. What is the timeout time exactly, do you know?
Firewall (Windows and Cisco) is configured same like it was for Skype 2015 so should be no issues there.
I managed to capture CLS logs when timeout happened and yeah, it shows that it sends the metrics (as it is SIP, it sends through the pool) but does not receive answer 202 OK, in all traffic captured when this one timeout occurs, this is the only SERVICE Out message that does not receive return message 202 OK, so it logs timeout.
I have NetMon logs, but will have to show it to a network guy to really understand it. I only see that one front end in US pool sends the metrics to another front end in the US pool (why?), maybe this other front end is then responsible for sending the metrics to Monitoring DB in pri site?
All servers ping DCs:)
In regards to issue nr1, I am still trying to reproduce, like I said it works sometimes for a few days then suddenly has a problem for 2-3 minutes, then back to normal, but nothing logged in SCOM from today yet.
So, I know more now, after understanding the flow of traffic.
It maybe that there are too many hops between new Skype 2019 pool in US and monitoring DB in Denmark.
To make it simple. User that made a call/received a call is still on Skype 2015, but for all mediation traffic it's Skype 2019 that is the primary pool now:
1. Skype 2019 in US initiates and controls the call until it ends
2. Since user calling/called is still on Skype 2015 in US, it sent the QoE metrics from First 2019 FE to Third 2019 FE in US (Hop 1)
3. Third 2019 FE sends QoE metrics to Second 2015 FE in US (Hop 2)
4. Second 2015 FE in US sends QoE metrics to whichever (don't have logs from this pool) 2015 FE in DK (Hop 3)
5. 2015 FE in DK sends data to Skype 2015 Monitoring DB
Since many calls (tens or hundreds) are happening on the mediation pool, and only few QoE metrics don't get delivered to the monitoring DB, it seems not to be a big issue and might be related to network latency since so many hops are in the path.
But the problem described as Nr 1 (with failing connection to backend) and PowerPoint sharing problems (let's call it problem nr 3) in Skype 2019 are still there. Apart from that we noticed another problem in Skype 2019 (let's call it problem nr 4):
4a) It starts with LS Protocol Stack Warning (EventID 14397):
Transport:TLS, IP address:0.0.0.0, Port:5061, Error:0xC3E93C0D(SIP_E_STACK_TRANSPORT_CERT_NOT_FOUND).
Ensure that a valid certificate is present in the local computer certificate store. Also ensure that the server has sufficient privileges to access the store.
Resolution:
Review and correct the certificate configuration, then start the service again.
Hi, Krzysztof, Did you ever find out the root cause of this problem. I have this issue on SFB Server 2015 latest CU.
Front End Service with LS Server ERROR (EventID 12303):The protocol stack reported a critical error: code 0xC3E93C0D (SIP_E_STACK_TRANSPORT_CERT_NOT_FOUND). The service has to stop.Did you ever find a solution for this? We're seeing identical behavior and Lync event log entries across all of our FE's (Skype 2019, Server 2016).
Thanks!
"4a) It starts with LS Protocol Stack Warning (EventID 14397):
A configured certificate could not be loaded from store. The serial number is attached for reference.Extended Error Code: 0x80092004(CRYPT_E_NOT_FOUND).4b) it then gets LS Protocol Stack ERROR (EventID 14623):A serious problem related to certificates is preventing Skype for Business Server from functioning.Unable to use a certificate as configured.
Transport:TLS, IP address:0.0.0.0, Port:5061, Error:0xC3E93C0D(SIP_E_STACK_TRANSPORT_CERT_NOT_FOUND).
Ensure that a valid certificate is present in the local computer certificate store. Also ensure that the server has sufficient privileges to access the store.Cause: The Skype for Business Server failed to initialize with the configured certificate.
Resolution:
Review and correct the certificate configuration, then start the service again.4c) then it goes into shutting down the Front End Service with LS Server ERROR (EventID 12303):The protocol stack reported a critical error: code 0xC3E93C0D (SIP_E_STACK_TRANSPORT_CERT_NOT_FOUND). The service has to stop.4d) it continues with series of few warnings and erros about not being able to open connection to Storage Service (LS Data Collection EventID 56726), and then after 3 minutes it starts working again and service is brought back up, and no more issues connecting to Certificates store."No, although it seems to happen very rarely now, we are running CU3 from March 2020, are you?
But like mentioned, it does happen from time to time, dunno why 😞
We're on CU2 from December. We checked CU3 but it seemed like the updates listed had nothing to do with our current situation. Like everyone else i suppose we just need to be patient. The world is not running optimally to say the least. Stay safe!
TomCullenI also see this on Skype for Business 2015 on Server 2016, exactly the same behaviour. No solution as of yet, although for me it appear to be happening more sporadically.
It's sporadic for us too; happens about every 2 - 7 days. can't predict when it'll happen again or on which FE. we've captured the event using cslogger, but so far MS hasn't come back with a solution or explanation. very frustrating these days especially when everyone is working from home. skype is more critical than ever and we can't afford to have meetings disrupted randomly/unexpectedly.
Krzysztof Sienkiewicz So far, I have not had the four questions you mentioned in my environment. I will continue to observe and respond to you in time, thank you.
Just to let you know.
Our Skype 2019 and SQL 2016 environment is built on Windows Server 2019
Office Online Server (latest build with latest updates) is built on Windows Server 2016
Krzysztof Sienkiewicz sorry to replay to you so late, I met an error witch occurred on one of the 3 SFB 2019 FE servers is very similar to your problem nr 4 ( Apart from that we noticed another problem in Skype 2019 ,let's call it problem nr 4).You can get the details from attach file.I am looking forward to find the real reason which lead to this.Any news I found,I will contact you.The only difference is my cert's error is "Untrusted Root" not "Not found".
Yes, multiple times on multiple occasions.
Did you check certificates of your 2019 pools?
I find some experience is similar to yours, please refer to below link and wish it will works.
http://terenceluk.blogspot.com/2016/12/skype-for-business-server-access-edge.html
Krzysztof Sienkiewicz Can you upload the CLS logs to some location and share the link?
For which of my described problems? Cause the one with QoE metrics, like I said, probably too many hops and after investigation it seems to be happening only sporadically, probably will go away when we move users to new Skype pool, if not, I will get back on that. Other problems I described are more of a show-stopper for us (PowerPoint presenting, services going down because Cert Store cannot be opened, etc.)
- Please post for the showstopper problems, along with detailed repro steps.
