Home

BRK2004 - Learn from large scale Skype for Business Voice deployments

BRK2004.PNG

 

Session information and recording with slide deck can be found on MyIgnite


Francois Doremieux opened up the session with some impressive statistics about Skype for Business (based on third party data):

  • About 15 million Skype for Business Enterprise Voice monthly active users
  • 92% of US EPG have or plan to deploy Skype for Business Enterprise Voice (33% have and 59% plan)
  • Microsoft is preferred as PBX vendor over Cisco by more than a 2:1 ratio
  • Lync/Skype for Business has 35% of the World wide Enterprise Voice lines shipped vs. 15% for the closest competitor (Cisco), who has been seeing its share decline.

 

Francois confirmed that interest in the room by asking the audience to raise their hand if they were contemplating or just starting a PBX replacement project using Skype for Business. Out of the 400 attendees, I would say that more than 3 quarters raised their hand!

 

Francois then discussed how Microsoft turned Skype for Business into a proven, mature, complete and comprehensive PBX replacement solution: “I am confident that we can succeed at any PBX replacement project; PBX replacement projects are complex and can be difficult, but I have yet to encounter a solution where the combination of Microsoft and our ecosystem was not able to meet the customer’s needs”.

 

In support of that claim, Francois asked four companies to describe their Large Enterprise Voice Deployments. It was clear when these companies shared the details, that they were all convinced they had made the right choice with Skype for Business Server. Still, these customers presented complex use cases, much beyond the basic and simple “white collar” use case; these deployments were far from straightforward. Tremendous work and dedication were all a part of each of these deployments, and each was unique in its own way.

 

 

BNSF Railway

 

BNSF Railway, represented by Silvina Petersilge, was first to talk about their deployment. They currently have 170+ Multivendor PBX sites with 30,000 digital and analog stations. The main driver for the project is their aging PBX infrastructure some of which are older than their youngest train engineers. Having moved 7 sites already with 4,700 Skype for Business voice lines, 300 faxes and 50 analog voice lines, BNSF still has a lot of work to complete. On top of consolidating PRIs to centralized SIP trunks and implementing a company-wide e911 system, BNSF plans to migrate the remaining sites through 2021, at which point all the PBXs will have been decommissioned and BNSF will be completely running on Skype for Business Server Enterprise Voice.

 

Along the way there were some challenges. BNSF highlighted vendor interop at the top of their list citing BTOE (Better Together over Ethernet), call recording and support for analog devices. True to the saying "The Railroad Never Sleeps", they also highlighted challenges patching in a 24x7 operation, stating that early on it was quite unsettling for users when their phones briefly lost service during the patching process. For their successes, BNSF highlighted that the users have really taken to desktop sharing and online meetings. They also have some users migrating to just headsets after a training program highlighted the benefits of being able to work from anywhere. BNSF added that they are seeing a continued increase in video usage.

 

Key to their successful rollouts so far have been usage of tools that give an end to end view of the network. BNSF emphasized in their lessons learned that following Microsoft's specs and baseline security recommendations are key to a successful rollout, especially when rolling out Virtual Machines.

 

 

Cargill

 

Cargill is a provider of food, agricultural, financial and industrial products. They are the largest privately owned company with 1200+ locations, operating in 70 countries. Currently Cargill has 160,000 employees, and they have been in business for 152 years.

 

Joe Milledge shared very impressive numbers about Skype for Business Enterprise Voice at Cargill:

  • 5 Pools Globally
  • 80,000 IM&P Clients
  • 8,000 Enterprise Voice users currently and 13,000 planned by July 1, 2018
  • 25 million minutes per month (using analytics to target new potential users)
  • Federated with 400+ companies
  • Installed 8 Skype native immersive Room systems in the last year (that have already paid for themselves)

 

Cargill’s voice progression started with proof of concept back in 2011. Started with low risk and low cost sites in 2012 and then proceeded to roll out in their "Open Office" environments at their headquarters in 2015. In 2017, they rolled out Skype for Business to 9 Large Plant locations replacing Avaya PBXs between March 2017 and July 2017. Total numbers for those 9 Large Plants were 3000 users, 1000 analog, 600 Common Area Phones and 20 Response Groups.

 

Key to the Cargill strategy was to utilize select employee champions to help with change management. This allowed for fewer IT resources when rolling out Enterprise Voice and was one of their key lessons learned. In total for each of their most recent Large Plant Conversions they had 4 people on-site, 2 people off-site on top of the assistance from the employees. Cargill also brought in AudioCodes to come on-site and augment the on-site staff.

 

Cargill found that 90% of the time there was no need to perform station reviews. One item that really helped was to set expectations with users regarding the fact that they were getting a new generation phone system that was 18 years newer than what they had before and it will function differently in some cases. They also hedged their deployment by connecting PRI to old PBX for a period of time to make sure nothing was missed or to allow for transition in chunks/sections rather than the entire site.

 

A gotcha that Cargill came across was the amount of time needed to setup shared phones for UM and Common Area Phones. They also noted that Response Groups were needed to perform basic hunting function. Cargill deployed high density gateways and found they were easy to register phones, but preferred to do analog administration in GUI in SfB. Cargill also found that BToE was difficult for IT to manage.

 

  

Bosch

 

Bosch has four different business sectors, with more than 440 subsidiary companies and regional subsidiaries in some 60 countries. Bosch started it's rollout in 2014 and has rolled out Skype for Business to 304,000 users with 210,000 of those active on a daily basis. Of those users rolled out, 74,374 are enabled for Enterprise Voice with plans to grow to 150,000 in the immediate future. Those users generate about 5.9 million minutes of PSTN calls per month. Bosch currently has 10 pools worldwide and plans to grow to 12.

 

Jasmin Gokovic also provided some stats on their conferencing usage racking up about 44,000 conference calls per day and 71 million minutes a month of usage. Key to Bosch's strategy is that they have developed a factory like rollout procedure with many vendors and internal departments involved. Bosch has partnered with AudioCodes and Avaya. Avaya has helped with speeding up the Enterprise Voice rollouts. Contact Center wise, Bosch is looking for a Call Center partner on Skype for Business, currently all of their contact centers are still being served by Avaya PBXs. Bosch is also looking at doing a pilot for Microsoft Teams. 

 

Overall Bosch has 3 hubs that have a DMZ, SIP Trunks, and SBCs servicing their users. They also stated they have open federation with 695 different federated domains.

 

 

ZF

 

ZF is a global leader in driveline and chassis technology as well as active and passive safety technology. ZF is running their Skype for Business system in Hybrid mode. There are 61,000 users on-premises of which 16,000 are using Enterprise Voice. ZF also have 33,000 users without Enterprise Voice homed to Office 365.

 

Eugene Kozoil provided details on what their sites looked like before migration. EMEA region had about 30,000 users on Siemen's HiPath and Unify Openscape. They had contact center, switchboard/attendant users, Boss/Admin scenarios, 1000s of analog devices, door openers, security intercoms and 10,000 DECT based extensions.

 

In North America, Cisco CUCM was deployed for about 4,000 users. In that region they also had, Cisco Contact Center Express, Jabber, Intercoms, and 1,000s of analog devices. Central America had about 1,000 users with a combination of Avaya and Nortel PBXs. Lastly, Asia and South America had about 7,000 users and used a combination of Siemen's Hipath and Unify Openscape Voice.

 

ZF started their rollout in March 2015 but there was a 4-5 year time period where they conceptualized the project and then 8-9 months of laying out the infrastructure. ZF completed the European sites in May of 2017 and have now kicked off their Phase 2 rollouts (January 2017) in Mexico, UK, Brazil, Thailand and India.

 

ZF has plans to Pilot CloudPBX, now known as Phone System and utilize Cloud Connector Edition as part of that Pilot. They also plan to pilot Microsoft Teams as well.

 

ZF stated reasons for their success included an already existing demand for UC and user mobility. They also devised a comprehensive communications plan with on-site and remote trainings provided to users. ZF also made sure that there was training documentation created for every feature in 7 different languages.

 

Finally ZF standardized their migration approach with an end device portfolio and defining a roles and responsibilities for migration.

 

Key challenges they had were missing PBX features such as Busy on Busy (now available), end to end monitoring and many dependencies to 3rd products such as firewalls, load balancers, proxies, databases, servers, hypervisors, end  devices, and WAN/LAN network gear and the QoS configurations on that network gear. Those dependencies demanded some evolution in maturity of operational rigor. Lessons learned for ZF had "Standardize, Standardize, Standardize" at the top of their list. The more you can standardize you can eliminate complexity in your environment. ZF tackled this during their first couple of sites.

 

 

AudioCodes

 

AudioCodes, represented by Eric Bauer, has been in the business of voice enabling Microsoft since OCS 2007 R2 days. AudioCodes works in a 100 different countries and in the last 5 years, they have deployed over 11 million VoIP gateway ports in support of SfB Enterprise Voice. They also stated that they voice enabled over 10 million Enterprise Voice seats, while also helping with analog and fax workloads. Coupled with their endpoint devices they play a critical role in filling in some of the gaps to complete the Microsoft Unified Communications ecosystem.

 

 

Q&A

 

Question for Cargill: Are the traders at Cargill using Skype for Business?

  • Cargill answer: Until SmartTap recording was deployed, Cargill would not allow the traders on Skype for Business, but now they are.

 

Question For Cargill: Is the UCC team or the Client supporting the Skype for Business platform?

  • Cargill answer: UCC team is responsible

 

Question for all: Was there a strategy for standardizing or managing codecs and bandwidth usage with relation to real-time media workloads like conferncing, peer to peer etc?

  • ZF answer: Out of the box the codecs were not modified
  • AudioCodes answer: Customers don't actively manage codecs.
  • Cargill answer: purpose built the WANs for the Codecs being used. Cargill noticed a decrease in the poor call percentage as a result of Silk.
  • ZF answer: Received feedback as to the increase in quality of the Silk codec from G.729.
  • Bosch answer: Bandwidth consumption has increased as Web conferencing was introduced in part due to higher usage. They used WebEx before in the cloud. When they did a network analysis, the bandwith increased double or triple what was experienced before as adoption increased.

 

Question for all: what is experience or challenges implementing softphones?

  • BNSF answer: Went with hard phones to help with transition. As BNSF gives users phone, they also give headset and train them on the experience. Slowly the users are transitioning to headsets. Headset with PC is much easier to support than deskphone.
  • Bosch answer: Strategy was soft client with 90% soft client and 10% deskphone, but the challenge was user acceptance. Bosch also had challenges fighting for resources on the local machine and headset providers software needing to have settings tweaked.
  • ZF answer: Anticipated 60/40 and ended up with 90% headsets. There was a delay with some site migrations because older laptops needed a hardware refresh.

 

Question for all: is anyone using VDI?

  • Bosch answer: VDI is available with a limited set of functionality. There is no enterprise voice yet.
  • ZF answer: got Citrix working but it wasn't a long term strategy for us.

 

Question for all: Is Exchange Online or on-premises used for Unified Messaging? Also are Auto Attendants used?

  • ZF answer: ZF has Exchange UM on-premises. Exchange team has started the hybrid cloud journey and has started the process of testing that out. For Auto Attendants pay attention to latency. ZF has their Exchange environment centrally located and for example some sites in Mexico experienced delays. Overall it was a great experience. ZF had concerns about WAN outages, but they worked with AudioCodes to route to a failover number to an operator extension.
  • Cargill answer: Unified Messaging is on-premises and about half their mailboxes (45,000) are in the cloud.
  • BNSF answer: Unified Messaging is in Office 365. BNSF Migrated Unified Messagine online at the same time that Exchage migrated to the cloud. Skype for Business is on-premises and Unified Messaging is in the cloud.

 

Question for all: Do you also use all the other features like whiteboard, video conferencing, file sharing and how did that affect your bandwidth?

  • BNSF answer: Our users use the online meeting but are not very keen on the video. BNSF does bandwidth calculation previous to the migration and they perform a site remediation. They make add bandwidth, make sure there is no Cat 3 wiring, etc.
  • Bosch answer: we use all those features, but one thing that helped us we introduced the Office Online server for PowerPoint presentations. That helps a lot with bandwidth consumption and they are pushing hard to educate users to present PowerPoints the right way and not use screen share.

 

Question for all: Faxing is a big issue coming from a hybrid environment. Have you had issues after porting fax machines?

  • ZF answer: We have a fax server solution we left in place. For stand alone fax systems we used AudioCodes MediaPacks. We had to play with them for a bit, but in the end it has worked out.
  • BNSF answer: There are some fax machines that are too old and we had to replace them. So far we've been able to make faxes work without much challenges.
  • Bosch answer: We left faxes out of Skype for Business and routing through AudioCodes.

 

Question for all: What is used for end-to-end monitoring for call quality?

  • ZF answer: We don't have end to end monitoring. It depends on how far you want to go. You have your Lync/Skype for Business monitoring reports, which is a good place to start.  There is also Statisics Manager from Microsoft, which is another tool that gets closer to that goal. We run AudioCodes Session Experience Manager and Microsoft's CQM/CQD. At some point you have to look at VMWare and look at the server performance. The challenge is to get all that together. To start out out in one place and then dive down in the different systems.
  • Bosch answer: The closest we got was AudioCodes EMS. We have to use many many different tools.
  • Cargill answer: Sometimes we use complaints from our users like a canary in a coal mine. They tend to fix all our WAN and LAN issues because it is Skype for Business that tends to uncover those issues first.


Jonathan McKinney is an Office Server and Services MVP specializing in Skype for Business and Microsoft Teams. You can find him on Twitter at @ucomsgeek and on LinkedIn. He can also be found at ucomsGeek blog. He currently is a Senior UC Architect for Time2Market and lives in Elizabeth, CO.