Migrating to the new web services based publishing for Office 365 IP Addresses and URLs

[Paul Andrew]

We announced in April that the Office 365 IP Addresses and URLs page was being replaced with access to the data from new web services. See http://aka.ms/ipurlblog for the announcement. I’ve heard a few requests for guidance on how to migrate to the new web services.

We publish network endpoint data for Office 365 across the five service instances and each has separate network connectivity. They are:

• Office 365 worldwide commercial
• Office 365 US Government GCC High
• Office 365 US Government DoD
• Office 365 Germany
• Office 365 operated by 21Vianet

For each of these we publish am HTML page with tables that list the URLs and IP Address ranges that must be accessible. We also publish an XML file for each and have an RSS feed which is updated to show changes.

These are all planned to be deprecated and no longer updated from October 2^nd, 2018. Depending on how you use the current data, you will have a different recommended path for migrating to the new web services which provide the data.

The new web services provide IP Address and URL network endpoint data in JSON format along with other attributes that were previously only available in the HTML tables and some new attributes. We will be providing new published HTML tables and RSS feeds that are based off the web services data.

Here are some existing scenarios where customers use the data along with how we recommend migrating to the new web services.

Using the XML file for firewall or proxy server configuration using a script

The XML files that we publish will no longer be updated from October 2^nd, 2018 so it’s particularly important to migrate off them. Typically, if you have script that is downloading these files it will be necessary to update it to use the JSON data from the web services. Although these are different formats, most scripting languages have native support for both, so the update should be straight forward.

We provide sample scripts for accessing the data in PowerShell and Python in the web services user guide.

Screen scraping data attributes from the HTML web page

Although Microsoft doesn’t support screen scraping of the HTML published data we understand that some customers do this to get additional attributes about Office 365 network endpoints from the HTML tables. The good news is that you don’t need to do this any longer. All supported attributes are available in the new web services so there will be no longer any need for screen scraping of the HTML tables. Please discontinue any screen scraping.

Classifying network traffic by the Office 365 application for reporting purposes

Some customers look at the destination IP Address and map it to the Office 365 Product name listed in the XML file. This Product list has inconsistencies, is missing dependencies, has out of date names and we do not support selection of individual Office 365 applications by network endpoint so we are removing it. We are replacing this with a shorter list of Service Areas for which selective enablement is supported. There are three service areas which can be selected independently and a fourth common service area. This table shows the mapping of the old Product to the new Service Areas. Two of the old Product names no longer exist and are shown mapped to <Removed>.

Old XML Product	New JSON ServiceArea
WAC	Common
Sway	Common
Planner	Common
ProPlus	Common
Ex-Fed	<Removed>
Yammer	Common
Teams	Skype
OfficeiPad	Common
OfficeMobile	Common
RCA	<Removed>
OneNote	Common
EXO	Exchange
SPO	SharePoint
Office365Video	Common
LYO	Skype
Identity	Common
CRLs	Common
o365	Common
EOP	Exchange

Although we recommend enabling all service areas, you can select which service areas to return data when calling the web services. The Common service area is always returned since the other service areas have a dependency on it. The specific service area name is also included in the web services attributes returned. Here’s an example PowerShell script command to get endpoint data for the Exchange Online service area and the Common service area:

PowerShell command getting endpoints 

Using the RSS publishing to have changes reviewed and then applied to networking devices

We will be replacing the current RSS feed so that customers who are familiar with using an RSS reader to get notification of changes can still do so. The old RSS feeds will all be removed after October 2^nd, 2018. The URLs for each RSS feed will be changing, and the structure of the text returned, and the number of updates will be improved on.

You can also use Microsoft Flow to create review and update processes to respond to IP Address and URL changes. We’re working on an article which demonstrates how to set this up. I’ll update this page with a link to that article once it is available.

Just searching for an IP Address

If you use the IP Addresses and URLs page just for searching for an IP Address that you see in a log file, then you can continue to do that. The current HTML tables of the network endpoints are manually edited, and we are going to be removing them. We will republish the HTML tables as generated from the web services. You will still need to search for the IP Address network (or CIDR) that contains the IP Address you want.

Summary

The new web services are improved over the old publishing, but there will still be some migration work required. The above recommendations are intended to help you to plan for the changes.

How do you use the existing Office 365 network endpoint data? Are there any other scenarios that you are using the existing published IP Address and URL data for? Let us know.

[#2]

Paul,

Reaching out to you as I couldn't find another avenue to ask about this. We've started using rest-based clientrequestID method of retrieving the IPs and URLs necessary for us to create firewall rules in our environment. Everything has been going well with the exception that sometimes secure.aadcdn.microsoftonline-p.com is blocked because the IP it is resolving to an IP that is not on the most up to date list. Examples would be 23.57.51.177 or 104.100.70.5.

Am I missing something? Should those IPs be on the list?

[#3]

Hello @Carlos Costa,

Taking a look for secure.aadcdn.microsoftonline-p.com I can see that it is in the Default network endpoint category. These two lines of PowerShell can be used to query it.

$e = invoke-restmethod -Uri ("https://endpoints.office.com/endpoints/WorldWide?noipv6&clientrequestid=" + ([GUID]::NewGuid()).Guid)
$e | Where-Object { "secure.aadcdn.microsoftonline-p.com" -in $_.urls }

We have a policy of not providing IP Addresses for Default category network endpoints and recommend that you direct network traffic for them to your default Internet egress point. No firewall rules should be required just as no firewall rules are required for other proxied connections.

Regards,

Paul

[#4]

I'm trying to understand the meaning of the following which is:

The ip set "13.70.151.216/32" (along with several others) appears in the results of the https://endpoints.office.com/changes/worldwide/0000000000?clientrequestid=b10c5ed1-bad1-445f-b386-b9... multiple times as a "remove ips" "remove" disposition type of update associated with a "urls" of "*.dc.trouter.io" in one update and associated with a "urls" of [*.skype.com","*.teams.microsoft.com", and "teams.microsoft.com"]  in a second "remove ips" "remove" disposition type of update and in a 3rd "remove ips" type of "change" disposition update but associated with no urls . In all three updates,  the "previous expressroute" state is listed as "true" but the "current expressroute" state is "false".

This is compared to the endpoints list (https://endpoints.office.com/endpoints/Worldwide?ClientRequestId=b10c5ed1-bad1-445f-b386-b919946339a... )  where i still see that ip listed multiple times in the endpoints list with ExpressRoute=true and i also see it listed in the following places (https://docs.microsoft.com/en-us/office365/enterprise/urls-and-ip-address-ranges?redirectSourcePath=... ) , https://support.office.com/en-us/o365ip/rss  and https://support.content.office.net/en-us/static/O365IPAddresses.xml  .

Do the updates in the changes results mean that the ip set is 1) no longer serviced by  Express Route and 2) no longer associated with the urls mentioned but still used by o365 services but not via ER and are the sites i listed that seem to still indicate that the ip set is both express routeable and associated with the urls mentioned just not in sync because they are being deprecated ?

Can you please clarify the intent of the updates mentioned (just search for 13.70.151.216/32 to see all the ones i was referencing) ?

Thanks

[#5]

What is the URL for new RSS feed? is it published or not yet??

[#6]

Hi @Abhinay Sharma, it's documented at http://aka.ms/ipurlblog. The Office 365 Worldwide Commercial RSS change feed is here: https://endpoints.office.com/version/worldwide?allversions=true&format=rss&clientrequestid=b10c5ed1-... 

Paul

[#7]

Hi @Paul Andrew- how to check which 0365 endpoint version we are using?

[#8]

Hi @Abhinay Sharma, not sure what you mean here. We publish the endpoints data and the version for each publishing. The version you are currently using depends on where you download the data to and how you store it. You'd need to store the version somewhere of the data you downloaded. If you use our Microsoft Flow sample, then your latest downloaded version is going to be in your SharePoint list.

Paul

[#9]

Hi @John Tullo,

The /changes output faithfully describes each and every change in the output of the /endpoints web method.

We're working on adding an additional field to the /changes output that will identify the severity of each change. Note that this IP Address is still part of Office 365 and none of these changes should be applied to a firewall as a removal. We would have some kind of change severity that indicates duplicate removal only.

Here's more detail on those three change sets for IP 13.70.151.216. 

Change Set ID 141 - this is deleting endpoint set 23 entirely. It was removed as it was a duplicate of another endpoint set ID 25. The other endpoint set is still published.

Change Set ID 152 - this is a change to endpoint set 69. We changed it to ExpressRoute not supported and removed all of the IP Addresses. This endpoint set is telemetry and is in the Default category which we recommend directing to a default Internet egress proxy server. This is an operation that does not require IP Addresses.

Change Set ID 167 - this is deleting endpoint set 24 entirely. It was also removed as a duplicate. This one is a duplicate of endpoint set ID 12.

Regards,

Paul

[#10]

Thanks Paul. That clears up my confusion.

[#11]

Hi @Paul Andrew- just need to know if the endpoint set ids are fixed or it can be change in future? like for Skype we have endpoint set id-11,12,13,14,15,16,17,18,19,20,22,25,26,27,29. So for latest changes in IPs we need to check these endpoint ids only?

[#12]

Hi @Abhinay Sharma, we wont change the endpoint set id numbers although we will add new numbers.

Regards,

Paul

[#13]

Thanks Paul

Lets say i want to get FQDN's and IP's for a single application say "YAMMER".

Documentation states - "notes - For optional endpoints, this text describes Office 365 functionality that will be missing if IP addresses or URLs in this endpoint set cannot be accessed at the network layer. Omitted if blank." 

Old XML had a product called "YAMMER" and FQDN's and IP's specific to YAMMER.

now for the achieving same effect Can i rely on notes field part of endpoints and extract FQDN's and IP's for "YAMMER" ?

And Can i assume the notes will be updated and maintained just like IPS's and FQDN's ?

[#14]

---

@jaffer.jobs jaffer.jobs wrote:

Lets say i want to get FQDN's and IP's for a single application say "YAMMER".

Documentation states - "notes - For optional endpoints, this text describes Office 365 functionality that will be missing if IP addresses or URLs in this endpoint set cannot be accessed at the network layer. Omitted if blank." 

Old XML had a product called "YAMMER" and FQDN's and IP's specific to YAMMER.

now for the achieving same effect Can i rely on notes field part of endpoints and extract FQDN's and IP's for "YAMMER" ?

And Can i assume the notes will be updated and maintained just like IPS's and FQDN's ?

---

Hi @jaffer.jobs jaffer.jobs,

We only support selective blocking of Office 365 network traffic by the ServiceArea attribute in the web service output. The previous listed products in the XML file are not supported for selective network blocking. You can read above in this article about how the old products are mapped to ServiceAreas. Separately, the notes field is provided for any Required: false endpoint sets to identify what Office 365 functionality would be missing if this endpoint set is blocked. The notes field doesn't follow any specific schema, doesn't group endpoint sets, and doesn't have an update or maintenance policy.

Regards,

Paul