Migrating to the new web services based publishing for Office 365 IP Addresses and URLs

Microsoft

We announced in April that the Office 365 IP Addresses and URLs page was being replaced with access to the data from new web services. See http://aka.ms/ipurlblog for the announcement. I’ve heard a few requests for guidance on how to migrate to the new web services.

 

We publish network endpoint data for Office 365 across the five service instances and each has separate network connectivity. They are:

  • Office 365 worldwide commercial
  • Office 365 US Government GCC High
  • Office 365 US Government DoD
  • Office 365 Germany
  • Office 365 operated by 21Vianet

For each of these we publish am HTML page with tables that list the URLs and IP Address ranges that must be accessible. We also publish an XML file for each and have an RSS feed which is updated to show changes.

 

These are all planned to be deprecated and no longer updated from October 2nd, 2018. Depending on how you use the current data, you will have a different recommended path for migrating to the new web services which provide the data.

 

The new web services provide IP Address and URL network endpoint data in JSON format along with other attributes that were previously only available in the HTML tables and some new attributes. We will be providing new published HTML tables and RSS feeds that are based off the web services data.

Here are some existing scenarios where customers use the data along with how we recommend migrating to the new web services.

 

Using the XML file for firewall or proxy server configuration using a script

The XML files that we publish will no longer be updated from October 2nd, 2018 so it’s particularly important to migrate off them. Typically, if you have script that is downloading these files it will be necessary to update it to use the JSON data from the web services. Although these are different formats, most scripting languages have native support for both, so the update should be straight forward.

 

We provide sample scripts for accessing the data in PowerShell and Python in the web services user guide.

 

Screen scraping data attributes from the HTML web page

Although Microsoft doesn’t support screen scraping of the HTML published data we understand that some customers do this to get additional attributes about Office 365 network endpoints from the HTML tables. The good news is that you don’t need to do this any longer. All supported attributes are available in the new web services so there will be no longer any need for screen scraping of the HTML tables. Please discontinue any screen scraping.

 

Classifying network traffic by the Office 365 application for reporting purposes

Some customers look at the destination IP Address and map it to the Office 365 Product name listed in the XML file. This Product list has inconsistencies, is missing dependencies, has out of date names and we do not support selection of individual Office 365 applications by network endpoint so we are removing it. We are replacing this with a shorter list of Service Areas for which selective enablement is supported. There are three service areas which can be selected independently and a fourth common service area. This table shows the mapping of the old Product to the new Service Areas. Two of the old Product names no longer exist and are shown mapped to <Removed>.

 

Old XML Product

New JSON ServiceArea

WAC

Common

Sway

Common

Planner

Common

ProPlus

Common

Ex-Fed

<Removed>

Yammer

Common

Teams 

Skype

OfficeiPad

Common

OfficeMobile 

Common

RCA 

<Removed>

OneNote 

Common

EXO 

Exchange

SPO 

SharePoint

Office365Video 

Common

LYO 

Skype

Identity

Common

CRLs

Common

o365

Common

EOP 

Exchange

 

Although we recommend enabling all service areas, you can select which service areas to return data when calling the web services. The Common service area is always returned since the other service areas have a dependency on it. The specific service area name is also included in the web services attributes returned. Here’s an example PowerShell script command to get endpoint data for the Exchange Online service area and the Common service area:

PowerShell command getting endpointsPowerShell command getting endpoints 

Using the RSS publishing to have changes reviewed and then applied to networking devices

We will be replacing the current RSS feed so that customers who are familiar with using an RSS reader to get notification of changes can still do so. The old RSS feeds will all be removed after October 2nd, 2018. The URLs for each RSS feed will be changing, and the structure of the text returned, and the number of updates will be improved on.

 

You can also use Microsoft Flow to create review and update processes to respond to IP Address and URL changes. We’re working on an article which demonstrates how to set this up. I’ll update this page with a link to that article once it is available.

 

Just searching for an IP Address

If you use the IP Addresses and URLs page just for searching for an IP Address that you see in a log file, then you can continue to do that. The current HTML tables of the network endpoints are manually edited, and we are going to be removing them. We will republish the HTML tables as generated from the web services. You will still need to search for the IP Address network (or CIDR) that contains the IP Address you want.

 

Summary

The new web services are improved over the old publishing, but there will still be some migration work required. The above recommendations are intended to help you to plan for the changes.

How do you use the existing Office 365 network endpoint data? Are there any other scenarios that you are using the existing published IP Address and URL data for? Let us know.

32 Replies

Part of our use case is checking ONLY the Updated ip addresses when there is an update. From what I can tell there is no "serviceArea" tag in the `/changes` endpoint. Is the only way to distinguish what Product is by going back to the `/endpoints` and comparing "id" from `/endpoints` to "endpointSetId" from `/changes` ?

 

for example:

this is from https://endpoints.office.com/endpoints/worldwide?ServiceAreas=Exchange&ClientRequestId=b10c5ed1-bad1...

  {
    "id": 9,
    "serviceArea": "Exchange",
    "serviceAreaDisplayName": "Exchange Online",
    "urls": [
      "*.protection.outlook.com"
    ],
    "ips": [
      "23.103.132.0/22",
      "23.103.136.0/21",
      "23.103.144.0/20",
      "23.103.198.0/23",
      "23.103.200.0/22",
      "40.92.0.0/14",
      "40.107.0.0/17",
      "52.100.0.0/14",
      "52.238.78.88/32",
      "65.55.88.0/24",
      "65.55.169.0/24",
      "94.245.120.64/26",
      "104.47.0.0/17",
      "157.55.234.0/24",
      "157.56.110.0/23",
      "157.56.112.0/24",
      "207.46.100.0/24",
      "207.46.163.0/24",
      "213.199.154.0/24",
      "213.199.180.128/26",
      "216.32.180.0/23",
      "2a01:111:f400:7c00::/54",
      "2a01:111:f400:fc00::/54",
      "2a01:111:f403::/48"
    ],
    "tcpPorts": "443",
    "expressRoute": true,
    "category": "Allow",
    "required": true
  },

 

Then if you go to https://endpoints.office.com/changes/worldwide/0000000000?ClientRequestId=b10c5ed1-bad1-445f-b386-b9... to get the latest update i can see:

  {
    "id": 3,
    "endpointSetId": 9,
    "disposition": "Change",
    "version": "2018072800",
    "remove": {
      "ips": [
        "23.103.144.0/20",
        "23.103.212.0/22",
        "40.107.128.0/18"
      ]
    }
  },

 

 

I'm just wondering if serviceArea would be something that could be added in the `/changes` endpoint to make them easier to distinguish. Or if this is the only way to separate the Products.

 

Thanks,

-Bryan

Hi @Deleted, yes, when you get the /changes data, you should also get the /endpoints data. You can lookup those other fields in /endpoints using the EndpointSetId attribute. Are you asking this question because you are retrieving the /changes data in some way that makes linking to an endpoint set difficult? What script language or client are you using?

Paul

No, Not necessarily difficult to get them that way, I was just wondering if that was the only way to distinguish between the two. But if that is how they are set up it should work for what we want to do. I just wanted to confirm that that really was the case. 

 

Another question is how accurate these CIDR's are? as we have gone through some of these ip addresses, specifically china page, we have noticed that the who is domain lookup results don't say anything about Microsoft. Some include Microsoft as a secondary owner but some do not. How accurate should these all be specific to those products?

 

Also, we will be using Java for our process.

 

Thanks,

-Bryan

 

@Deleted The IP Addresses are accurate. Office 365 in China is operated by our partner 21Vianet.

Could you please answer the below questions?

 

*Question

①”ClientRequestId” It seems that the same response as that used in the past is returned.

Do I need to use PowerShell every time Or will the update page, such as the URLs below, be available every time after October 2, 2018?

<Office 365 URLs and IP address ranges>

URL:

https://docs.microsoft.com/en-us/office365/enterprise/urls-and-ip-address-ranges?redirectSourcePath=...

 

② What is the instance? If the base in Japan, can I specify it "Office 365 worldwide commercial" of five service instances?

 

>> ER: This is Yes if the endpoint set is supported over Azure ExpressRoute with Office 365 >>route prefixes. The BGP community that includes the route prefixes shown aligns with the >>service area listed. When ER is No, this means that ExpressRoute is not supported for this >>endpoint set. However, it should not be assumed that no routes are advertised for an >>endpoint set where ER is No.

 

If the response of ”expressRoute” is ”True” in the endpoints web method,

Is the URL or IP available only on expressRoute?

 

>>However, it should not be assumed that no routes are advertised for an endpoint set >>where ER is No.

 

Assuming the above, should an endpoint be allowed or deleted even if ER is No?

 

⑤ Is it possible to get data from all the past specific time endpoint? 

     If possible, please let me know how.

 

⑥ Most of the data so far have been reflected after the disclosure of information has been activated, but is there any problem in the future as well?

 


@Kyounghwan Lee wrote:

①”ClientRequestId” It seems that the same response as that used in the past is returned.

Do I need to use PowerShell every time Or will the update page, such as the URLs below, be available every time after October 2, 2018?

Paul >>> The web page will be updated with changes.

 

② What is the instance? If the base in Japan, can I specify it "Office 365 worldwide commercial" of five service instances?

Paul >>> That is the correct one to use.

 

If the response of ”expressRoute” is ”True” in the endpoints web method,

Is the URL or IP available only on expressRoute?

Paul >>> No. All endpoints are available over the Internet.

 

Assuming the above, should an endpoint be allowed or deleted even if ER is No?

Paul >>> Allowed to do what?

 

⑤ Is it possible to get data from all the past specific time endpoint? 

     If possible, please let me know how.

Paul >>> No. Please only work with the current endpoint list. Old data should not be applied to network devices.

 

⑥ Most of the data so far have been reflected after the disclosure of information has been activated, but is there any problem in the future as well?

Paul >>> I don't understand this question, can you please explain more what you want to know?

 

Regards,

Paul


 

Thank you very much for answering all of my questions. 

We have checked your up-to-date information that it will be replaced new RSS feed and HTML tables soon, right?

we will expect for that.  


Regards,
Kyounghwan Lee

Hi @Kyounghwan Lee, the HTML tables update has already published. See http://aka.ms/o365ip. The updated RSS feed is coming soon.

 

Regards,

Paul

https://docs.microsoft.com/en-us/office365/enterprise/urls-and-ip-address-ranges?redirectSourcePath=...

 

The ID described on the Web page of the above URL is "ENDPOINTSETID". Separately from that, there is also "ID" in the CSV output data.

 

I could not find a sentence describing the ID on the CSV side.

 

I have checked, these two IDs are "the numbers are different and the contents data are also different ".

So, I am confused because these two IDs are mixed.

 

Question:

 

1) What are the differences between the two IDs? Please provide me with a detailed explanation.

 

2) Why are there two types of IDs? Please provide me with the method of using properly.

Can we have some consistency in the data please? eg. Categories returned by the endpoint web method begin with a capital letter, categories in the changes web method begin with lower case. There are also duplicate sets, and the list of "allow" and "optimize" URLs for the PAC file do not match the list in the PAC file separates required list here https://support.office.com/en-gb/article/Managing-Office-365-endpoints-99cab9d4-ef59-4207-9f2b-3728e... (broadcast.skype.com is missing but it is included in set 13)

Hi @Ian Williams, I'll get the capitalization inconsistency with categories taken care of, thanks for reporting this. We're going to replace the example PAC files with a more automated solution soon so that issue should go away. In the interim, please use the web services as the supported data.

Regards,

Paul

Hello @Kyounghwan Lee, there are three web methods which provide for CSV output and it is not clear which you are referring to. I will assume you are referring to the endpoints web method CSV which is located here: https://endpoints.office.com/endpoints/worldwide?format=csv&clientrequestid=b10c5ed1-bad1-445f-b386-...

 

If you compare that to the doc page that you referenced (here), the two ID fields refer to the same endpoint set and can be used for correlation.

 

Regards,

Paul

Great to see this @Lucian Franghiu. We see a lot of customers who completely miss changes or don't respond in time. I'd be keen to hear your feedback on what can help to get changes deployed more reliably and faster. Also take a look at my Flow for managing change which uses SharePoint Online instead of Azure blobs that you use.

Regards,

Paul 

Paul,

I agree with Bryan, It would be useful to have the serviceArea in the changes web method as if the disposition is "remove" then the endpointset may not exist in the current endpoint list.

eg. change id 141 has endpointset 23 which does not exist in the endpoint list. There are also others - change ids (146, 151, 162, 167). - Suppose it depends on when the endpoint set is removed but would be useful to know what serviceArea the change was when looking back in the history.

 

Paul,

Just to confirm how to determine future updates.

If current version is 2018083000. For future updates. eg new url will be added on 01/10/2018, will entries appear in the changes web method for 2018093000 sometime before 01/10?

 

Hi @Ian Williams, we create the version number from the date when public publishing is done. The effectiveDate attribute in the changes web method for add operations is the date that the engineering team who owns the endpoint said that the service was or will be live.

 

Can you share a scenario where the serviceArea is needed in the changes web method? If you have the previous version, then this isn't necessary as you can just look it up. If you don't have the previous version, then the changes are of no use.

 

Regards,

Paul

Thank you for your prompt reply.

 

As you mentioned, I had referred to that page.

So, let me change the question, I wonder that how the difference between the following "id" and "endpointSetId".

I wanted to know the meaning of each ID on the three reference pages below.

 

 

 

 

I have finally figured out that each ID can be a serial number or an endpointsetId.

In other words, only the ID of the pages below has the meaning of endpointsetId.

<https://endpoints.office.com/endpoints/worldwide.>

Is it correct that I have understood?

 

Also, is it correct to understand that the other IDs are used as serial numbers?

 

 

Regards,

Kyounghwan Lee

Hi @Kyounghwan Lee, the /changes/ web method has an ID which refers to the change set and not the endpoint set so that is a different number. In this image of the output from the /changes/ API you can see an id field which is the id of the change set. That is followed by an endpointSetId field which identifies the endpoint set that had the change. These two id numbers cannot be used interchangeably as they refer to different tables.2018-09-06_12-03-57.png

Regards,

 

Paul