Blog Post

Project Support Blog
6 MIN READ

Reading Project Online OData with Azure Data Factory

Brian-Smith's avatar
Brian-Smith
Former Employee
Sep 02, 2022

The first problem we hear from customers moving to Azure Data Factory (ADF), who have been using SQL Server Integration Services (SSIS) to get their Project Online OData, is that the authentication and authorization is not straightforward.  There isn't a simple choice to login to Project Online, so you have to make a call to get a token which can then be used in the REST calls to OData.  The following post steps through the process.  I'm not going deep into the details of ADF and won't cover all the steps of making an App Registration - there are plenty of resources out there, and this concentrates on the authentication then pulls in some Project level data.  It gets more complicated obviously when you also want tasks and assignments, but the same approaches used with SSIS will work just as well in ADF.

 

TL;DR - if you know all about ADF and Project Online and App Registrations and just want the auth piece - jump to the M365Login section - just about halfway down, or just take a look at https://github.com/LunchWithaLens/adf which has definitions for the whole pipeline.

 

What you will need:

  • An App Registration in Azure Active Directory that allows you to read the Project reporting data.  You will need your Tenant ID and also the Client ID and registered secret of the App Registration

The require App Registration Settings

  • A user account that just needs Access to Project Server reporting service.  You will need the account name and password.  The authentication will use the Resource Owner Password Credential (ROPC).  This method of authentication is not recommended when other approaches are available (see Microsoft identity platform and OAuth 2.0 Resource Owner Password Credentials ) but as there is no "app-only" authentication options for Project Online this is one such occasion when this is the only way.  To ensure this is as secure as possible we will be storing the username and password in Azure Key Vault (AKV).  

Minimum user settings for the account (although they don't need Team Member)

In this example they are also a team member, but that is not necessary.

  • An Azure Data Factory resource in Azure
  • Somewhere to write the data.  In this example I cover both saving out as Json to blob storage in Azure, as well as saving to SQL Server (in this case hosted in Azure.  You will need connection strings for whatever storage you are using
  • If using SQL Server you will need stored procedures that will do the data handling - more details later

Once you have all these pieces in place, we can continue with ADF to:

  • Add Linked Services 
  • Add Datasets
  • Build a pipeline

 

Linked Services

We need 4 linked services

  1. An Azure Key Vault where we will be storing our account details and App Registration secret
  2. A REST linked service - basically our OData endpoint
  3. Azure Blob Storage (not necessary - but I found it useful in debugging before I added it all into SQL Server)
  4. SQL Server

To keep this blog relatively short, I'm not going into all the details of setting up AKV, just that using a managed identity makes it fairly easy to use in AFD.  

The REST linked literally just needs the base URL configured - and this will be the URL for your PWA instance's OData feed, along with any select options to limit the returned fields.  As an example, I used:

https://<tenantname>.sharepoint.com/sites/pwa/_api/ProjectData/Projects?$select=ProjectId,ProjectActualCost,ProjectActualDuration,ProjectActualFinishDate,ProjectActualStartDate,ProjectActualWork,ProjectCost,ProjectCreatedDate,ProjectCurrency,ProjectDescription,ProjectDuration,ProjectFinishDate,ProjectIdentifier,ProjectLastPublishedDate,ProjectModifiedDate,ProjectName,ProjectPercentCompleted,ProjectPercentWorkCompleted,ProjectStartDate,ProjectStatusDate,ProjectWork

This limited the columns returned to just those I needed.  The authentication type was left as anonymous as I was handling this latter with a bearer token.

The Azure Blog storage isn't a necessity - if you want to use one then easy to configure but I won't go into the full details here.  Ping me in the comments if you can't find good resources to help.

Finally the SQL Server, and mine was a database I was already using for something else to which I just added a couple of tables and sprocs.  In an earlier attempt I'd configured a more expensive SQL Server instance than I'd realised - and blown through my monthly allowance...  The SQL Server linked service allows easy connectivity to an AKV to get the connection string - for a secure configuration.

 

Datasets

The datasets match up to 3 of the linked services.  My "RestResource1" to link to my REST, my "ProjectTable" to match up to my SQL database and a specific table, and my "json1" that I use to connect to my blob storage to save a file.  Again, configuring these I leave as an exercise for the reader 🙂 , but the GitHub repo has definitions for all of these so you can see how they hang together.  The pipeline will help them make more sense too - which comes next.

 

The Pipeline

To help visualize where we are headed, first we can look at the final short pipeline:

The full end-to-end pipeline

The first column of activities is reading the required data from AKV.  The names should make it obvious what the data is, the username and password, the ClientId and secret for the app registration, then finally the scope for the authentication call.  This isn't strictly a 'secret' but I put in in the AKV as it helps when demonstrating (or recording) the solution to be able to show the values.  Exposing the scope is no big deal and avoids having to redact stuff in any recording I do.

The only part defined for these activities are the settings - and the scope one is a good example:

Example KeyVault settings

The most interesting step, and maybe the only one you are interested in, is the one I called M365Login - and that is just my name - there isn't a special activity, it is just a web activity.  The settings for this one are as follows:

Web call settings to get token

The URL is of the form https://login.microsoftonline.com/<tenantid>/oauth2/v2.0/token and the method is POST and the headers configured as shown above with Content-Type application/x-www-form-urlencoded, Accept */* and Connection keep-alive.  The Body is the key part - and is using the concatenation function and brings in the values from the previous calls to AKV.  The full form looks something like the following, where I have used specific names for my AKV activities - yours may vary.

 

@concat('grant_type=password&client_id=',activity('AKVPjoClientId').output.value,'&client_secret=',activity('AKVPjoODataSecret').output.value,'&scope=',activity('AKVPjoScope').output.value,'&username=',activity('AKVUserName').output.value,'&password=',activity('AKVUserPassword').output.value)
 
Basically it is using the output.value property of the previous steps to complete the "grant_type" body needed for an ROPC call.
 
I then use a Set variable action to take the response and keep the token for later use.
 
Variable setting for token
The full string used in the Value is @activity('M365Login').output.access_token
 
Now I have my token I can use that to make my REST call to Project Online's OData endpoint using a Copy data activity.  First I use a Stored procedure activity to clear out my staging table.  Take a look at the GitHub for more details, but it is just a 'delete from' call.
The copy data activity has a source and sink (destination) and I use one to read and then write to blob storage, then another to read and write to SQL.  I'll concentrate on the second, which has Source settings configured like this:
Source data settings
The source dataset is my REST dataset, I add the header Authorization with a Value of  
 
@concat('Bearer ',variables('token'))
 
 which gets the token from my variable called token, and I have also set the Pagination rulesRFC5988 with a Value True (although that isn't in the above screenshot.
The Sink settings are as follows:
Sink data settings
with the sink dataset as my SQL dataset 'ProjectsTable'.  The magic happens on the Mappings tab - and I had created a table that matched the columns I was returning from REST - so just a 1:1 mapping.  You can get more adventurous here if you need to do anything fancy:
Data mapping from OData to my SQL table
 
Once that is complete, we have a populated Project staging table with the current projects read from OData.  The final steps are then just 3 stored procedure steps that remove deleted projects from the live project table (by deleting if they do not now exist in staging). also deleting any projects that have been updated (the modified date is newer in the staging table) and then finally copying in the updated and new plans from staging to the live table. 
As mentioned, this is just the basics and only looks at Projects - but the main focus here was the authentication steps of getting the token with ROPC, then using the token in the REST call. 
 
I appreciate I have glossed over a lot of the detail here so happy to fill in some of the gaps if required in the comments section or another blog if needed.  However, if you know ADF and already use SSIS - the authentication piece was probably all you came for.
Updated Sep 02, 2022
Version 1.0

10 Comments

  • Great question Skyuk and a Plan 3 would be sufficient.  I actually had a Plan 5 in my testing, but Plan 3 is enough.

    Best regards,

    Brian

  • Skyuk's avatar
    Skyuk
    Copper Contributor

    Great Article. Out of interest which license did you apply to the user account? and what would be the minimum license required? i.e. Project Plan 3/5 or something else? 

  • TDawggg it is correct that it has to use a delegated permission and no Application permission is available and no plans to introduce one.  Whether the account is an actual user, or a specific named account added to your system and given the right permissions is up to you.

  • TDawggg's avatar
    TDawggg
    Copper Contributor

    Thanks for your response Brian-Smith, in the case of querying POL Odata end-points I thought we had to use an actual user account as the usual application authentication is not supported (still have no idea why this is not supported by Microsoft). Is there any way around this? 

  • PriceMIT's avatar
    PriceMIT
    Copper Contributor

    Most of the https://pricemit.com/price-of-zoho-crm-zoho-expense/ There is often challenges in getting REST call to ODATA.

  • Correct TDawggg - MFA will cause a problem here.  It would be best practice to not use an actual user account (and thus have to avoid MFA for that person) and use a specific account for that purpose.  You may also want to add additional protection in terms of Conditional Access to limit use of that account.

  • TDawggg's avatar
    TDawggg
    Copper Contributor

    Hi Brian-Smith, great article and is something we have been trying to crack for years. We've just managed to get this working, however, it seems to only work with accounts without MFA enabled. Looking on the MS Docs it looks like MFA may not be supported using this authentication method but just wanted to confirm if this is something you have managed to get working yourself? Paul_Mather keen to get your opinion on this too. Thank you kindly for any assistance you may provide.

  • Scott_Clausen's avatar
    Scott_Clausen
    Copper Contributor

    Thank you Brian-Smith for always keeping a step ahead.

     

    A note for the novice - I was receiving "Rest call failed with client error, status code 401 Unauthorized" until I corrected the &scope (AKVPjoScope) value to:

    https://{replace.sharepoint.com/ProjectWebAppReporting.Read