Microsoft Secure Tech Accelerator
Apr 03 2024, 07:00 AM - 11:00 AM (PDT)
Microsoft Tech Community

Azure AD SCIM Validator is in General Availability (GA) Status

Microsoft

You can now validate the compatibility of your SCIM provisioning endpoint and Azure AD code base using our Azure AD SCIM Validator. This tool can be used by ISVs who want to build SCIM compatible servers either for gallery app or generic app and developers building their line of business SCIM apps. https://learn.microsoft.com/azure/active-directory/app-provisioning/scim-validator-tutorial

45 Replies

Where's the best place to ask questions about the results I'm getting from the validator? My POST /Users request is failing validation but I'm not getting a reason for the failure

Turns out the status code should be 201, but the validator doesn't indicate that's the problem

@owinoakelo 

 

Can the SCIM Validator be made open source or at least have a location where issues can be filed against it? We've encountered a couple problems with the SCIM Validator and would like some way of reporting issues or discussing misunderstandings with the developers.

 

The first is that it does not read the schema correctly. For instance, our endpoint describes the 'email' 'primary' attribute as boolean, but SCIM Validator sets it to string, apparently ignoring our schema.

 

Another is a failure during group testing where it re-creates a user that already exists. It seems not to expect a 409 reply.

 

It is very hard to tell *what* the tester expects without a diagnostic error or insight into its source code.

@Vincent_Drake_Cerberus I'm so glad I'm not alone, and want you to know that you're not alone either! I've spent the past few days trying to figure out what I'm doing wrong. I finally reached the same conclusion that the validator doesn't expect a 409 to be returned after looking through my server logs.

 

The validator makes a patch request to change a username from 'x' to 'y' as part of a completely different test, and then as part of the failing test, the first thing it tries to do is submit a post request for a user with username 'y'... I can get it to pass by removing the duplicate user throws an error code (and just returning the user instead), but then the Create a duplicate User test obviously fails.

 

Very strange that the otherwise randomly generated test data is kept between tests.

I'm also seeing the 409 for 'Patch Group - Remove Member'
I'm also having trouble getting 'Patch Group - Replace Attributes' to work. It looks like it's trying to make a group a member of another group, which is not something we support. We've adjusted the schema to say we only support users as members of a group but that doesn't seem to help

This feedback is noted @Doogal  @Vincent_Drake_Cerberus  and  @JameyAbdelnoorJameyAbdelnoor . We are working to address these, and I will share the individual updates.

@owinoakelo , same issue as @Vincent_Drake_Cerberus with phone primary attribute being sent as random phone number string instead of boolean value.

"Can the SCIM Validator be made open source or at least have a location where issues can be filed against it?"

Did anything like this happen? I'm searching around for any docs / issue tracker and not coming up with anything.

I am also having issues understanding the errors I'm being shown:

1. When I look at the "Show Details" for any of my failed requests they are all showing as POST requests, even though I'm 90% certain they are using other HTTP verbs (DELETE, PATCH, PUT)
2. When I look at the url under "Show Details" it appears to always be the top level URL that I entered into the test field, not the specific URL hit by the test (eg: Update group membership does not show what is used as the id for the path parameter to identify the group)

Looks like there was recently a big update on the Validator. The "Discover Schema" now sets the correct type (string/boolean). Thank you!

However, now we're seeing a new complication. It appears the Validator is trying to add new entries using replace with a value selection filter (valuePath). This was discussed in 708183: SCIM PATCH of Complex Multi-Valued Attribute Includes Filter and Sub Attribute in Path in the context of patching with add. And that discussion makes sense as the spec is a little vague.

However, with replace, RFC 7644 is quite specific in 3.5.2.3:

   o  If the target location is a multi-valued attribute for which a
      value selection filter ("valuePath") has been supplied and no
      record match was made, the service provider SHALL indicate failure
      by returning HTTP status code 400 and a "scimType" error code of
      "noTarget".

If the valuePath does not match anything, then you need to return 400 noTarget. Microsoft Azure AD SCIM Validator does not appear to expect that reply and expects us to add new entries on replace.
I disagree that:

 o  If the target location path specifies an attribute that does not
      exist, the service provider SHALL treat the operation as an "add".

would change this interpretation as that does not consider the filter; plus the way these are written they get more and more specific as you go down the list.

A specific example:
The current value of emails:

 

 

 "emails": [                     
   {                             
     "type": "work",             
     "value": "some_primary_email",  
     "primary": true             
   }                             

 

 

and the replace operation:

 

 

"Operations": [{
  "op": "replace",
  "path": "emails[type eq \"home\"].value",
  "value": "some_email"
}]

 

NOTE: values do not show email addresses as those get removed (privacy checker) - so they are replaced with "some_primary_email" and "some_email"

This should fail as there is no email of type = home to replace.

@Vincent_Drake_Cerberus We released a fix that addresses the read schema issues. The new release also addressed the duplicate creation the triggers the 409 error. 

To the question on having the making the validator open source, we do not have that in the current plan. We are working on a channel to enable you raise issues.

@PatM_Cerberus. The feedback above is correct and our implementation of this capability is not consistent with the standard. We are incorrectly using replace to update multivalued attribute with a filter instead of add. We have this item in our backlog of things to fix. The validator is however supposed to test the compatibility with our provisioning service as is and for that matter an existing scim incorrectness will also show up here. Hope this is clear and to enable the provisioning service to work for your connector, we have to build with the incompatibility in place for now.

@owinoakelo Thank you for your fast reply.

To clarify, are you saying that using PATCH replace to add multi-valued entries is:

a.) what Azure's SCIM implementation currently does as of Feb 2023 (we have not seen this in practice)

or

b.) an issue with the Azure AD SCIM Validator only

 

If option "a", is this affected by the feature flag _aadOptscim062020_? And if so, does the Validator accept/use those feature flags? (I wasn't able to see a difference with/without the flag after the last update.)

@PatM_Cerberus Yes, the azure provisioning service currently uses replace for patch operations and expects the endpoint to add if needed. That's why the scim validator makes the same request and it supports the capability of the feature flag.

@owinoakelo We have been working to allow Azure's specific replace semantics.

We've noticed when using Schema Discovery, the system now correctly captures that primary is a Boolean.

       phoneNumbers[type eq "work"].primary      boolean

However, when it replaces phoneNumbers, we're seeing:

{
  "Operations": [
    {
      "op": "replace",
      "path": "phoneNumbers[type eq \"work\"].value",
      "value": "767.814.5465"
    },
    {
      "op": "replace",
      "path": "phoneNumbers[type eq \"work\"].primary",
      "value": "(974)693-6276 x6968"
    }
  ],
  "schemas": [ "urn:ietf:params:scim:api:messages:2.0:PatchOp" ]
}

 Notice that the boolean primary value is a text string. We specifically have handlers for Azure sending "True" or "False", but this value is not acceptable and we're failing it.

I believe this is a problem with the Azure AD SCIM Validator, but curious if Azure AD would try to send such a bad value.

Email values now seem to work properly - we're getting true/false booleans for primary in those.

@PatM_Cerberus , you are right. This is an issue with the SCIM validator. The Azure code base will respond with a Boolean as expected. We have added this in our backlog of fixes and will address it. Thanks for highlighting this.

@owinoakelo 

I suspect I found a bug in SCIM Validator. What should I do?

Please refer to the image below.

norfas_0-1678974855411.png

 

Just running through the validator, and noticed that for CREATE operations, when the status code returns 200 it fails the test, however the test result doesn't inform us that it's looking for a 201 status (we had to try a lot of things before we figured it out)

@norfas Thanks for highlighting this. This has to do with the communication of the error. Question, when you did the patch operation, did you commit the updates that we sent over to your db? This error is mostly likely coz you did not commit the updates and so what you returned to us in the shown response is not the true state of your db.

@owinoakelo
Thank you for your response, it helped!
FYI I did commit the updates to our db, but the Validator were issuing an extra HTTP GET request that was processed by our Cache layer that still had an old value.
It would be nice if your Validator were more transparent as to what it does exactly (or maybe open-sourced?)...
But this specific problem is resolved for me now, thank you again.