A seeming illogical design\test strategy?

[PeteGomersall]

This post is looking for answers from people in the Microsoft Edge design\testing team.

I have been in IT system administration and software development for over 30 years but I just don't get your design and testing strategy specifically related to who gets to see new features and on what devices.

Here is my gripe, I run Edge Canary on multiple systems, each using the same Microsoft account and none of installations exhibit\expose the same feature set. So how in the world are we suppose to be involved in validating features that say, are supposed to sync but are only exposed on one install? I can totally understand A\B testing, but the same setup should apply to the user across all their devices - either they see the feature on everything or not at all.

Say for example the feature "Payment instruments are now synced across devices" that I saw appear today. This shows on just one device; this seems to me to be bluntly illogical? In another area, I have been able to set 2 rows of quicklink icons on the new tab page on all but 1 device; it is hugely inconsistent and to be honest frustrating.

It would seem sensible to make the same Microsoft account get all the same testing features rather than as it seems so randomly assigned.

Regardless, I am sure there are more users who would benefit from an explanation and look forward to an explanation.

Lastly looking for a real explanation, not supposition.

Pete

[#2]

@PeteGomersall 

I totally agree:

---

@PeteGomersall wrote:

I can totally understand A\B testing, but the same setup should apply to the user across all their devices - either they see the feature on everything or not at all.

The A/B test should consider the user's account and display the same features across all installations for that account.

[#3]

@josh_bodner
It would be nice to see some response from MSFT even if it is to only comment that the discussion has been noted.

[#4]

The main reason we don't set experiments per user is because signing into the browser isn't required. Whatever system we use needs to work regardless of a user's identity, so in order for experiment results to be statistically valid, we can't change the way we set experiment populations for any reason, including whether or not the user has an identity that is known across other devices.

For what it's worth, the best way around this is to use flags for any feature that you want to explicitly turn on, although that does require that we actually create said flag.

[#5]

@josh_bodner,
Thanks for the info I appreciate the reply. I can totally understand the approach you detail for what one could call "global features" but not for features that provide/require an account to actually work, like those related to sync.