Blog Post

Viva Engage Blog
4 MIN READ

Yammer Explained: How data and testing shapes Yammer

benfreeman's avatar
benfreeman
Icon for Microsoft rankMicrosoft
Mar 22, 2022

Have you ever been curious about how we build Yammer?  We want to take you behind the scenes of our product team to show what goes into building the features that you see every day in your Yammer communities.

 

The best way to ask tens of millions of Yammer users their opinion is to run tests at scale.  Using data to create deeper engagement and adoption is a cornerstone of the product development process at Yammer and at Microsoft in general. To do this, we try different product variations in order to understand the experiences of many users without sitting next to them. These variations could relate to how and where we surface a product feature, the impacts of a new launch, or optimization of the recommendations we show in the Home Feed, among others.

 

Not everything works... taking a hypothesis driven approach

 

When we build Yammer, our product team makes hypotheses about how users will react and respond to different changes and feature sets. However, we know based on past experiences that our hypothesis may not always be correct, and there may be unexpected impact from a change. We can’t assume we will always know how you will react and respond to these changes. To bridge the gap between our expectations and reality, we run experiments using variations of Yammer.

 

We collect anonymized data on user behavior from these experiments to inform decisions about where we take the product. We run most experiments for a few weeks in order to gather a statistically significant sample, but some may go longer in order to determine lasting vs. novelty effects. For our testing to be valid, we choose a random set of users from across millions of people . The answers we gain from experiments let us learn from you and help to inform decisions. These interactions and learnings help us evolve the features we deliver for Yammer.

 

For example, our data shows that conversations with images or videos attached get twice as many reactions/replies. So, we recently launched an experiment to add an image icon to the publisher Based on some qualitative insights we collected from feedback forms, we learned that users don’t have a clear entry point to add images. The paperclip icon we have indicates 'attach a file' to users, but users told us they didn’t know to click on it to add an image. Our hypothesis was that if we make the entry point for adding images/videos clearer, then more people will add images/video to their posts.

We tend to be cautious about big changes like this, but in this case, we saw positive results and our hypothesis was correct. This change resulted in an increase in attachment clicks, growth in the total number of employees creating conversations, and growth in total conversations with an image created. Interestingly, we saw more users choose the photo icon to add an image than the paper clip icon.

 

New designs are not always tested with the explicit purpose of increasing engagement.   Often times, we are brainstorming ways to enhance the experience for our customers, and want to ensure that a new design does so without introducing a new set of issues or complexity.   Below, you can see two such examples.

 

Enhanced Feed Explanations

 

In this variant of the explanations that appear at the top of content cards in the home feed, we wanted to be provide users with an annotation that let them know whether or not suggested content was from a community they followed.  In the case where it was not, like the example above, we added a button to enable them to join communities.  This change both increased transparency on why content was being suggested and helped us ensure our recommendations are working as intended.  The change did not result in significantly increased engagement (Home Feed Views decreased by ~1%), but we shipped it anyway because it was a better user experience and helped us to identify and fix errors more easily.

 

Communities App with no Left Navigation pane

 

Sometimes, our hypothesis is proved incorrect.  In this case, we experimented with removing the left-side navigation that shows Communities you are a member of from our Communities in Teams app, to better align with the Teams user interface styling.  The results indicated that users were confused and had a difficult time navigating to their communities--messages views, community visits, and active engagement in communities were all down by 10%+, with high statistical significance.  In this case, the cleaner interface and alignment with Teams was not enough to offset the significant drop in engagement, which told us users were having a hard time navigating to their communities.  We did not ship the treated version, and our Communities experience today looks very similar to what you see on Yammer.com.

 

Educate your employees

 

We are always exploring variations of Yammer to help you and your employees be more productive and efficient. Since changes to our product are inevitable, it might be useful to help your end users understand that their Yammer interface might not always look the same as their co-workers'. And depending on the changes we release, you may need to update your documentation.

 

Here is some sample copy that you can use to share with your Yammer 101 community to help your employees understand why Yammer instances may look slightly different over time.   

 

The Yammer team at Microsoft explores different variations of Yammer in order to determine what’s effective and productive for you. For example, you may see different placements or colors of buttons within the Yammer interface. Yammer explores variations of the product to put their own assumptions to the test and to see which features create the best experience. Your interactions and use of Yammer help to improve the future product for everyone.

 

Stay connected and know what’s coming next

 

Learn about the new features coming and when by following the M365 Roadmap. You can filter by Yammer and by product stage.  Additionally, as features come to production environment of Yammer, are sent to admins and the Yammer blog is updated with new features as they are announced and released. If you have technical issues, please contact Microsoft Support .

 

Learn more about our Microsoft 365 Change management and communication policies here.

 

-The Yammer Team

 

 

Updated Apr 19, 2022
Version 2.0
  • Thanks for the info and insight. I appreciate where the Yammer team is coming from with regards to these tests and changes, however the guidance "what you see may not be what I see" is woefully lacking in the real world.

     

    It is embarrassing to be the Yammer advocate in my company and not know whether something someone is seeing is a bug or a ... possible new feature. It is embarrassing to have to say "I don't know what you're talking about" when someone asks you about a change they see. Your Yammer advocates and network admins are supposed to be the experts at Yammer and not knowing about these A/B tests only serves to undercut our expertise and status.

     

    The Yammer Community example above is something I'd look at and expect to be a bug. No sidebar? Very disorienting. I'd never have expected that to be a feature.

     

    If the Yammer team insists on continuing this practice it would be extraordinarily helpful for a notice to be sent to Yammer network admins in some way (via the Message Center, or a post in the MTC, or via a Twitter account, etc.). Create a special opt-in mailing list. I don't really care how, but please let us know so that we don't look stupid in front of our colleagues and customers.

  • KevinCrossman We discussed proactively communicating about these experiments. There were a few challenges with doing so. First, sending communications out has the potential to influence the experiment. Second, we run so many experiments at a given time, our team isn't currently setup to be able to communicate about each and every experiment and then do follow up about the outcomes. Not to mention the iteration that often times happens as we learn along the way. You're always welcome to inquire in the Yammer Customer Connection community, as you do today. 

  • wroot's avatar
    wroot
    Silver Contributor

    Honestly, i don't care. I have to deal so much with real bugs in software and now with hidden experiments. We are not you test lab rats. We are paying customers and deserve some respect. "Oh, lets remove navigation and see how they struggle to get to things". Do one experiment, ask every user and admin if they want to be part of such unexpected experiment. Guess what majority would say. This has to be done differently, but MS just switched gears a few years ago and decided, screw QA, we will just use our users to test. And i would get if i am an Insider, things can break unexpectedly. But not for regular users.

  • wroot's avatar
    wroot
    Silver Contributor

    I agree with above comment. This doesn't seem transparent enough. There should be some indication that user is in a variation UI (badge somewhere) and an option to opt out. Maybe just early release users should be getting variants.

  • I know I'm very late to the party but I was only just guided to this post and I want to make one observation: You did run a test with no left navigation pane in the Communities app and stated in detail that this test was a failure because the engagement in communities dropped signifcantly. Why then did you decide to launch the Viva Engage app with no left navigation pane? This doesn't make any sense.

  • wroot If we add indicators that someone is in a special treatment, that could have an adverse effect on the experiment results.