We believe that Yammer is the best way for teams to collaborate openly to connect their organizations as networks, and that email plays a key role in the adoption and continued use of Yammer. In talking to many users and looking at our qualitative data, it became clear that emails from Yammer suffered from two primary issues that were hindering network adoption:  we sent WAY too many emails, which often caused signal to become lost amongst noise and  our emails did not find the right balance between enabling consumption from email and providing reasons to engage in the community. The problem with  is that it encourages people to turn off or ignore Yammer emails entirely and the problem with  is that the user doesn’t see the full value of engaging in Yammer and the network misses on those potential contributions.
Our goal for this project was to improve how we notify users about important content on Yammer so we can engage those users over time and improve the overall health and quality of their networks. By pulling users into Yammer, they’ll be able to experience and understand the benefits of Yammer sooner (that ‘aha’ moment). As more users view content, post and join groups etc., the network improves and a virtuous cycle is established.
This project is part of a larger effort to improve how Yammer notifies users of relevant content through email and mobile phone push notifications. In this project, we planned significant changes to both the design and the delivery rules for the email we send when a user receives a high priority message in their Yammer Inbox.
On the Design side:
- make it easier to parse and triage the Yammer notification
- make it explicit why people actually receive the email
- highlight who triggered the email
- increase the prominence of the group name and the network name
- create an onramp to Yammer that leads to more active network engagement
- make it obvious that the email originated from Yammer
- include a strong call to action (CTA) providing users with very clear direction for what to do with the email
- balance the amount of content shown in the email between providing enough context and leading the user to engage in their Yammer network
On the Delivery side:
- increase the signal and decrease the noise by delivering only messages that we believe are highly relevant and interesting
When we first enabled this experience for internal Microsoft users, the feedback was immediate and just as impassioned as it was on the O365 network. After gathering feedback from internal users and Microsoft MVPs, we realized that our original truncation limit (60 characters) failed to provide enough context for people to make an informed decision regarding the notification. So we increased the truncation limit from 60 characters to up to 200 (if the message is 200 characters or fewer, it is displayed in its entirety; if it exceeds 200 characters, we display the first 150 characters). Based on the feedback, we also decided to test a second version of the email which included the entire message body (no truncation).
Because of the impact of this email and the concern expressed amongst the O365 community, we are holding this feature to a considerably higher bar as compared to typical feature changes. To ensure that the test wasn't simply click-bait, we let this test run for eight weeks (typically we run tests for about two). Additionally, we segmented the results in many ways, including looking at many specific networks that expressed concern along the way. We also looked deeply at new users and their behavior since they wouldn't have been biased by any prior email configuration. As the results have been consistent from week to week, we feel confident making a decision on this feature test.
The email with the shortened message body has outperformed the version with the full message body and the original email (control) in pretty much every metric that leads to healthier networks:
- New user retention – users who activated their accounts during the experiment were more likely to come back to Yammer in weeks 2, 3, and all the way through to week 7.
- Days engaged – both new and existing visited Yammer more often.
- Posting – more people posted a message during the experiment and those that posted also posted more often.
- Group activity – people visited more groups, they saw more group conversations, and they joined more groups. This suggests that people did not simply visit the conversation they were being notified on, but they explored more of Yammer as well.
The lifts we saw in these metrics were high, significant, and sustained across the eight weeks of the test. Again, if this were a normal project, we would have made the decision to ship it after two weeks. But we wanted to be sure that the lifts we’re seeing are indeed the result of users getting more value out of Yammer and not feeling like they had no choice but to click through to see content.
Additionally, users in the treatment group received 34% fewer emails. It's notable that this test drove the above results while also reducing the number of emails users received.
Based on the overwhelmingly positive results for both new and existing users, we have determined that the shorter email is the best experience, but we are not shipping it to all users immediately. (More on this below.)
Since we started this experiment, we've receive lots of great feedback on a variety of channels, and we have listened to and considered every comment. Based on that feedback, we believe that there is still room for improvement in this experience. To that end we will staff projects to build on the gains we've made in this project and to address some of the feedback we've received. Some of the issues we plan to address include:
- The current design favors the network name over the group name, truncating the group name at 24 characters. We'll test changes to include the full group name.
- We've heard frustration that our design emphasizes meta-data about a conversation at the expense of actual content. We will test some design variations to emphasize the message more.
- We know there's concern that users are missing important updates because we are not sending an email for every message in a conversation. We agree that it's important to let people know about ongoing messages in a conversation, and we plan to provide summary updates of unread conversations on a daily basis.
We know this is a significant change and we don't underestimate the impact of changes like this to your network. For that reason, we will not ship this new email to all users until we’ve had a chance to test the changes described above. We genuinely appreciate the feedback from this community and will use it to improve on this email even further. "
[NFM> There was a variety of mixed opinions in response to this, but it was great to have the detail provided to understand why choices have been made, based on real data and usage stats.]
@Ron Blandford posted a subsequent update - July 5 2016 -
- I want to provide an update regarding our plans for this email. In my thread starter, I stated that we would hold off on shipping the new email to 100% of our users until we’ve had a chance to make some improvements to our new email. As we looked more closely at this plan, a few problems became clear:
* As the current email test is a three part test (the original email and the two variations we tested), layering on additional, sequential tests (the improvements we want to make) is quite complicated and decreases our confidence in interpreting the future test results.
* The experiment design described above would limit the number of people who see the improvements; thus we'd have to let the experiments run for a much longer period (and thus colleagues within the same team would have substantially different experiences for a much longer period).
* We’d have to delay shipping the improvements we’ve made in our delivery pipeline (including some bug fixes) until after the additional tests have completed.
When we look at the complexity and risks of the original proposal, we think it's best for our users to ship this email to all of our users now. This will provide cleaner, more reliable experiment results in a shorter period of time.
We are still pushing forward quickly with the email improvements mentioned in my thread starter. We are already underway on the first of those improvements. From a development perspective, the changes are small, so my expectation is that we can complete these additional improvements quickly (on the order of a few weeks) and get results within our normal timeframe. Thank you for all the feedback in this thread."