Get api/v1/messages.json ommits 40% of messages

[Ben Hudson]

When I compare the data returned by get api/v1/messages.json to the data retrieved using the export in yammer. get api/v1/messages.json returns 6,300 records. The data export returns 10,469 records.

To get my incomplete 6,300 rows I...
   1. Use the export API to get a list of all of the groups
   2. Loop through that list of groups and download the messages for each group using https://www.yammer.com/api/v1/messages/in_group/:group_id.json
   3. Then I use https://www.yammer.com/api/v1/messages.json with the older_than parameter to get all of the messages in the All Company feed.

The issue appears to be with step 3.

I would appreciate any insight on what the issue might be with the get api/v1/messages.json API endpoint, and how I might modify my code to resolve this issue.

[#2]

In your export, do a sort by deleted_at date, and then do a secondary sort by suspended_at date.  Then, count how many rows have data in either of those columns, vs. how many rows have null in those columns.

Once you've done that, do a tertiary sort by in_private_conversation, and separate those which are listed as TRUE.  These are PMs.

I'm hoping this finds your issue - exports will include deleted data, and the messages.json won't.  And, of course, you won't find private messages through querying groups either.  

Finally, you'll need the token you're using for the messages.json to be a verified admin set in Private Content Mode (not the default) in order to get the messages in private groups.

Please let us know what you find. 

[#3]

Thanks for your response Tom K.

I had zero deleted posts in the export (which is odd b/c I have deleted posts from yammer...I guess they're not included in the export).

After removing private messages I had 9,860 posts in the yammer export.

My API download does include posts from private groups.

[#4]

Make sure your date is properly early - I use like 1980 to make sure all my data is there.

Maybe you have your network configured for hard delete, which would REALLY delete any messages deleted.

Is the API executing as a verified admin with private content mode turned on?

[#5]

Microsoft phone support recognized this as a known issue, and said that v2 of the API resolves this issue.   But, they were unable to provide an estimated release date for v2 of the API.  

Separately, I had a Yammer support tech reach out to me on stackoverflow and reply similarly that omitting portions of the data was an intended result by design.

[#6]

Wow, by design?  What is the design goal?