Server-side auto-response

I have enabled auto-response for three users in the Office 365 Exchange Server admin centre. I know it's enabled because when the user opens their Outlook, they get a message at the top of the screen...