After a bit of research it seems that it depends on the Linux distribution as to which characters are allowed. But the POSIX specification allows for the period in the name. Also the Azure portal doesn't allow periods, but deploying via an ARM template does (the latter being what we are doing).
I can remove the period from most of our usernames, the exception being the main admin account which was created by the Azure processes. I've not been able to find a means to change this. If I update the ARM template it tells me this option can't be changed.
Any recommendations would be great. Either with an update to the policy script to allow for periods as this is allowable on some Linux distributions. Or a means to change the admin username that is supplied in the ARM template.