Calling MGGraph V2 without -AsPlainText parameter

Like many other people after Graph was updated to V2 I started getting this error

Cannot bind parameter 'AccessToken'. Cannot convert the *** value of type "System.String" to type "System.Security.SecureString".

And to fix it was told to use something like this 

Connect-MgGraph -AccessToken ($graphApiToken |ConvertTo-SecureString -AsPlainText -Force)

Which does work. However my problem is that every solution I've found requires using the -AsPlainText parameter which of course presents an obvious security issue and Microsoft recommends not using it. 

But if I just remove that parameter from my code and use something like this 

 -token ($token | ConvertTo-SecureString -Force) 

I get this error 

Exception : System.FormatException: Input string was not in a
correct format.

Is there a way to convert this token correctly without using -AsPlainText?