I don't know why I can't edit my post but the previous one has misinformation/typo, so here we go again.
@FreakyNL , @The_Exchange_Team
I have both exchange 2016 and 2019 servers.
Edge server is 2016, the edge server's exchange tool box works just fine with $ account. (The user's name and display name does not have $, the server is in DMZ, a separate domain)
Both exchange mailbox server 2016 and 2019's exchange toolbox does not work when used with $ account (this account's name and display name has $, it still does not work even after I changed name and display name)
Both exchange mailbox server 2016 and 2019's exchange toolbox works when used an account w/o $.
I re-installed the patch to one of exchange 2016 and 2019 each, and tested, the behavior is the same.
Because our organization does not allow PowerShell to be executed remotely, I can't test the PowerShell remote-ability.
All exchange mailbox server 2016 and 2019's local powershell works even if I used $ account.