I don't think it's possible with Teams yet. It's possible with Outlook and the other Office apps, including SfB. But Teams is "special" :D
What's even more annoying is that often times information is missing on the contact card. Especially when it comes to guest users - they don't even display an email address. And I've even seen some cases where no names were displayed, fun.