Hmm, notification support must not be working as I didn't get pinged about your comment.
Azure Key Vault should be happy generating certs for you, but you can also create them locally and upload the Key Vault. For instance in Windows:
$cert = New-SelfSignedCertificate -Type Custom -Subject "CN=MySelfSignedCertificate" -TextExtension @("2.5.29.37={text}1.3.6.1.5.5.7.3.3") -KeyUsage DigitalSignature -KeyAlgorithm RSA -KeyLength 2048 -NotAfter (Get-Date).AddYears(2) -CertStoreLocation "Cert:\CurrentUser\My"
Followed by exporting (with private key) through MMC, and uploading to portal. You could separately export the public key. It's self-signed, so if it needs to be trusted by a public CA that's not going to work.
(Details here: https://techcommunity.microsoft.com/t5/azure-developer-community-blog/generating-azure-ad-quot-look-alike-quot-tokens/ba-p/1163098#)