We are trying to access Outlook on the Web, enforce MFA on all accounts, and have no access to cellphones (and behind an IVR for landlines). We have Business Essentials licenses and do not have Azure conditional access licenses.
We can access our email through other tools using app passwords, but those do not work for Outlook on the Web. I have tried to connect the machines via Powershell + MSOnline module without success, which I believe is related to these being non-admin accounts.
What options do I have for accessing Outlook on the Web?
Not sure what exactly you are trying to achieve here. OWA supports MFA, as it's a web based application that directly leverages the Azure AD experience. You do NOT need app passwords for OWA. Conditional access is also not needed, as long as you have enforced MFA for the user.