That scenario is not an easy one, and MS is working in some kind of spHttpGraphClient to allow it.
I didn't succeed with AdalAngular. The most far I got was using Hello.js and this sample: https://github.com/SharePoint/sp-dev-fx-webparts/tree/dev/samples/angular-msgraph
However, the sample is using Azure AD V2 endpoint, and if you registered your App using Azure portal, it needs the V1 endpoint, but you can change that in the "aad.js" file, with the auth and grant endpoint URLs (you can get the URLs from Azure portal, something like: https://login.windows.net/common/oauth2/authorize
Anyway, although I made it work, I've never used it in a production scenario, and I saw some issues with the renegotiation of the token once it expires...