Azure Virtual Desktop joined to Azure AD (not Azure ADDS), Azure file permissioning?

Hi All

I have a Azure Virtual Desktop session host running Win 11 multi session joined to Azure AD (not Azure ADDS), all users are on Azure AD only, is it possible to have file permissions on Azure Files?

At the moment I've created a file share and mapped it to the session host but everyones got access to all files, ideally want to restrict for example the HR folder to only HR users.

Advice would be much appreciated on how to go about doing this, I've read about Azure AD Kerberos authentication but getting confused as to if this is what I need.

Thanks