Site - Limited Access

Hi all,

I know I can use Conditional Access and Sharepoint to make a site read only or limited access for unmanaged devices.

Is it possible to make a site read only/limited access for everyone bar a subset of internal users? So even if they have a compliant device they will still get a limited view?

My goal is a site that will prevent download for most users bar a very small group.

Thanks