Filter Pane Issues

The filter panes in our document library never finish loading the managed metadata for "normal" users on the site.  It appears they only load for people who are term store admins.  Is this how the filter pane should work?  Is there another setting that I'm not aware of to make this work for anyone who has access to this site?  Any help is appreciated!