How to remove SharePoint Site Owner from a specific Document Library

Hello KimHai,

"wouldn't that mean having to create and administer another Site"

Yes, that is exactly what i am suggesting. A Site Collection administrator has access to all files in a given Site Collection. If you don't want a Site Collection administrator to have any access to a specific file, you'll have to move that file out of the Site Collection.
( You only have that problem with Site Collection administrators as you can take away permissions for a given file/folder from every other non-admin user-type )

If you worry about the administrative overhead with a second site collection then you have the same overhead whether you administer permissions for a folder or for a site... You just do it on another level

"Our intend is to reduce the number of sites and have everyone Access the same departmental site"

Reducing the number of sites is a good thing, but putting everything into a single Site Collection is bad. Over the time you will get very, very complicated permission structures and side effects that are hard to debug and explain.

It is better to think about a site collection as single application with a clearly defined scope, both in terms of requirements and permissions.
If you get new requirements, you need to match them to the scope of your existing site and create a new site if they don't really match that scope.
You will end up with multiple sites, but each will be maintainable.

Best Regards,
Sven