How to set access for a specific file to specific users

Hello, I'm an Office 365 user as part of a company account where I work. I want to create a "secure" folder in our SharePoint under "documents" so that each file in that folder will be accessible by some specific user e.g. user1 can access file1 and user2 can access file2 - both under that "secure" folder, but other users in the company should not be able to access either file.

I could not find how to do it because regardless of any shared links I create for specific people, each file shows, under "direct access", that all members of the company can access it. I tested it and indeed that is the case.

Any idea how to setup the access I require?