Subsite Document Library Unique Permissions Not Working

Hello there, I am the site owner for the main site as well as the subsite. Here's a rundown of what's going on.

In the main site, we'll call it X, the permission level is set to read for an AD group users_all, (basically everyone in the org), then full control as myself the site owner. In the subsite, we'll call it Y, the same read level access is set for the site, users_all. I confirmed both X and Y have the exact same permission settings, with full control being delegated to myself the site owner. In subsite Y, I have two document libraries I'm trying to restrict access to. I revoked inherit permissions from parent site for both libraries, and removed all access except for myself having full control as the site owner. I then added one AD group to have read access for each one, each group only contains roughly 3-4 people

The problem is even after all of this, everyone in the org can still view the doc libraries, download the files, etc. I tried repeating the entire process by resetting the perms for the doc libraries, revoke inherit from parent site, remove all other perms except myself as site owner, then adding the two AD groups again. I waited multiple sync cycles, waited 24 hours, same issue is still occurring. I read something online about anonymous access being enabled for the parent site, but couldn't find any setting related to this

Hope someone can help