Stop Sharing Button Not Working

I am trying to remove group access (Company Data Visitor) for a folder within a document library however the "Remove" button does not work. 

The steps i take to remove access is :

-Click on 3 dots for the folder i want to manage and select Manage Access.

-Under Direct Access select Company Data Visitors and click on Stop Sharing.

-When promoted with the following pop-up "Want to remove Company Data Visitor's Access?" select Remove. This then removes the group access for the folder. 

The above steps have worked fine for all folder except one. When i follow the above steps and get to the last stage, When promoted with the following pop-up "Want to remove Company Data Visitor's Access?" select Remove, the Remove button does not work.

When i click on Remove it just does nothing! This parent folder has MANY sub folders and then sub-sub folder... you get what i mean. I have checked all the sub folders and removed access to Company Data Visitors. I thought maybe this is stopping the parent folder from removing access to Company Data Visitors as the sub folder has permission to it, but this has not worked either.

I have not gone into each sub folder and then into each folder within the sub folder as this will take days!. I want to remove Data Visitors Access from the parent folder but cannot.. please can you advice on how i can achieve this.