All Users gets access to a sharepoint site

Hi, 

 SharePoint sites show All Users (windows) to  few sites. How do i investigate why it keeps on happening? I removed it and it got assigned again.