Controls added to the header using SPFx extensions are visible on different site after navigation

Controls/customizations added to the SharePoint online site header using SPFx extension are visible on different SharePoint site after navigation.

Even if the SPFx extension is not added on the different site.

I have one SharePoint site without any SPFx extension, which looks like below.

I have another site with SPFx extension added, which shows a toggle button after search box and has some custom css, that site looks like below.

When I navigate from SiteWithExtension site to the site - ‘SiteWithoutExtension’ where SPFx extension is not added, still header customization related to search is visible. Please check below image.

Could you please help me, how to get rid of these custom controls/ css when navigating to another site?