Delete a managed metadata site column from a library only

In SPO, i am trying to delete a Managed Metadata Site Column from a library. I'm using the below script to delete:

$SiteURL = "https://Abc.sharepoint.com/sites/SiteName"
$ListName = "Site Pages"
$ColumnName = "CandC_Tax_6" #Internal Name
Try {
#Connect to PnP Online
Connect-PnPOnline -Url $SiteURL -Useweblogin
Set-PnPField -List $ListName -Identity $ColumnName -Values @{Required=$False;Hidden=$False} 
Remove-PnPField -Identity $ColumnName -List $ListName -Force   
}
catch {
write-host "Error: $($_.Exception.Message)" -foregroundcolor Red
}

When i execute the script, i get the below error:

"Error: Cannot change Hidden attribute for this field"

I'm able to delete few other columns in the library using this script but it is displaying error when trying to delete this particular Sitecolumn.