Remove unused labels

Hello...is it possible to remove unused labels? I only want to use 3 and by default planner shows me 6 labels, so now it shows "My Label1", "My Label2", "My Label3", Green, Blue, Purple. I'd like to get rid of the last 3. Is it possible?

Thanks

Ezequiel