Can I delete or reorder the un-used labels in Planner?

I noticed that there are a lot more color labels in Planner and picked the brighter colors that are lower in the drop-down list. When I use "Group By Labels" to view my planner according to these col...