What is the point of Categories in Outlook

Your observation is completely valid. Many people expect Categories to function as an advanced way of organizing contacts, but that isn't really their primary purpose.

Outlook Categories were designed to act as tags that can be applied across multiple types of items, including emails, calendar appointments, tasks, contacts, and notes. This allows related information to be organized without moving items into different folders. 

When it comes to contacts specifically, Categories are mainly intended for classification, filtering, and creating custom views. For example, you can tag contacts as "Customers," "Vendors," or "Project Team Members" and then filter or group your contact list accordingly.

I agree that there are some limitations. In many Outlook versions, Categories are not available as a filter in the recipient selection dialog, and they are not always preserved in certain export formats. This can make them feel less useful for contact management.

In practice, Categories tend to provide the most value when used with emails, calendar items, and tasks rather than as a contact-grouping mechanism. If your goal is to send messages to a specific set of people, Contact Groups or Microsoft 365 Groups are usually a better fit.

So while Categories may seem underpowered for contacts alone, they become much more useful