Planner unfortunately creates office 365 groups. Which in turn creates an email box. Using comments or what not will send email to that group. Most likely when you don’t have s lot of contacts it pulls from recent list from cache or graph and displays them there.
Only thing you can really try is go into powershell and use the hide from gal flag on the group mailbox and see if they will suppress it from showing but you have to do this every single time someone creates a planner plan so probabaly something you can’t really prevent.
