Schedule a Meeting but only invite certain team members

Hi, I want to schedule a meeting and connect it to a Channel within my Team but I only want certain members of my team to attend, not the whole team.

I know I could create the meeting without connecting to a Teams Channel and just invite the relevant attendees but I want to add meeting notes which I want to be stored in the Team channel.

Is there a way to do this or is the only option, when linking to a channel, for all team members to be invited?

Thanks