The email to channel functionality uses a "phantom" tenant to route the messages, and we have pretty much zero visibility of what happens there. If might be that the different attachments take different amount of time to process for anti-spam and anti-malware checks, or it might be something completely random. There's really no way for us to know as we simply don't have access to that "phantom" tenant in order to run troubleshooting tools. You can open a support case and request more info.
I've seen such delays with regular, no attachment mails as well, and overall my experience with the feature has been on the negative side. I would suggest finding another solution if possible, for example using Flow to save the attachments directly into some SPO library, etc.
