I agree, that is how it should work, but that is not how it actually appears to be working in the wild.
Enabled, Direct Send appears to bypass all connector restrictions, which appears to be as designed. "“Anonymous” in this context means that the messages are not attributed to any mail flow connector when they are sent to Exchange Online."
However, disabled, Direct Send starts blocking valid messages on the same restricted connectors, most apparently so far, when they are both to & from valid tenant mailbox addresses as if it has some undocumented anti-spoofing logic is built in???
The real troubleshooting challenge is that so far, I have not found a way to identify the incoming connector that a message traverses or doesn't in the case of direct send. It does not appear to be logged in the headers, and the "connector_id" field in an extended Report Message trace does not appear to correlate to an actual Mail Flow connector in any intelligible way...
I am willing to open a support ticket with our MSP, but in my experience, it will be impossible to escalate it to the level required to find an technician/engineer capable of understanding and troubleshooting something at this level of complexity and newness.