Exchange 2016 email queue stuck with smarthost

Recently I've been seeing email constantly stuck on our SmartHostConnectorDelivery and status alwasys in Retry. 

Our environment is exchange 2016. email routing are as below: 

External email > send connector using smarthost (our security gateway) > external users 

For the past week, email are stuck on queue and unable to deliver to next hop domain which is our security gateway. 

What was done so far:

1. suspend the queue, resume and retry queue - no luck 

2. change the send connector to deliver via MX record - mail flow start going out 

3. change it back to smarthost and things start working 

This symptoms come and goes, have anyone experience the same and could share some experience tips on this 

Thank you guys!