Open in explorer option is not working in IE 11 for document library

Please try following below:

1) Check if the value of the following Registry key is “1”:
 right-click or tap-and-hold the Start button and then choose Run, input “regedit” 
and then press Enter to open the Registry Editor.
2) Go to HKey_Local_Machine > Software > Microsoft > Internet Explorer >

Main and switch the value to 1.
3) Have you enable the ActiveX controls in Internet Explorer? You may refer to: http://windows.microsoft.com/en-US/windows/help/genuine/ie-activex

4). Reset your Internet Explorer via Internet Options > Advanced > Reset Internet Explorer settings

Please also refer Microsoft troubleshooting article for SPO

https://support.microsoft.com/en-us/help/2629108/how-to-use-the-open-with-explorer-command-to-troubleshoot-issues-in-sh

If you still have issues go with Onedrive Sync option.