Understanding security and privacy of Delve and intelligent experiences in Office 365

Yes, you can turn this off or just opt-out but it's not necessarily that simple.  First, check out this: 

Office Delve for Office 365 admins

In the 'Control access to the Office Graph' section, you will see the steps to disable Office Graph in the tenant, which powers Delve. Bear in mind, Office Graph does a lot more than just Delve and you loose considerable functionality by taking this path. 

If you read some of the resources in this discussion, you will see Delve never changes any permissions and only surfaces what users have already been given access to.  By disabling Office Graph, users can still access these documents by other means, so it's better to look at the underlying issues and deal with those. 

Check that documents haven't been saved to the 'Shared with everyone' folder in OneDrive, as that would open them up to everyone.  Otherwise, there must be some permissions to grant access or have been shared in some way for the contractor to have been able to see the documents.  If there is no indication on further inspection why this is happening, I'd consider logging a service request, as it just doesn't sound right.

Another option is to opt-out of Delve as covered here -

Are my documents safe in Office Delve?

Hope you get to the bottom of this.  I know it can be a shock with these incidents but Delve is really a great feature once understood and everyone knows what to expect from it.