BinderHub is a cloud-based, multi-server technology used for hosting repoducible computing environments and interactive Jupyter Notebooks built from code repositories.
This blog will walk you through getting started and deploying a BinderHub onto Microsoft Azure, and connect a DockerHub container registry, so that you can host your own Binder service.
We would like to acknowledge and thank the following people for their contributions to this project:
- Tim Greaves (@tmbgreaves)
- Gerard Gorman (@ggorman)
- Tania Allard (@trallard)
- Diego Alonso Alvarez (@dalonsoa)
- Sarah Gibson (@sgibson91)
You will require a Microsoft Azure account and subscription. A Free Trial subscription can be obtained here.
If you are building a BinderHub as a service for an organisation, your institution may already have an Azure account. You should contact your IT Services for further information regarding permissions and access (see the Service Principal Creation section below). There is a documented repo which contains a set of scripts to automatically deploy a BinderHub onto Microsoft Azure, and connect a DockerHub container registry, so that you can host your own Binder service.
"Deploy to Azure" Button
"Platform as a Service" through the "Deploy to Azure" button you can quickly deploy a BinderHub to Azure in a single click (and some form-filling), use the deploy button below.
Service Principal Creation
You will be asked to provide a Service Principal in the form launched when you click the "Deploy to Azure" button above.
[NOTE: The following instructions can also be run in a local terminal session. They will require the Azure command line to be installed, so make sure to run setup.sh first.]
To create a Service Principal, go to the Azure Portal (and login!) and open the Cloud Shell:
You may be asked to create storage when you open the shell. This is expected, click "Create".
Make sure the shell is set to Bash, not PowerShell.
Set the subscription you'd like to deploy your BinderHub on.
az account set -s <subscription>
This image shows the command being executed for an "Azure Pass - Sponsorship" subscription.
You will need the subscription ID, which you can retrieve by running:
az account list --refresh --output table
Next, create the Service Principal with the following command. Make sure to give it a sensible name!
az ad sp create-for-rbac --name binderhub-sp --role contributor --scopes /subscriptions/<subscription ID from above>
The fields appId, password and tenant are the required pieces of information. These should be copied into the "Service Principal App ID", "Service Principal App Key" and "Service Principal Tenant ID" fields in the form, respectively.
Keep this information safe as the password cannot be recovered after this step!
Monitoring Deployment Progress
To monitor the progress of the blue-button deployment, go to the Azure portal and select "Resource Groups" from the left hand pane. Then in the central pane select the resource group you chose to deploy into.
This will give you a right hand pane containing the resources within the group. You may need to "refresh" until you see a new container instance.
When it appears, select it and then in the new pane go to "Settings->Containers". You should see your new container listed.
Select it, then in the lower right hand pane select "Logs". You may need to "refresh" this to display the logs until the container starts up. The logs are also not auto-updating, so keep refreshing them to see progress.
Retrieving Deployment Output from Azure
When BinderHub is deployed using the "Deploy to Azure" button (or with a local container), output logs, YAML files, and ssh keys are pushed to an Azure storage account to preserve them once the container exits. The storage account is created in the same resource group as the Kubernetes cluster, and files are pushed into a storage blob within the account.
Both the storage blob name and the storage account name are derived from the name you gave to your BinderHub instance, but may be modified and/or have a random seed appended. To find the storage account name, navigate to your resource group by selecting "Resource Groups" in the left-most panel of the Azure Portal, then clicking on the resource group containing your BinderHub instance. Along with any pre-existing resources (for example, if you re-used an existing resource group), you should see three new resources: a container instance, a Kubernetes service, and a storage account. Make a note of the name of the storage account (referred to in the following commands as ACCOUNT_NAME) then select this storage account.
In the new pane that opens, select "Blobs" from the "Services" section. You should see a single blob listed. Make a note of the name of this blob, which will be BLOB_NAME in the following commands.
The Azure CLI can be used to fetch files from the blob (either in the cloud shell in the Azure Portal, or in a local terminal session if you've run setup.sh first). Files are fetched into a local directory, which must already exist, referred to as OUTPUT_DIRECTORY in the following commands.
You can run setup.sh to install the Azure CLI or use the cloud shell on the Azure Portal.
To fetch all files:
az storage blob download-batch --account-name <ACCOUNT_NAME> --source <BLOB_NAME> --pattern "*" -d "<OUTPUT_DIRECTORY>"
The --pattern argument can be used to fetch particular files, for example all log files:
az storage blob download-batch --account-name <ACCOUNT_NAME> --source <BLOB_NAME> --pattern "*.log" -d "<OUTPUT_DIRECTORY>"
To fetch a single file, specify REMOTE_FILENAME for the name of the file in blob storage, and LOCAL_FILENAME for the filename it will be fetched into:
az storage blob download --account-name <ACCOUNT_NAME> --container-name <BLOB_NAME> --name <REMOTE_FILENAME> --file <LOCAL_FILENAME>
For full documentation, see the az storage blob documentation.
Accessing your BinderHub after Deployment
Once the deployment has succeeded and you've downloaded the log files, visit the IP address of your Binder page to test it's working.
The Binder IP address can be found by running the following:
cat <OUTPUT_DIRECTORY>/binder-ip.log
A good repository to test your BinderHub with is binder-examples/requirements
Customising your BinderHub Deployment
Customising your BinderHub deployment is as simple as editing config.yaml and/or secret.yaml and then upgrading the BinderHub Helm Chart. The Helm Chart can be upgraded by running upgrade.sh (make sure you have the CLIs installed by running setup.sh first).
The Jupyter guide to customising the underlying JupyterHub can be found here.
The BinderHub guide for changing the landing page logo can be found here.
Additional Resources
A full technical walkthrough on Binderhub on Azure is available at the following repo see https://github.com/alan-turing-institute/binderhub-deploy/blob/master/README.md