Dataset
As part of this tutorial, we will be loading the AMEX data - integer dtypes - parquet format | Kaggle available on kaggle.
Setting up your Kaggle API key
Kaggle username and API key. To create a key:
- Go to your kaggle account → Settings → Account → Create a new API token.
- A kaggle.json file will be downloaded and it will contain your username and API key. Keep this somewhere safe!
Connect Kaggle data in Azure
Go to Azure Portal.
Click on Create a Resource -> Search for Machine Learning.
Click Create and follow the steps until you reach the following page.
Click on notebooks from the sidebar and click create a new notebook.
Create a Compute resource (if one does not exist already).
Once the compute has been created
Next, open the terminal.
On the terminal window, pip install kaggle.
Set Kaggle username and API key (from the json file) as environment variables in the terminal:
export KAGGLE_USERNAME=xxxxxxxx
export KAGGLE_KEY=xxxxxxxxxxxxxx
Finally, you are ready to download your Kaggle dataset via the command line in the terminal. The API command to do so is available on the Kaggle dataset page itself. Click on the three dots next to New Notebook and select ‘copy API command’.
Next paste (or CTRL +V) on the terminal window
kaggle datasets download -d raddar/amex-data-integer-dtypes-parquet-format
Note: This might take a while as you can see the file is approx 4GB in size
Voila…. you will see your dataset will be downloaded (as a zip file) in your current working directory onto your Azure workspace.
Alternatively, you can specify a folder where the files should be downloaded using optional arguments in the API call (for more info, see Kaggle documentation here). For example:
kaggle datasets download -p images/train/
The following code goes in your Notebook.
import os
import zipfile# name of the zip file you want to unzip
local_zip = 'amex-data-integer-dtypes-parquet-format.zip'# opening a file with mode parameter 'r' : read existing file
zip_ref = zipfile.ZipFile(local_zip, 'r')# extract all contents of the zip file
zip_ref.extractall('')# close the file
zip_ref.close()
And there you have it. All your data would be unzipped into a new folder, which will be sitting in your current working directory on Azure.
Microsoft