This blog post has been co-authored by Microsoft and Bala RameshBabu from NetApp.
Windows containers are on the rise and one question customers running enterprise workloads have is: How does one go about provisioning SMB file shares for Windows containers on Azure Kubernetes Service (AKS)?
This technical blog explains a reference architecture that uses AKS and Azure NetApp Files for providing persistent storage to containerized Windows applications.
An overview of Azure NetApp Files and Astra Control
Azure NetApp Files is an enterprise grade native managed file storage service on Azure, powered by NetApp. It provides the following benefits:
- Three performance tiers. Pay for the tier you need, get guaranteed throughput per volume.
- NFS and CIFS volumes can be created in seconds, allowing enterprise applications to access high performance, shared file-storage built for Azure.
- Deep integration with Azure enables a seamless and highly secure Azure experience.
- Leading certifications, including SAP HANA, GDPR, and HIPAA, enable migration of the most demanding workloads to Azure.
NetApp Astra provides orchestration and data connectivity for Kubernetes applications.
- Dynamic volume creation: Automatically create Persistent Volumes (PVs) on Azure NetApp Files for your Kubernetes stateful applications.
- Protection: Snapshot or clone your volumes
Pre-Requisites
To perform the actions in this blog, you will need an Azure subscription with access to the following services:
- Azure Kubernetes Service
- Azure NetApp Files
- An Active Directory (AD) instance. This blog uses Azure Active Directory Domain Services
Understanding the environment
The key components of this environment will be:
- A Kubernetes cluster: Create an AKS cluster that contains Windows nodes and uses the “azure” network plugin.
- An Active Directory Domain: Azure Active Directory Domain Services (AADDS) provides a convenient way of spinning up an AD domain. If pre-created AD domains exist, this step can be safely skipped.
- An Azure NetApp Files account.
- Astra Trident: NetApp’s dynamic storage orchestrator, used to provision SMB volumes through Kubernetes.
Steps Involved
Below we will describe the steps to get the above solution up and running.
Create an AKS cluster
Create a Kubernetes cluster with Windows nodes.
The following objects are defined:
- A resource group named `windows-cluster-rg`.
- A VNet named `prod-vnet`.
- It contains two subnets. `k8s-subnet` is used for the nodes in the AKS cluster. `anf-subnet` is used for the ANF volumes.
- An Azure NetApp Files account named `smb-netapp-account` in East US. A 4TiB Ultra capacity pool is created in this account.
- An AKS cluster named `prod-cluster`. A username and password are provided as arguments. The desired subnet is also provided with `vnet-subnet-id`.
- A Windows nodepool for the AKS cluster.
#Create a resource group
az group create --name windows-cluster-rg --location eastus
# Create a VNet, two subnets, and delegate one subnet
az network vnet create --name prod-vnet --resource-group windows-cluster-rg --address 10.1.0.0/16 --subnet-name anf-subnet --subnet-prefixes 10.1.4.0/24
az network vnet subnet create -g windows-cluster-rg --vnet-name prod-vnet -n anf-subnet --address-prefixes 10.1.4.0/24
az network vnet subnet create -g windows-cluster-rg --vnet-name prod-vnet -n k8s-subnet --address-prefixes 10.1.3.0/24
az network vnet subnet update --resource-group windows-cluster-rg --name anf-subnet --vnet-name prod-vnet --delegations Microsoft.NetApp/volumes
# Create an ANF account and a capacity pool
az netappfiles account create -a smbaccount -g windows-cluster-rg -l eastus
az netappfiles pool create -a smbaccount -g windows-cluster-rg -n ultra-pool --service-level ultra --size 4
#Store the Subnet ID into a variable to be used later
SUBNET_ID=$(az network vnet subnet show --resource-group windows-cluster-rg --vnet-name prod-vnet --name k8s-subnet --query id -o tsv)
# Create an AKS cluster
az aks create \
--resource-group windows-cluster-rg \
--name wincluster \
--node-count 2 \
--enable-addons monitoring \
--generate-ssh-keys \
--windows-admin-username produser \
--vm-set-type VirtualMachineScaleSets \
--vnet-subnet-id $SUBNET_ID \
--network-plugin azure
# Create a Windows nodepool
az aks nodepool add \
--resource-group windows-cluster-rg \
--cluster-name wincluster \
--os-type Windows \
--name npwin \
--node-count 1
Create an Azure Active Directory Domain Services managed domain
The next requirement is to have an Active Directory Domain that SMB shares can be authenticated against. Multiple options are available when it comes to Active Directory management.
This blog uses Azure Active Directory Domain Services (Azure ADDS) to define an Azure-managed domain. The domain will be used for mounting and authenticating SMB shares to workloads on the AKS cluster.
Azure provides a handy, how-to guide on how to deploy Azure ADDS: https://learn.microsoft.com/en-us/azure/active-directory-domain-services/tutorial-create-instance. You can leverage this example and provide the following parameters:
- The subscription and resource group
- Domain name
- Region and SKU
- Virtual Network and Subnet: Use “prod-vnet” and create a new subnet named `aadds-subnet`.
This takes a while, typically between 20 to 45 minutes. Once the domain is healthy and running, the Properties tab provides some details, such as:
- the domain name.
- IP Addresses, i.e., DNS Servers. We will need to add these values as DNS Servers to `prod-vnet`.
# Add DNS Servers from Azure AD Domain Services to VNet
az network vnet update -g windows-cluster-rg -n prod-vnet --dns-servers 10.1.0.5 10.1.0.4
Deploy Astra Trident and create a SMB backend
With our AD domain in the works, let us switch gears to deploy Astra Trident on the AKS cluster created earlier. The steps involved will be to:
- Install Astra Trident: Follow the instructions in the documentation; we install Trident using its operator. Once the operator is deployed, we create a TridentOrchestrator with spec.windows set to ‘true’.
apiVersion: trident.netapp.io/v1
kind: TridentOrchestrator
metadata:
name: trident
spec:
debug: false
namespace: trident
imagePullPolicy: IfNotPresent
windows: true
- Create a backend: Here is where we specify SMB as the protocol of choice. A basic backend definition looks like this:
{
"version": 1,
"backendName": "smb-backend",
"nasType": "smb",
"storageDriverName": "azure-netapp-files",
"subscriptionID": "randwxyz-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"tenantID": " tenawxyz-xxxx-xxxx-xxxx-xxxxxxxxxxxx ",
"clientID": "cliewxyz-xxxx-xxxx-xxxx-xxxxxxxxxxxx ",
"clientSecret": "SeCreTjS0g3r5VIogcl.4JJCdZSHzv~vh3daUE",
"location": "eastus",
"resourceGroups": ["windows-cluster-rg"],
"netappAccounts": ["smbaccount"],
"serviceLevel": "Ultra",
"virtualNetwork": "prod-vnet",
"subnet": "anf-subnet"
}
tridentctl create backend -f backend-smb-anf.json -n trident
+-------------+--------------------+--------------------------------------+--------+---------+
| NAME | STORAGE DRIVER | UUID | STATE | VOLUMES |
+-------------+--------------------+--------------------------------------+--------+---------+
| smb-windows | azure-netapp-files | 2f65c789-12fc-423f-8f66-4ce182a7f5ef | online | 0 |
+-------------+--------------------+--------------------------------------+--------+---------+
Create an AD user
Going back to Azure Active Directory, let us now create a user profile. We will use this user to authenticate SMB connections.
First, add a custom domain name. Head over to Azure Active Directory and select the “Custom domain names” sub-menu. Provide the name of the AD domain in use. This will ensure the domain shows up when a new user is created.
With this in place, a new user can be created from Active Directory -> Users -> Create new user.
Provide a user principal name (make sure to select the domain that was just added) and a password. Store the credentials, as we will use them later.
Under the “Assignments” tab, add the “AAD DC Administrators” group. Members of this group are granted administrative permissions on VMs that are domain-joined to the managed domain. Go ahead and create the user.
Next, reset the password for this user. This is a necessary requirement. Follow the instructions provided here.
Add Active Directory connection to Azure NetApp Files
The next step is to add an Active Directory connection to the ANF account. It is self-explanatory; select the “Active Directory connections” tab and populate the form:
- Primary DNS and AD DNS Domain Name can be obtained from Active Directory instance.
- AD Site Name is set to the desired AD site.
- Organizational Unit Path is set to “OU=AADDC Computers”.
- Username and an updated password for the AD user.
- AES encryption is enabled.
To learn more about the requirements for configuring an AD connection to Azure NetApp Files, take a look at the how-to guide available here: https://learn.microsoft.com/en-us/azure/azure-netapp-files/create-active-directory-connections.
Create a Kubernetes Secret
The next step is to create a Kubernetes secret. It contains the username and password used to authenticate SMB connections. The secret is created in all namespaces that require SMB volumes provisioned by Trident.
Let’s create a new namespace (“smb-test”) and a secret within it. The username is of the format “Domain/username”.
# Create a namespace and a secret within the namespace
---
apiVersion: v1
kind: Namespace
metadata:
labels:
kubernetes.io/metadata.name: smb-test
name: smb-test
spec:
finalizers:
- kubernetes
---
apiVersion: v1
stringData:
password: '<Password>'
username: 'wincluster.onmicrosoft.com\smbuser'
kind: Secret
metadata:
name: smb-ad-credential
namespace: smb-test
type: Opaque
Wrapping it up
Now let’s create a storageClass. Provide the name of the secret from the previous step, as well as the namespace the secret is present in.
# SMB storageClass
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: anf-smb
provisioner: csi.trident.netapp.io
parameters:
backendType: "azure-netapp-files"
trident.netapp.io/nasType: "smb"
csi.storage.k8s.io/node-stage-secret-name: "smb-ad-credential"
csi.storage.k8s.io/node-stage-secret-namespace: "smb-test"
The example above references a secret present in a single namespace. Different secrets can be used (one per PVC) in a namespace. Unique secrets per namespace is another option. Look at the examples to learn how that works.
Hurray! With all the pieces in place, go ahead and try creating a PVC and a Windows pod. Use the example spec provided below:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test
labels:
app: windows-app
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 100Gi
storageClassName: anf-smb
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: windows-app
labels:
app: windows-app
spec:
replicas: 1
template:
metadata:
name: windows-app
labels:
app: windows-app
spec:
nodeSelector:
"kubernetes.io/os": windows
containers:
- name: windows-app
image: mcr.microsoft.com/dotnet/framework/samples:aspnetapp
resources:
limits:
cpu: 1
memory: 800M
ports:
- containerPort: 80
volumeMounts:
- mountPath: "C:\\Data"
name: volume
volumes:
- name: volume
persistentVolumeClaim:
claimName: test
selector:
matchLabels:
app: windows-app
---
apiVersion: v1
kind: Service
metadata:
name: windows-app
spec:
type: LoadBalancer
ports:
- protocol: TCP
port: 80
selector:
app: windows-app
Within minutes, Trident creates an SMB volume and mounts it to the Windows app pod.
Things to consider
- Mounting SMB volumes requires CSI Proxy v1.0.2 or above. V1.0.2 is generally available in all Azure regions. Older AKS clusters will require the addition of a new Windows node pool. Newly created node pools use v1.0.2 or later.
- When encrypting traffic using Kerberos, care must be taken to ensure the appropriate encryption types are enabled in Active Directory. The steps required to add an Active Directory connection to Azure NetApp Files are well documented here.
Get started today!
Sign up for Azure NetApp Files and start provisioning volumes! Click here to learn more about Astra and its benefits.
