Using Trident to Automate Azure NetApp Files from OpenShift

Published May 21 2021 03:38 AM 1,733 Views
Microsoft

Dynamic Storage provisioning with NetApp Trident for ARO with ANF

 

This post has been written with the collaboration of Rizul Khanna.

 

Azure RedHat OpenShift (ARO) is a managed OpenShift platform that runs a Kubernetes-powered container engine to run and manage applications in an agile way. OpenShift leverages Kubernetes technologies to provide the required infrastructure to run workloads, including the necessary storage for stateful applications.

There are multiple options that can be used with Azure RedHat OpenShift to provide storage, being Azure NetApp Files (ANF) one of them. ANF-backed volumes provide multiple advantages, for example:

  • Kubernetes “ReadWriteMany” access mode, meaning that the same volume can be mounted simultaneously by multiple application instances
  • Dynamic allocation of volumes to different performance classes, without having to unmount the volumes
  • No impact on the maximum number of disks or the maximum IOPS limits of Azure Virtual Machines (Azure Disks do count against these limits)

With the Trident software component, some additional functionality are available for all Kubernetes and OpenShift clusters using Azure NetApp Files: volumes can be dynamically created in Azure as they are demanded by the application. Instead of having to go to the Azure portal and manage the volumes in ANF accounts, Trident will take care of the lifecycle of the ANF volumes: it will create them when they are required, and delete them when they are not. Operational complexity is reduced, and application deployments are simplified.

The following diagram summarizes the overall architecture of Trident:

 

JoseMoreno_5-1632754730440.png

 

In essence, an OpenShift pod will mount a persistent volume (PV), that is based on a persistent volume claim (PVC). This PVC is the Kubernetes representation of an ANF volume: when a PVC is created, it is based on a storage class (SC) that will tell Trident how to create the volume in Azure. When the PVC is deleted, Trident will delete the volume from the ANF account.

In this blog we will cover some of the new features of Trident version 21.7.1, the latest at the time of this writing, such as service level selectors and specifying a list of capacity pools where the NetApp volumes will be provisioned.

ANF and Trident backend

As a first step, the Azure NetApp Files accounts and the capacity pools need to be created. In this example we will work with one single account, and three capacity pools, which appear like this in the Azure portal:

JoseMoreno_7-1632755188443.png

The first thing after installing Trident in the OpenShift cluster (with the command “tridentctl install”) is creating the backend, that will instruct Trident which account and capacity pools to use when:

 

{
  "backendName": "$anf_name",
  "version": 1,
  "storageDriverName": "azure-netapp-files",
  "subscriptionID": "$subscription_id",
  "tenantID": "$tenant_id",
  "clientID": "$sp_app_id",
  "clientSecret": "$sp_app_secret",
  "location": "$anf_location",
  "serviceLevel": "$anf_sku",
  "virtualNetwork": "$vnet_name",
  "subnet": "$anf_subnet_name",
  "nfsMountOptions": "vers=3,proto=tcp,timeo=600",
  "limitVolumeSize": "4Ti",
  "capacityPools": [ "${anf_name}-Standard", "${anf_name}-Premium", "${anf_name}-Ultra" ],
  "defaults": {
    "exportRule": "0.0.0.0/0",
    "size": "200Gi"
  },
 "storage": [{
      "labels": {
        "performance": "gold"
      },
      "serviceLevel": "Ultra"
    },
    {
      "labels": {
        "performance": "silver"
      },
      "serviceLevel": "Premium"
    },
    {
      "labels": {
        "performance": "bronze"
      },
      "serviceLevel": "Standard"
    }
  ]
}

 

There are a few things to notice:

  • The different service levels at the end allow for specifying which ANF capacity pool to use, depending on the performance of the persistent volume being requested in Kubernetes.
  • Another new feature is the possibility of specifying a list of valid ANF pools where Trident will choose from when creating new volumes. Otherwise ANF would just pick any ANF capacity pool that matches the required ANF service level from your Azure subscription. In this case we only have 3 pools, but if you have more it is a good practice to define from which pools you want your OpenShift volumes to be provisioned from.
  • The Service Principal ID and secret should have permissions in Azure to modify the ANF pools, so that Trident can manage the volumes

Storage Classes and Persistent Volume Claims

After the Trident backend has been created, storage classes can be created. Storage classes are the Kubernetes resource that instructs the control plane how to create persistent volume claims as they are demanded by the applications:

 

# Storage classes
perf_tiers=(bronze silver gold)           # To create a SC per tier
for perf_tier in "${perf_tiers[@]}"
do
  cat <<EOF | kubectl apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: anf-$perf_tier
provisioner: csi.trident.netapp.io
parameters:
  backendType: "azure-netapp-files"
  fsType: "nfs"
  selector: "performance=$perf_tier"  # Matching labels in the backends...
allowVolumeExpansion: true            # To allow volume resizing. This parameter is optional
mountOptions:
  - nconnect=16
EOF
done

 

There are some things to note in the previous code:

  • It is essentially a “for” loop in bash, that creates three different storage classes (bronze, silver and gold)
  • For each storage class, the “csi.trident.netapp.io” Trident provisioner is specified as backend­­­. Note that this version is based on CSI, instead of on the old in-tree drivers
  • In the parameters, a “selector” tag matches the labels in the provisioner backends “performance=bronze”, “performance=silver” and “performance=gold”. So when a PVC is created using a certain storage class, the class knows which provisioner backend to use
  • The option “allowVolumeExpansion” enables increasing volume sizes without having to unmount the NFS shares
  • In the mount options, nconnect=16 is specified for best performance, to increase the number of parallel NFS threads

After creating the storage classes, they can be inspected with the kubectl command:

 

❯ kubectl get storageclass
NAME                        PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
anf-bronze                  csi.trident.netapp.io      Delete          Immediate              true                   88s
anf-gold                    csi.trident.netapp.io      Delete          Immediate              true                   80s
anf-silver                  csi.trident.netapp.io      Delete          Immediate              true                   84s
managed-premium (default)   kubernetes.io/azure-disk   Delete          WaitForFirstConsumer   true                   60m

 

In ARO, the storage classes are visible from the console as well:

 

JoseMoreno_6-1632754813279.png

 

After having the storage classes, Persistent Volume Claims that reference them can be deployed. For example, this bash loop would create three PVCs, one per storage tier (gold/silver/bronze). Each PVCs refers to the specific storage class (“anf-gold”, “anf-silver” or “anf-bronze”) created previously:

 

# PVCs
perf_tiers=(gold silver bronze)           # To create a SC per tier
for perf_tier in "${perf_tiers[@]}"
do
  cat <<EOF | kubectl apply -f -
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: anf-$perf_tier
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 4000Gi
  storageClassName: anf-$perf_tier
EOF
done

 

Again, some things to notice:

  • The PVCs are created with 4TB to get good performance, since further down we will have a look at how the volumes perform.
  • The PVCs are created with the “ReadWriteMany” access mode (ANF and Trident support ReadWriteOnce too)

At this point, the volumes will be created in the Azure NetApp Files accounts. You can inspect them via the CLI (“az netappfiles volume list”) or the Azure portal:

 

JoseMoreno_8-1632755239818.png

 

And of course, the PVCs are visible in Kubernetes and the OpenShift console:

 

❯ k get pvc
NAME         STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
anf-bronze   Pending                                      anf-bronze     2m36s
anf-gold     Pending                                      anf-gold       2m38s
anf-silver   Pending                                      anf-silver     2m37s

 

 

JoseMoreno_9-1632755296515.png

Using the PVCs

Once the PVCs are available, Kubernetes pods can mount Persistent Volumes based on them. This code deploys a sample application (a Python-based API that I often use for testing) that leverages one of the PVCs created earlier on, in this particular example in the gold class:

 

# Deployment & Service
perf_tier=gold
name=api-$perf_tier
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: $name
  labels:
    app: $name
    deploymethod: trident
spec:
  replicas: 2
  selector:
    matchLabels:
      app: $name
  template:
    metadata:
      labels:
        app: $name
        deploymethod: trident
    spec:
      containers:
      - name: $name
        image: erjosito/sqlapi:1.0
        ports:
        - containerPort: 8080
        volumeMounts:
        - name: disk01
          mountPath: /mnt/disk
      volumes:
      - name: disk01
        persistentVolumeClaim:
          claimName: anf-$perf_tier
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: $name
  name: $name
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: $name
  type: LoadBalancer
EOF

 

Performance

As the previous Azure screenshot of the volumes shows, the volume in the gold class should support 512 MiB/s. If we install fio in the pod and do a quick test against the mounted volume, we will see the performance offered by Azure NetApp Files:

 

 

pod_name=$(kubectl get pod -o json -l app=api-${perf_tier} | jq -r '.items[0].metadata.name')
kubectl exec $pod_name -- /bin/bash -c "apt update && apt install -y fio ioping --fix-missing"
kubectl exec $pod_name -- /bin/bash -c "cd /mnt/disk && fio --name=8kseqreads --rw=read --direct=1 --ioengine=libaio --bs=64k --numjobs=16 --iodepth=1 --size=1G --runtime=60 --group_reporting --fallocate=none"
[...]
   READ: bw=516MiB/s (542MB/s), 516MiB/s-516MiB/s (542MB/s-542MB/s), io=16.0GiB (17.2GB), run=31723-31723msec

 

 

We can measure the storage latency as well, which is really great (sub-millisecond) for an NFS-mounted volume (consider that Azure Disks is targeting at “single-digit millisecond latency for most I/O operations”):

 

 

❯ kubectl exec $pod_name -- /bin/bash -c "cd /mnt/disk && ioping -c 20 ."
[...]
--- . (nfs 192.168.0.196:/anf-1788fdae0a034ae28e9c85ef126bc1db) ioping statistics ---
19 requests completed in 21.2 ms, 76 KiB read, 897 iops, 3.50 MiB/s
generated 20 requests in 19.0 s, 80 KiB, 1 iops, 4.21 KiB/s
min/avg/max/mdev = 779.7 us / 1.11 ms / 4.25 ms / 746.0 us

 

Conclusion

Azure NetApp Files is a great way to provide persistent storage to Kubernetes and OpenShift workloads. Trident includes important functionality such as managing the whole lifecycle of ANF volumes using the Kubernetes APIs, dynamically creating and deleting ANF volumes as demanded by the applications running on the OpenShift cluster.

%3CLINGO-SUB%20id%3D%22lingo-sub-2367351%22%20slang%3D%22en-US%22%3EUsing%20Trident%20to%20Automate%20Azure%20NetApp%20Files%20from%20OpenShift%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2367351%22%20slang%3D%22en-US%22%3E%3CH1%20id%3D%22toc-hId-1330999660%22%20id%3D%22toc-hId-1331862580%22%3EDynamic%20Storage%20provisioning%20with%20NetApp%20Trident%20for%20ARO%20with%20ANF%3C%2FH1%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CEM%3EThis%20post%20has%20been%20written%20with%20the%20collaboration%20of%20%3CA%20href%3D%22https%3A%2F%2Fae.linkedin.com%2Fin%2Frizul-khanna-a3b08139%22%20target%3D%22_self%22%20rel%3D%22nofollow%20noopener%20noreferrer%22%3ERizul%20Khanna%3C%2FA%3E%3C%2FEM%3E%3CEM%3E.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fazure%2Fopenshift%2F%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%22%3EAzure%20RedHat%20OpenShift%20(ARO)%3C%2FA%3E%26nbsp%3Bis%20a%20managed%20%3CA%20href%3D%22https%3A%2F%2Fcloud.redhat.com%2Flearn%2Fwhat-is-openshift%22%20target%3D%22_self%22%20rel%3D%22nofollow%20noopener%20noreferrer%22%3EOpenShift%3C%2FA%3E%26nbsp%3Bplatform%20that%20runs%20a%20Kubernetes-powered%20container%20engine%20to%20run%20and%20manage%20applications%20in%20an%20agile%20way.%20OpenShift%20leverages%20Kubernetes%20technologies%20to%20provide%20the%20required%20infrastructure%20to%20run%20workloads%2C%20including%20the%20necessary%20storage%20for%20stateful%20applications.%3C%2FP%3E%0A%3CP%3EThere%20are%20multiple%20options%20that%20can%20be%20used%20with%20Azure%20RedHat%20OpenShift%20to%20provide%20storage%2C%20being%20%3CA%20href%3D%22https%3A%2F%2Fazure.microsoft.com%2Fservices%2Fnetapp%2F%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%22%3EAzure%20NetApp%20Files%20(ANF)%3C%2FA%3E%26nbsp%3Bone%20of%20them.%20ANF-backed%20volumes%20provide%20multiple%20advantages%2C%20for%20example%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EKubernetes%20%E2%80%9CReadWriteMany%E2%80%9D%20access%20mode%2C%20meaning%20that%20the%20same%20volume%20can%20be%20mounted%20simultaneously%20by%20multiple%20application%20instances%3C%2FLI%3E%0A%3CLI%3EDynamic%20allocation%20of%20volumes%20to%20different%20performance%20classes%2C%20without%20having%20to%20unmount%20the%20volumes%3C%2FLI%3E%0A%3CLI%3ENo%20impact%20on%20the%20maximum%20number%20of%20disks%20or%20the%20maximum%20IOPS%20limits%20of%20Azure%20Virtual%20Machines%20(Azure%20Disks%20do%20count%20against%20these%20limits)%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3EWith%20the%20%3CA%20href%3D%22https%3A%2F%2Fnetapp-trident.readthedocs.io%2F%22%20target%3D%22_self%22%20rel%3D%22nofollow%20noopener%20noreferrer%22%3ETrident%3C%2FA%3E%20software%20component%2C%20some%20additional%20functionality%20are%20available%20for%20all%20Kubernetes%20and%20OpenShift%20clusters%20using%20Azure%20NetApp%20Files%3A%20volumes%20can%20be%20dynamically%20created%20in%20Azure%20as%20they%20are%20demanded%20by%20the%20application.%20Instead%20of%20having%20to%20go%20to%20the%20Azure%20portal%20and%20manage%20the%20volumes%20in%20ANF%20accounts%2C%20Trident%20will%20take%20care%20of%20the%20lifecycle%20of%20the%20ANF%20volumes%3A%20it%20will%20create%20them%20when%20they%20are%20required%2C%20and%20delete%20them%20when%20they%20are%20not.%20Operational%20complexity%20is%20reduced%2C%20and%20application%20deployments%20are%20simplified.%3C%2FP%3E%0A%3CP%3EThe%20following%20diagram%20summarizes%20the%20overall%20architecture%20of%20Trident%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22JoseMoreno_5-1632754730440.png%22%20style%3D%22width%3A%20583px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F313111i6BC6DAC97ADC480F%2Fimage-dimensions%2F583x274%3Fv%3Dv2%22%20width%3D%22583%22%20height%3D%22274%22%20role%3D%22button%22%20title%3D%22JoseMoreno_5-1632754730440.png%22%20alt%3D%22JoseMoreno_5-1632754730440.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIn%20essence%2C%20an%20OpenShift%20pod%20will%20mount%20a%20persistent%20volume%20(PV)%2C%20that%20is%20based%20on%20a%20persistent%20volume%20claim%20(PVC).%20This%20PVC%20is%20the%20Kubernetes%20representation%20of%20an%20ANF%20volume%3A%20when%20a%20PVC%20is%20created%2C%20it%20is%20based%20on%20a%20storage%20class%20(SC)%20that%20will%20tell%20Trident%20how%20to%20create%20the%20volume%20in%20Azure.%20When%20the%20PVC%20is%20deleted%2C%20Trident%20will%20delete%20the%20volume%20from%20the%20ANF%20account.%3C%2FP%3E%0A%3CP%3EIn%20this%20blog%20we%20will%20cover%20some%20of%20the%20new%20features%20of%20Trident%20version%2021.7.1%2C%20the%20latest%20at%20the%20time%20of%20this%20writing%2C%20such%20as%20service%20level%20selectors%20and%20specifying%20a%20list%20of%20capacity%20pools%20where%20the%20NetApp%20volumes%20will%20be%20provisioned.%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId-2021561134%22%20id%3D%22toc-hId-2022424054%22%3EANF%20and%20Trident%20backend%3C%2FH2%3E%0A%3CP%3EAs%20a%20first%20step%2C%20the%20Azure%20NetApp%20Files%20accounts%20and%20the%20capacity%20pools%20need%20to%20be%20created.%20In%20this%20example%20we%20will%20work%20with%20one%20single%20account%2C%20and%20three%20capacity%20pools%2C%20which%20appear%20like%20this%20in%20the%20Azure%20portal%3A%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22JoseMoreno_7-1632755188443.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F313113i839CD7AFA911FCD9%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%22JoseMoreno_7-1632755188443.png%22%20alt%3D%22JoseMoreno_7-1632755188443.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3EThe%20first%20thing%20after%20installing%20Trident%20in%20the%20OpenShift%20cluster%20(with%20the%20command%20%E2%80%9Ctridentctl%20install%E2%80%9D)%20is%20creating%20the%20backend%2C%20that%20will%20instruct%20Trident%20which%20account%20and%20capacity%20pools%20to%20use%20when%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-bash%22%3E%3CCODE%3E%7B%0A%20%20%22backendName%22%3A%20%22%24anf_name%22%2C%0A%20%20%22version%22%3A%201%2C%0A%20%20%22storageDriverName%22%3A%20%22azure-netapp-files%22%2C%0A%20%20%22subscriptionID%22%3A%20%22%24subscription_id%22%2C%0A%20%20%22tenantID%22%3A%20%22%24tenant_id%22%2C%0A%20%20%22clientID%22%3A%20%22%24sp_app_id%22%2C%0A%20%20%22clientSecret%22%3A%20%22%24sp_app_secret%22%2C%0A%20%20%22location%22%3A%20%22%24anf_location%22%2C%0A%20%20%22serviceLevel%22%3A%20%22%24anf_sku%22%2C%0A%20%20%22virtualNetwork%22%3A%20%22%24vnet_name%22%2C%0A%20%20%22subnet%22%3A%20%22%24anf_subnet_name%22%2C%0A%20%20%22nfsMountOptions%22%3A%20%22vers%3D3%2Cproto%3Dtcp%2Ctimeo%3D600%22%2C%0A%20%20%22limitVolumeSize%22%3A%20%224Ti%22%2C%0A%20%20%22capacityPools%22%3A%20%5B%20%22%24%7Banf_name%7D-Standard%22%2C%20%22%24%7Banf_name%7D-Premium%22%2C%20%22%24%7Banf_name%7D-Ultra%22%20%5D%2C%0A%20%20%22defaults%22%3A%20%7B%0A%20%20%20%20%22exportRule%22%3A%20%220.0.0.0%2F0%22%2C%0A%20%20%20%20%22size%22%3A%20%22200Gi%22%0A%20%20%7D%2C%0A%20%22storage%22%3A%20%5B%7B%0A%20%20%20%20%20%20%22labels%22%3A%20%7B%0A%20%20%20%20%20%20%20%20%22performance%22%3A%20%22gold%22%0A%20%20%20%20%20%20%7D%2C%0A%20%20%20%20%20%20%22serviceLevel%22%3A%20%22Ultra%22%0A%20%20%20%20%7D%2C%0A%20%20%20%20%7B%0A%20%20%20%20%20%20%22labels%22%3A%20%7B%0A%20%20%20%20%20%20%20%20%22performance%22%3A%20%22silver%22%0A%20%20%20%20%20%20%7D%2C%0A%20%20%20%20%20%20%22serviceLevel%22%3A%20%22Premium%22%0A%20%20%20%20%7D%2C%0A%20%20%20%20%7B%0A%20%20%20%20%20%20%22labels%22%3A%20%7B%0A%20%20%20%20%20%20%20%20%22performance%22%3A%20%22bronze%22%0A%20%20%20%20%20%20%7D%2C%0A%20%20%20%20%20%20%22serviceLevel%22%3A%20%22Standard%22%0A%20%20%20%20%7D%0A%20%20%5D%0A%7D%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThere%20are%20a%20few%20things%20to%20notice%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EThe%20different%20service%20levels%20at%20the%20end%20allow%20for%20specifying%20which%20ANF%20capacity%20pool%20to%20use%2C%20depending%20on%20the%20performance%20of%20the%20persistent%20volume%20being%20requested%20in%20Kubernetes.%3C%2FLI%3E%0A%3CLI%3EAnother%20new%20feature%20is%20the%20possibility%20of%20specifying%20a%20list%20of%20valid%20ANF%20pools%20where%20Trident%20will%20choose%20from%20when%20creating%20new%20volumes.%20Otherwise%20ANF%20would%20just%20pick%20any%20ANF%20capacity%20pool%20that%20matches%20the%20required%20ANF%20service%20level%20from%20your%20Azure%20subscription.%20In%20this%20case%20we%20only%20have%203%20pools%2C%20but%20if%20you%20have%20more%20it%20is%20a%20good%20practice%20to%20define%20from%20which%20pools%20you%20want%20your%20OpenShift%20volumes%20to%20be%20provisioned%20from.%3C%2FLI%3E%0A%3CLI%3EThe%20Service%20Principal%20ID%20and%20secret%20should%20have%20permissions%20in%20Azure%20to%20modify%20the%20ANF%20pools%2C%20so%20that%20Trident%20can%20manage%20the%20volumes%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CH2%20id%3D%22toc-hId-214106671%22%20id%3D%22toc-hId-214969591%22%3EStorage%20Classes%20and%20Persistent%20Volume%20Claims%3C%2FH2%3E%0A%3CP%3EAfter%20the%20Trident%20backend%20has%20been%20created%2C%20storage%20classes%20can%20be%20created.%20Storage%20classes%20are%20the%20Kubernetes%20resource%20that%20instructs%20the%20control%20plane%20how%20to%20create%20persistent%20volume%20claims%20as%20they%20are%20demanded%20by%20the%20applications%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-bash%22%3E%3CCODE%3E%23%20Storage%20classes%0Aperf_tiers%3D(bronze%20silver%20gold)%20%20%20%20%20%20%20%20%20%20%20%23%20To%20create%20a%20SC%20per%20tier%0Afor%20perf_tier%20in%20%22%24%7Bperf_tiers%5B%40%5D%7D%22%0Ado%0A%20%20cat%20%26lt%3B%3CEOF%3E%3C%2FEOF%3E%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThere%20are%20some%20things%20to%20note%20in%20the%20previous%20code%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EIt%20is%20essentially%20a%20%E2%80%9Cfor%E2%80%9D%20loop%20in%20bash%2C%20that%20creates%20three%20different%20storage%20classes%20(bronze%2C%20silver%20and%20gold)%3C%2FLI%3E%0A%3CLI%3EFor%20each%20storage%20class%2C%20the%20%E2%80%9Ccsi.trident.netapp.io%E2%80%9D%20Trident%20provisioner%20is%20specified%20as%20backend%C2%AD%C2%AD%C2%AD.%20Note%20that%20this%20version%20is%20based%20on%20CSI%2C%20instead%20of%20on%20the%20old%20in-tree%20drivers%3C%2FLI%3E%0A%3CLI%3EIn%20the%20parameters%2C%20a%20%E2%80%9Cselector%E2%80%9D%20tag%20matches%20the%20labels%20in%20the%20provisioner%20backends%20%E2%80%9Cperformance%3Dbronze%E2%80%9D%2C%20%E2%80%9Cperformance%3Dsilver%E2%80%9D%20and%20%E2%80%9Cperformance%3Dgold%E2%80%9D.%20So%20when%20a%20PVC%20is%20created%20using%20a%20certain%20storage%20class%2C%20the%20class%20knows%20which%20provisioner%20backend%20to%20use%3C%2FLI%3E%0A%3CLI%3EThe%20option%20%E2%80%9CallowVolumeExpansion%E2%80%9D%20enables%20increasing%20volume%20sizes%20without%20having%20to%20unmount%20the%20NFS%20shares%3C%2FLI%3E%0A%3CLI%3EIn%20the%20mount%20options%2C%20nconnect%3D16%20is%20specified%20for%20best%20performance%2C%20to%20increase%20the%20number%20of%20parallel%20NFS%20threads%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3EAfter%20creating%20the%20storage%20classes%2C%20they%20can%20be%20inspected%20with%20the%20kubectl%20command%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-bash%22%3E%3CCODE%3E%E2%9D%AF%20kubectl%20get%20storageclass%0ANAME%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20PROVISIONER%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20RECLAIMPOLICY%20%20%20VOLUMEBINDINGMODE%20%20%20%20%20%20ALLOWVOLUMEEXPANSION%20%20%20AGE%0Aanf-bronze%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20csi.trident.netapp.io%20%20%20%20%20%20Delete%20%20%20%20%20%20%20%20%20%20Immediate%20%20%20%20%20%20%20%20%20%20%20%20%20%20true%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%2088s%0Aanf-gold%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20csi.trident.netapp.io%20%20%20%20%20%20Delete%20%20%20%20%20%20%20%20%20%20Immediate%20%20%20%20%20%20%20%20%20%20%20%20%20%20true%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%2080s%0Aanf-silver%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20csi.trident.netapp.io%20%20%20%20%20%20Delete%20%20%20%20%20%20%20%20%20%20Immediate%20%20%20%20%20%20%20%20%20%20%20%20%20%20true%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%2084s%0Amanaged-premium%20(default)%20%20%20kubernetes.io%2Fazure-disk%20%20%20Delete%20%20%20%20%20%20%20%20%20%20WaitForFirstConsumer%20%20%20true%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%2060m%0A%0A%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIn%20ARO%2C%20the%20storage%20classes%20are%20visible%20from%20the%20console%20as%20well%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22JoseMoreno_6-1632754813279.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F313112iACD6D622F3C32CB3%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%22JoseMoreno_6-1632754813279.png%22%20alt%3D%22JoseMoreno_6-1632754813279.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAfter%20having%20the%20storage%20classes%2C%20Persistent%20Volume%20Claims%20that%20reference%20them%20can%20be%20deployed.%20For%20example%2C%20this%20bash%20loop%20would%20create%20three%20PVCs%2C%20one%20per%20storage%20tier%20(gold%2Fsilver%2Fbronze).%20Each%20PVCs%20refers%20to%20the%20specific%20storage%20class%20(%E2%80%9Canf-gold%E2%80%9D%2C%20%E2%80%9Canf-silver%E2%80%9D%20or%20%E2%80%9Canf-bronze%E2%80%9D)%20created%20previously%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-bash%22%3E%3CCODE%3E%23%20PVCs%0Aperf_tiers%3D(gold%20silver%20bronze)%20%20%20%20%20%20%20%20%20%20%20%23%20To%20create%20a%20SC%20per%20tier%0Afor%20perf_tier%20in%20%22%24%7Bperf_tiers%5B%40%5D%7D%22%0Ado%0A%20%20cat%20%26lt%3B%3CEOF%3E%3C%2FEOF%3E%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAgain%2C%20some%20things%20to%20notice%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EThe%20PVCs%20are%20created%20with%204TB%20to%20get%20good%20performance%2C%20since%20further%20down%20we%20will%20have%20a%20look%20at%20how%20the%20volumes%20perform.%3C%2FLI%3E%0A%3CLI%3EThe%20PVCs%20are%20created%20with%20the%20%E2%80%9CReadWriteMany%E2%80%9D%20access%20mode%20(ANF%20and%20Trident%20support%20ReadWriteOnce%20too)%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3EAt%20this%20point%2C%20the%20volumes%20will%20be%20created%20in%20the%20Azure%20NetApp%20Files%20accounts.%20You%20can%20inspect%20them%20via%20the%20CLI%20(%E2%80%9Caz%20netappfiles%20volume%20list%E2%80%9D)%20or%20the%20Azure%20portal%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22JoseMoreno_8-1632755239818.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F313114iDBD0B6339E1C0E46%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%22JoseMoreno_8-1632755239818.png%22%20alt%3D%22JoseMoreno_8-1632755239818.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAnd%20of%20course%2C%20the%20PVCs%20are%20visible%20in%20Kubernetes%20and%20the%20OpenShift%20console%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-bash%22%3E%3CCODE%3E%E2%9D%AF%20k%20get%20pvc%0ANAME%20%20%20%20%20%20%20%20%20STATUS%20%20%20%20VOLUME%20%20%20CAPACITY%20%20%20ACCESS%20MODES%20%20%20STORAGECLASS%20%20%20AGE%0Aanf-bronze%20%20%20Pending%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20anf-bronze%20%20%20%20%202m36s%0Aanf-gold%20%20%20%20%20Pending%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20anf-gold%20%20%20%20%20%20%202m38s%0Aanf-silver%20%20%20Pending%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20anf-silver%20%20%20%20%202m37s%0A%0A%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22JoseMoreno_9-1632755296515.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F313115i989157DC9C439CB2%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%22JoseMoreno_9-1632755296515.png%22%20alt%3D%22JoseMoreno_9-1632755296515.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId--1593347792%22%20id%3D%22toc-hId--1592484872%22%3EUsing%20the%20PVCs%3C%2FH2%3E%0A%3CP%3EOnce%20the%20PVCs%20are%20available%2C%20Kubernetes%20pods%20can%20mount%20Persistent%20Volumes%20based%20on%20them.%20This%20code%20deploys%20a%20sample%20application%20(a%20Python-based%20API%20that%20I%20often%20use%20for%20testing)%20that%20leverages%20one%20of%20the%20PVCs%20created%20earlier%20on%2C%20in%20this%20particular%20example%20in%20the%20gold%20class%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-bash%22%3E%3CCODE%3E%23%20Deployment%20%26amp%3B%20Service%0Aperf_tier%3Dgold%0Aname%3Dapi-%24perf_tier%0Acat%20%26lt%3B%3CEOF%3E%3C%2FEOF%3E%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId-894165041%22%20id%3D%22toc-hId-895027961%22%3EPerformance%3C%2FH2%3E%0A%3CP%3EAs%20the%20previous%20Azure%20screenshot%20of%20the%20volumes%20shows%2C%20the%20volume%20in%20the%20gold%20class%20should%20support%20512%20MiB%2Fs.%20If%20we%20install%20fio%20in%20the%20pod%20and%20do%20a%20quick%20test%20against%20the%20mounted%20volume%2C%20we%20will%20see%20the%20performance%20offered%20by%20Azure%20NetApp%20Files%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-bash%22%3E%3CCODE%3Epod_name%3D%24(kubectl%20get%20pod%20-o%20json%20-l%20app%3Dapi-%24%7Bperf_tier%7D%20%7C%20jq%20-r%20'.items%5B0%5D.metadata.name')%0Akubectl%20exec%20%24pod_name%20--%20%2Fbin%2Fbash%20-c%20%22apt%20update%20%26amp%3B%26amp%3B%20apt%20install%20-y%20fio%20ioping%20--fix-missing%22%0Akubectl%20exec%20%24pod_name%20--%20%2Fbin%2Fbash%20-c%20%22cd%20%2Fmnt%2Fdisk%20%26amp%3B%26amp%3B%20fio%20--name%3D8kseqreads%20--rw%3Dread%20--direct%3D1%20--ioengine%3Dlibaio%20--bs%3D64k%20--numjobs%3D16%20--iodepth%3D1%20--size%3D1G%20--runtime%3D60%20--group_reporting%20--fallocate%3Dnone%22%0A%5B...%5D%0A%20%20%20READ%3A%20bw%3D516MiB%2Fs%20(542MB%2Fs)%2C%20516MiB%2Fs-516MiB%2Fs%20(542MB%2Fs-542MB%2Fs)%2C%20io%3D16.0GiB%20(17.2GB)%2C%20run%3D31723-31723msec%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWe%20can%20measure%20the%20storage%20latency%20as%20well%2C%20which%20is%20really%20great%20(sub-millisecond)%20for%20an%20NFS-mounted%20volume%20(consider%20that%20Azure%20Disks%20is%20targeting%20at%20%E2%80%9Csingle-digit%20millisecond%20latency%20for%20most%20I%2FO%20operations%E2%80%9D)%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-bash%22%3E%3CCODE%3E%E2%9D%AF%20kubectl%20exec%20%24pod_name%20--%20%2Fbin%2Fbash%20-c%20%22cd%20%2Fmnt%2Fdisk%20%26amp%3B%26amp%3B%20ioping%20-c%2020%20.%22%0A%5B...%5D%0A---%20.%20(nfs%20192.168.0.196%3A%2Fanf-1788fdae0a034ae28e9c85ef126bc1db)%20ioping%20statistics%20---%0A19%20requests%20completed%20in%2021.2%20ms%2C%2076%20KiB%20read%2C%20897%20iops%2C%203.50%20MiB%2Fs%0Agenerated%2020%20requests%20in%2019.0%20s%2C%2080%20KiB%2C%201%20iops%2C%204.21%20KiB%2Fs%0Amin%2Favg%2Fmax%2Fmdev%20%3D%20779.7%20us%20%2F%201.11%20ms%20%2F%204.25%20ms%20%2F%20746.0%20us%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId--913289422%22%20id%3D%22toc-hId--912426502%22%3EConclusion%3C%2FH2%3E%0A%3CP%3EAzure%20NetApp%20Files%20is%20a%20great%20way%20to%20provide%20persistent%20storage%20to%20Kubernetes%20and%20OpenShift%20workloads.%20Trident%20includes%20important%20functionality%20such%20as%20managing%20the%20whole%20lifecycle%20of%20ANF%20volumes%20using%20the%20Kubernetes%20APIs%2C%20dynamically%20creating%20and%20deleting%20ANF%20volumes%20as%20demanded%20by%20the%20applications%20running%20on%20the%20OpenShift%20cluster.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-2367351%22%20slang%3D%22en-US%22%3E%3CP%3EAzure%20NetApp%20Files%20is%20a%20reliable%20option%20for%20persistent%20storage%20in%20Azure%20RedHat%20OpenShift%2C%20and%20Trident%20makes%20it%20easier%20to%20use%20than%20ever.%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-2367351%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3ECloud%20Native%20Apps%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Co-Authors
Version history
Last update:
‎Sep 27 2021 08:23 AM
Updated by: