Validating Change Requests with Kubernetes Admission Controllers

Microsoft

Jun 17, 2025

Promoting an application or infrastructure change into production often comes with a requirement to follow a change control process. This ensures that changes to production are properly reviewed and that they adhere to required approvals, change windows and QA process. Often this change request (CR) process will be conducted using a system for recording and auditing the change request and the outcome.

When deploying a release, there will often be places in the process to go through this change control workflow. This may be as part of a release pipeline, it may be managed in a pull request or it may be a manual process. Ultimately, by the time the actual changes are made to production infrastructure or applications, they should already be approved. This relies on the appropriate controls and restrictions being in place to make sure this happens.

When it comes to the point of deploying resources into production Kubernetes clusters, they should have already been through a CR process. However, what if you wanted a way to validate that this is the case, and block anything from being deployed that does not have an approved CR, providing a backstop to ensure that no unapproved resources get deployed? Let's take a look at how we can use an Admission Controller to do this.

Admission Controllers

A Kubernetes Admission Controller is a mechanism to provide a checkpoint during a deployment that validates resources and applies rules and policies before this resource is accepted into the cluster. Any request to create, update or delete (CRUD) a resource is first run through any applicable admission controllers to check if it violates any of the required rules. Only if all admission controllers allow the request is it then processed. Kubernetes includes some built-in admission controllers, but you can also create your own.

Admission controllers are essentially webhooks that are registered with the Kubernetes API server. When a CRUD request is processed by the API server, it calls any of these webhooks that are registered, and processes the response. When creating your own Admission controller, you would usually implement the webhook as a pod running in the cluster.

There are three types of Admission Controller webhooks:

MutatingAdmissionWebhook: Can modify the incoming object before it is persisted (e.g., injecting sidecars).
ValidatingAdmissionWebhook: Can only approve or reject the request based on validation logic.
ValidatingAdmissionPolicy: Validation logic is embedded in the API, rather than requiring a separate web service

For our scenario we are going to look at using a ValidatingAdmissionWebhook, as we only want to approve or reject a request based on its change request status.

Sample Code

In this article, we are not going to go line by line through the code for this admission controller, however you can see an example implementation of this in this repo.

In this example, we do not build out the full web service for validating change requests themselves. We have some pre-defined CR IDs with pre-configured statuses returned by the application. In a real world implementation your web service would call out to your change management solution to get the current status of the change request. This does not impact how you would build the Admission Controller, just the business logic inside your controller.

Components

Our Admission Controller consists of several components:

Application

Our actual admission controller application, which runs a HTTP service that receives the request from the API Server calling the webhook, processes it and applies business logic, and returns a response. In our example this service has been written in GO, but you can use whatever language you like. Your service must meet the API contract defined for the admission webhook.

Our application does the following:

Reads the incoming change body YAML and extracts the Change ID from the change.company.com/id annotation that should be applied to the resource. We also support the argocd.argoproj.io/change-id and deployment.company.com/change-id annotations.

func extractChangeID(req *admissionv1.AdmissionRequest) string {
	// Try to extract change ID from object annotations
	obj := req.Object.Raw
	var objMap map[string]interface{}
	if err := json.Unmarshal(obj, &objMap); err != nil {
		return ""
	}

	if metadata, ok := objMap["metadata"].(map[string]interface{}); ok {
		if annotations, ok := metadata["annotations"].(map[string]interface{}); ok {
			// Look for change ID in various annotation formats
			if changeID, ok := annotations["change.company.com/id"].(string); ok {
				return changeID
			}
			if changeID, ok := annotations["argocd.argoproj.io/change-id"].(string); ok {
				return changeID
			}
			if changeID, ok := annotations["deployment.company.com/change-id"].(string); ok {
				return changeID
			}
		}
	}
	return ""
}

If it does not find the required annotation, it immediately fails the validation, as no CR is present.

if changeID == "" {
   // Reject resources without change ID annotation
   klog.Infof("No change ID found, rejecting request")
   ac.respond(w, &admissionReview, false, "Change ID annotation is required")
   return
}

If the CR is present, it validates it. In our demo application this is checked against a hard-coded list of CRs, but in the real world, this is where you would make a call out to your external change management solution to get the CR with that ID. There are 3 possible outcomes here:

The CR ID does not match an ID in our system, the validation fails
The CR does match an ID in our system, but this CR is not approved, the validation fails
The CR does match an ID in our system and this CR has been approved, the validation passes and the resources are created.

changeRecord, err := ac.changeService.ValidateChange(changeID)
if err != nil {
   klog.Errorf("Change validation failed: %v", err)
   ac.respond(w, &admissionReview, false, fmt.Sprintf("Change validation failed: %v", err))
   return
}

if !changeRecord.Approved {
   klog.Infof("Change %s is not approved (status: %s)", changeID, changeRecord.Status)
   ac.respond(w, &admissionReview, false,
       fmt.Sprintf("Change %s is not approved (status: %s)", changeID, changeRecord.Status))
   return
}

klog.Infof("Change %s is approved, allowing deployment", changeID)
ac.respond(w, &admissionReview, true,
   fmt.Sprintf("Change %s approved by %s", changeID, changeRecord.Requester))

Container

To run our Admission Controller inside the AKS cluster we need to create a Docker container that runs our application. In the sample code you will find a Docker file used to build this container. We then push the container to a Docker registry, so we can consume the image when we run the webhook service.

Kubernetes Resources

To run our Docker container and setup a URL that the API server can call we will deploy:

A Kubernetes Deployment
A Kubernetes Service
A set of RBAC roles and bindings to grant access to the Admission Controller

Finally, we will deploy the actual ValidatingAdmissionWebhook resource itself. This resource tells the API servers:

Where to call the webhook
Which operations should require calling the webhook - in our demo application we look at create and delete operations. If you wanted to validate delete operations had a CR, you could also add that
Which resource types need to be validated - in our demo we are looking at Deployments, Services and Configmaps, but you could make this as wide or narrow as you require
Which namespaces to validate - we added a condition that only applies this validation to namespaces that have a label of changeValidation set to enabled, this way we can control where this is applied and avoid applying it to things like system namespaces. This is very important to ensure you don't break your core Kubernetes infrastructure. This also allows for differentiation between development and production namespaces, where you likely would not want to require Change Requests in development.
Finally, we define what happens when the validation fails. There are two options:
- fail which blocks the resource creation
- ignore which ignores the failure and allows the resource to be created

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionWebhook
metadata:
name: change-validation-webhook
spec:
clientConfig:
  service:
    name: admission-controller
    namespace: admission-controller
    path: "/admit"
rules:
- operations: ["CREATE", "UPDATE"]
  apiGroups: ["apps"]
  apiVersions: ["v1"]
  resources: ["deployments"]
- operations: ["CREATE", "UPDATE"]
  apiGroups: [""]
  apiVersions: ["v1"]
  resources: ["services", "configmaps"]
namespaceSelector:
  matchLabels:
    change-validation: "enabled"
admissionReviewVersions: ["v1", "v1beta1"]
sideEffects: None
failurePolicy: Fail

Admission Controller In Action

Now that we have our admission controller setup, let's attempt to make a change to a resource. Using a Kubernetes Deployment resource, we will attempt to change the number of replicas from three to two. For this resource, the change.company.com/id annotation is set to CHG-2025-000 which is a change request that doesn't exist in our change management system.

apiVersion: apps/v1
kind: Deployment
metadata:
name: demo-app
namespace: demo
annotations:
  change.company.com/id: "CHG-2025-000"
labels:
  app: demo-app
  environment: development
spec:
replicas: 2
selector:
  matchLabels:
    app: demo-app

Once we attempt to deploy this, we will quickly see that the the request to update the resource is denied:

one or more objects failed to apply, reason: error when patching "/dev/shm/1236013741": 
admission webhook "change-validation.company.com" denied the request: 
Change validation failed: change record not found,admission webhook "change-validation.company.com" denied the request: 
Change validation failed: change record not found.

Similarly, if we change the annotation to CHG-2025-999 which is a change request that does exist, but has not been approved, we again see that the request is denied, but this time the error is clear that it is not approved:

one or more objects failed to apply, reason: error when patching "/dev/shm/28290353":
 admission webhook "change-validation.company.com" denied the request: 
Change CHG-2025-999 is not approved (status: pending),admission webhook "change-validation.company.com" denied the request: 
Change validation failed: change record not found.

Finally, we update the annotation to CHG-2025-002, which has been approved. This time our deployment update succeeds and the number of replicas has been reduced to two.

Next Steps

What we have created so far works as a Proof of Concept to confirm that using an Admission Controller for this job will work. To move this into production use, we'd need to take a few more steps:

Update our web API to call out to our external change management solution and retrieve real change requests
Implement proper security for the Admission Controller with SSL certificates and network restrictions inside the cluster
Implement high availability with multiple replicas to ensure the service is always able to respond to requests
Implement monitoring and log collection for our service to ensure we are aware of any issues
Automate the build and release of this solution, including implementing it's own set of change controls!

Conclusions

Controlling updates into production through a change control process is vital for a stable, secure and audited production environments. Ideally these CR processes will happen early in the release pipeline in a clear, automated process that avoids getting to the point where anyone tries to deploy unapproved changes into production. However, if you want to ensure that this cannot happen, and put some safeguards to ensure that unapproved changes are always blocked, then the use of Admission Controllers is one way to do this.

Creating a custom Admission Controller is relatively straightforward and it allows you to integrate your business processes into the decision on whether a resource can be deployed or not. A change control Admission Controller should not be your only change control process, but it can form part of your layers of control and audit.