%3CLINGO-SUB%20id%3D%22lingo-sub-1813887%22%20slang%3D%22en-US%22%3EmTLS%20between%20AKS%20and%20APIM%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1813887%22%20slang%3D%22en-US%22%3E%3CH2%20id%3D%22user-content-introduction%22%20id%3D%22toc-hId--1215041490%22%20id%3D%22toc-hId--1214892664%22%3EMutual%20TLS%20Authentication%20between%20Azure%20Kubernetes%20Service%20and%20API%20Management%3C%2FH2%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EBy%26nbsp%3B%3CSPAN%3E(alphabetically)%3A%20Akinlolu%20Akindele%2C%20Dan%20Balma%2C%20Maarten%20Van%20De%20Bospoort%2C%20Erin%20Corson%2C%20Nick%20Drouin%2C%20Heba%20Elayoty%2C%20Andrei%20Ermilov%2C%20David%20Giard%2C%20Michael%20Green%2C%20Alfredo%20Chavez%20Hernandez%2C%20Hao%20Luo%2C%20Maggie%20Marxen%2C%20Siva%20Mullapudi%2C%20Nsikan%20Udoyen%2C%20William%20Zhang%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22user-content-introduction%22%20id%3D%22toc-hId-1272471343%22%20id%3D%22toc-hId-1272620169%22%3EIntroduction%3C%2FH2%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWe%20have%20two%20goals%20in%20this%20doc%3A%3C%2FP%3E%0A%3COL%3E%0A%3CLI%3EHow%20to%20set%20up%20AKS%20cluster%20for%20mutual%20TLS%20(mTLS)%20authentication%20between%20Azure%20Kubernetes%20Service%20(AKS)%20NGINX%20ingress%20controller%20and%20a%20client%20app%20such%20as%20curl%3F%20If%20you%20are%20not%20using%20a%20gateway%20for%20your%20microservices%20or%20using%20a%20gateway%20other%20than%20Azure%20API%20Management%20(APIM)%2C%20this%20portion%20is%20what%20might%20interests%20you.%3C%2FLI%3E%0A%3CLI%3EHow%20to%20set%20up%20APIM%20for%20mTLS%20between%20AKS%20and%20APIM%3F%20This%20covers%20the%20case%20in%20which%20APIM%20is%20used%20as%20the%20API%20gateway%20for%20REST%20services%20hosted%20in%20AKS%20cluster.%3C%2FLI%3E%0A%3C%2FOL%3E%0A%3CP%3EThis%20is%20a%20sister%20doc%20to%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fazure-paas-blog%2Fuse-mitreid-connect-for-oauth2-authorization-in-api-management%2Fba-p%2F1717430%22%20target%3D%22_blank%22%20rel%3D%22noopener%22%3EUse%20MITREid%20Connect%20for%20OAuth2%20Authorization%20in%20API%20Management%3C%2FA%3E%3A%20one%20covers%20securing%20AKS%20via%20mTLS%20between%20AKS%20and%20APIM%20while%20the%20other%20covers%20securing%20APIM%20via%20OAuth2%20and%20OpenID%20Connect%20across%20APIM%2C%20Identity%20Provider%20and%20clients.%3C%2FP%3E%0A%3CP%3EOur%20goal%20is%20for%20AKS%20(as%20a%20service)%20to%20authenticate%20APIM%20(as%20a%20client)%20so%20that%20only%20calls%20from%20APIM%20with%20a%20valid%20client%20cert%20with%20private%20key%20can%20get%20thru.%20Therefore%20what%20we%20need%20is%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CEM%3Enot%3C%2FEM%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3ETLS%20between%20APIM%20and%20AKS%20which%20is%20for%20client%20(APIM%20in%20our%20case)%20to%20authenticate%20server%20(AKS%20in%20our%20case).%20What%20we%20need%20is%20mutual%20TLS.%3C%2FP%3E%0A%3CP%3EAs%20a%20reference%20and%20also%20for%20context%2C%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fapi-management%2Fapi-management-howto-mutual-certificates%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ethis%3C%2FA%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3Eand%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fapp-service%2Fapp-service-web-configure-tls-mutual-auth%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ethis%3C%2FA%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3Edocuments%20provide%20mTLS%20authentication%20between%20APIM%20and%20Azure%20App%20Service.%3C%2FP%3E%0A%3CP%3EThe%20steps%3A%3C%2FP%3E%0A%3COL%3E%0A%3CLI%3EHTTPS%20calls%20from%20APIM%20is%20intercepted%20by%20frontend%20load%20balancer%20of%20App%20Service%3C%2FLI%3E%0A%3CLI%3EApp%20Service%20frontend%20load%20balancer%20injects%20an%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CEM%3EX-ARR-ClientCert%3C%2FEM%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3Erequest%20header%20with%20the%20client%20certificate%20(base64%20string)%20as%20the%20header%20value%2C%20before%20forwarding%20the%20request%20to%20application%20code.%3C%2FLI%3E%0A%3CLI%3EApplication%20code%20retrieves%20the%20cert%20string%20such%20as%20headers%5B%22X-ARR-ClientCert%22%5D%20and%20converts%20it%20to%20an%20X.509%20cert.%3C%2FLI%3E%0A%3CLI%3EApplication%20code%20parses%20the%20cert%20and%20verifies%20the%20attributes%20and%20claims%20as%20client%20authentication.%3C%2FLI%3E%0A%3C%2FOL%3E%0A%3CP%3EAs%20you%20can%20see%2C%20the%20approach%20for%20mTLS%20between%20APIM%20and%20App%20Service%20is%20not%20as%20good%20as%20we%20wish%3A%3C%2FP%3E%0A%3COL%3E%0A%3CLI%3EThe%20HTTP%20header%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CEM%3EX-ARR-ClientCert%3C%2FEM%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3Eseems%20to%20be%20Microsoft-specific%2C%20instead%20of%20any%20open%20spec%20(maybe%20there%20is%20no%20such%20spec%3F)%20What%20is%20the%20story%20for%20AKS%3F%20While%20APIM%20is%20Microsoft%2C%20AKS%20is%20internally%20Kubernetes.%3C%2FLI%3E%0A%3CLI%3EThe%20client%20cert%20authentication%20happens%20inside%20application%20code.%20This%20defeats%20the%20purpose%20of%20using%20APIM%20as%20API%20gateway.%3C%2FLI%3E%0A%3C%2FOL%3E%0A%3CP%3EOur%20goal%20is%20to%20achieve%20mTLS%20between%20APIM%20and%20AKS%20without%20custom%20security%20code%20in%20applications%20in%20AKS%20pods.%20Rather%20we%20hope%20to%20rely%20on%20AKS%20NGINX%20ingress%20controller%20and%20ingress%20resources%20to%20perform%20client%20cert%20authentication%20at%20infrastructure%20level.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22user-content-prerequisites%22%20id%3D%22toc-hId--534983120%22%20id%3D%22toc-hId--534834294%22%3EPrerequisites%3C%2FH2%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3E%3CP%3Ekubectl.%20Minimum%20version%20required%20is%20v%201.18.%20To%20find%20your%20kubectl%20client%20version%3A%3C%2FP%3E%0A%3CPRE%3E%3CCODE%3Ekubectl%20version%20--client%0A%3C%2FCODE%3E%3C%2FPRE%3E%0A%3C%2FLI%3E%0A%3CLI%3E%3CP%3Eopenssl%20for%20preparing%20certificates.%20Or%20if%20you%20prefer%2C%20you%20can%20use%20other%20tools%20for%20creating%20self-signed%20certs.%3C%2FP%3E%0A%3C%2FLI%3E%0A%3CLI%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2Fhelm%2Fhelm%2Freleases%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehelm%3C%2FA%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E.%20(Windows%2010%20users%20can%20just%20put%20the%20unzipped%20folder%20anywhere%20and%20add%20the%20corresponding%20PATH%20variable.)%3C%2FP%3E%0A%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22user-content-prepare-dns%22%20id%3D%22toc-hId-1952529713%22%20id%3D%22toc-hId-1952678539%22%3EPrepare%20DNS%3C%2FH2%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ESince%20our%20plan%20is%20not%20to%20use%20VNET%20to%20enclose%20both%20AKS%20and%20APIM%2C%20we%20need%20to%20have%20a%20DNS-resolvable%20domain%20name.%20This%20domain%20name%20will%20be%20mapped%20to%20AKS%20NGINX%20ingress%20controller%20load%20balancer%20static%20IP.%20For%20this%20we%20need%20to%20first%20register%20a%20domain.%20As%20an%20example%2C%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CA%20href%3D%22http%3A%2F%2Faksingress.com%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3Eaksingress.com%3C%2FA%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3Eis%20registered%20and%20its%20subdomain%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CA%20href%3D%22http%3A%2F%2Fdev.aksingress.com%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3Edev.aksingress.com%3C%2FA%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3Ewill%20be%20used%20in%20this%20document.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22user-content-prepare-x.509-certificates%22%20id%3D%22toc-hId-145075250%22%20id%3D%22toc-hId-145224076%22%3EPrepare%20X.509%20Certificates%3C%2FH2%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ESelf-signed%20certs%20can%20be%20used%20for%20dev%2Ftest.%20OpenSSL%20can%20be%20used%20for%20creating%20self-signed%20certs.%3C%2FP%3E%0A%3CP%3EWe%20need%20the%20following%20three%20certs%20in%20certain%20file%20formats%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CTABLE%3E%0A%3CTHEAD%3E%0A%3CTR%3E%0A%3CTH%3EName%3C%2FTH%3E%0A%3CTH%3EPurpose%3C%2FTH%3E%0A%3CTH%3EEnvironment%3C%2FTH%3E%0A%3CTH%3EPrivate%20Key%20Required%3C%2FTH%3E%0A%3CTH%3ERequired%20Formats%3C%2FTH%3E%0A%3C%2FTR%3E%0A%3C%2FTHEAD%3E%0A%3CTBODY%3E%0A%3CTR%3E%0A%3CTD%3ECA%3C%2FTD%3E%0A%3CTD%3ECertificate%20Authority%3C%2FTD%3E%0A%3CTD%3EKubernetes%20Secrets%3C%2FTD%3E%0A%3CTD%3ENo%3C%2FTD%3E%0A%3CTD%3E.crt%2C%20.cer%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%3EServer%3C%2FTD%3E%0A%3CTD%3EServer%20Certificate%3C%2FTD%3E%0A%3CTD%3EKubernetes%20Secrets%3C%2FTD%3E%0A%3CTD%3EYes%3C%2FTD%3E%0A%3CTD%3E.crt%2C%20.key%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%3EClient%3C%2FTD%3E%0A%3CTD%3EClient%20Certificate%3C%2FTD%3E%0A%3CTD%3EAPIM%2C%20test%20client%3C%2FTD%3E%0A%3CTD%3EYes%3C%2FTD%3E%0A%3CTD%3E.crt%2C%20.key%2C%20.pfx%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3C%2FTBODY%3E%0A%3C%2FTABLE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ENOTES%3A%3C%2FP%3E%0A%3COL%3E%0A%3CLI%3ERelying%20on%20legacy%20Common%20Name%20for%20cert%20validation%20will%20be%20deprecated%20in%20Kubernetes.%20It%20is%20recommended%20to%20use%20Subject%20Alternate%20Names%20(SANs)%20instead.%3C%2FLI%3E%0A%3CLI%3EFor%20TLS%2C%20the%20server%20cert%20SAN%20must%20match%20the%20FQDN%20of%20server%20backend%2C%20which%2C%20in%20our%20case%2C%20is%20the%20AKS%20ingress%20resource%20host%20name.%20This%20host%20name%20will%20pair%20with%20the%20static%20IP%20of%20AKS%20NGINX%20ingress%20controller%20we%20will%20create%20later%20on.%3C%2FLI%3E%0A%3CLI%3EThe%20private%20key%20of%20CA%20is%20NOT%20installed%20anywhere%3A%20either%20in%20Kubernetes%20secret%20or%20API%20Management.%3C%2FLI%3E%0A%3C%2FOL%3E%0A%3CP%3EFirst%20let's%20create%20configuration%20files%20for%20both%20client%20and%20server%20certs%3A%3C%2FP%3E%0A%3CP%3EFile%3A%20server_dev.cnf%3C%2FP%3E%0A%3CPRE%3E%3CCODE%3E%5B%20req%20%5D%0Adefault_bits%20%3D%204096%0Aprompt%20%3D%20no%0Aencrypt_key%20%3D%20no%0Adefault_md%20%3D%20sha256%0Adistinguished_name%20%3D%20dn%0Areq_extensions%20%3D%20req_ext%0A%0A%5B%20dn%20%5D%0ACN%20%3D%20dev.aksingress.com%0AemailAddress%20%3D%20acp%40microsoft.com%0AO%20%3D%20Microsoft%0AOU%20%3D%20CSE%0AL%20%3D%20Redmond%0AST%20%3D%20WA%0AC%20%3D%20US%0A%0A%5B%20req_ext%20%5D%0AsubjectAltName%20%3D%20%40alt_names%0A%5Balt_names%5D%0ADNS.1%20%20%20%3D%20dev.aksingress.com%0A%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3EFile%3A%20client_dev.cmf%3C%2FP%3E%0A%3CPRE%3E%3CCODE%3E%5B%20req%20%5D%0Adefault_bits%20%3D%204096%0Aprompt%20%3D%20no%0Aencrypt_key%20%3D%20no%0Adefault_md%20%3D%20sha256%0Adistinguished_name%20%3D%20dn%0Areq_extensions%20%3D%20req_ext%0A%0A%5B%20dn%20%5D%0ACN%20%3D%20gateway.com%0AemailAddress%20%3D%20acp%40microsoft.com%0AO%20%3D%20Microsoft%0AOU%20%3D%20CSE%0AL%20%3D%20Redmond%0AST%20%3D%20WA%0AC%20%3D%20US%0A%0A%5B%20req_ext%20%5D%0AsubjectAltName%20%3D%20%40alt_names%0A%5Balt_names%5D%0ADNS.1%20%20%20%3D%20gateway.com%0A%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3EBelow%20we%20assume%20the%20existence%20of%20a%20subfolder%20.%5CmTLS%20under%20openssl%20command.%3C%2FP%3E%0A%3CP%3EOpenssl%20commands%3A%3C%2FP%3E%0A%3CPRE%3E%3CCODE%3E%23%20Create%20CA%0Aopenssl%20req%20-x509%20-sha256%20-newkey%20rsa%3A4096%20-keyout%20mTLS%5Cca.key%20-out%20mTLS%5Cca.crt%20-days%203650%20-nodes%20-subj%20%22%2FCN%3DMy%20Cert%20Authority%22%0A%0A%23%20Generate%20the%20Server%20Key%2C%20and%20Certificate%20and%20Sign%20with%20the%20CA%20Certificate%0Aopenssl%20req%20-out%20mTLS%5Cserver_dev.csr%20-newkey%20rsa%3A4096%20-nodes%20-keyout%20mTLS%5Cserver_dev.key%20-config%20mTLS%5Cserver_dev.cnf%0Aopenssl%20x509%20-req%20-sha256%20-days%203650%20-in%20mTLS%5Cserver_dev.csr%20-CA%20mTLS%5Cca.crt%20-CAkey%20mTLS%5Cca.key%20-set_serial%2001%20-out%20mTLS%5Cserver_dev.crt%0A%0A%23%20Generate%20the%20Client%20Key%2C%20and%20Certificate%20and%20Sign%20with%20the%20CA%20Certificate%0Aopenssl%20req%20-out%20mTLS%5Cclient_dev.csr%20-newkey%20rsa%3A4096%20-nodes%20-keyout%20mTLS%5Cclient_dev.key%20-config%20mTLS%5Cclient_dev.cnf%0Aopenssl%20x509%20-req%20-sha256%20-days%203650%20-in%20mTLS%5Cclient_dev.csr%20-CA%20mTLS%5Cca.crt%20-CAkey%20mTLS%5Cca.key%20-set_serial%2002%20-out%20mTLS%5Cclient_dev.crt%0A%0A%23%20to%20verify%20CSR%20and%20show%20SAN%0Aopenssl%20req%20-text%20-in%20mTLS%5Cserver_dev.csr%20-noout%20-verify%0Aopenssl%20req%20-text%20-in%20mTLS%5Cclient_dev.csr%20-noout%20-verify%0A%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3ESince%20APIM%20expects%20certs%20in%20Microsoft%20format%20such%20as%20.pfx%20and%20.cer%2C%20and%20Kubernetes%20expects%20certs%20in%20.crt%20and%20.key%20format%2C%20we%20need%20the%20following%20conversion.%3C%2FP%3E%0A%3CPRE%3E%3CCODE%3E%23%20Convert%20.crt%20%2B%20.key%20to%20.pfx%0Aopenssl%20pkcs12%20-export%20-out%20mTLS%5Cca.pfx%20-inkey%20mTLS%5Cca.key%20-in%20mTLS%5Cca.crt%0Aopenssl%20pkcs12%20-export%20-out%20mTLS%5Cclient_dev.pfx%20-inkey%20mTLS%5Cclient_dev.key%20-in%20mTLS%5Cclient_dev.crt%0Aopenssl%20pkcs12%20-export%20-out%20mTLS%5Cserver_dev.pfx%20-inkey%20mTLS%5Cserver_dev.key%20-in%20mTLS%5Cserver_dev.crt%0A%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CH2%20id%3D%22toc-hId--1662379213%22%20id%3D%22toc-hId--1662230387%22%3E%26nbsp%3B%3C%2FH2%3E%0A%3CH2%20id%3D%22user-content-create-aks-cluster%22%20id%3D%22toc-hId-825133620%22%20id%3D%22toc-hId-825282446%22%3ECreate%20AKS%20Cluster%3C%2FH2%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ETo%20leverage%20the%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Faks%2Fmanaged-aad%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EAKS-managed%20Azure%20Active%20Directory%20integration%3C%2FA%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3Efeature%2C%20we%20can%20use%20the%20following%20CLI%20to%20create%20an%20AKS%20cluster%20with%20AKS-managed%20AAD%20integration.%3C%2FP%3E%0A%3CPRE%3E%3CCODE%3E%23%20parameters%20used%20for%20creating%20AKS%0Atenant_id%3D%221aaaabcc-73b2-483c-a2c7-b9146631c677%22%0Aaks_admin_group_name%3D%22aks-admin-group%22%0Aaks_api_group_name%3D%22aks-api-group%22%0Aresource_group_name%3D%22rg-aks%22%0Aaks_cluster_name%3D%22aks-cluster-04%22%0A%0Aecho%20%22display%20current%20AAD%20groups%22%0Aaz%20ad%20group%20list%20-o%20table%0A%23%20echo%20%22Create%20a%20group%20for%20AKS%20cluster%20admins%22%0A%23%20az%20ad%20group%20create%20--display-name%20%24aks_admin_group_name%20--mail-nickname%20myalias%0A%0A%23%20echo%20%22Create%20resource%20group%20%24resource_group_name%22%0A%23%20az%20group%20create%20--name%20%24resource_group_name%20--location%20centralus%0A%0Aecho%20%22get%20aks-admin-group%20object%20ID%20for%20%24aks_admin_group_name%3A%22%0Aaks_admin_group_object_id%3D%24(az%20ad%20group%20show%20--group%20%24aks_admin_group_name%20--query%20objectId%20-o%20tsv)%0Aecho%20%24aks_admin_group_object_id%0Aecho%20%22get%20aks-api-group%20object%20ID%20for%20%24aks_api_group_name%3A%22%0Aaks_api_group_object_id%3D%24(az%20ad%20group%20show%20--group%20%24aks_api_group_name%20--query%20objectId%20-o%20tsv)%0Aecho%20%24aks_api_group_object_id%0A%0Aecho%20%22Create%20an%20AAD-managed%20AKS%20cluster%22%0Aaz%20aks%20create%20--resource-group%20%24resource_group_name%20%5C%0A%20%20%20%20%20%20%20%20%20%20%20%20--name%20%24aks_cluster_name%20%5C%0A%20%20%20%20%20%20%20%20%20%20%20%20--node-count%201%20%5C%0A%20%20%20%20%20%20%20%20%20%20%20%20--enable-aad%20%5C%0A%20%20%20%20%20%20%20%20%20%20%20%20--aad-admin-group-object-ids%20%24aks_admin_group_object_id%20%5C%0A%20%20%20%20%20%20%20%20%20%20%20%20--aad-tenant-id%20%24tenant_id%20%0A%20%20%20%20%20%20%20%20%20%20%20%20%23--generate-ssh-keys%0A%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CH2%20id%3D%22toc-hId--982320843%22%20id%3D%22toc-hId--982172017%22%3E%26nbsp%3B%3C%2FH2%3E%0A%3CH2%20id%3D%22user-content-creating-kubernetes-secrets%22%20id%3D%22toc-hId-1505191990%22%20id%3D%22toc-hId-1505340816%22%3ECreating%20Kubernetes%20Secrets%3C%2FH2%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EFirst%20make%20sure%20we%20are%20working%20with%20the%20correct%20AKS%20cluster%20context.%3C%2FP%3E%0A%3CPRE%3E%3CCODE%3Eecho%20%22Ensure%20you%20have%20the%20right%20credential.%20It%20will%20update%20C%3A%5CUsers%5C%5Buserid%5D%5C.kube%5Cconfig%20with%20the%20new%20cluster%20context.%22%0Aaz%20aks%20get-credentials%20-g%20rg-aks%20-n%20aks-cluster-04%0A%0Aecho%20%22Display%20the%20current%20AKS%20cluster%20context%22%0Akubectl%20config%20current-context%0A%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3EAssume%20the%20ca.crt%2C%20server_dev.crt%20and%20server_dev.key%20files%20are%20in%20a%20sub-folder%20named%20mTLS.%3C%2FP%3E%0A%3CPRE%3E%3CCODE%3E%23%20Add%20server.crt%2C%20server.key%20and%20ca.crt%20into%20Kubernetes%20secret%20named%20ingress-secret%0Akubectl%20create%20secret%20generic%20ingress-secret-dev%20--from-file%3Dtls.crt%3D%22mTLS%5Cserver_dev.crt%22%20--from-file%3Dtls.key%3D%22mTLS%5Cserver_dev.key%22%20--from-file%3Dca.crt%3D%22mTLS%5Cca.crt%22%0A%0A%23%20Display%20the%20secret%0Akubectl%20get%20secret%20ingress-secret-dev%0A%23%20List%20all%20secrets%20in%20the%20cluster%0Akubectl%20get%20secrets%0A%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CH2%20id%3D%22toc-hId--302262473%22%20id%3D%22toc-hId--302113647%22%3E%26nbsp%3B%3C%2FH2%3E%0A%3CH2%20id%3D%22user-content-creating-an-nginx-ingress-controller%22%20id%3D%22toc-hId--435947197%22%20id%3D%22toc-hId--435798371%22%3ECreating%20an%20NGINX%20Ingress%20Controller%3C%2FH2%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAn%20ingress%20controller%20is%20required%20to%20work%20with%20Kubernetes%20ingress%20resources.%20We%20will%20define%20client%20authentication%20and%20TLS%20configurations%20in%20an%20ingress%20resource.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWe%20can%20put%20ingress%20controller%20either%20in%20the%20default%20namespace%20or%20a%20custom%20namespace.%3C%2FP%3E%0A%3CPRE%3E%3CCODE%3E%20%20%20%20%23%20Create%20a%20Helm%20repo%0A%20%20%20%20helm%20repo%20add%20ingress-nginx%20https%3A%2F%2Fkubernetes.github.io%2Fingress-nginx%0A%20%20%20%20%23%20To%20see%20the%20Helm%20repo%0A%20%20%20%20helm%20repo%20list%0A%20%20%20%20%23%20Use%20Helm%20to%20deploy%20an%20NGINX%20ingress%20controller%0A%20%20%20%20helm%20install%20nginx-ingress%20ingress-nginx%2Fingress-nginx%20%5C%0A%20%20%20%20%20%20%20%20--namespace%20default%20%5C%0A%20%20%20%20%20%20%20%20--set%20controller.replicaCount%3D1%20%5C%0A%20%20%20%20%20%20%20%20--set%20controller.nodeSelector.%22beta%5C.kubernetes%5C.io%2Fos%22%3Dlinux%20%5C%0A%20%20%20%20%20%20%20%20--set%20defaultBackend.nodeSelector.%22beta%5C.kubernetes%5C.io%2Fos%22%3Dlinux%0A%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3EDetails%20can%20be%20found%20in%20this%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Faks%2Fingress-basic%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Edoc%3C%2FA%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22user-content-have-a-test-app-and-add-its-container-to-acr%22%20id%3D%22toc-hId-2051565636%22%20id%3D%22toc-hId-2051714462%22%3EHave%20a%20Test%20App%20and%20Add%20its%20Container%20to%20ACR%3C%2FH2%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThis%20is%20beyond%20the%20scope%20of%20this%20document.%3C%2FP%3E%0A%3CP%3EIdeally%20for%20better%20test%20result%2C%20the%20REST%20API%20app%20should%20have%20the%20following%3A%3C%2FP%3E%0A%3COL%3E%0A%3CLI%3EAt%20least%20three%20methodts%3A%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CEM%3Eget%2C%20post%2C%20delete%3C%2FEM%3E.%20This%20would%20allow%20us%20to%20test%20RBAC%2C%20such%20as%20only%20a%20specific%20role%20can%20delete%20while%20anyone%20can%20do%20get%2Fcreate.%20RBAC%20is%20out%20of%20the%20scope%20of%20this%20document.%3C%2FLI%3E%0A%3CLI%3EThe%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CEM%3Eget%3C%2FEM%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3Emethod%20should%20return%20the%20full%20request%20headers%20as%20part%20of%20the%20response%20so%20that%20we%20can%20see%20the%20headers%20received%20by%20the%20application%20code.%20As%20a%20request%20goes%20through%20OAuth2%20and%20then%20mTLS%2C%20some%20additional%20headers%20will%20be%20added%20and%20are%20available%20to%20application%20code.%3C%2FLI%3E%0A%3C%2FOL%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22user-content-deploy-the-container-to-aks-pods%22%20id%3D%22toc-hId-244111173%22%20id%3D%22toc-hId-244259999%22%3EDeploy%20the%20Container%20to%20AKS%20Pods%3C%2FH2%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ECreate%20a%20YAML%20file%20and%20save%20it%20with%20the%20name%20%22tinyrest_container.yml.%3C%2FP%3E%0A%3CPRE%3E%3CCODE%3EapiVersion%3A%20apps%2Fv1%0Akind%3A%20Deployment%0Ametadata%3A%0Aname%3A%20tinyrest%0Alabels%3A%0A%20%20%20%20app%3A%20tinyrest%0Aspec%3A%0Areplicas%3A%201%0Aselector%3A%0A%20%20%20%20matchLabels%3A%0A%20%20%20%20app%3A%20tinyrest%0Atemplate%3A%0A%20%20%20%20metadata%3A%0A%20%20%20%20labels%3A%0A%20%20%20%20%20%20%20%20app%3A%20tinyrest%0A%20%20%20%20spec%3A%0A%20%20%20%20containers%3A%0A%20%20%20%20-%20name%3A%20tinyrest%0A%20%20%20%20%20%20%20%20image%3A%20myacr.azurecr.io%2Ftinyrest%3Alatest%0A%20%20%20%20%20%20%20%20ports%3A%0A%20%20%20%20%20%20%20%20-%20containerPort%3A%203000%0A%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Faks%2Fcluster-container-registry-integration%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EAuthenticate%20with%20Azure%20Container%20Registry%20from%20Azure%20Kubernetes%20Service%3C%2FA%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3Eby%20running%20a%20command%20like%20below%3A%3C%2FP%3E%0A%3CPRE%3E%3CCODE%3Eecho%20%22ACR%20integration%20with%20AKS%22%0Aaz%20aks%20update%20--name%20aks-cluster-04%20--resource-group%20rg-aks%20--attach-acr%20myacr%0A%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3EDeploy%20the%20container%20by%20running%20the%20following%20kubectl%20commands%3A%3C%2FP%3E%0A%3CPRE%3E%3CCODE%3Eecho%20%22Deploy%20container%20from%20ACR%20to%20AKS%22%0Akubectl%20apply%20-f%20.%2Faks_bash%2Ftinyrest_container.yml%0Akubectl%20get%20deploy%0Akubectl%20get%20pods%0A%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CH2%20id%3D%22toc-hId--1563343290%22%20id%3D%22toc-hId--1563194464%22%3E%26nbsp%3B%3C%2FH2%3E%0A%3CH2%20id%3D%22user-content-deploy-a-service-to-expose-the-pods%22%20id%3D%22toc-hId-924169543%22%20id%3D%22toc-hId-924318369%22%3EDeploy%20a%20Service%20to%20Expose%20the%20Pods%3C%2FH2%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ECreate%20a%20YAML%20for%20service%3A%3C%2FP%3E%0A%3CPRE%3E%3CCODE%3EapiVersion%3A%20v1%0Akind%3A%20Service%0Ametadata%3A%0Aname%3A%20tinyrest-svc%0Aspec%3A%0Aports%3A%0A-%20port%3A%208080%0A%20%20%20%20targetPort%3A%203000%0A%20%20%20%20protocol%3A%20TCP%0A%20%20%20%20name%3A%20http%0Aselector%3A%0A%20%20%20%20app%3A%20tinyrest%0A%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3EDeploy%20the%20service%20by%20running%20the%20following%20kubectl%20commands%3A%3C%2FP%3E%0A%3CPRE%3E%3CCODE%3Eecho%20%22Deploy%20AKS%20service%22%0Akubectl%20apply%20-f%20.%2Faks_bash%2Ftinyrest_service.yml%0Akubectl%20get%20svc%0A%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3EThe%20second%20command%20should%20show%20the%20NGINX%20ingress%20controller%20as%20a%20LoadBalancer%20in%20addition%20to%20the%20service%20you%20just%20added.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22user-content-deploy-an-ingress-resource-with-security-configurations%22%20id%3D%22toc-hId--883284920%22%20id%3D%22toc-hId--883136094%22%3EDeploy%20an%20Ingress%20Resource%20with%20Security%20Configurations%3C%2FH2%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ECreate%20a%20YAML%20file%20for%20an%20ingress%20resource%3A%3C%2FP%3E%0A%3CPRE%3E%3CCODE%3EapiVersion%3A%20networking.k8s.io%2Fv1beta1%0Akind%3A%20Ingress%0Ametadata%3A%0A%20%20%20%20annotations%3A%0A%20%20%20%20%20%20%20%20nginx.ingress.kubernetes.io%2Fauth-tls-verify-client%3A%20%22on%22%0A%20%20%20%20%20%20%20%20nginx.ingress.kubernetes.io%2Fauth-tls-secret%3A%20%22default%2Fingress-secret-dev%22%0A%20%20%20%20name%3A%20tinyrest-ingress-dev%0A%20%20%20%20namespace%3A%20default%0Aspec%3A%0A%20%20%20%20rules%3A%0A%20%20%20%20-%20host%3A%20dev.aksingress.com%0A%20%20%20%20%20%20%20%20http%3A%0A%20%20%20%20%20%20%20%20paths%3A%0A%20%20%20%20%20%20%20%20-%20backend%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20serviceName%3A%20tinyrest-svc%0A%20%20%20%20%20%20%20%20%20%20%20%20servicePort%3A%208080%0A%20%20%20%20%20%20%20%20%20%20%20%20path%3A%20%2F%0A%20%20%20%20tls%3A%0A%20%20%20%20-%20hosts%3A%0A%20%20%20%20%20%20%20%20-%20dev.aksingress.com%0A%20%20%20%20%20%20%20%20secretName%3A%20ingress-secret-dev%0A%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3EIn%20this%20ingress%20resource%2C%20we%20have%20specified%20the%20following%20for%20mTLS%20authentication%3A%3C%2FP%3E%0A%3COL%3E%0A%3CLI%3EThe%20ingress%20must%20verify%20a%20client%20cert%20(server%20authenticating%20client)%3B%3C%2FLI%3E%0A%3CLI%3EUse%20the%20Kubernetes%20secret%20(named%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CEM%3Eingress-secret-dev%3C%2FEM%3E)%20as%20the%20source%20for%20server%20cert%2C%20server%20key%20and%20CA%20cert.%3C%2FLI%3E%0A%3CLI%3EThe%20hostname%20specified%20(%3CA%20href%3D%22http%3A%2F%2Fdev.aksingress.com%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3Edev.aksingress.com%3C%2FA%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3Ein%20our%20example)%20must%20match%20the%20SAN%20in%20server%20cert.%3C%2FLI%3E%0A%3C%2FOL%3E%0A%3CP%3EDeploy%20the%20ingress%20resource%20with%20rules%20via%20the%20following%20kubectl%20commands%3A%3C%2FP%3E%0A%3CPRE%3E%3CCODE%3Eecho%20%22Deploy%20ingress%20resource%20with%20rules%22%0Akubectl%20apply%20-f%20.%2Faks_bash%2Ftinyrest_ingress_rules.yml%0Akubectl%20get%20ingress%0Akubectl%20describe%20ingress%20tinyrest-ingress-dev%0A%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3EMake%20sure%20you%20see%20a%20static%20external%20IP%20address%20after%20deploying%20the%20ingress%20service.%20There%20might%20be%20a%20short%20delay%20after%20running%20the%20deploy%20command%20before%20the%20static%20IP%20shows%20up.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22user-content-add-*a-record*-to-dns%22%20id%3D%22toc-hId-1604227913%22%20id%3D%22toc-hId-1604376739%22%3EAdd%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CEM%3EA%20Record%3C%2FEM%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3Eto%20DNS%3C%2FH2%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ENow%20the%20static%20IP%20of%20the%20AKS%20ingress%20controller%20is%20available.%20You%20can%20map%20it%20to%20the%20domain%20(%3CA%20href%3D%22http%3A%2F%2Fdev.aksingress.com%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3Edev.aksingress.com%3C%2FA%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E)%20in%20your%20DNS%20setup.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22user-content-testing-aks-configuration-for-mtls-authentication%22%20id%3D%22toc-hId--203226550%22%20id%3D%22toc-hId--203077724%22%3ETesting%20AKS%20Configuration%20for%20mTLS%20Authentication%3C%2FH2%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWith%20client%20cert%20authentication%20and%20CA%20cert%20configured%20in%20AKS%20ingress%20resource%2C%20we%20can%20test%20it%20using%20curl%20client.%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3E%3CP%3EIf%20you%20call%20the%20ingress%20without%20supplying%20the%20client%20cert%20or%20client%20key%2C%20you%20will%20get%20the%20following%20error%3C%2FP%3E%0A%3CP%3E%24%20curl%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CA%20href%3D%22https%3A%2F%2Fdev.aksingress.com%2Fresource%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdev.aksingress.com%2Fresource%3C%2FA%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E-k%3C%2FP%3E%0A%20%3CTITLE%3E400%20No%20required%20SSL%20certificate%20was%20sent%3C%2FTITLE%3E%20%3CCENTER%3E%0A%3CH1%20id%3D%22toc-hId--2139763732%22%20id%3D%22toc-hId--2139614906%22%3E400%20Bad%20Request%3C%2FH1%3E%0A%3C%2FCENTER%3E%3CCENTER%3ENo%20required%20SSL%20certificate%20was%20sent%3C%2FCENTER%3E%3CHR%20%2F%3E%3CCENTER%3Enginx%2F1.19.2%3C%2FCENTER%3E%20%3C%2FLI%3E%0A%3CLI%3E%3CP%3EMutual%20TLS%20authentication%20between%20AKS%20and%20curl%20client%20can%20be%20achieved%20by%20supplying%20client%20cert%2C%20client%20key%20and%20CA%20cert%2C%20as%20shown%20below.%3C%2FP%3E%0A%3CPRE%3E%3CCODE%3E%20curl%20--verbose%20https%3A%2F%2Fdev.aksingress.com%2Fresource%20--cert%20%22mTLS%5Cclient_dev.crt%22%20--key%20%22mTLS%5Cclient_dev.key%22%20--cacert%20%22mTLS%5Cca.crt%22%0A%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3EIf%20our%20test%20application%20returns%20the%20incoming%20headers%2C%20it%20looks%20like%20below%3A%3C%2FP%3E%0A%3CPRE%3E%3CCODE%3E%20%22request_header%22%3A%20%7B%0A%20%20%20%20%20%22host%22%3A%20%22dev.aksingress.com%22%2C%0A%20%20%20%20%20%22ssl-client-verify%22%3A%20%22SUCCESS%22%2C%0A%20%20%20%20%20%22ssl-client-subject-dn%22%3A%20%22C%3DUS%2CST%3DIL%2CL%3DLibertyville%2COU%3DCSE%2CO%3DMicrosoft%2CemailAddress%3Dacp%40microsoft.com%2CCN%3Dgateway.com%22%2C%0A%20%20%20%20%20%22ssl-client-issuer-dn%22%3A%20%22CN%3DMy%20Cert%20Authority%22%2C%0A%20%20%20%20%20%22x-request-id%22%3A%20%22556a994d6f9949eef44189a18294080e%22%2C%0A%20%20%20%20%20%22x-real-ip%22%3A%20%2210.244.0.1%22%2C%0A%20%20%20%20%20%22x-forwarded-for%22%3A%20%2210.244.0.1%22%2C%0A%20%20%20%20%20%22x-forwarded-proto%22%3A%20%22https%22%2C%0A%20%20%20%20%20%22x-forwarded-host%22%3A%20%22dev.aksingress.com%22%2C%0A%20%20%20%20%20%22x-forwarded-port%22%3A%20%22443%22%2C%0A%20%20%20%20%20%22x-scheme%22%3A%20%22https%22%2C%0A%20%20%20%20%20%22user-agent%22%3A%20%22curl%2F7.68.0%22%2C%0A%20%20%20%20%20%22accept%22%3A%20%22*%2F*%22%0A%20%7D%0A%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3EIn%20its%20response%2C%20in%20addition%20to%20the%20correct%20response%20from%20the%20AKS%20pods%2C%20the%20following%20verbose%20section%20indicates%20client%20authentication%20of%20server%20cert%20is%20successful.%3C%2FP%3E%0A%3CPRE%3E%3CCODE%3E%20*%20Server%20certificate%3A%0A%20*%20%20subject%3A%20CN%3Ddev.aksingress.com%3B%20emailAddress%3Dacp%40microsoft.com%3B%20O%3DMicrosoft%3B%20OU%3DCSE%3B%20L%3DLibertyville%3B%20ST%3DIL%3B%20C%3DUS%0A%20*%20%20start%20date%3A%20Sep%2029%2013%3A10%3A18%202020%20GMT%0A%20*%20%20expire%20date%3A%20Sep%2027%2013%3A10%3A18%202030%20GMT%0A%20*%20%20common%20name%3A%20dev.aksingress.com%20(matched)%0A%20*%20%20issuer%3A%20CN%3DMy%20Cert%20Authority%0A%20*%20%20SSL%20certificate%20verify%20ok.%0A%3C%2FCODE%3E%3C%2FPRE%3E%0A%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CH2%20id%3D%22toc-hId-476831820%22%20id%3D%22toc-hId-476980646%22%3E%26nbsp%3B%3C%2FH2%3E%0A%3CH2%20id%3D%22user-content-configuring-mtls-in-apim%22%20id%3D%22toc-hId--632460702%22%20id%3D%22toc-hId--632311876%22%3EConfiguring%20mTLS%20in%20APIM%3C%2FH2%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EDetails%20can%20be%20found%20in%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fapi-management%2Fapi-management-howto-mutual-certificates%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EHow%20to%20secure%20back-end%20services%20using%20client%20certificate%20authentication%20in%20Azure%20API%20Management%3C%2FA%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22user-content-end-to-end-test%22%20id%3D%22toc-hId-1855052131%22%20id%3D%22toc-hId-1855200957%22%3EEnd-to-End%20Test%3C%2FH2%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ETo%20perform%20end-to-end%20test%2C%20we%20also%20need%20to%20follow%20the%20other%20document%20to%20configure%20OAuth2.%3C%2FP%3E%0A%3CP%3EThe%20end-to-end%20test%20covers%20two%20security%20loops%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EOAuth2%3C%2FSTRONG%3E%2C%20which%20covers%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3Eclient%20app%20(either%20public%20or%20private%20client)%3C%2FLI%3E%0A%3CLI%3EIdentity%20Provider%20(any%20OAuthe2-compliant%20Identity%20Provider%20such%20as%20Azure%20AD%20or%20MITREid%20Connect)%3C%2FLI%3E%0A%3CLI%3EAPI%20gateway%20(APIM)%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3E%3CSTRONG%3EmTLS%3C%2FSTRONG%3E%2C%20which%20covers%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EClient%20(APIM)%20authenticating%20server%20(AKS)%3C%2FLI%3E%0A%3CLI%3EServer%20(AKS)%20authenticating%20client%20(APIM)%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3EThe%20end-to-end%20security%20can%20be%20illustrated%20by%20the%20diagram%20below.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22security_oauth2.drawio.png%22%20style%3D%22width%3A%20745px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F228926i0F822572FCAAD427%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%22security_oauth2.drawio.png%22%20alt%3D%22security_oauth2.drawio.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20OAuth2%20Test%20Tool%20(%3CA%20href%3D%22http%3A%2F%2Faka.ms%2Fott%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttp%3A%2F%2Faka.ms%2Fott%3C%2FA%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E)%20can%20be%20used%20for%20the%20test.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIf%20your%20REST%20API%20used%20for%20test%20returns%20the%20incoming%20HTTP%20headers%20in%20its%20response%20body%2C%20the%20headers%20in%20its%20response%20should%20look%20like%20below%3A%3C%2FP%3E%0A%3CPRE%3E%3CCODE%3E%22request_header%22%3A%20%7B%0A%20%20%20%20%20%20%20%20%22host%22%3A%20%22aksingress.com%22%2C%0A%20%20%20%20%20%20%20%20%22ssl-client-verify%22%3A%20%22SUCCESS%22%2C%0A%20%20%20%20%20%20%20%20%22ssl-client-subject-dn%22%3A%20%22CN%3Dgateway.com%22%2C%0A%20%20%20%20%20%20%20%20%22ssl-client-issuer-dn%22%3A%20%22CN%3DMy%20Cert%20Authority%22%2C%0A%20%20%20%20%20%20%20%20%22x-request-id%22%3A%20%22a1e62e86b490b1afc29f5fd3fbfa802c%22%2C%0A%20%20%20%20%20%20%20%20%22x-real-ip%22%3A%20%2210.244.0.1%22%2C%0A%20%20%20%20%20%20%20%20%22x-forwarded-for%22%3A%20%2210.244.0.1%22%2C%0A%20%20%20%20%20%20%20%20%22x-forwarded-proto%22%3A%20%22https%22%2C%0A%20%20%20%20%20%20%20%20%22x-forwarded-host%22%3A%20%22aksingress.com%22%2C%0A%20%20%20%20%20%20%20%20%22x-forwarded-port%22%3A%20%22443%22%2C%0A%20%20%20%20%20%20%20%20%22x-scheme%22%3A%20%22https%22%2C%0A%20%20%20%20%20%20%20%20%22x-original-forwarded-for%22%3A%20%2267.186.69.18%22%2C%0A%20%20%20%20%20%20%20%20%22x-correlation-id%22%3A%20%2223a8237a-d16b-4471-8c19-058717c982cf%22%2C%0A%20%20%20%20%20%20%20%20%22origin%22%3A%20%22https%3A%2F%2Fnpmwebapp.azurewebsites.net%22%2C%0A%20%20%20%20%20%20%20%20%22sec-fetch-site%22%3A%20%22cross-site%22%2C%0A%20%20%20%20%20%20%20%20%22sec-fetch-mode%22%3A%20%22cors%22%2C%0A%20%20%20%20%20%20%20%20%22sec-fetch-dest%22%3A%20%22empty%22%2C%0A%20%20%20%20%20%20%20%20%22content-type%22%3A%20%22application%2Fjson%22%2C%0A%20%20%20%20%20%20%20%20%22accept%22%3A%20%22*%2F*%22%2C%0A%20%20%20%20%20%20%20%20%22accept-encoding%22%3A%20%22gzip%2Cdeflate%2Cbr%22%2C%0A%20%20%20%20%20%20%20%20%22accept-language%22%3A%20%22en-US%2Cen%3Bq%3D0.9%22%2C%0A%20%20%20%20%20%20%20%20%22authorization%22%3A%20%22Bearer%20%5Btoken%5D%22%2C%0A%20%20%20%20%20%20%20%20%22referer%22%3A%20%22https%3A%2F%2Fnpmwebapp.azurewebsites.net%2F%22%2C%0A%20%20%20%20%20%20%20%20%22user-agent%22%3A%20%22Mozilla%2F5.0%20(Windows%20NT%2010.0%3B%20Win64%3B%20x64)%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%20Chrome%2F85.0.4183.121%20Safari%2F537.36%22%0A%20%20%20%20%7D%0A%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3EThe%20first%20half%20indicates%20client%20(APIM)%20has%20successfully%20authenticated%20the%20server%20(AKS)%20cert%20and%20forwarded%20the%20request%20to%20the%20server%20(%3CA%20href%3D%22http%3A%2F%2Faksingress.com%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3Eaksingress.com%3C%2FA%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E)%2C%20which%20performs%20its%20own%20authentication%20of%20the%20client.%20The%20second%20half%20shows%20the%20JWT%20used%20for%20OAuth2%20authorization.%20The%20sec-fetch-*%20headers%20indicate%20this%20is%20a%20CORS%20call%20and%20preflight%20is%20required%20(client%20domain%3A%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CA%20href%3D%22http%3A%2F%2Fnpmwebapp.azurewebsites.net%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Enpmwebapp.azurewebsites.net%3C%2FA%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%2C%20API%20gateway%20domain%3A%26nbsp%3B%5Bapim-svc-name%5D.azure-api.net).%20The%20client%20cert%20CN%20(in%20our%20case%20aksingress.com)%20is%20different%20from%20APIM%20FQDN.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22user-content-troubleshooting%22%20id%3D%22toc-hId-47597668%22%20id%3D%22toc-hId-47746494%22%3ETroubleshooting%3C%2FH2%3E%0A%3CH3%20id%3D%22toc-hId--1630774076%22%20id%3D%22toc-hId--1630625250%22%3E%26nbsp%3B%3C%2FH3%3E%0A%3CH3%20id%3D%22user-content-log-of-nginx-ingress-controller%22%20id%3D%22toc-hId-856738757%22%20id%3D%22toc-hId-856887583%22%3ELog%20of%20NGINX%20Ingress%20Controller%3C%2FH3%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EReading%20the%20log%20of%20the%20NGINX%20ingress%20controller%20is%20an%20effective%20way%20to%20troubleshoot.%20You%20can%20retrieve%20the%20ingress%20controller%20log%20via%20the%20following%20kubectl%20commands%3A%3C%2FP%3E%0A%3CPRE%3E%3CCODE%3E%23%20get%20the%20name%20of%20NGINX%20ingress%20controller%0Akubectl%20get%20pods%20-n%20default%20%7C%20grep%20nginx-ingress%0A%23%20get%20the%20log%20for%20the%20NGINX%20ingress%20controller%0Akubectl%20logs%20-n%20default%20nginx-ingress-ingress-nginx-controller-7cb87487f5-jg8xw%0A%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3EBelow%20is%20a%20sample%20error%20entry%20in%20such%20log%3A%3C%2FP%3E%0A%3CPRE%3E%3CCODE%3EW0923%2016%3A30%3A28.571719%20%20%20%20%20%20%206%20controller.go%3A1146%5D%20Unexpected%20error%20validating%20SSL%20certificate%20%22default%2Fingress-secret%22%20for%20server%20%22aksingress.com%22%3A%20x509%3A%20certificate%20relies%20on%20legacy%20Common%20Name%20field%2C%20use%20SANs%20or%20temporarily%20enable%20Common%20Name%20matching%20with%20GODEBUG%3Dx509ignoreCN%3D0%0A%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3Ewhile%20a%20successful%20request%20can%20look%20like%20below%3A%3C%2FP%3E%0A%3CPRE%3E%3CCODE%3E10.244.0.1%20-%20-%20%5B23%2FSep%2F2020%3A22%3A30%3A38%20%2B0000%5D%20%22GET%20%2Fresource%20HTTP%2F2.0%22%20200%20459%20%22-%22%20%22curl%2F7.68.0%22%2038%200.002%20%5Bdefault-tinyrest-svc-8080%5D%20%5B%5D%2010.244.0.13%3A3000%20459%200.000%20%0A200%2069ad615ba1e85defdaba5a0ba57529df%0A%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CH3%20id%3D%22user-content-ingress-resource-setup%22%20id%3D%22toc-hId--950715706%22%20id%3D%22toc-hId--950566880%22%3EIngress%20Resource%20Setup%3C%2FH3%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAnother%20thing%20to%20check%20is%20the%20ingress%20resource%20setup%3A%3C%2FP%3E%0A%3CPRE%3E%3CCODE%3E%24%20kubectl%20describe%20ingress%20tinyrest-ingress-dev%0AName%3A%20%20%20%20%20%20%20%20%20%20%20%20%20tinyrest-ingress-dev%0ANamespace%3A%20%20%20%20%20%20%20%20default%0AAddress%3A%20%20%20%20%20%20%20%20%20%2052.154.41.113%0ADefault%20backend%3A%20%20default-http-backend%3A80%20(%26lt%3B%20endpoints%3D%22%22%26gt%3B)%0ATLS%3A%0Aingress-secret-dev%20terminates%20dev.aksingress.com%0ARules%3A%0AHost%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20Path%20%20Backends%0A----%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20----%20%20--------%0Adev.aksingress.com%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%2F%20%20%20tinyrest-svc%3A8080%20(10.244.0.13%3A3000)%0AAnnotations%3A%20%20%20%20%20%20%20%20%20%20nginx.ingress.kubernetes.io%2Fauth-tls-secret%3A%20default%2Fingress-secret-dev%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20nginx.ingress.kubernetes.io%2Fauth-tls-verify-client%3A%20on%0AEvents%3A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%3CNONE%3E%0A%3C%2FNONE%3E%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3ENotice%20that%20since%20we%20have%20configured%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CA%20href%3D%22http%3A%2F%2Fnginx.ingress.kubernetes.io%2Fauth-tls-verify-client%3A%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3Enginx.ingress.kubernetes.io%2Fauth-tls-verify-client%3A%3C%2FA%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3Eon%2C%20the%20error%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CEM%3Eendpoints%20%22default-http-backend%22%20not%20found%3C%2FEM%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3Eis%20expected.%3C%2FP%3E%0A%3CH3%20id%3D%22user-content-missing-client-cert-for-server-authentication-of-client%22%20id%3D%22toc-hId-1536797127%22%20id%3D%22toc-hId-1536945953%22%3EMissing%20Client%20Cert%20for%20Server%20Authentication%20of%20Client%3C%2FH3%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIf%20error%20indicates%20missing%20client%20cert%2C%20please%20check%20the%20API%20inbound%20policy%20in%20APIM.%20In%20order%20for%20APIM%20to%20supply%20client%20cert%20to%20AKS%20ingress%20resource%20for%20authenticating%20the%20client%2C%20the%20inbound%20processing%20policy%20must%20contain%20the%20following%20node%3C%2FP%3E%0A%3CPRE%3E%3CCODE%3E%3CAUTHENTICATION-CERTIFICATE%20thumbprint%3D%2205F6B958079A4FC88978946FB3DA65B37F0F9E4E%22%3E%3C%2FAUTHENTICATION-CERTIFICATE%3E%0A%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3EMake%20sure%20the%20thumbprint%20matches%20with%20the%20thumbprint%20of%20the%20client%20cert%20you%20installed%20on%20APIM.%3C%2FP%3E%0A%3CH3%20id%3D%22user-content-ingress-secret-cannot-be-found%22%20id%3D%22toc-hId--270657336%22%20id%3D%22toc-hId--270508510%22%3EIngress%20Secret%20Cannot%20be%20Found%3C%2FH3%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ECheck%20the%20YAML%20file%20for%20the%20ingress%20resource%20to%20make%20sure%20the%20secret%20name%20and%20namespace%20are%20correct.%20You%20can%20use%20kubectl%20to%20describe%20the%20Kubernetes%20secret%20and%20should%20see%20the%20following%20three%20certs%2Fkey%3A%3C%2FP%3E%0A%3CPRE%3E%3CCODE%3E%24%20kubectl%20describe%20secret%20ingress-secret-dev%0AName%3A%20%20%20%20%20%20%20%20%20ingress-secret%0ANamespace%3A%20%20%20%20default%0ALabels%3A%20%20%20%20%20%20%20%3CNONE%3E%0AAnnotations%3A%20%20%3CNONE%3E%0A%0AType%3A%20%20Opaque%0A%0AData%0A%3D%3D%3D%3D%0Atls.crt%3A%20%201675%20bytes%0Atls.key%3A%20%203272%20bytes%0Aca.crt%3A%20%20%201809%20bytes%3C%2FNONE%3E%3C%2FNONE%3E%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-1813887%22%20slang%3D%22en-US%22%3E%3CP%3EProvide%20detailed%20steps%20for%20configuring%20mTLS%20between%20AKS%20and%20API%20Management.%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1813887%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20API%20management%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Microsoft

Mutual TLS Authentication between Azure Kubernetes Service and API Management

 

By (alphabetically): Akinlolu Akindele, Dan Balma, Maarten Van De Bospoort, Erin Corson, Nick Drouin, Heba Elayoty, Andrei Ermilov, David Giard, Michael Green, Alfredo Chavez Hernandez, Hao Luo, Maggie Marxen, Siva Mullapudi, Nsikan Udoyen, William Zhang

 

Introduction

 

We have two goals in this doc:

  1. How to set up AKS cluster for mutual TLS (mTLS) authentication between Azure Kubernetes Service (AKS) NGINX ingress controller and a client app such as curl? If you are not using a gateway for your microservices or using a gateway other than Azure API Management (APIM), this portion is what might interests you.
  2. How to set up APIM for mTLS between AKS and APIM? This covers the case in which APIM is used as the API gateway for REST services hosted in AKS cluster.

This is a sister doc to Use MITREid Connect for OAuth2 Authorization in API Management: one covers securing AKS via mTLS between AKS and APIM while the other covers securing APIM via OAuth2 and OpenID Connect across APIM, Identity Provider and clients.

Our goal is for AKS (as a service) to authenticate APIM (as a client) so that only calls from APIM with a valid client cert with private key can get thru. Therefore what we need is not TLS between APIM and AKS which is for client (APIM in our case) to authenticate server (AKS in our case). What we need is mutual TLS.

As a reference and also for context, this  and this  documents provide mTLS authentication between APIM and Azure App Service.

The steps:

  1. HTTPS calls from APIM is intercepted by frontend load balancer of App Service
  2. App Service frontend load balancer injects an X-ARR-ClientCert request header with the client certificate (base64 string) as the header value, before forwarding the request to application code.
  3. Application code retrieves the cert string such as headers["X-ARR-ClientCert"] and converts it to an X.509 cert.
  4. Application code parses the cert and verifies the attributes and claims as client authentication.

As you can see, the approach for mTLS between APIM and App Service is not as good as we wish:

  1. The HTTP header X-ARR-ClientCert seems to be Microsoft-specific, instead of any open spec (maybe there is no such spec?) What is the story for AKS? While APIM is Microsoft, AKS is internally Kubernetes.
  2. The client cert authentication happens inside application code. This defeats the purpose of using APIM as API gateway.

Our goal is to achieve mTLS between APIM and AKS without custom security code in applications in AKS pods. Rather we hope to rely on AKS NGINX ingress controller and ingress resources to perform client cert authentication at infrastructure level.

 

Prerequisites

 

  • kubectl. Minimum version required is v 1.18. To find your kubectl client version:

    kubectl version --client
    
  • openssl for preparing certificates. Or if you prefer, you can use other tools for creating self-signed certs.

  • helm . (Windows 10 users can just put the unzipped folder anywhere and add the corresponding PATH variable.)

 

Prepare DNS

 

Since our plan is not to use VNET to enclose both AKS and APIM, we need to have a DNS-resolvable domain name. This domain name will be mapped to AKS NGINX ingress controller load balancer static IP. For this we need to first register a domain. As an example, aksingress.com  is registered and its subdomain dev.aksingress.com  will be used in this document.

 

Prepare X.509 Certificates

 

Self-signed certs can be used for dev/test. OpenSSL can be used for creating self-signed certs.

We need the following three certs in certain file formats:

 

Name Purpose Environment Private Key Required Required Formats
CA Certificate Authority Kubernetes Secrets No .crt, .cer
Server Server Certificate Kubernetes Secrets Yes .crt, .key
Client Client Certificate APIM, test client Yes .crt, .key, .pfx

 

NOTES:

  1. Relying on legacy Common Name for cert validation will be deprecated in Kubernetes. It is recommended to use Subject Alternate Names (SANs) instead.
  2. For TLS, the server cert SAN must match the FQDN of server backend, which, in our case, is the AKS ingress resource host name. This host name will pair with the static IP of AKS NGINX ingress controller we will create later on.
  3. The private key of CA is NOT installed anywhere: either in Kubernetes secret or API Management.

First let's create configuration files for both client and server certs:

File: server_dev.cnf

[ req ]
default_bits = 4096
prompt = no
encrypt_key = no
default_md = sha256
distinguished_name = dn
req_extensions = req_ext

[ dn ]
CN = dev.aksingress.com
emailAddress = acp@microsoft.com
O = Microsoft
OU = CSE
L = Redmond
ST = WA
C = US

[ req_ext ]
subjectAltName = @alt_names
[alt_names]
DNS.1   = dev.aksingress.com

File: client_dev.cmf

[ req ]
default_bits = 4096
prompt = no
encrypt_key = no
default_md = sha256
distinguished_name = dn
req_extensions = req_ext

[ dn ]
CN = gateway.com
emailAddress = acp@microsoft.com
O = Microsoft
OU = CSE
L = Redmond
ST = WA
C = US

[ req_ext ]
subjectAltName = @alt_names
[alt_names]
DNS.1   = gateway.com

Below we assume the existence of a subfolder .\mTLS under openssl command.

Openssl commands:

# Create CA
openssl req -x509 -sha256 -newkey rsa:4096 -keyout mTLS\ca.key -out mTLS\ca.crt -days 3650 -nodes -subj "/CN=My Cert Authority"

# Generate the Server Key, and Certificate and Sign with the CA Certificate
openssl req -out mTLS\server_dev.csr -newkey rsa:4096 -nodes -keyout mTLS\server_dev.key -config mTLS\server_dev.cnf
openssl x509 -req -sha256 -days 3650 -in mTLS\server_dev.csr -CA mTLS\ca.crt -CAkey mTLS\ca.key -set_serial 01 -out mTLS\server_dev.crt

# Generate the Client Key, and Certificate and Sign with the CA Certificate
openssl req -out mTLS\client_dev.csr -newkey rsa:4096 -nodes -keyout mTLS\client_dev.key -config mTLS\client_dev.cnf
openssl x509 -req -sha256 -days 3650 -in mTLS\client_dev.csr -CA mTLS\ca.crt -CAkey mTLS\ca.key -set_serial 02 -out mTLS\client_dev.crt

# to verify CSR and show SAN
openssl req -text -in mTLS\server_dev.csr -noout -verify
openssl req -text -in mTLS\client_dev.csr -noout -verify

Since APIM expects certs in Microsoft format such as .pfx and .cer, and Kubernetes expects certs in .crt and .key format, we need the following conversion.

# Convert .crt + .key to .pfx
openssl pkcs12 -export -out mTLS\ca.pfx -inkey mTLS\ca.key -in mTLS\ca.crt
openssl pkcs12 -export -out mTLS\client_dev.pfx -inkey mTLS\client_dev.key -in mTLS\client_dev.crt
openssl pkcs12 -export -out mTLS\server_dev.pfx -inkey mTLS\server_dev.key -in mTLS\server_dev.crt

 

Create AKS Cluster

 

To leverage the AKS-managed Azure Active Directory integration  feature, we can use the following CLI to create an AKS cluster with AKS-managed AAD integration.

# parameters used for creating AKS
tenant_id="1aaaabcc-73b2-483c-a2c7-b9146631c677"
aks_admin_group_name="aks-admin-group"
aks_api_group_name="aks-api-group"
resource_group_name="rg-aks"
aks_cluster_name="aks-cluster-04"

echo "display current AAD groups"
az ad group list -o table
# echo "Create a group for AKS cluster admins"
# az ad group create --display-name $aks_admin_group_name --mail-nickname myalias

# echo "Create resource group $resource_group_name"
# az group create --name $resource_group_name --location centralus

echo "get aks-admin-group object ID for $aks_admin_group_name:"
aks_admin_group_object_id=$(az ad group show --group $aks_admin_group_name --query objectId -o tsv)
echo $aks_admin_group_object_id
echo "get aks-api-group object ID for $aks_api_group_name:"
aks_api_group_object_id=$(az ad group show --group $aks_api_group_name --query objectId -o tsv)
echo $aks_api_group_object_id

echo "Create an AAD-managed AKS cluster"
az aks create --resource-group $resource_group_name \
            --name $aks_cluster_name \
            --node-count 1 \
            --enable-aad \
            --aad-admin-group-object-ids $aks_admin_group_object_id \
            --aad-tenant-id $tenant_id 
            #--generate-ssh-keys

 

Creating Kubernetes Secrets

 

First make sure we are working with the correct AKS cluster context.

echo "Ensure you have the right credential. It will update C:\Users\[userid]\.kube\config with the new cluster context."
az aks get-credentials -g rg-aks -n aks-cluster-04

echo "Display the current AKS cluster context"
kubectl config current-context

Assume the ca.crt, server_dev.crt and server_dev.key files are in a sub-folder named mTLS.

# Add server.crt, server.key and ca.crt into Kubernetes secret named ingress-secret
kubectl create secret generic ingress-secret-dev --from-file=tls.crt="mTLS\server_dev.crt" --from-file=tls.key="mTLS\server_dev.key" --from-file=ca.crt="mTLS\ca.crt"

# Display the secret
kubectl get secret ingress-secret-dev
# List all secrets in the cluster
kubectl get secrets

 

Creating an NGINX Ingress Controller

 

An ingress controller is required to work with Kubernetes ingress resources. We will define client authentication and TLS configurations in an ingress resource.

 

We can put ingress controller either in the default namespace or a custom namespace.

    # Create a Helm repo
    helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
    # To see the Helm repo
    helm repo list
    # Use Helm to deploy an NGINX ingress controller
    helm install nginx-ingress ingress-nginx/ingress-nginx \
        --namespace default \
        --set controller.replicaCount=1 \
        --set controller.nodeSelector."beta\.kubernetes\.io/os"=linux \
        --set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux

Details can be found in this doc .

 

Have a Test App and Add its Container to ACR

 

This is beyond the scope of this document.

Ideally for better test result, the REST API app should have the following:

  1. At least three methodts: get, post, delete. This would allow us to test RBAC, such as only a specific role can delete while anyone can do get/create. RBAC is out of the scope of this document.
  2. The get method should return the full request headers as part of the response so that we can see the headers received by the application code. As a request goes through OAuth2 and then mTLS, some additional headers will be added and are available to application code.

 

Deploy the Container to AKS Pods

 

Create a YAML file and save it with the name "tinyrest_container.yml.

apiVersion: apps/v1
kind: Deployment
metadata:
name: tinyrest
labels:
    app: tinyrest
spec:
replicas: 1
selector:
    matchLabels:
    app: tinyrest
template:
    metadata:
    labels:
        app: tinyrest
    spec:
    containers:
    - name: tinyrest
        image: myacr.azurecr.io/tinyrest:latest
        ports:
        - containerPort: 3000

Authenticate with Azure Container Registry from Azure Kubernetes Service  by running a command like below:

echo "ACR integration with AKS"
az aks update --name aks-cluster-04 --resource-group rg-aks --attach-acr myacr

Deploy the container by running the following kubectl commands:

echo "Deploy container from ACR to AKS"
kubectl apply -f ./aks_bash/tinyrest_container.yml
kubectl get deploy
kubectl get pods

 

Deploy a Service to Expose the Pods

 

Create a YAML for service:

apiVersion: v1
kind: Service
metadata:
name: tinyrest-svc
spec:
ports:
- port: 8080
    targetPort: 3000
    protocol: TCP
    name: http
selector:
    app: tinyrest

Deploy the service by running the following kubectl commands:

echo "Deploy AKS service"
kubectl apply -f ./aks_bash/tinyrest_service.yml
kubectl get svc

The second command should show the NGINX ingress controller as a LoadBalancer in addition to the service you just added.

 

Deploy an Ingress Resource with Security Configurations

 

Create a YAML file for an ingress resource:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
    annotations:
        nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
        nginx.ingress.kubernetes.io/auth-tls-secret: "default/ingress-secret-dev"
    name: tinyrest-ingress-dev
    namespace: default
spec:
    rules:
    - host: dev.aksingress.com
        http:
        paths:
        - backend:
            serviceName: tinyrest-svc
            servicePort: 8080
            path: /
    tls:
    - hosts:
        - dev.aksingress.com
        secretName: ingress-secret-dev

In this ingress resource, we have specified the following for mTLS authentication:

  1. The ingress must verify a client cert (server authenticating client);
  2. Use the Kubernetes secret (named ingress-secret-dev) as the source for server cert, server key and CA cert.
  3. The hostname specified (dev.aksingress.com  in our example) must match the SAN in server cert.

Deploy the ingress resource with rules via the following kubectl commands:

echo "Deploy ingress resource with rules"
kubectl apply -f ./aks_bash/tinyrest_ingress_rules.yml
kubectl get ingress
kubectl describe ingress tinyrest-ingress-dev

Make sure you see a static external IP address after deploying the ingress service. There might be a short delay after running the deploy command before the static IP shows up.

 

Add A Record to DNS

 

Now the static IP of the AKS ingress controller is available. You can map it to the domain (dev.aksingress.com ) in your DNS setup.

 

Testing AKS Configuration for mTLS Authentication

 

With client cert authentication and CA cert configured in AKS ingress resource, we can test it using curl client.

  • If you call the ingress without supplying the client cert or client key, you will get the following error

    $ curl https://dev.aksingress.com/resource  -k

    <html> <head><title>400 No required SSL certificate was sent</title></head> <body>

    400 Bad Request

    No required SSL certificate was sent

    nginx/1.19.2
    </body> </html>
  • Mutual TLS authentication between AKS and curl client can be achieved by supplying client cert, client key and CA cert, as shown below.

     curl --verbose https://dev.aksingress.com/resource --cert "mTLS\client_dev.crt" --key "mTLS\client_dev.key" --cacert "mTLS\ca.crt"
    

    If our test application returns the incoming headers, it looks like below:

     "request_header": {
         "host": "dev.aksingress.com",
         "ssl-client-verify": "SUCCESS",
         "ssl-client-subject-dn": "C=US,ST=IL,L=Libertyville,OU=CSE,O=Microsoft,emailAddress=acp@microsoft.com,CN=gateway.com",
         "ssl-client-issuer-dn": "CN=My Cert Authority",
         "x-request-id": "556a994d6f9949eef44189a18294080e",
         "x-real-ip": "10.244.0.1",
         "x-forwarded-for": "10.244.0.1",
         "x-forwarded-proto": "https",
         "x-forwarded-host": "dev.aksingress.com",
         "x-forwarded-port": "443",
         "x-scheme": "https",
         "user-agent": "curl/7.68.0",
         "accept": "*/*"
     }
    

    In its response, in addition to the correct response from the AKS pods, the following verbose section indicates client authentication of server cert is successful.

     * Server certificate:
     *  subject: CN=dev.aksingress.com; emailAddress=acp@microsoft.com; O=Microsoft; OU=CSE; L=Libertyville; ST=IL; C=US
     *  start date: Sep 29 13:10:18 2020 GMT
     *  expire date: Sep 27 13:10:18 2030 GMT
     *  common name: dev.aksingress.com (matched)
     *  issuer: CN=My Cert Authority
     *  SSL certificate verify ok.
    

 

Configuring mTLS in APIM

 

Details can be found in How to secure back-end services using client certificate authentication in Azure API Management .

 

End-to-End Test

 

To perform end-to-end test, we also need to follow the other document to configure OAuth2.

The end-to-end test covers two security loops:

 

OAuth2, which covers

  • client app (either public or private client)
  • Identity Provider (any OAuthe2-compliant Identity Provider such as Azure AD or MITREid Connect)
  • API gateway (APIM)

mTLS, which covers

  • Client (APIM) authenticating server (AKS)
  • Server (AKS) authenticating client (APIM)

The end-to-end security can be illustrated by the diagram below.

 

security_oauth2.drawio.png

 

The OAuth2 Test Tool (http://aka.ms/ott ) can be used for the test.

 

If your REST API used for test returns the incoming HTTP headers in its response body, the headers in its response should look like below:

"request_header": {
        "host": "aksingress.com",
        "ssl-client-verify": "SUCCESS",
        "ssl-client-subject-dn": "CN=gateway.com",
        "ssl-client-issuer-dn": "CN=My Cert Authority",
        "x-request-id": "a1e62e86b490b1afc29f5fd3fbfa802c",
        "x-real-ip": "10.244.0.1",
        "x-forwarded-for": "10.244.0.1",
        "x-forwarded-proto": "https",
        "x-forwarded-host": "aksingress.com",
        "x-forwarded-port": "443",
        "x-scheme": "https",
        "x-original-forwarded-for": "67.186.69.18",
        "x-correlation-id": "23a8237a-d16b-4471-8c19-058717c982cf",
        "origin": "https://npmwebapp.azurewebsites.net",
        "sec-fetch-site": "cross-site",
        "sec-fetch-mode": "cors",
        "sec-fetch-dest": "empty",
        "content-type": "application/json",
        "accept": "*/*",
        "accept-encoding": "gzip,deflate,br",
        "accept-language": "en-US,en;q=0.9",
        "authorization": "Bearer [token]",
        "referer": "https://npmwebapp.azurewebsites.net/",
        "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"
    }

The first half indicates client (APIM) has successfully authenticated the server (AKS) cert and forwarded the request to the server (aksingress.com ), which performs its own authentication of the client. The second half shows the JWT used for OAuth2 authorization. The sec-fetch-* headers indicate this is a CORS call and preflight is required (client domain: npmwebapp.azurewebsites.net , API gateway domain: [apim-svc-name].azure-api.net). The client cert CN (in our case aksingress.com) is different from APIM FQDN.

 

Troubleshooting

 

Log of NGINX Ingress Controller

 

Reading the log of the NGINX ingress controller is an effective way to troubleshoot. You can retrieve the ingress controller log via the following kubectl commands:

# get the name of NGINX ingress controller
kubectl get pods -n default | grep nginx-ingress
# get the log for the NGINX ingress controller
kubectl logs -n default nginx-ingress-ingress-nginx-controller-7cb87487f5-jg8xw

Below is a sample error entry in such log:

W0923 16:30:28.571719       6 controller.go:1146] Unexpected error validating SSL certificate "default/ingress-secret" for server "aksingress.com": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0

while a successful request can look like below:

10.244.0.1 - - [23/Sep/2020:22:30:38 +0000] "GET /resource HTTP/2.0" 200 459 "-" "curl/7.68.0" 38 0.002 [default-tinyrest-svc-8080] [] 10.244.0.13:3000 459 0.000 
200 69ad615ba1e85defdaba5a0ba57529df

Ingress Resource Setup

 

Another thing to check is the ingress resource setup:

$ kubectl describe ingress tinyrest-ingress-dev
Name:             tinyrest-ingress-dev
Namespace:        default
Address:          52.154.41.113
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
ingress-secret-dev terminates dev.aksingress.com
Rules:
Host                Path  Backends
----                ----  --------
dev.aksingress.com
                    /   tinyrest-svc:8080 (10.244.0.13:3000)
Annotations:          nginx.ingress.kubernetes.io/auth-tls-secret: default/ingress-secret-dev
                    nginx.ingress.kubernetes.io/auth-tls-verify-client: on
Events:               <none>

Notice that since we have configured nginx.ingress.kubernetes.io/auth-tls-verify-client:  on, the error endpoints "default-http-backend" not found is expected.

Missing Client Cert for Server Authentication of Client

 

If error indicates missing client cert, please check the API inbound policy in APIM. In order for APIM to supply client cert to AKS ingress resource for authenticating the client, the inbound processing policy must contain the following node

<authentication-certificate thumbprint="05F6B958079A4FC88978946FB3DA65B37F0F9E4E" />

Make sure the thumbprint matches with the thumbprint of the client cert you installed on APIM.

Ingress Secret Cannot be Found

 

Check the YAML file for the ingress resource to make sure the secret name and namespace are correct. You can use kubectl to describe the Kubernetes secret and should see the following three certs/key:

$ kubectl describe secret ingress-secret-dev
Name:         ingress-secret
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
tls.crt:  1675 bytes
tls.key:  3272 bytes
ca.crt:   1809 bytes