%3CLINGO-SUB%20id%3D%22lingo-sub-1819490%22%20slang%3D%22en-US%22%3EManaging%20and%20Rotating%20Secrets%20with%20Azure%20Key%20Vault%2C%20Managed%20Services%2C%20and%20some%20automation%20%E2%80%93%20Part%202%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1819490%22%20slang%3D%22en-US%22%3E%3CP%3EIn%20the%20previous%20%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fazure-architecture-blog%2Fmanaging-and-rotating-secrets-with-azure-keyvault-managed%2Fba-p%2F1800612%22%20target%3D%22_blank%22%20rel%3D%22noopener%22%3Epost%3C%2FA%3E%20I%20discussed%20the%20bigger%20picture%20for%20Harpocrates%E2%80%99%20existence%20and%20goals.%20In%20this%20post%2C%20I%20will%20begin%20by%20diving%20into%20the%20constructs%20defined%20by%20Harpocrates%20and%20how%20they%20can%20be%20leveraged%20to%20achieve%20the%20desired%20configuration%20for%20your%20environment.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAs%20discussed%20before%2C%20Harpocrates%20addresses%20the%20need%20for%20a%20periodic%20rotation%20of%20secrets.%20And%20as%20such%2C%20we%20need%20a%20primitive%20to%20describe%20this%20%E2%80%9Cperiod%E2%80%9D.%20To%20allow%20for%20support%20of%20other%20governing%20constructs%20in%20the%20future%2C%20Harpocrates%20utilizes%20a%20concept%20of%20a%20%E2%80%9CPolicy%E2%80%9D.%20This%20primitive%20is%20meant%20to%20enable%20the%20system%20administrator%20to%20define%20a%20set%20of%20time-based%20rotation%20intervals%20that%20can%20then%20be%20applied%20to%20other%20constructs%20in%20the%20system.%20At%20the%20time%20of%20writing%20of%20this%20post%2C%20the%20Policy%20is%20used%20solely%20to%20define%20a%20rotation%20interval%2C%20though%20it%20could%20be%20extended%20to%20add%20additional%20governing%20properties.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EPolicies%20are%20applied%20to%20specific%20services.%20This%20means%20that%20a%20specific%20storage%20account%20can%20have%20a%20%E2%80%9C15-day%E2%80%9D%20rotation%20policy%2C%20while%20another%20one%20can%20have%20a%20%E2%80%9C90-day%E2%80%9D%20policy%20applied%20to%20it.%20This%20enables%20the%20administrator%20to%20describe%20how%20frequently%20a%20particular%20service%20needs%20to%20have%20its%20credentials%20rotated.%20This%20brings%20us%20to%20the%20next%20primitive%3A%20%E2%80%9CService%E2%80%9D.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EA%20%E2%80%9CService%E2%80%9D%20primitive%20is%20used%20to%20describe%20a%20specific%20service%20you%20want%20Harpocrates%20to%20manage.%20For%20example%2C%20you%20may%20want%20to%20enable%20Harpocrates%20to%20rotate%20keys%20for%20%E2%80%9CStorage%20Account%20A%E2%80%9D%2C%20and%20as%20such%2C%20you%20would%20create%20a%20%E2%80%9CService%E2%80%9D%20definition%20for%20such%20service.%20%E2%80%9CService%E2%80%9D%20definition%20would%20also%20provide%20additional%20metadata%20about%20the%20given%20service%2C%20such%20as%20the%20type%20of%20the%20service%20this%20is%20(i.e.%20storage%20account%2C%20cosmosdb%20account%2C%20Redis%20cache%2C%20etc)%3B%20and%20lastly%2C%20a%20%E2%80%9CService%E2%80%9D%20will%20be%20configured%20w%2F%20a%20%E2%80%9Cconnection%20string%E2%80%9D%20that%20would%20describe%20how%20this%20particular%20service%20could%20be%20managed%20by%20the%20secret%20management%20provider.%20In%20most%20cases%2C%20this%20connection%20string%20would%20simply%20contain%20a%20resource%20group%20and%20the%20service%20uri.%20However%2C%20it%20is%20possible%20to%20add%20additional%20tokens%20to%20this%20piece%20of%20metadata%2C%20as%20long%20as%20the%20syntax%20used%20to%20define%20it%20is%20understood%20by%20the%20respective%20secret%20management%20provided.%20For%20example%2C%20for%20Azure%20Storage%20Account%20secret%20management%20provider%2C%20the%20connection%20string%20looks%20like%20this%3A%20%E2%80%9CAccountEndpoint%3D%3CA%20href%3D%22https%3A%2F%2Fstorageaccounta.core.windows.net%3BResourceGroup%3Dmyrg%3B%E2%80%9D%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fstorageaccounta.core.windows.net%3BResourceGroup%3Dmyrg%3B%E2%80%9D%3C%2FA%3E.%20If%20you%20had%20extended%20the%20secret%20management%20provider%20by%20adding%20your%20own%20custom%20provider%2C%20the%20connection%20string%20used%20to%20configure%20such%20service%20would%20simply%20need%20to%20match%20the%20syntax%20your%20customer%20provider%20knows%20how%20to%20interpret.%20At%20the%20time%20of%20writing%20of%20this%20article%2C%20%E2%80%9CService%E2%80%9D%20configuration%20information%20is%20not%20encrypted%2C%20therefore%20you%20should%20avoid%20storing%20secrets%20with%20its%20definitions.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EHarpocrates%20uses%20Azure%20Key%20Vault%20as%20the%20secure%20store%20for%20service%20secrets%20as%20well%20as%20the%20eventing%20mechanism%20to%20signal%20when%20a%20specific%20secret%20is%20to%20be%20rotated.%20In%20Key%20Vault%2C%20secrets%20are%20identified%20by%20a%20unique%20Uri.%20Here%E2%80%99s%20an%20example%20of%20a%20Uri%20for%20KV%20Secret%20named%20%E2%80%9Cmy-secret%E2%80%9D%3A%20%3CA%20href%3D%22https%3A%2F%2Fmykevault.vault.azure.net%2Fsecrets%2Fmy-secret%2F%257bversion%2520id%257d%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fmykevault.vault.azure.net%2Fsecrets%2Fmy-secret%2F%7Bversion%20id%7D%3C%2FA%3E.%20Since%20each%20KV%20secret%20has%20a%20unique%2C%20well%20defined%20uri%2C%20we%20can%20use%20that%20Uri%20to%20correlate%20secrets%20stored%20in%20KV%20to%20the%20services%20we%20defined%20in%20Harpocrates.%20To%20do%20this%2C%20we%20need%20a%20third%20primitive%3A%20%E2%80%9CSecret%E2%80%9D.%20As%20you%20have%20probably%20guessed%2C%20%E2%80%9CSecret%E2%80%9D%20will%20provide%20%E2%80%9Cmapping%20information%E2%80%9D%2C%20allowing%20Harpocrates%20to%20%E2%80%9Ctranslate%E2%80%9D%20a%20KV%20uri%20to%20a%20service%20we%20are%20trying%20to%20manage.%20In%20addition%20to%20this%20important%20piece%20of%20information%2C%20a%20%E2%80%9CSecret%E2%80%9D%20also%20contains%20information%20about%20the%20%E2%80%9Ctype%E2%80%9D%20of%20secret%20it%20is%2C%20as%20well%20as%20the%20format%20in%20which%20this%20secret%20is%20to%20be%20stored%20in%20KV%20(more%20on%20this%20later).%20There%20are%20currently%20two%20types%20of%20secrets%20supported%20by%20Harpocrates%3A%20Attached%20%26amp%3B%20Dependency.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%E2%80%9CAttached%E2%80%9D%20secrets%20represent%20the%20entries%20in%20KV%20that%20store%20raw%20values%20of%20the%20downstream%20service%20secret.%20For%20example%2C%20when%20creating%20management%20metadata%20for%20%E2%80%9CStorageAccountA%E2%80%9D%2C%20one%20would%20create%20a%20Key%20Vault%20secret%20named%20%E2%80%9Cmy-storage-account-a-master-key%E2%80%9D.%20This%20KV%20Secret%20would%20be%20associated%20with%20a%20Harpocrates%20%E2%80%9CSecret%E2%80%9D%20%E2%80%9CStorageAccountASecret%E2%80%9D%20that%20would%20in%20turn%20be%20associated%20with%20the%20%E2%80%9CStorageAccountA%E2%80%9D%20service%20definition%20and%20be%20of%20type%20%E2%80%9CAttached%E2%80%9D.%20Since%20%E2%80%9CStorageAccountA%E2%80%9D%20service%20definition%20is%20associated%20with%20a%20particular%20%E2%80%9CPolicy%E2%80%9D%2C%20that%20policies%20will%20be%20used%20to%20set%20the%20expiration%20date%20of%20the%20key%20vault%20secret%20named%20%E2%80%9Cmy-storage-account-a-master-key%E2%80%9D%2C%20and%20as%20such%2C%20would%20cause%20this%20secret%20to%20expire%2C%20firing%20a%20KV%20event%20that%20Harpocrates%20can%20respond%20to%20by%20rotating%20the%20storage%20account%20key%20%26amp%3B%20updating%20the%20associated%20KV%20secret.%20The%20above%20described%20flow%20is%20depicted%20below%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22flow-attached.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F229293i09A6016F4B67C16D%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%22flow-attached.png%22%20alt%3D%22flow-attached.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3EFigure%201%20%E2%80%9CAttached%E2%80%9D%20secret%20process%20flow%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%E2%80%9CDependency%E2%80%9D%20secrets%20represent%20entries%20in%20Key%20Vault%20that%20depend%20on%20the%20value%20of%20another%20secret.%20This%20type%20of%20a%20secret%20would%20typically%20be%20used%20to%20define%20a%20secret%20consumed%20by%20an%20application%20that%20needs%20credentials%20to%20access%20one%20of%20the%20services%20managed%20by%20Harpocrates.%20Most%20application%20do%20not%20store%20secrets%20in%20plain%20form%20but%20rather%20use%20a%20specific%20format%2C%20such%20as%20a%20connection%20string.%20In%20a%20typical%20application%2C%20one%20would%20likely%20see%20a%20configuration%20entry%20in%20a%20form%20of%20a%20database%20connection%20string%2C%20rather%20than%20multiple%20entries%20each%20representing%20the%3A%20server%20name%2C%20username%2C%20user%20password%2C%20etc.%20As%20such%2C%20%E2%80%9CDependency%E2%80%9D%20secrets%20support%20a%20concept%20of%20a%20Format%20Expression.%20When%20setting%20up%20a%20%E2%80%9CDependency%E2%80%9D%20secret%2C%20a%20system%20admin%20would%20provide%20Harpocrates%20with%20an%20expression%20that%20matches%20the%20expectations%20of%20the%20consuming%20application%20and%20may%20look%20something%20like%20this%3A%20%E2%80%9CServer%3D%7B%7Bserver-secret-id%7D%7D%3BUser%3D%7B%7Buser-secret-id%7D%7D%3BPassword%3D%7B%7Bpassword-secret-id%7D%7D%E2%80%9D.%20The%20values%20specified%20in%20the%20%E2%80%9C%7B%7B%E2%80%9C%20%E2%80%9C%7D%7D%E2%80%9D%20are%20system%20generated%20ids%20of%20the%20secrets%20managed%20by%20Harpocrates%20that%20can%20be%20resolved%20at%20time%20of%20dependency%20update%2C%20thus%20allowing%20the%20actual%20KV%20secret%20value%20to%20be%3A%20%E2%80%9CServer%3DmyDbServer%3BUser%3DMyDbUserName%3BPassword%3DP%40ssword%401%E2%80%9D.%3C%2FP%3E%0A%3CP%3ENow%20that%20we%20have%20the%20language%20to%20describe%20a%20Policy%2C%20Service%2C%20and%20Secrete%2C%20we%20need%20one%20last%20construct%20to%20simplify%20the%20dependency%20secret%20resolution%3A%20Secret%20Dependency.%20This%20is%20not%20as%20much%20of%20a%20primitive%20as%20just%20a%20relationship%20that%20is%20defined%20within%20the%20system.%20When%20a%20secret%20has%20dependencies%20and%20that%20secret%20has%20its%20associated%20service%20keys%20rotated%2C%20the%20system%20can%20schedule%20updates%20to%20all%20other%20secrets%20that%20depend%20on%20it.%20Since%20dependencies%20do%20not%20technically%20require%20that%20they%20depend%20on%20%E2%80%9Cattached%E2%80%9D%20secrets%2C%20one%20could%20design%20an%20extensive%20cascading%20secret%20rotation%20strategy%20to%20be%20executed%20by%20Harpocrates.%20The%20above%20described%20flow%20is%20depicted%20in%20the%20diagram%20below%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22flow-dep.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F229294iE302D98DA19FE545%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%22flow-dep.png%22%20alt%3D%22flow-dep.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3EFigure%202%20%E2%80%9CDependency%E2%80%9D%20secret%20process%20flow%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ENow%20that%20we%20have%20the%20vocabulary%20required%20to%20describe%20the%20behavior%20we%20want%20Harpocrates%20to%20automate%2C%20we%20can%20discuss%20the%20runtime%20requirements%20and%20assumptions%20made%20by%20Harpocrates.%20We%20will%20do%20this%20in%20the%20next%20installment%20of%20this%20blog%20series.%20As%20always%2C%20your%20feedback%20is%20welcomed%20and%20greatly%20appreciated.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-1819490%22%20slang%3D%22en-US%22%3E%3CP%3ESecret%20rotation%20is%20not%20a%20new%20problem.%20In%20cloud-based%20environments%20many%20services%20implement%20secret-based%20authentication%20schemes.%20For%20many%20organizations%2C%20these%20secrets%20must%20be%20rotated%20on%20a%20regular%20schedule.%20In%20addition%20to%20the%20actual%20problem%20of%20rotating%20the%20access%20keys%2C%20there%20exists%20a%20problem%20of%20how%20these%20newly%20rotated%20credentials%20are%20propagated%20to%20all%20the%20applications%20and%20systems%20that%20utilize%20them.%20In%20this%20multi-part%20blog%20series%2C%20I%20will%20discuss%20a%20solution%20that%2C%20through%20automation%2C%20addresses%20both%20the%20scheduled%20rotation%20and%20dependency%20notification%2Fupdate%20requirements.%3C%2FP%3E%3C%2FLINGO-TEASER%3E
Microsoft

In the previous post I discussed the bigger picture for Harpocrates’ existence and goals. In this post, I will begin by diving into the constructs defined by Harpocrates and how they can be leveraged to achieve the desired configuration for your environment.

 

As discussed before, Harpocrates addresses the need for a periodic rotation of secrets. And as such, we need a primitive to describe this “period”. To allow for support of other governing constructs in the future, Harpocrates utilizes a concept of a “Policy”. This primitive is meant to enable the system administrator to define a set of time-based rotation intervals that can then be applied to other constructs in the system. At the time of writing of this post, the Policy is used solely to define a rotation interval, though it could be extended to add additional governing properties.

 

Policies are applied to specific services. This means that a specific storage account can have a “15-day” rotation policy, while another one can have a “90-day” policy applied to it. This enables the administrator to describe how frequently a particular service needs to have its credentials rotated. This brings us to the next primitive: “Service”.

 

A “Service” primitive is used to describe a specific service you want Harpocrates to manage. For example, you may want to enable Harpocrates to rotate keys for “Storage Account A”, and as such, you would create a “Service” definition for such service. “Service” definition would also provide additional metadata about the given service, such as the type of the service this is (i.e. storage account, cosmosdb account, Redis cache, etc); and lastly, a “Service” will be configured w/ a “connection string” that would describe how this particular service could be managed by the secret management provider. In most cases, this connection string would simply contain a resource group and the service uri. However, it is possible to add additional tokens to this piece of metadata, as long as the syntax used to define it is understood by the respective secret management provided. For example, for Azure Storage Account secret management provider, the connection string looks like this: “AccountEndpoint=https://storageaccounta.core.windows.net;ResourceGroup=myrg;”. If you had extended the secret management provider by adding your own custom provider, the connection string used to configure such service would simply need to match the syntax your customer provider knows how to interpret. At the time of writing of this article, “Service” configuration information is not encrypted, therefore you should avoid storing secrets with its definitions.

 

Harpocrates uses Azure Key Vault as the secure store for service secrets as well as the eventing mechanism to signal when a specific secret is to be rotated. In Key Vault, secrets are identified by a unique Uri. Here’s an example of a Uri for KV Secret named “my-secret”: https://mykevault.vault.azure.net/secrets/my-secret/{version id}. Since each KV secret has a unique, well defined uri, we can use that Uri to correlate secrets stored in KV to the services we defined in Harpocrates. To do this, we need a third primitive: “Secret”. As you have probably guessed, “Secret” will provide “mapping information”, allowing Harpocrates to “translate” a KV uri to a service we are trying to manage. In addition to this important piece of information, a “Secret” also contains information about the “type” of secret it is, as well as the format in which this secret is to be stored in KV (more on this later). There are currently two types of secrets supported by Harpocrates: Attached & Dependency.

 

“Attached” secrets represent the entries in KV that store raw values of the downstream service secret. For example, when creating management metadata for “StorageAccountA”, one would create a Key Vault secret named “my-storage-account-a-master-key”. This KV Secret would be associated with a Harpocrates “Secret” “StorageAccountASecret” that would in turn be associated with the “StorageAccountA” service definition and be of type “Attached”. Since “StorageAccountA” service definition is associated with a particular “Policy”, that policies will be used to set the expiration date of the key vault secret named “my-storage-account-a-master-key”, and as such, would cause this secret to expire, firing a KV event that Harpocrates can respond to by rotating the storage account key & updating the associated KV secret. The above described flow is depicted below:

 

flow-attached.png

Figure 1 “Attached” secret process flow

 

“Dependency” secrets represent entries in Key Vault that depend on the value of another secret. This type of a secret would typically be used to define a secret consumed by an application that needs credentials to access one of the services managed by Harpocrates. Most application do not store secrets in plain form but rather use a specific format, such as a connection string. In a typical application, one would likely see a configuration entry in a form of a database connection string, rather than multiple entries each representing the: server name, username, user password, etc. As such, “Dependency” secrets support a concept of a Format Expression. When setting up a “Dependency” secret, a system admin would provide Harpocrates with an expression that matches the expectations of the consuming application and may look something like this: “Server={{server-secret-id}};User={{user-secret-id}};Password={{password-secret-id}}”. The values specified in the “{{“ “}}” are system generated ids of the secrets managed by Harpocrates that can be resolved at time of dependency update, thus allowing the actual KV secret value to be: “Server=myDbServer;User=MyDbUserName;Password=P@ssword@1”.

Now that we have the language to describe a Policy, Service, and Secrete, we need one last construct to simplify the dependency secret resolution: Secret Dependency. This is not as much of a primitive as just a relationship that is defined within the system. When a secret has dependencies and that secret has its associated service keys rotated, the system can schedule updates to all other secrets that depend on it. Since dependencies do not technically require that they depend on “attached” secrets, one could design an extensive cascading secret rotation strategy to be executed by Harpocrates. The above described flow is depicted in the diagram below:

 

flow-dep.png

Figure 2 “Dependency” secret process flow

 

Now that we have the vocabulary required to describe the behavior we want Harpocrates to automate, we can discuss the runtime requirements and assumptions made by Harpocrates. We will do this in the next installment of this blog series. As always, your feedback is welcomed and greatly appreciated.