There are several products available that allow you to access keys, secrets, and certificates at run-time such as HashiCorp Vault and Azure Key Vault (KV). The benefit of programmatically accessing secrets and keys is the ability to rotate them and still be able to reference the same secrets from multiple locations instead of updating multiple secret stores.
We, at OSS Group, were approached by a large Telco Provider to create two Cloud Landing Zones in AWS and Azure. Using the strengths of each offering, there was a requirement to use Azure for secret and key management. I’m referencing this multi-cloud solution where Terraform Enterprise (TFE) and Ansible Tower (AT) are running on VMs in AWS and accessing Azure Key Vault via privatelink in Azure. The Certificate Authority lives in AWS. Certificates are sync’d to Azure Key Vault, but that’s an entirely different article!
The customer required a single source of truth that could be easily enforced with policies, ie rotation of secrets and keys while not affecting its consumers.
The following requirements and pre-requisites were met prior to integration.
Since we’ve already created the KV with TFE, we can just reference the module used to create it in order to access secrets and keys. You can simply reference the Terraform workspace used to create the Key Vault in other workspaces in order to use the Key Vault ID to access secrets at runtime.
In this example, we’ve already created the KV using TFE, and we can reference it using module.key_vault.id:
In fact, why don’t we use it to get the Ansible Tower API token needed to make API calls to Tower to create something that has no Terraform Provider (MS DNS)?
That was all too easy! Now let’s have a look at Ansible Tower.
The first thing to set up in Ansible Tower is your Key Vault Credentials. These can then be used to obtain client secrets and passwords in other Credentials.
Create a new Credential type using the built-in Microsoft Azure Key Vault:
Using the SP credentials Key Vault URL created, we create a new Credential in Ansible Tower:
You can test access to the KV by using a secret name in the Test function:
Now that we have that setup, we can use those Credentials to obtain secrets and keys used by other Credentials.
In this example, we obtain the password used by Ansible to authenticate to both Windows and Linux VMs using an AD DS Domain service account, with admin/sudoer permissions for configuration management purposes.
Create a new Credential of Machine type:
For the PASSWORD, we’re going to use the Azure Key Vault Credential we created prior to this:
That’s all we need to do to access a password at runtime.
We can actually use the Key Vault for any of these secrets or keys: Username, SSH Private Key, Private Key Passphrase, etc.
The above example was pretty straightforward. However, there’s no way to use that Key Vault credential within an Ansible playbook itself. In order to do that, we need to create an Azure RM (Resource Manager) Credential that is able to read KV secrets and keys.
Create a new Credential of Microsoft Azure Resource Manager type. Use the same Client subscription, ID, tenant, and secret that was used for the Key Vault. We also gave that Service Principal reader permissions in the Azure Subscription we’re using along with its Key Vault access policy.
This Credential, when assigned to a template, is available within a playbook at runtime. These variables can be used along with the azure_keyvault_secret Lookup plugin.
Add the MS Azure RM Credential to your template:
Within the playbook, define those variables needed to use the lookup plugin:
The Azure vars are set using the environment variable lookup plugin above.
The following code will look up a password used to Domain Join Linux servers at provisioning time:
The lookup plugin ‘azure_keyvault_secret’ used above is referencing the Environment Variables we set in the playbook that’s calling this role. This is the MS Azure RM Credentials in Tower!
Now that one wasn’t as straight forward as the previous example.
This blog article showed how Azure Key Vault secrets and keys can be accessed from multiple locations. The business outcome was a secure, private, cloud-based solution that is enforced using Azure Policies, providing password/secret/certificate/key access from multiple locations. The benefits include:
No Comments Yet
Let us know what you think