Creating Business Advantage

Azure Key Vault, Terraform Enterprise and Ansible Tower Integration

Written by Scotty Laing | Apr 28, 2021 11:14:00 PM

Accessing keys, secrets, and certificates at runtime

There are several products available that allow you to access keys, secrets, and certificates at run-time such as HashiCorp Vault and Azure Key Vault (KV). The benefit of programmatically accessing secrets and keys is the ability to rotate them and still be able to reference the same secrets from multiple locations instead of updating multiple secret stores.

We, at OSS Group, were approached by a large Telco Provider to create two Cloud Landing Zones in AWS and Azure. Using the strengths of each offering, there was a requirement to use Azure for secret and key management. I’m referencing this multi-cloud solution where Terraform Enterprise (TFE) and Ansible Tower (AT) are running on VMs in AWS and accessing Azure Key Vault via privatelink in Azure. The Certificate Authority lives in AWS. Certificates are sync’d to Azure Key Vault, but that’s an entirely different article!

Customer Requirements

The customer required a single source of truth that could be easily enforced with policies, ie rotation of secrets and keys while not affecting its consumers.

  • Create the Key Vault using Infrastructure as Code in Terraform Enterprise/Azure DevOps.
  • Obtain secrets and keys at runtime for use with authentication and setting passwords/keys for IAM.
  • TFE for IaC, and AT for configuration management.

Technical Requirements

The following requirements and pre-requisites were met prior to integration.

  • IAM Service Principal in place to create resources in Azure.
  • KV created in Azure using TFE along with policy-based access, network policies, and privatelink.
  • IAM Service Principal in place to read secrets/keys/certificates from AT.

Terraform Enterprise Integration

Since we’ve already created the KV with TFE, we can just reference the module used to create it in order to access secrets and keys. You can simply reference the Terraform workspace used to create the Key Vault in other workspaces in order to use the Key Vault ID to access secrets at runtime.

In this example, we’ve already created the KV using TFE, and we can reference it using module.key_vault.id:

In fact, why don’t we use it to get the Ansible Tower API token needed to make API calls to Tower to create something that has no Terraform Provider (MS DNS)?

That was all too easy! Now let’s have a look at Ansible Tower.

Ansible Tower Integration

The first thing to set up in Ansible Tower is your Key Vault Credentials. These can then be used to obtain client secrets and passwords in other Credentials.

Create a new Credential type using the built-in Microsoft Azure Key Vault:

Using the SP credentials Key Vault URL created, we create a new Credential in Ansible Tower:

You can test access to the KV by using a secret name in the Test function:
 

Now that we have that setup, we can use those Credentials to obtain secrets and keys used by other Credentials.

In this example, we obtain the password used by Ansible to authenticate to both Windows and Linux VMs using an AD DS Domain service account, with admin/sudoer permissions for configuration management purposes.

Create a new Credential of Machine type:

For the PASSWORD, we’re going to use the Azure Key Vault Credential we created prior to this:

That’s all we need to do to access a password at runtime.
We can actually use the Key Vault for any of these secrets or keys: Username, SSH Private Key, Private Key Passphrase, etc.

 

Looking up secrets and keys within a Playbook or Role

The above example was pretty straightforward. However, there’s no way to use that Key Vault credential within an Ansible playbook itself. In order to do that, we need to create an Azure RM (Resource Manager) Credential that is able to read KV secrets and keys.

Create a new Credential of Microsoft Azure Resource Manager type. Use the same Client subscription, ID, tenant, and secret that was used for the Key Vault. We also gave that Service Principal reader permissions in the Azure Subscription we’re using along with its Key Vault access policy.


This Credential, when assigned to a template, is available within a playbook at runtime. These variables can be used along with the azure_keyvault_secret Lookup plugin.

Add the MS Azure RM Credential to your template:

Within the playbook, define those variables needed to use the lookup plugin:

The Azure vars are set using the environment variable lookup plugin above.

The following code will look up a password used to Domain Join Linux servers at provisioning time:

The lookup plugin ‘azure_keyvault_secret’ used above is referencing the Environment Variables we set in the playbook that’s calling this role. This is the MS Azure RM Credentials in Tower!

Now that one wasn’t as straight forward as the previous example.

 

Summary

This blog article showed how Azure Key Vault secrets and keys can be accessed from multiple locations. The business outcome was a secure, private, cloud-based solution that is enforced using Azure Policies, providing password/secret/certificate/key access from multiple locations. The benefits include:

  • Create secrets, keys, and certificates in one location,
  • Rotate keys and secrets in one spot, this allows for automation,
  • Have one set of Network and Access policies to protect your Key Vault,
  • Use Azure policies to:
    - enforce secret rotation,
    - ensure certificates use the correct CA,
    - ensure the private link is used to bring services internally.

Read more about HashiCorp

Read more about Microsoft