Environment configuration variable substitution fails for ACR authentication in Kubernetes deployments

ALM Azure DevOps
, , , , , , , ,
1

Our Kubernetes deployment pipeline is failing during the variable substitution phase when trying to authenticate with Azure Container Registry. The environment-specific variables for ACR credentials aren’t being properly substituted into the imagePullSecrets configuration.

The pipeline works fine in our dev environment but fails consistently in staging and production. I’ve verified the service connection exists and the variables are defined at the environment scope. The error suggests the substitution syntax isn’t resolving the secret variables correctly.

Variable definition in environment:

ACR_USERNAME: $(acr-service-principal-id)
ACR_PASSWORD: $(acr-service-principal-secret)
ACR_SERVER: myregistry.azurecr.io

Deployment manifest snippet:

imagePullSecrets:
- name: acr-secret
  dockerconfigjson: $(ACR_SERVER)/$(ACR_USERNAME):$(ACR_PASSWORD)

The actual deployed secret shows literal variable names instead of values. Is there a specific order or syntax required for environment variable scoping with secret variables in Azure Pipelines?

2

Here’s the complete solution addressing all the configuration aspects:

ACR Service Connection Configuration: First, ensure your service connection is properly configured with service principal authentication. Go to Project Settings → Service connections → Your ACR connection. The service principal must have AcrPull role on the registry. However, don’t rely on automatic credential exposure - you’ll need explicit mapping.

Secret Variable Scoping in Environments: Define your variables at the environment level (Pipelines → Environments → Select environment → Add variable), but mark them as secret. The key issue is that secret variables require explicit reference in the pipeline context:

variables:
- group: acr-credentials  # Link variable group if using
- name: acrServer
  value: myregistry.azurecr.io

Environment-scoped secrets are only available within deployment jobs that reference that environment.

Kubernetes imagePullSecrets Setup: You cannot directly substitute secret variables into YAML manifests. Instead, create the Kubernetes secret dynamically:

steps:
- task: KubernetesManifest@0
  inputs:
    action: 'createSecret'
    secretType: 'dockerRegistry'
    secretName: 'acr-secret'
    dockerRegistryEndpoint: 'acr-service-connection'
    namespace: 'production'

This task uses the service connection directly, bypassing variable substitution issues.

Variable Substitution Syntax: If you must use variable substitution in manifests, use a two-step process:

  1. Create secret in script task:
kubectl create secret docker-registry acr-secret \
  --docker-server=$(acrServer) \
  --docker-username=$(ACR_USERNAME) \
  --docker-password=$(ACR_PASSWORD)
  1. Reference the secret name (not credentials) in manifest:
imagePullSecrets:
- name: acr-secret

The dockerconfigjson field should never contain variable references - it’s a base64-encoded credential blob.

Service Principal RBAC Permissions: Verify the service principal has:

  • AcrPull role on ACR (minimum)
  • Contributor role on AKS cluster (for secret creation)
  • Reader role on resource group (for validation)

Use this command to verify:

az role assignment list --assignee <service-principal-id> \
  --scope /subscriptions/<sub-id>/resourceGroups/<rg-name>

Complete Working Pipeline Pattern:

stages:
- stage: Deploy
  jobs:
  - deployment: DeployToAKS
    environment: production
    variables:
    - group: acr-secrets
    strategy:
      runOnce:
        deploy:
          steps:
          - task: KubernetesManifest@0
            inputs:
              action: 'createSecret'
              secretType: 'dockerRegistry'
              secretName: 'acr-auth'
              dockerRegistryEndpoint: 'acr-connection'

          - task: KubernetesManifest@0
            inputs:
              action: 'deploy'
              manifests: 'k8s/deployment.yaml'
              imagePullSecrets: 'acr-auth'

This approach eliminated our substitution failures. The key insight is that ACR authentication should use the KubernetesManifest task’s built-in secret creation, not manual variable substitution. The service connection handles credential management automatically, and environment scoping ensures the correct connection is used per stage.

3

I ran into this last month. The issue is that environment variables don’t automatically inherit from service connection credentials. Even though your ACR service connection is configured, the pipeline doesn’t expose those credentials as variables unless you explicitly map them.

You need to use the Docker task or a script task to create the Kubernetes secret dynamically, pulling credentials from the service connection at runtime.

4

Secret variables are available, but the substitution timing matters. If you’re using variable substitution in a Kubernetes manifest file, the secrets might not be decrypted at the time of file processing. You need to use the replacetokens task or construct the secret in a script task where secret variables are properly decrypted in the execution context.

5

Secret variables have special handling in Azure Pipelines and can’t be used directly in certain contexts. The dockerconfigjson field expects a base64-encoded JSON structure, not raw variable substitution. You need to construct the secret in a pipeline task before applying it to Kubernetes.

6

The environment scope is also critical here. If your variables are defined at the environment level but your deployment task isn’t explicitly referencing that environment, the variables won’t be in scope. Make sure your deployment job uses environment: staging or environment: production to pull in those environment-scoped variables.

7

That makes sense about the service connection not auto-exposing credentials. But I’ve defined the ACR_USERNAME and ACR_PASSWORD as explicit secret variables in the environment configuration. Shouldn’t those be available for substitution regardless of the service connection?