Synapse Analytics ML pipeline fails on scheduled trigger but works manually

Cloud Platforms Microsoft Azure
, , , , , , , ,
1

Running into a frustrating authentication issue with our Synapse Analytics ML pipeline. The pipeline executes perfectly when triggered manually through the Synapse Studio interface, but consistently fails with authentication errors when running on a scheduled trigger (daily at 3 AM).

The pipeline calls an Azure ML workspace to train a model, then stores results back to a dedicated SQL pool. Error message from failed runs:


Linked service 'AzureMLWorkspace' authentication failed
Error code: AuthorizationFailed

Our linked service uses a service principal with credentials stored in Key Vault. The SP has Contributor access to both Synapse workspace and ML workspace. We’ve verified the Key Vault secret hasn’t expired. Why would authentication work for manual triggers but fail on scheduled triggers? This is blocking our automated ML retraining workflow.

2

Don’t recreate yet. First, verify the managed identity was added correctly - it can take 5-10 minutes to propagate. Also check if you’re using Azure RBAC for Key Vault instead of access policies (newer vaults use RBAC by default). If using RBAC, you need to assign the ‘Key Vault Secrets User’ role to the Synapse managed identity. Another common issue: make sure your linked service connection is set to use the correct authentication method in the pipeline trigger configuration.

3

That makes sense - I hadn’t considered the different authentication contexts. So even though our linked service specifies a service principal, the scheduled trigger might be trying to use the workspace managed identity instead? How do I verify which identity is actually being used during scheduled runs?

4

I’ve dealt with this exact scenario. The issue is usually Key Vault access policies. Your service principal can read the secret when you’re logged in (because you have Key Vault permissions), but the Synapse managed identity can’t access it during scheduled runs. Go to your Key Vault, check Access Policies, and add the Synapse workspace managed identity with Secret Get permission. After that, your scheduled triggers should work properly.