Code Engine job fails to access Cloud Object Storage due to missing credentials

Cloud Platforms IBM Cloud
, , , , , , , ,
1

I’m running a batch processing job in Code Engine that needs to read files from Cloud Object Storage. The job worked fine in our test environment, but after deploying to production it’s failing with NoCredentialsError: Unable to locate credentials. I’ve verified the service binding exists in Code Engine and the COS bucket has the correct access policies. The job logs show it’s trying to connect but can’t find the credentials. I’m not sure if this is an environment variable issue or something with how the service binding is configured. Has anyone encountered this before? The job runs every 6 hours and we’re missing critical data processing windows.

2

Thanks for the tip. I checked and you’re right - the binding isn’t listed in the job configuration. I thought creating the binding at the project level would be enough. I see the binding exists (ibmcloud ce project binding list shows it), but it’s not connected to the job. Do I need to update the job definition to reference it, or should I recreate the job?

3

That makes sense. So the binding creates environment variables, but my code needs to explicitly read from CE_SERVICES. Looking at my code, I’m using the AWS SDK for Python (boto3) and just calling s3 = boto3.client('s3') without any credential parameters. I assumed it would auto-discover them. Do I need to parse CE_SERVICES and manually configure the boto3 client?

4

Check if the service binding is actually mounted to your job. Code Engine jobs don’t automatically inherit bindings from the project - you need to explicitly reference them in the job configuration. Run ibmcloud ce job get --name YOUR_JOB and look for the service bindings section. If it’s empty, that’s your problem.

5

Exactly right. The boto3 client won’t automatically parse CE_SERVICES - that’s a Code Engine specific format. You need to extract the credentials from that JSON and pass them explicitly. Also worth noting that if you’re using HMAC credentials for COS, you’ll need both the access key ID and secret access key from the binding. Some teams create a small helper function to parse CE_SERVICES once at startup and configure all their cloud service clients from it. Makes the code cleaner and you only parse the JSON once.

6

I had the exact same issue last month. Here’s the complete solution:

First, update your Code Engine job to include the service binding:


ibmcloud ce job update --name data-processor \
  --service-binding cos-binding

Then modify your Python code to parse the CE_SERVICES environment variable and configure boto3 properly:

import json, os, boto3

ce_services = json.loads(os.environ.get('CE_SERVICES', '{}'))
cos_creds = ce_services['cloud-object-storage'][0]['credentials']

s3 = boto3.client('s3',
    endpoint_url=cos_creds['endpoints'],
    aws_access_key_id=cos_creds['cos_hmac_keys']['access_key_id'],
    aws_secret_access_key=cos_creds['cos_hmac_keys']['secret_access_key'])

Key points for service binding configuration: The binding name you use in the job update command must match the binding created at the project level. For environment variable setup, CE_SERVICES is a JSON object where each service type is an array (even if you have only one binding). For COS credential management, make sure you’re using HMAC credentials, not IAM API keys directly - COS service credentials should have the ‘Include HMAC Credential’ option enabled when created.

One gotcha: If you have multiple COS bindings, you’ll need to identify the right one by checking the ‘instance_name’ or ‘credentials.resource_instance_id’ fields. Also, the endpoints field in credentials contains multiple URLs - use the ‘direct’ endpoint for better performance within IBM Cloud, or ‘public’ if you need external access. After updating your code, redeploy the job and the NoCredentialsError should be resolved. The credentials are now properly extracted from the service binding and passed to boto3.

7

You can update the existing job without recreating it. Use ibmcloud ce job update --name YOUR_JOB --service-binding YOUR_BINDING_NAME. The binding will then be available as environment variables prefixed with CE_SERVICES. Your application code needs to parse this JSON structure to extract the COS credentials. I’ve seen people miss this step and wonder why their code can’t find the credentials even though the binding is there. The CE_SERVICES variable contains all the connection details including endpoints, access keys, and bucket information.