Cloud Storage signed URL expiry breaks ERP invoice attachments for external customers

Cloud Platforms Google Cloud Platform (GCP)
, , , , , , , ,
1

Our ERP system generates signed URLs for invoice PDF attachments stored in Cloud Storage buckets. External customers receive these URLs via email to download their invoices. Recently, customers have been reporting ‘Access Denied’ errors when clicking the links, even though the URLs are generated just before sending the email.

We’re using the Cloud Storage JSON API to create signed URLs with a 7-day expiry, which should be more than enough time. Here’s how we generate them:


SignedUrlOptions options = SignedUrlOptions.withV4Signature()
  .setExpiration(7, TimeUnit.DAYS);
URL signedUrl = storage.signUrl(blobInfo, options);

The strange part is that some URLs work fine while others fail within hours. We’ve verified the service account has Storage Object Viewer permissions and the bucket is not public. Is there something about signed URL expiry configuration that we’re missing? Could this be related to how Cloud Storage handles customer access to ERP documents?

2

I found one of the issues - our object names do have special characters and we weren’t properly URL-encoding them before generating the signed URL. I’ve updated the code to encode the blob name. However, I’m still seeing some failures. The time sync suggestion is interesting - I’ll check our server clocks.

One more thing I noticed: some of our invoice objects have lifecycle policies that move them to Nearline storage after 30 days. Could that affect signed URL access?

3

Lisa, that’s a good point. I checked and we do have multiple service account keys - looks like our deployment pipeline creates a new key each time we deploy, and we’re at 8 keys now. But wouldn’t all URLs fail if the key was deleted, not just some?

I’m also noticing in our logs that the failures seem to correlate with invoices that have special characters in the filename (like ‘Invoice_#2024-001.pdf’). Could URL encoding be part of the issue?

4

Special characters in object names definitely cause signed URL issues. The URL encoding must match exactly between generation and access. If your email client or the customer’s browser re-encodes the URL, the signature becomes invalid.

Also check if you’re setting the correct content-disposition header when generating the signed URL. For customer downloads, you want ‘attachment; filename=“invoice.pdf”’ so browsers download instead of trying to render inline, which can trigger CORS issues.

5

Another common issue: time synchronization. Signed URLs include a timestamp in the signature calculation. If your application server’s clock is skewed by more than a few minutes from GCP’s time servers, the URLs can fail validation immediately or after a short period. Run ntpdate -q time.google.com to check your server’s time accuracy. We’ve seen 3-5 minute clock drifts cause exactly this pattern of intermittent failures.

6

Storage class changes shouldn’t affect signed URL access - the URL signature is based on object name and bucket, not storage class. However, if your lifecycle policy is deleting objects or changing their metadata, that would break the URLs.

For your service account key rotation issue, implement a key management strategy: keep maximum 2 keys active, use key ID explicitly in your signing code, and rotate keys during maintenance windows after verifying all old URLs have expired.

7

The intermittent failures suggest your service account key might be getting rotated or you’re using different keys for generation. Signed URLs are cryptographically tied to the specific key used to sign them. If the key rotates or is deleted, all URLs signed with that key become invalid immediately, regardless of the expiry time set.

Check your service account keys in IAM console and verify you’re not hitting the 10-key limit which triggers automatic deletion of old keys.

8

Here’s a comprehensive solution addressing all three focus areas for Cloud Storage signed URLs with ERP document access:

Cloud Storage Signed URLs - Proper Generation: The core issue is combining proper URL encoding with stable key management. Update your signing code to handle special characters correctly:


String encodedName = URLEncoder.encode(blobName, "UTF-8");
BlobInfo blobInfo = BlobInfo.newBuilder(bucketName, encodedName).build();
URL signedUrl = storage.signUrl(blobInfo, options);

Critically, specify which service account key to use by providing the key ID explicitly rather than relying on default credentials. This prevents issues when multiple keys exist.

URL Expiry Configuration Best Practices: Your 7-day expiry is reasonable, but implement these safeguards:

  1. Time Synchronization: Ensure your application servers sync with NTP servers. Clock skew >5 minutes will cause signature validation failures. Configure NTP with Google’s time servers: time1.google.com through time4.google.com.

  2. Key Rotation Strategy: Maintain exactly 2 active service account keys maximum. Before rotating keys, verify all URLs signed with the old key have passed their expiry time. Document which key ID is active in your configuration management system.

  3. Expiry Buffer: Set expiry to 8 days instead of 7 to account for timezone differences and customer email delays. Add logic to regenerate URLs if accessed after 6 days to provide fresh links.

  4. Metadata Headers: Include content-disposition in your signed URL options:


Map<String, String> headers = new HashMap<>();
headers.put("Content-Disposition", "attachment; filename=invoice.pdf");
SignedUrlOptions options = SignedUrlOptions.withV4Signature()
  .setExpiration(8, TimeUnit.DAYS)
  .setHeaders(headers);

Customer Access to ERP Documents - Security & Reliability:

  1. Access Logging: Enable Cloud Storage access logs to track successful and failed URL accesses. Filter for 403 errors to identify patterns in failures. Log entries will show whether failures are due to expired signatures, wrong keys, or encoding issues.

  2. Fallback Mechanism: Implement a URL validation endpoint in your ERP system. When customers report access issues, they can request a fresh URL through a secure portal using their invoice number and email verification.

  3. Object Lifecycle Coordination: If using lifecycle policies for cost optimization, ensure they don’t conflict with URL expiry. Set lifecycle transitions to occur after maximum URL expiry (e.g., move to Nearline after 10 days if URLs expire at 8 days).

  4. Email Template Updates: In your email notifications, include: URL expiry date, instructions for requesting new URLs, and a note that clicking the link multiple times won’t consume downloads (common customer concern).

  5. Monitoring: Create Cloud Monitoring alerts for signed URL generation failures and track the ratio of 403 errors to successful accesses in Cloud Storage logs. Threshold alert if 403 rate exceeds 5%.

The combination of proper URL encoding for special characters, explicit key ID usage to prevent rotation issues, and time synchronization will resolve the intermittent access failures your customers are experiencing. The special characters in filenames like ‘Invoice_#2024-001.pdf’ were causing signature mismatches when email clients re-encoded the URLs.