Object Storage upload fails for large files from OCI Compute instance

Cloud Platforms Oracle Cloud
, , , , , , , ,
1

We’re experiencing consistent failures when uploading large backup files (50GB+) from our OCI Compute instance to Object Storage. Small files under 5GB upload fine using the OCI CLI, but anything larger fails partway through.

We’re running a VM.Standard2.4 instance in the Phoenix region, trying to upload to a bucket in the same region. The uploads typically fail after 20-30 minutes with a timeout error. I’ve tried increasing the timeout values in the CLI config, but it doesn’t seem to help.


oci os object put -bn backup-bucket -f /data/backup_20241208.tar.gz
ServiceError: {"code":"RequestTimeout","message":"Request timeout after 1800 seconds"}

Our backup process generates these large files nightly and we need a reliable way to get them into Object Storage. Network connectivity seems fine - we can access other OCI services without issues. Is there a better approach for handling these large file uploads?

2

Let me provide a comprehensive solution that addresses the multipart upload configuration, network bandwidth optimization, and CLI options for your large file backup scenario.

Complete Solution for Large File Uploads:

1. Multipart Upload Configuration: The key issue is that default CLI settings aren’t optimized for your 50GB+ files. Configure explicit multipart parameters:


oci os object put -bn backup-bucket \
  --file /data/backup_20241208.tar.gz \
  --part-size 128 \
  --parallel-upload-count 5

Part Size Guidelines:

  • 50-100GB files: use 128MB part size
  • 100GB+ files: use 256MB part size
  • Maximum 10,000 parts per object, so calculate accordingly

2. Network Bandwidth Optimization:

First, verify your network configuration is optimized:

  • Service Gateway: Ensure your VCN has a service gateway for Object Storage
  • Add route rule: Destination = Object Storage service CIDR, Target = Service Gateway
  • This keeps traffic on OCI backbone (no internet egress)

Check current bandwidth utilization:


# Monitor during upload
oci monitoring metric-data summarize-metrics-data \
  --namespace oci_computeagent \
  --query-text 'NetworksBytesOut[1m].mean()'

VM.Standard2.4 provides up to 4.1 Gbps network bandwidth. If consistently hitting limits, consider:

  • Upgrading to VM.Standard3 or E4 shapes (higher bandwidth)
  • Scheduling uploads during off-peak hours
  • Splitting very large files if possible

3. Enhanced CLI Options and Configuration:

Create an OCI CLI configuration profile optimized for large uploads:


# ~/.oci/config
[BACKUP_PROFILE]
user=ocid1.user...
fingerprint=xx:xx:...
key_file=~/.oci/oci_api_key.pem
tenancy=ocid1.tenancy...
region=us-phoenix-1

# Add these for large file handling
connection_timeout=600
read_timeout=600
max_retries=5

Use the profile with enhanced retry and timeout:


oci os object put \
  --profile BACKUP_PROFILE \
  --bucket-name backup-bucket \
  --file /data/backup_20241208.tar.gz \
  --part-size 128 \
  --parallel-upload-count 5 \
  --no-multipart-md5-verification

4. Production-Grade Backup Script:

For nightly automated backups, implement proper error handling:

#!/bin/bash
# Backup upload with retry logic
MAX_ATTEMPTS=3
ATTEMPT=1

while [ $ATTEMPT -le $MAX_ATTEMPTS ]; do
  oci os object put \
    --bucket-name backup-bucket \
    --file /data/backup_$(date +%Y%m%d).tar.gz \
    --part-size 128 \
    --parallel-upload-count 5

  if [ $? -eq 0 ]; then
    echo "Upload successful"
    break
  fi

  ATTEMPT=$((ATTEMPT+1))
  sleep 300
done

5. Monitoring and Validation:

  • Enable Object Storage logging to track upload operations
  • Set up alarms for failed uploads using OCI Monitoring
  • Verify upload integrity with MD5 checksums
  • Monitor Object Storage metrics for your bucket

Additional Recommendations:

  • Compression: If not already compressed, use compression before upload to reduce transfer size
  • Lifecycle Policies: Configure Object Storage lifecycle policies for automated backup retention
  • Archive Storage: Consider Archive Storage tier for long-term backups (lower cost)
  • Parallel Uploads: If you have multiple backup files, upload them in parallel to maximize throughput

For Your Immediate Issue:

  1. Verify service gateway is configured in your VCN
  2. Use the multipart command with 128MB part size and 5 parallel uploads
  3. Test with one backup file first before automating
  4. Monitor network metrics during the upload
  5. If still failing, check OCI service limits for Object Storage in your tenancy

This comprehensive approach addresses all three focus areas: proper multipart upload configuration for large files, network bandwidth optimization through service gateway, and optimal CLI options with retry logic. Your nightly backup uploads should now complete reliably.

3

For files over 5GB, you should definitely be using multipart uploads instead of the standard put command. The OCI CLI has built-in multipart support but you need to configure it properly. Try using the --part-size parameter to control chunk sizes. Also check your instance’s network bandwidth - Standard2 shapes have decent bandwidth but sustained large uploads can sometimes hit limits.

4

I’ve dealt with similar backup scenarios. Beyond the multipart configuration, make sure you’re not running into Object Storage request rate limits. With large multipart uploads, you’re making many API calls rapidly. Also consider implementing retry logic - the OCI SDK has better built-in retry handling than the CLI for long-running operations. For production backup workflows, I’d actually recommend using the Python SDK instead of CLI for better control and error handling.

5

The CLI does handle multipart automatically, but the default settings aren’t always optimal for very large files. For 50GB+ files, I’d recommend setting part-size to 128MB or even 256MB. You can also increase parallel upload threads. Check network metrics in the OCI Console under the Compute instance metrics - look at NetworkBytesOut over the time period when uploads fail. If you’re consistently maxing out, you might need a higher bandwidth shape or consider using FastConnect for large data transfers.

6

One thing people often overlook is the network path. Even though you’re in the same region, your traffic might be routing through the internet gateway rather than staying on the OCI backbone. Verify your compute instance’s subnet routing and ensure you have a service gateway configured for Object Storage. This keeps traffic on Oracle’s private network and significantly improves reliability for large transfers. Also check if you have any network security lists or security groups that might be interfering with sustained connections.

7

I wasn’t aware there were specific multipart options. I thought the CLI handled that automatically. What part size would you recommend for 50GB files? And how do I check if we’re hitting network bandwidth limits on the compute instance?