Your export timeout failures are caused by inefficient single-file upload strategy and inadequate timeout configuration for large dataset transfers. I’ll provide a comprehensive solution addressing all three focus areas to ensure reliable exports of large analytics datasets.
Export Timeout Configuration: The 300-second timeout is insufficient for large file uploads, especially considering network variability and cloud storage API overhead. Reconfigure your workflow timeouts at multiple levels:
First, increase the workflow step timeout to allow for large transfers:
Workflow Definition > Export Step > Timeout: 3600 seconds (1 hour)
Workflow Definition > Export Step > Retry on Timeout: Enabled
Workflow Definition > Export Step > Max Retries: 3
Second, configure the Azure SDK timeout settings in your cloud storage integration:
azure.storage.upload.timeout=1800000
azure.storage.connection.timeout=60000
azure.storage.read.timeout=300000
Upload timeout of 30 minutes allows for complete transfer of large files. Connection timeout of 60 seconds handles initial connection establishment. Read timeout of 5 minutes handles Azure API response delays.
Third, implement adaptive timeout calculation based on file size:
long fileSize = exportFile.length();
long estimatedUploadTime = (fileSize / 1024 / 1024) * 2; // 2 seconds per MB
long timeout = Math.max(600, estimatedUploadTime * 2); // Minimum 10 min, 2x estimated time
uploadClient.setTimeout(timeout);
This dynamically adjusts timeout based on actual file size, preventing timeouts on legitimately large transfers while still catching stuck uploads.
Data Chunking Strategy: Replace the single-file upload approach with Azure’s block blob multipart upload for dramatically improved performance and reliability:
Implement block-based upload in your workflow:
BlobClient blobClient = containerClient.getBlobClient(fileName);
BlockBlobClient blockBlobClient = blobClient.getBlockBlobClient();
long blockSize = 8 * 1024 * 1024; // 8MB blocks
List<String> blockIds = new ArrayList<>();
try (FileInputStream fis = new FileInputStream(exportFile)) {
int blockNum = 0;
byte[] buffer = new byte[(int)blockSize];
int bytesRead;
while ((bytesRead = fis.read(buffer)) > 0) {
String blockId = Base64.getEncoder().encodeToString(
String.format("block-%05d", blockNum++).getBytes()
);
blockBlobClient.stageBlock(blockId,
new ByteArrayInputStream(buffer, 0, bytesRead), bytesRead);
blockIds.add(blockId);
}
blockBlobClient.commitBlockList(blockIds);
}
This splits the file into 8MB blocks and uploads them sequentially. Each block upload has its own timeout, so a timeout only affects one block rather than the entire file.
For even better performance, implement parallel block uploads:
ExecutorService executor = Executors.newFixedThreadPool(4);
List<Future<String>> futures = new ArrayList<>();
for (int i = 0; i < totalBlocks; i++) {
final int blockNum = i;
futures.add(executor.submit(() -> {
String blockId = generateBlockId(blockNum);
byte[] blockData = readBlock(exportFile, blockNum, blockSize);
blockBlobClient.stageBlock(blockId,
new ByteArrayInputStream(blockData), blockData.length);
return blockId;
}));
}
List<String> blockIds = futures.stream()
.map(f -> f.get())
.collect(Collectors.toList());
blockBlobClient.commitBlockList(blockIds);
executor.shutdown();
This uploads 4 blocks in parallel, reducing total upload time by approximately 75% for large files.
Cloud Storage API Limits: Optimize your workflow to work within Azure’s throughput limits and implement proper error handling:
First, implement compression before upload to reduce data transfer volume:
File compressedFile = new File(exportFile.getPath() + ".gz");
try (FileInputStream fis = new FileInputStream(exportFile);
FileOutputStream fos = new FileOutputStream(compressedFile);
GZIPOutputStream gzos = new GZIPOutputStream(fos)) {
byte[] buffer = new byte[8192];
int len;
while ((len = fis.read(buffer)) > 0) {
gzos.write(buffer, 0, len);
}
}
CSV files typically compress to 15-25% of original size, dramatically reducing upload time and bandwidth costs.
Second, implement file partitioning for datasets exceeding 1M rows:
Workflow Configuration > Export Settings > Partition Strategy: Time-based
Workflow Configuration > Export Settings > Partition Size: 250000 rows
Workflow Configuration > Export Settings > Parallel Uploads: 3
This creates multiple smaller files (approximately 150MB each when compressed) that can be uploaded in parallel. Configure the workflow to track completion of all partitions:
List<String> partitionFiles = partitionDataset(exportData, 250000);
List<CompletableFuture<Void>> uploadFutures = partitionFiles.stream()
.map(file -> CompletableFuture.runAsync(() -> uploadToAzure(file)))
.collect(Collectors.toList());
CompletableFuture.allOf(uploadFutures.toArray(new CompletableFuture[0])).join();
Third, implement retry logic with exponential backoff for transient Azure API failures:
int maxRetries = 3;
int retryDelay = 1000;
for (int attempt = 0; attempt < maxRetries; attempt++) {
try {
blockBlobClient.stageBlock(blockId, blockData, blockSize);
break;
} catch (StorageException e) {
if (attempt == maxRetries - 1) throw e;
Thread.sleep(retryDelay * (1 << attempt)); // Exponential backoff
}
}
Finally, monitor Azure storage account metrics to detect throttling:
Azure Portal > Storage Account > Metrics > Add Metric: “Throttling Errors”
Alert Rule: If throttling errors > 10 in 5 minutes, notify operations team
If you’re consistently hitting throttling limits, consider upgrading your Azure Storage account tier or implementing rate limiting in your workflow to stay within Azure’s throughput targets (approximately 20,000 requests per second for standard accounts).
Implement progress tracking and resumable uploads to handle partial failures:
String checkpointFile = exportFile.getPath() + ".checkpoint";
Set<String> completedBlocks = loadCheckpoint(checkpointFile);
for (int i = 0; i < totalBlocks; i++) {
String blockId = generateBlockId(i);
if (completedBlocks.contains(blockId)) continue;
uploadBlock(blockId, blockData);
completedBlocks.add(blockId);
saveCheckpoint(checkpointFile, completedBlocks);
}
This allows the workflow to resume from the last successfully uploaded block if a timeout occurs, rather than restarting the entire upload.
After implementing these changes, your workflow will reliably export datasets of 2M+ rows. The combination of compression (reducing file size by 75-85%), parallel block uploads (reducing upload time by 70-80%), and proper timeout configuration (allowing up to 1 hour for large transfers) ensures exports complete successfully even for the largest datasets. Monitor the workflow execution logs to verify block uploads are completing successfully and total export time is within acceptable limits for your nightly processing window.
This draft is based on general Adobe Experience Cloud knowledge. It has not been verified against your specific version and environment. Practitioners: verify the steps and share your experience below.