Data stream firmware update fails due to payload size exceeded during chunked upload

IOT Oracle IoT Cloud
, , , , , , , ,
1

Our firmware update rollout is completely blocked because edge devices fail to download firmware packages via data streams. The updates work fine for smaller packages (under 5MB), but our latest firmware is 12MB and consistently fails with payload size errors.

Error from device logs:


DataStreamClient: Payload size 12582912 exceeds limit 10485760
FirmwareDownload: Failed to retrieve package chunk
HTTP 413: Request Entity Too Large

We need chunked upload support to handle larger firmware files, but I can’t find clear documentation on payload size configuration for data streams in oiot-pm. The firmware compression we applied only reduced the size to 12MB from 18MB original, which isn’t enough.

Is there a way to configure the data stream payload limits or enable proper chunked transfer for firmware downloads? This is blocking critical updates to 300+ edge devices in production.

2

We faced this exact issue. Instead of using data streams for large firmware files, we switched to using signed URLs with direct S3/Object Storage downloads. The IoT platform generates a temporary download URL, and devices fetch the firmware directly from object storage. This bypasses the data stream payload limits entirely and is much more efficient for large files. Data streams are better suited for small configuration updates, not multi-megabyte firmware images.

3

The signed URL approach sounds promising, but our edge devices are in restricted network environments with limited outbound connectivity. They can reach the IoT platform endpoints but not general internet/cloud storage. That’s why we’re using data streams. Can chunked upload work within the data stream framework itself?

4

I’ve implemented chunked firmware delivery over data streams for similar constraints. Here’s the complete solution:

Understanding Payload Size Configuration: The 10MB limit in oiot-pm is enforced at the message broker level and cannot be increased through configuration. However, chunked upload support is the designed approach for large payloads.

Server-Side Implementation:

Prepare firmware package with chunking:

# Split firmware into chunks
chunk_size = 8 * 1024 * 1024  # 8MB chunks
firmware_chunks = split_firmware(firmware_file, chunk_size)
total_chunks = len(firmware_chunks)

# Publish chunks via data stream
for idx, chunk in enumerate(firmware_chunks):
    message = {
        "chunkIndex": idx,
        "totalChunks": total_chunks,
        "data": base64.encode(chunk),
        "checksum": sha256(chunk)
    }
    data_stream.publish(device_id, "firmware-update", message)

Device-Side Implementation:

Device receives and assembles chunks:

// Device firmware update handler
void onDataStreamMessage(message) {
    int chunkIndex = message.get("chunkIndex");
    byte[] chunkData = base64.decode(message.get("data"));

    // Verify chunk integrity
    if (!verifyChecksum(chunkData, message.get("checksum"))) {
        requestChunkRetransmit(chunkIndex);
        return;
    }

    // Store chunk
    firmwareBuffer.writeChunk(chunkIndex, chunkData);
}

Firmware Compression Strategy:

Your compression from 18MB to 12MB isn’t optimal. Apply compression per-chunk:

  1. Split firmware into logical sections (bootloader, application, resources)
  2. Compress each section separately using LZMA or XZ (better ratios than gzip)
  3. This typically achieves 40-60% size reduction for firmware binaries
  4. Chunk the compressed sections into 8MB pieces

Chunked Upload Support - Complete Flow:

  1. Initiation Phase:

    • Server sends metadata message with total chunks, firmware version, full hash
    • Device acknowledges and prepares storage

  2. Transfer Phase:

    • Server publishes chunks sequentially with 2-second intervals
    • Device acknowledges each chunk or requests retransmit
    • Implement sliding window protocol (allow 3 chunks in-flight)

  3. Verification Phase:

    • Device assembles all chunks
    • Validates complete firmware hash
    • Sends confirmation or requests missing chunks

  4. Installation Phase:

    • Device applies firmware update
    • Reboots and validates new version

Payload Size Configuration Workaround:

While you can’t change the 10MB broker limit, optimize chunk size based on network conditions:

# Adaptive chunk sizing
if device.network_quality == "excellent":
    chunk_size = 9 * 1024 * 1024  # 9MB chunks
elif device.network_quality == "good":
    chunk_size = 6 * 1024 * 1024  # 6MB chunks
else:
    chunk_size = 3 * 1024 * 1024  # 3MB chunks for poor networks

Error Handling:

  • Implement exponential backoff for failed chunks
  • Store chunk state on device (persist across restarts)
  • Set overall timeout (e.g., 24 hours) for complete download
  • Allow resume from last successful chunk

Testing Recommendations:

  1. Test with network disruptions (simulate packet loss)
  2. Verify behavior when device restarts mid-download
  3. Validate chunk reassembly with various firmware sizes
  4. Monitor memory usage during chunk buffering

Production Deployment:

For your 300+ devices:

  • Roll out to 10 devices first (pilot group)
  • Monitor chunk transfer success rates
  • Adjust chunk size based on observed network performance
  • Implement rate limiting (update 50 devices concurrently max)

This approach has successfully delivered 25MB+ firmware updates to thousands of edge devices in restricted networks. The key is robust chunk management and per-chunk compression to stay well under the 10MB limit.

5

The 10MB limit is hardcoded for single data stream messages in oiot-pm. You need to implement chunked upload on both server and client side. The firmware package should be split into chunks before transmission, not sent as a single payload.

6

Yes, chunked upload can work with data streams, but you need to implement the chunking logic in your update service. Split the firmware into 8MB chunks, send each chunk as a separate data stream message with sequence metadata, and have devices reassemble them. We do this for our industrial controllers and it works reliably. The key is proper error handling for partial downloads and chunk verification.