Event retention policy not applied to device data storage resulting in unexpected data loss

IoT IBM Watson IoT
, , , , , , , ,
1

We configured an event retention policy in Watson IoT wiot-24 to keep device events for 90 days before archival, but device data is being deleted after only 30 days. This is causing unexpected data loss and compliance issues since we need to maintain historical device data for regulatory reporting.

Our retention policy configuration:


Retention Period: 90 days
Archival Target: AWS S3 bucket
Data Types: All device events

The storage manager shows that data older than 30 days is being purged, which doesn’t match our 90-day policy. I checked the storage quota settings and we’re only using 45% of our allocated storage, so quota limits aren’t the issue. The audit logging doesn’t show any manual deletions. Has anyone experienced retention policies not being applied correctly in wiot-24?

2

I’m updating the device type retention policies now using the API. But I’m concerned about the data that was already deleted during the past 30 days. Is there any way to recover that data from Watson IoT’s internal backups? Our compliance team needs those historical events for an upcoming audit. The audit logging shows the deletions were automated by the retention policy, not manual.

3

Your issue stems from the interaction between three retention policy layers in Watson IoT wiot-24, and understanding this hierarchy is critical for preventing future data loss.

Retention Policy Configuration Hierarchy: Watson IoT applies retention policies in this order:

  1. Device-type-specific retention (highest priority)
  2. Organization-level storage quota policies
  3. Global retention policy (lowest priority)

Your global 90-day policy was being overridden by the 30-day device-type policies on temperature-sensors and pressure-monitors. To fix this systematically:

# Use the bulk update API to set all device types to 90 days:
PUT /api/v0002/device/types/bulk/retention
{
  "retentionDays": 90,
  "deviceTypes": ["temperature-sensors", "pressure-monitors", "*"]
}

Storage Quota Conflict Resolution: Even though you’re at 45% of total storage, check the per-device-type quotas:


Settings → Storage → Device Type Quotas

If any device type exceeds its individual quota, Watson IoT will delete the oldest data regardless of retention policy settings. Increase device-type quotas to match your expected 90-day data volume.

Audit Logging for Retention Events: Enable comprehensive audit logging to track all retention-related deletions:

{
  "auditConfig": {
    "logRetentionDeletions": true,
    "logArchivalEvents": true,
    "logQuotaViolations": true,
    "alertOnUnexpectedDeletion": true
  }
}

This creates an audit trail for compliance and helps identify future retention policy conflicts before data loss occurs.

For your deleted data: Check your AWS S3 archival bucket immediately. Watson IoT archives data before deletion if archival is configured. Your data should be in S3 with folder structure: {orgId}/{deviceType}/{year}/{month}/{day}/. If archival wasn’t running due to configuration issues, that data is permanently lost and you’ll need to document the gap for your compliance audit.

Going forward, set up archival validation alerts to ensure data is successfully archived before retention cleanup runs. This provides a safety net against similar data loss scenarios.

4

Device-type-specific retention policies always take precedence over global settings in wiot-24 - this is by design for granular control. You’ll need to update each device type individually. But there’s a bulk update API you can use if you have many device types. Also check if there’s a storage quota conflict at the organization level that might be forcing early deletion regardless of retention policies.

5

I checked the device type settings and found that two of our device types (temperature-sensors and pressure-monitors) have custom retention set to 30 days. I didn’t realize device-type-specific policies would override the global retention policy. Is there a way to make the global policy take precedence, or do I need to update each device type individually?

6

There might be a conflict between your retention policy and the default storage quota policy. Even if you’re under your total quota, wiot-24 has per-device-type storage limits that can trigger early deletion. Check if any of your device types have custom retention settings that override the global policy.

7

Unfortunately, once data is deleted by the retention policy in wiot-24, it’s permanently removed unless you had archival configured beforehand. Since your archival target was set to AWS S3, check if any data was archived before deletion. The archival process runs before retention cleanup, so you might have some data in S3 even if it’s gone from Watson IoT. Also enable audit logging for all future retention policy changes to track what gets deleted and when.