IoT Hub data storage integration vs external data lake for long-term telemetry retention

IOT Microsoft Azure IoT
, , , , , , ,
1

We’re architecting our long-term telemetry storage strategy and evaluating IoT Hub’s built-in storage endpoints versus routing data to an external Azure Data Lake. Our requirement is 7+ years of retention for compliance, with frequent analytics queries on historical data. I’m trying to understand the tradeoffs between using IoT Hub’s native storage capabilities versus the scalability of Data Lake, and how each approach affects analytics integration downstream. Would appreciate insights from anyone who’s made this decision at scale.

2

IoT Hub’s built-in storage endpoints (Event Hub-compatible endpoint) only retain data for 1-7 days depending on your tier. For 7-year retention, you must route to external storage. We use IoT Hub message routing to send telemetry directly to Azure Data Lake Storage Gen2, which gives us unlimited retention at low cost (~$0.02/GB/month for cool tier).

3

Data Lake scalability is unmatched for long-term storage. We’re storing 50TB of IoT telemetry with 7-year retention, and query performance with Azure Synapse Analytics is excellent. The key is organizing data properly - partition by date and device type, use Parquet format for efficient compression and query performance. Our queries that took 10+ minutes on Azure SQL now run in under 30 seconds on Data Lake with Synapse.

4

Having architected storage strategies for multiple large-scale IoT deployments, here’s my comprehensive analysis of the three focus areas:

Built-in Storage Endpoints:

IoT Hub provides an Event Hub-compatible endpoint for telemetry consumption, but it’s not designed for long-term storage:

Capabilities:

  • Retention: 1-7 days depending on IoT Hub tier (S1: 1 day, S2: 1 day, S3: 7 days)
  • Format: Raw messages in Event Hub format
  • Access: Stream processing only (Event Hub SDK, Stream Analytics)
  • Cost: Included in IoT Hub pricing
  • Limitations: No direct query capability, no archival features, limited retention

Use Cases:

  • Real-time stream processing
  • Short-term buffering for downstream systems
  • Hot path analytics (current state, recent trends)

For 7-year retention, IoT Hub’s built-in storage is NOT suitable. It’s designed as a transient buffer, not a data warehouse.

Alternative: Message Routing to Storage: IoT Hub can route messages to Azure Storage (Blob/Data Lake), but this is still considered ‘external’ storage from IoT Hub’s perspective. The routing happens within IoT Hub’s infrastructure, but the data lives in your Storage account.

Data Lake Scalability:

Azure Data Lake Storage Gen2 is purpose-built for massive-scale analytics:

Scalability Characteristics:

  • Storage: Exabyte-scale capacity, no practical limits for IoT scenarios
  • Throughput: 15 GB/s per storage account, can scale to 100+ GB/s with multiple accounts
  • File Size: No limits (supports both small IoT messages and large aggregated files)
  • Partitioning: Hierarchical namespace enables efficient organization by date/device/sensor
  • Format Support: JSON, Parquet, Avro, CSV - choose based on query patterns

Cost Optimization: Lifecycle management policies automatically move data through tiers:

  • Hot tier (frequent access): $0.0184/GB/month - first 30 days
  • Cool tier (infrequent access): $0.0100/GB/month - days 31-365
  • Archive tier (rare access): $0.00099/GB/month - year 2+

For 7-year retention of 100GB/day telemetry:

  • Year 1: ~$5,000 (hot/cool tiers)
  • Years 2-7: ~$2,500/year (archive tier)
  • Total 7-year cost: ~$20,000 for 255TB of data

Compare to Azure SQL Database: $50,000+/year just for year 1.

Optimal Data Organization: Partition structure for efficient queries:


/telemetry/
  /year=2025/
    /month=11/
      /day=01/
        /hour=00/
          device-001.parquet
          device-002.parquet

This structure enables partition pruning in queries, dramatically improving performance.

Format Recommendation:

  • Parquet for analytical queries (10x compression, columnar storage)
  • JSON for ad-hoc queries and debugging (human-readable)
  • Avro for schema evolution scenarios (built-in schema registry)

For most IoT scenarios, Parquet provides the best balance of compression, query performance, and analytics tool compatibility.

Analytics Integration:

Data Lake serves as the central hub for multiple analytics platforms:

1. Azure Synapse Analytics:

  • Serverless SQL pool queries Data Lake directly (no data movement)
  • Dedicated SQL pool for high-performance data warehousing
  • Spark pools for complex transformations and ML workloads
  • Cost: $5-15 per TB queried (serverless), or fixed cost for dedicated pools

Example query performance: 1TB scan in 30-60 seconds with proper partitioning.

2. Azure Data Explorer (ADX):

  • Time-series optimized database
  • Ingest from Data Lake for hot-path analytics
  • Keep recent data (90 days) in ADX, historical data in Data Lake
  • Query federation allows joining ADX and Data Lake data
  • Cost: $0.10-0.20 per GB ingested + storage

3. Power BI:

  • Direct Query to Synapse or ADX for real-time dashboards
  • Import mode for aggregated reports (faster, lower cost)
  • Dataflows for ETL from Data Lake to Power BI datasets
  • Cost: Included in Power BI Premium capacity

4. Azure Machine Learning:

  • Data Lake as training data source
  • Parquet format optimizes data loading for ML pipelines
  • Feature store backed by Data Lake for reproducible ML
  • Cost: Compute-based, storage is negligible

5. Databricks:

  • Delta Lake format on top of Data Lake for ACID transactions
  • Unified batch and streaming analytics
  • Advanced ML and AI capabilities
  • Cost: $0.15-0.55 per DBU (compute unit)

Recommended Architecture:

For 7-year retention with analytics requirements:

Tier 1: Hot Path (Real-time)

  • IoT Hub → Stream Analytics → Azure Data Explorer
  • Retention: 90 days
  • Use Case: Real-time dashboards, alerting, operational monitoring
  • Cost: ~$500-1,000/month for 100GB/day ingestion

Tier 2: Warm Path (Recent History)

  • IoT Hub → Data Lake (Parquet, hot tier)
  • Retention: 1 year
  • Use Case: Trend analysis, reporting, ad-hoc queries
  • Cost: ~$500/month for 36TB storage

Tier 3: Cold Path (Long-term Archive)

  • Data Lake lifecycle policy → Cool tier (year 2) → Archive tier (years 3-7)
  • Retention: 6 years in archive
  • Use Case: Compliance, historical analysis, ML training
  • Cost: ~$200/month for 219TB archive storage

Data Flow:


IoT Hub → Message Routing → Data Lake (JSON, real-time)
  ↓
Azure Functions → Convert to Parquet (hourly batch)
  ↓
Data Lake (Parquet, partitioned)
  ↓
Synapse Analytics (on-demand queries)
  ↓
Power BI / Databricks / ML (analytics consumers)

Compliance Considerations:

For 7-year retention compliance:

  1. Immutable Storage: Enable WORM (Write Once, Read Many) policy on Data Lake
    az storage account blob-service-properties update \
  --account-name <storage> \
  --enable-versioning true \
  --enable-change-feed true



2. **Legal Hold:** Prevent deletion even by administrators
   ```bash
   az storage container legal-hold set \
     --account-name <storage> \
     --container-name telemetry \
     --tags compliance audit
  1. Time-based Retention: Automatically enforce retention period
    az storage container immutability-policy create \
  --account-name <storage> \
  --container-name telemetry \
  --period 2555 # 7 years in days



4. **Audit Logging:** Enable diagnostic settings to track all access
   ```bash
   az monitor diagnostic-settings create \
     --name storage-audit \
     --resource <storage-resource-id> \
     --logs '[{"category":"StorageRead","enabled":true},{"category":"StorageWrite","enabled":true}]' \
     --workspace <log-analytics-workspace>

Performance Optimization:

For frequent analytics queries on historical data:

  1. Materialized Views: Pre-aggregate data in Synapse for common queries
  2. Caching: Use Azure Redis for frequently accessed aggregations
  3. Indexing: Create secondary indexes in Data Explorer for complex filters
  4. Partitioning: Align partition strategy with query patterns (typically date-based)
  5. Compression: Parquet with Snappy compression provides 10:1 ratio for IoT telemetry

Cost Comparison (7-year retention, 100GB/day):

  • IoT Hub Built-in Storage: Not feasible (max 7 days retention)
  • Azure SQL Database: ~$350,000 total (prohibitively expensive)
  • Data Lake + Synapse: ~$35,000 total (10x cheaper than SQL)
  • Data Lake + ADX (hybrid): ~$50,000 total (best query performance)

The Data Lake approach is clearly superior for long-term retention at scale. The built-in IoT Hub storage is only suitable for short-term buffering and real-time stream processing.

5

Cost difference is significant at scale. IoT Hub storage is included for short retention (1-7 days), but extending retention would require keeping data in the Event Hub-compatible endpoint, which isn’t designed for long-term storage. Data Lake with lifecycle management policies (move to cool tier after 30 days, archive tier after 1 year) brings our storage cost to under $200/month for 30TB of telemetry. Equivalent retention in a database would be $5,000+/month.

6

Don’t forget about the analytics integration aspect. If you’re using Power BI, Azure Data Explorer, or Synapse Analytics, they all have native connectors for Data Lake but not for IoT Hub’s built-in storage. Routing to Data Lake upfront saves you from building ETL pipelines later. We learned this the hard way after trying to backfill 2 years of data from Event Hub archives.