Our team has been refining our RDS monitoring strategy and I wanted to share our approach while getting feedback from the community. We manage 40+ RDS instances across MySQL, PostgreSQL, and Aurora, and we’ve evolved from basic CPU/memory alerts to a more sophisticated monitoring framework.
We’ve learned that default CloudWatch alarms often create alert fatigue-too many false positives from static thresholds that don’t account for workload patterns. We’re now using anomaly detection and composite alarms, but I’m curious what metrics others prioritize for early detection of performance degradation before it impacts users.
What CloudWatch alarm strategies have worked well for your RDS environments? Particularly interested in approaches that balance sensitivity with reducing noise.
Based on working with numerous RDS deployments, here’s a comprehensive CloudWatch alarm strategy framework:
CloudWatch Alarms Architecture: Implement three alarm severity levels with different response protocols. Critical alarms trigger PagerDuty for immediate response, warning alarms go to team Slack channels, and informational alarms log to dashboards for trend analysis. This prevents alert fatigue while ensuring urgent issues get immediate attention.
RDS Monitoring Metrics Priority: Focus on these key metrics in order of importance:
-
Connection Health: DatabaseConnections approaching max_connections (alert at 80%). This is your canary metric-connection exhaustion causes immediate application failures. Use composite alarms: (DatabaseConnections > 80% AND CPUUtilization > 70%) indicates genuine load issues versus connection leaks.
-
Memory Pressure: FreeableMemory below 15% of total memory. More importantly, monitor SwapUsage-any swap usage above 128MB warrants investigation. RDS performance degrades dramatically when swapping occurs. Set FreeableMemory anomaly detection with 2 standard deviation bands to catch gradual memory leaks.
-
Latency Patterns: Use anomaly detection on ReadLatency and WriteLatency rather than static thresholds. Latency varies dramatically by workload type and time of day. Set anomaly bands at 2 standard deviations and evaluate over 10-minute periods to filter transient spikes. For Aurora, AuroraReplicaLag above 1000ms is critical-indicates writer overwhelm or network issues.
-
Storage and I/O: DiskQueueDepth sustained above 10 indicates I/O bottleneck. FreeStorageSpace below 20% triggers expansion planning. For provisioned IOPS, monitor ReadIOPS and WriteIOPS against provisioned capacity-consistently hitting limits suggests under-provisioned storage.
Incident Response Integration: Configure alarm actions to trigger automated responses where appropriate. For example, high connection count can trigger Lambda to identify and kill long-running queries. CPU spikes can automatically capture Enhanced Monitoring snapshots for post-incident analysis. This reduces mean-time-to-resolution significantly.
Anomaly Detection Best Practices: Exclude deployment windows, maintenance periods, and known batch job times from training data to prevent false baselines. Use 2 standard deviation bands for latency and throughput metrics, 3 standard deviations for more volatile metrics like CPU. Retrain models quarterly to adapt to workload evolution. Combine anomaly detection with threshold alarms-alert when anomaly detected AND absolute threshold exceeded.
Composite Alarm Patterns: Create multi-signal alarms that reduce false positives:
- Performance degradation: (CPUUtilization > 80% AND WriteLatency anomaly AND DatabaseConnections > 70%)
- Memory pressure: (FreeableMemory < 512MB OR SwapUsage > 256MB)
- Replica health: (AuroraReplicaLag > 5000ms AND ReplicaLagMaximum > 10000ms)
Composite alarms provide context-high CPU alone might be normal during batch jobs, but high CPU with latency anomalies and connection pressure indicates genuine issues requiring intervention.
Dashboard and Correlation: Build CloudWatch dashboards showing metric correlations. Often performance issues manifest across multiple metrics simultaneously-seeing CPU, connections, and latency together reveals patterns that individual alarms miss. Include application-level metrics (APM tools, custom CloudWatch metrics) alongside RDS metrics for complete visibility.
The goal is signal-to-noise optimization. Too few alarms and you miss critical issues; too many and teams ignore them. Start conservative with critical-only alarms, then gradually add warning-level alarms as you tune thresholds based on actual incident patterns.
For anomaly detection, we use 2 standard deviations as the threshold rather than the default 3. This increases sensitivity but we found it catches issues earlier without excessive false positives. We also exclude known maintenance windows and deployment periods from the anomaly detection training data to avoid skewing the baselines.
One pattern that’s worked well: composite alarms combining multiple signals. For example, alarm when (CPUUtilization > 80% AND DatabaseConnections > 75% of max AND WriteLatency anomaly) all occur simultaneously. This reduces false positives from temporary spikes while catching genuine performance degradation. Single metric alarms are too noisy; correlated metrics provide better signal.
We use a tiered alerting approach. Tier 1 alarms (page immediately): DatabaseConnections approaching max_connections, FreeableMemory below 512MB, CPUUtilization sustained above 90% for 10 minutes. Tier 2 alarms (email/Slack): ReadLatency/WriteLatency anomalies, DiskQueueDepth trends. The key is using anomaly detection on latency metrics rather than static thresholds-workload patterns vary too much between day/night and weekday/weekend.
Great topic. One metric we found invaluable but often overlooked: SwapUsage. When RDS starts swapping, performance degrades rapidly, but it’s often a leading indicator before memory actually runs out. We alarm on any swap usage above 256MB. Also, for Aurora specifically, AuroraReplicaLag is critical-we’ve had cases where replica lag spiked to hours due to long-running transactions on the writer, and we didn’t notice until reads were severely stale.