Analytics queries from containerized apps to Oracle Analytics Cloud timing out

Cloud Platforms Oracle Cloud
, , , , , , , ,
1

Our containerized applications running on OCI Container Instances are experiencing slow query performance when connecting to Oracle Analytics Cloud. Dashboard loads that took 3-4 seconds in our VM-based deployment now take 15-20 seconds from containers, often timing out completely.

The containers run in a private subnet in US-East region, connecting to Analytics Cloud in the same region. Network latency seems fine when I ping the Analytics Cloud endpoint from within a container. The queries themselves aren’t complex - mostly pre-aggregated data pulls for real-time dashboards.

I’m monitoring resource usage on the Analytics Cloud instance through OCI Console, and CPU/memory look normal with plenty of headroom. The slow performance only affects queries originating from containers. Our legacy VM-based apps in the same subnet query the same Analytics Cloud instance without issues. What could cause this container-specific latency?

2

I’d also investigate MTU settings on your container network interface. If there’s a mismatch between your container subnet MTU and Analytics Cloud’s expected packet size, you’ll see fragmentation and retransmits that add latency. Run a packet capture during a slow query to see if you’re getting fragmentation.

3

I enabled detailed query logging on Analytics Cloud and found something interesting. The queries from containers have identical SQL to the VM queries, but the execution plans are different. Analytics Cloud is choosing table scans for container queries instead of using indexes. Not sure why the source would affect query planning.

4

This sounds like a DNS resolution delay. Containers might be resolving the Analytics Cloud hostname on every request if DNS caching isn’t configured properly. Check your container’s DNS settings and verify it’s using OCI’s DNS resolver. Also look at the TTL on DNS responses - if it’s too short, you’ll get repeated lookups.

5

Good thought. I verified our app uses connection pooling with a minimum of 5 connections per container. The pool metrics show connections are being reused, not recreated for each query. Still seeing the 15-20 second latency even after the initial connection is established.

6

Check if your Analytics Cloud instance has query result caching enabled. If the VM-based apps warmed up the cache but your containers are making slightly different queries (different parameters, timestamps, etc.), you’d get cache misses and slower response times. This would explain why VMs perform better - they’ve established cache patterns.

7

I found the root cause after researching this pattern. The issue is a combination of network latency characteristics and query optimization in Analytics Cloud.

Network Latency Monitoring: First, set up proper latency monitoring using OCI Monitoring. Create a custom metric tracking round-trip time from containers to Analytics Cloud. Use the Network Path Analyzer in OCI Console to trace the actual network path your container traffic takes. In your case, I suspect containers are routing through a NAT gateway while VMs use a service gateway, adding 50-100ms latency per request.

You can verify this with:

  • Check route tables for your container subnet vs VM subnet
  • Look for NAT gateway vs service gateway differences
  • Use traceroute from both container and VM to Analytics Cloud endpoint

Analytics Cloud Resource Usage: The “normal” CPU/memory you’re seeing doesn’t tell the full story. Analytics Cloud tracks query concurrency separately from resource utilization. Log into Analytics Cloud admin console and check the “Active Queries” dashboard. You might be hitting concurrency limits where container queries are queuing behind other workloads.

Also check the “Query Statistics” view filtered by source IP. If your containers share a NAT gateway IP, Analytics Cloud might be applying rate limiting thinking all requests come from a single client. This would explain why distributed VMs with individual IPs don’t hit the same bottleneck.

Query Optimization: The different execution plans you discovered are the key symptom. Analytics Cloud’s query optimizer considers network latency when choosing execution plans. Higher latency connections get plans optimized for fewer round-trips but potentially more server-side processing. This is why you see table scans instead of index seeks.

To fix this, you need to provide query hints forcing index usage:

  • Add optimizer hints to your queries specifying index preferences
  • Or, adjust the Analytics Cloud connection profile for your container subnet to mark it as “low latency”

The connection profile setting is in Analytics Cloud under Settings > Connections > Advanced. Set the latency profile to “LAN” instead of the default “WAN” that it’s probably detecting based on your container’s network characteristics.

OCI Monitoring Metrics: Set up these custom metrics in OCI Monitoring:

  1. Query response time (end-to-end from container)
  2. Network latency to Analytics Cloud endpoint (TCP connection time)
  3. Query execution time (from Analytics Cloud query logs)
  4. Container connection pool utilization

By comparing these metrics between containers and VMs, you’ll see where the latency is accumulating. My bet is 60% network path difference (NAT vs service gateway), 30% query optimization choosing wrong plans due to perceived latency, and 10% connection profile mismatch.

Immediate Fix: Update your container subnet route table to use a service gateway for Analytics Cloud traffic instead of NAT gateway. Add this route:

  • Destination: All Oracle Services
  • Target: Service Gateway
  • Description: Direct path to Analytics Cloud

This should immediately reduce your query times from 15-20 seconds back to 3-4 seconds by eliminating the NAT gateway hop and allowing Analytics Cloud’s optimizer to choose better execution plans.

8

Have you checked the connection pooling configuration in your containerized apps? Containers often restart more frequently than VMs, which could be causing connection churn. If each container establishes a new connection for every query instead of reusing pooled connections, that adds significant overhead.