VPC peering blocks ERP application traffic despite open security groups

Cloud Platforms Amazon Web Services (AWS)
, , , , , , , ,
1

Our ERP application spans two VPCs connected via VPC peering, but traffic between them is failing. The application server in VPC-A (10.0.0.0/16) needs to communicate with the database in VPC-B (10.1.0.0/16).

Security groups are configured to allow all traffic:


VPC-A SG: Inbound 0.0.0.0/0 all traffic
VPC-B SG: Inbound 10.0.0.0/16 port 3306

The peering connection shows as active, and we’ve added routes in both VPC route tables pointing to the peering connection. However, connection attempts from the app server timeout. Testing with telnet to the database port fails, and traceroute shows packets being dropped. We’ve verified the security groups multiple times and they appear correct. What else could be blocking traffic in a VPC peering setup?

2

Are your route tables associated with the correct subnets? You need routes in both the main route table AND any custom route tables associated with your subnets. A common mistake is adding the route to the main route table but the subnets are using a custom route table that doesn’t have the peering route.

3

Your security group in VPC-A is too permissive with 0.0.0.0/0. That’s not causing your issue, but it’s a security risk. For the actual problem, verify your route table associations. Run aws ec2 describe-route-tables and check which subnets each route table is associated with. The route tables containing your peering routes must be explicitly associated with the subnets where your instances live.

4

Your issue likely involves multiple configuration layers. Let me address all three key areas:

Security Group Rules: Your security groups need refinement. While they technically allow traffic, verify the following:


VPC-A App Server SG:
  Outbound: 10.1.0.0/16 TCP 3306 (to VPC-B)
  Inbound: 10.1.0.0/16 ephemeral ports (for return traffic)

VPC-B Database SG:
  Inbound: 10.0.0.0/16 TCP 3306 (from VPC-A)
  Outbound: 10.0.0.0/16 ephemeral ports (for return traffic)

Security groups are stateful, so return traffic is automatically allowed, but being explicit helps with auditing. The 0.0.0.0/0 rule in VPC-A should be removed - it’s a security risk and unnecessary.

Route Table Configuration: This is likely your main issue. You mentioned adding routes to “main route tables,” but you need to verify subnet associations:

  1. Check which route table is associated with your app server subnet:
aws ec2 describe-route-tables \
  --filters "Name=association.subnet-id,Values=subnet-xxxxx"
  1. Verify that route table contains:

Destination: 10.1.0.0/16
Target: pcx-xxxxxxxxx
  1. Repeat for the database subnet in VPC-B (route to 10.0.0.0/16).

Critical point: If your subnets use custom route tables (not the main route table), adding routes to the main route table has no effect. Each subnet must have its associated route table configured.

VPC Peering DNS Settings: If your ERP application uses DNS names (like database.internal.company.com), enable DNS resolution:

  1. Go to VPC Peering Connection settings
  2. Edit DNS settings
  3. Enable “DNS resolution from requester VPC to accepter VPC”
  4. Enable “DNS resolution from accepter VPC to requester VPC”

Without these settings, DNS queries for private hosted zones don’t work across the peering connection.

Additional Checks:

  • Network ACLs: Run aws ec2 describe-network-acls and verify your subnet NACLs allow traffic on port 3306 and ephemeral ports (1024-65535) in both directions
  • Peering connection must be accepted in both accounts (if cross-account)
  • No Transit Gateway or other routing devices interfering
  • Test with IP address directly to rule out DNS issues: `telnet 10.1.x.x 3306 Testing Methodology:
  1. From app server, test with IP: `telnet 10.1.x.x 3306
  2. Check VPC Flow Logs for REJECT entries
  3. Verify route propagation: ip route show on the instance
  4. Use VPC Reachability Analyzer to test the path

Most VPC peering issues stem from incorrect route table associations or missing DNS settings. Verify subnet-to-route-table mappings first, then enable DNS resolution if using hostnames.

5

I suspect DNS resolution issues. If your ERP app is using a hostname instead of an IP address, you need to enable DNS resolution for the peering connection. By default, VPC peering doesn’t resolve DNS hostnames across VPCs. Go to the peering connection settings and enable both ‘DNS resolution from accepter VPC to requester VPC’ and vice versa. This is often overlooked.

6

Check for overlapping CIDR blocks. If you have any overlapping IP ranges between the VPCs, peering won’t work properly even if the connection appears active. Also verify that your route table entries specify the exact peering connection ID (pcx-xxxxx) as the target, not just the destination CIDR.

7

Have you checked Network ACLs? Security groups are stateful, but NACLs are stateless and could be blocking return traffic. The default NACL allows all traffic, but if you have custom NACLs on your subnets, you need explicit allow rules for both inbound and outbound traffic. Also check if you have any AWS Network Firewall or third-party firewall appliances in the path.

8

No overlapping CIDRs - VPC-A is 10.0.0.0/16 and VPC-B is 10.1.0.0/16. I double-checked the route tables and the routes are there with the correct pcx-ID. Both are in the main route tables for each VPC. Still seeing timeouts. Could there be an issue with the peering connection itself even though it shows active?