Firewall policy blocking internal VPC traffic between subnets in same zone

Cloud Platforms IBM Cloud
, , , , , , , ,
1

We have a multi-tier application deployed across three subnets in the same availability zone within our VPC. Since implementing a new firewall policy yesterday, the web tier (subnet 10.240.10.0/24) can no longer communicate with the application tier (subnet 10.240.11.0/24), causing complete service disruption.

Both subnets are in the same VPC and zone, so internal routing should work automatically. The firewall policy was intended to restrict external traffic, but it seems to be affecting internal VPC communication as well. Here’s the policy rule structure:


Rule 1: Allow inbound HTTPS (443) from 0.0.0.0/0
Rule 2: Allow inbound HTTP (80) from 0.0.0.0/0
Rule 3: Deny all other inbound traffic

The application tier listens on port 8080 for requests from the web tier. I assumed internal VPC traffic wouldn’t be subject to firewall policies, but clearly something is blocking it. How should firewall rules be configured to allow internal subnet communication while restricting external access? This is blocking our production environment.

2

I updated the firewall policy to add explicit allow rules for internal traffic before the deny rule, and that fixed the immediate issue. The web tier can now reach the app tier on port 8080. I’ll look into the security group approach for better long-term management.

3

One thing to be careful about - make sure you’re configuring the firewall policy on the correct resource. IBM Cloud has both security groups and network ACLs that can act as firewalls. Security groups are stateful and attached to network interfaces. Network ACLs are stateless and attached to subnets.

If you’re using network ACLs, you need rules for both inbound AND outbound traffic, including return traffic on ephemeral ports. That’s a common gotcha that breaks internal communication even when inbound rules look correct.

4

Another consideration for multi-subnet applications - instead of manually managing firewall rules for every subnet pair, consider using security groups with remote references. You can create a security group that allows traffic from another security group, which makes it easier to manage as your application scales.

For example, create a web-tier security group and an app-tier security group. In the app-tier group, add a rule that allows traffic from the web-tier security group ID rather than a specific CIDR range. This way, if you add more web tier instances or change subnet ranges, the rules still work.

5

Firewall policies in IBM Cloud VPCs apply to all traffic entering the protected resource, including traffic from other subnets within the same VPC. Your Rule 3 ‘deny all’ is blocking the internal traffic on port 8080.

You need to add explicit allow rules for your internal subnet CIDR ranges before the deny rule. Firewall rules are processed in order, so put your internal allow rules at the top with lower priority numbers.

6

Excellent resolution! Let me provide comprehensive guidance on configuring firewall policies for internal VPC traffic:

1. Firewall Policy Configuration Principles: IBM Cloud VPCs use a combination of security groups and network ACLs for traffic control. Understanding the processing order is critical:

  • Security Groups (Stateful): Attached to network interfaces, evaluate rules in any order (all allow rules are ORed together), return traffic is automatically allowed
  • Network ACLs (Stateless): Attached to subnets, evaluate rules in priority order (lowest number first), require explicit rules for both directions
  • Processing Order: Traffic entering a subnet first hits the network ACL, then the security group on the destination instance’s network interface

Your issue stems from a deny-all rule blocking internal traffic. The fix requires explicit allow rules for internal subnet communication.

2. Subnet CIDR Management for Internal Traffic: When configuring firewall rules for multi-subnet applications, follow this pattern:

a) Identify all internal subnet CIDR ranges:


ibmcloud is subnets --output json | jq -r '.[] | "\(.name): \(.ipv4_cidr_block)"'

Document the CIDR ranges for each tier: web (10.240.10.0/24), app (10.240.11.0/24), data (10.240.12.0/24)

b) Create allow rules for internal traffic BEFORE deny rules:

If using network ACLs:


# Allow web -> app tier (port 8080)
ibmcloud is network-acl-rule-add ACL_ID allow inbound tcp \
  --source-cidr 10.240.10.0/24 \
  --destination-cidr 10.240.11.0/24 \
  --destination-port-min 8080 --destination-port-max 8080 \
  --priority 10

# Allow app -> data tier (port 5432)
ibmcloud is network-acl-rule-add ACL_ID allow inbound tcp \
  --source-cidr 10.240.11.0/24 \
  --destination-cidr 10.240.12.0/24 \
  --destination-port-min 5432 --destination-port-max 5432 \
  --priority 20

c) Add outbound rules for return traffic (network ACLs only):


# Allow return traffic from app -> web tier
ibmcloud is network-acl-rule-add ACL_ID allow outbound tcp \
  --source-cidr 10.240.11.0/24 \
  --destination-cidr 10.240.10.0/24 \
  --source-port-min 8080 --source-port-max 8080 \
  --priority 30

3. Rule Priority Troubleshooting: Priority determines evaluation order (lower number = higher priority):

  • Best Practice: Use priority increments of 10 (10, 20, 30…) to leave room for insertions
  • Internal traffic rules: Priority 10-100
  • External allow rules: Priority 100-200
  • Deny rules: Priority 900-999 (evaluated last)

To reorder rules, you must delete and recreate them with new priorities. There’s no direct “update priority” command:


ibmcloud is network-acl-rule-delete ACL_ID RULE_ID
ibmcloud is network-acl-rule-add ACL_ID allow inbound tcp --priority 10 ...

4. Security Group Alternative (Recommended): For internal application traffic, security groups are easier to manage than network ACLs:

a) Create security groups for each tier:


ibmcloud is security-group-create web-tier-sg VPC_ID
ibmcloud is security-group-create app-tier-sg VPC_ID
ibmcloud is security-group-create data-tier-sg VPC_ID

b) Add rules using security group references:


# Allow app tier to receive traffic from web tier on port 8080
ibmcloud is security-group-rule-add app-tier-sg inbound tcp \
  --port-min 8080 --port-max 8080 \
  --remote web-tier-sg

# Allow data tier to receive traffic from app tier on port 5432
ibmcloud is security-group-rule-add data-tier-sg inbound tcp \
  --port-min 5432 --port-max 5432 \
  --remote app-tier-sg

c) Attach security groups to instances:


ibmcloud is instance-network-interface-update INSTANCE_ID NIC_ID \
  --security-groups web-tier-sg

5. Validation and Testing: After configuring firewall policies:

a) Test connectivity between tiers:


# From web tier instance
telnet 10.240.11.5 8080
curl http://10.240.11.5:8080/health

b) Review effective rules:


ibmcloud is network-acl-rules ACL_ID --output json
ibmcloud is security-group-rules SG_ID --output json

c) Monitor for denied connections:


ibmcloud is flow-logs # Check VPC flow logs for dropped packets

Key Takeaways:

  • Internal VPC traffic is subject to firewall policies just like external traffic
  • Always place internal allow rules before deny-all rules
  • Use security groups with remote references for application tier communication (stateful, easier)
  • Reserve network ACLs for subnet-level policies and compliance requirements (stateless, complex)
  • Document your subnet CIDR ranges and firewall rule priorities
  • Test thoroughly after changes - firewall misconfigurations can cause complete service outages

Your specific issue was resolved by adding explicit allow rules for 10.240.10.0/24 → 10.240.11.0/24 traffic on port 8080 before the deny-all rule. For production environments, I strongly recommend migrating to security groups for application traffic control instead of network ACLs.