VM SSH connection times out due to restrictive NSG rules after scale-out

Cloud Platforms Microsoft Azure
, , , , , , , ,
1

After scaling out our VM scale set for payroll processing, we can’t SSH into the new VMs. Existing VMs work fine, but all new instances time out on port 22. The scale-out added 5 new VMs to handle end-of-month workload.

Connection attempt:


ssh admin@10.50.2.15
Connection timed out

All VMs are in the same subnet with an NSG attached. The NSG has an inbound rule allowing SSH from our jump box subnet (10.50.0.0/24). Priority 100, source 10.50.0.0/24, destination Any, port 22, protocol TCP, action Allow.

The new VMs have public IPs assigned but we connect through private IPs via site-to-site VPN. Azure portal shows the VMs as running and healthy. NSG flow logs aren’t enabled yet. This is blocking our payroll processing team from configuring the new instances and we’re under time pressure.

2

Check if there’s a second NSG attached at the NIC level on the new VMs. Scale sets can have NSG configurations that differ from the subnet NSG. Also verify the effective security rules on one of the new VMs - sometimes there are deny rules with higher priority that you’re not seeing at the subnet level. The Azure portal’s ‘Effective security rules’ view under the VM’s Network Interface will show you all rules being applied.

3

That explains it. I removed the NIC-level NSG from the scale set model configuration and redeployed the affected VMs. SSH access works now on all instances. Is there a way to prevent this from happening again? We have multiple teams managing infrastructure and someone must have added that NIC NSG thinking it would enhance security.

4

NSG evaluation is separate for subnet and NIC - they don’t share a priority space. Traffic is evaluated by subnet NSG first (inbound), then NIC NSG. Both must allow the traffic. Your subnet NSG allows SSH from jump box subnet, but the NIC NSG denies all VirtualNetwork traffic. Since 10.50.0.0/24 is part of VirtualNetwork service tag, the NIC NSG deny rule blocks it even though subnet NSG allows it. You need to either remove the NIC-level deny rule or add a higher priority allow rule (lower number) for SSH on the NIC NSG.