File storage mount fails on Linux with NFS 'permission denied' error in ic-2019 storage module

I’m trying to mount IBM Cloud File Storage on a RHEL 8 virtual server instance, but I keep getting ‘permission denied’ errors. The file storage was provisioned successfully and I can see it in the console with the mount point information, but the actual mount command fails.

I’m using the mount command provided in the console:


sudo mount -t nfs4 -o sec=sys fsf-dal1001a-fz.adn.networklayer.com:/IBM01SEV123456_1 /mnt/data

Error output:


mount.nfs4: access denied by server while mounting

The mount target shows as ‘active’ in the console, and I’ve verified the VSI has network connectivity to the storage endpoint (can ping it). I suspect this might be related to NFS export permissions or security group configuration, but I’m not sure what specific settings to check. Has anyone successfully mounted IBM Cloud File Storage on Linux and can point me in the right direction?

Let me provide a comprehensive solution for resolving NFS mount issues with IBM Cloud File Storage.

Root Cause Analysis: The ‘access denied by server’ error with IBM Cloud File Storage typically results from a combination of NFS export permissions, security group configuration, and mount command syntax issues.

NFS Export Permissions:

IBM Cloud File Storage uses IP-based access control lists (ACLs) to determine which hosts can mount an NFS export. Critical requirements:

  1. Authorized Hosts Configuration:

    • Navigate to Storage > File Storage in IBM Cloud console
    • Select your volume and find ‘Authorized Hosts’
    • Add your VSI’s private IP address (not public IP)
    • Format: Individual IP (10.240.0.15) or subnet (10.240.0.0/24)
    • Changes propagate within 1-2 minutes
  2. IP Address Verification:

    • On your VSI, run: ip addr show or `ifconfig
    • Use the IP from the private network interface (usually eth0 or ens3)
    • Verify this exact IP is in the authorized hosts list
    • Subnets must match: /32 for single host or appropriate CIDR for range

Security Group Configuration:

Security groups in IBM Cloud VPC act as virtual firewalls. For NFS connectivity:

  1. Required Outbound Rules:

    
    Protocol: TCP, Port: 2049, Destination: 10.0.0.0/8
    Protocol: TCP, Port: 111, Destination: 10.0.0.0/8
    Protocol: UDP, Port: 111, Destination: 10.0.0.0/8
    
  2. Why These Ports:

    • Port 2049: Primary NFSv4 communication
    • Port 111: RPC portmapper for NFS handshake
    • 10.0.0.0/8: IBM Cloud private network range (includes file storage endpoints)
  3. Common Mistake:

    • Not having a default ‘allow all outbound’ rule
    • Forgetting to apply security group changes to the VSI
    • Using public IP ranges instead of private network ranges

Mount Command Syntax:

The correct mount command format for IBM Cloud File Storage:

sudo mount -t nfs4 -o sec=sys,vers=4.1 \
  fsf-dal1001a-fz.adn.networklayer.com:/IBM01SEV123456_1 \
  /mnt/data

Key Parameters:

  • -t nfs4: Specifies NFSv4 protocol
  • -o sec=sys: Uses standard Unix authentication
  • vers=4.1: Explicitly sets NFS version (recommended)
  • Mount path: Use exact path from console (case-sensitive, no escaping needed)
  • Local mount point: Must exist before mounting (mkdir -p /mnt/data)

Troubleshooting Steps:

  1. Verify Network Connectivity:
    ping fsf-dal1001a-fz.adn.networklayer.com
    telnet fsf-dal1001a-fz.adn.networklayer.com 2049
    
    

   If telnet fails, security group rules are blocking traffic.

2. **Check NFS Client:**
   ```bash
   # Ensure NFS client is installed
   sudo yum install nfs-utils -y  # RHEL/CentOS
   sudo systemctl start rpcbind
   sudo systemctl enable rpcbind
   
  1. Test Mount with Verbose Output:
    sudo mount -t nfs4 -o sec=sys,vers=4.1,rw,bg,hard,timeo=600,rsize=65536,wsize=65536 \
      fsf-dal1001a-fz.adn.networklayer.com:/IBM01SEV123456_1 /mnt/data -vvv
    
    

   The `-vvv` flag provides detailed error information.

4. **Verify Mount Success:**
   ```bash
   df -h | grep /mnt/data
   ls -la /mnt/data
   

Permanent Mount Configuration:

Add to /etc/fstab for automatic mounting on boot:


fsf-dal1001a-fz.adn.networklayer.com:/IBM01SEV123456_1  /mnt/data  nfs4  sec=sys,vers=4.1,rw,bg,hard,timeo=600,rsize=65536,wsize=65536,_netdev  0  0

The _netdev option ensures the mount waits for network availability.

Complete Resolution Checklist:

  • [ ] VSI private IP added to File Storage authorized hosts
  • [ ] Security group allows TCP 2049 and TCP/UDP 111 outbound
  • [ ] Security group changes applied to VSI
  • [ ] NFS client packages installed (nfs-utils)
  • [ ] rpcbind service running
  • [ ] Local mount point directory exists
  • [ ] Mount command uses correct hostname and export path from console
  • [ ] NFSv4.1 explicitly specified in mount options

Expected Timeline:

  • Authorized host changes: 1-2 minutes
  • Security group changes: Immediate
  • Mount operation: 5-10 seconds

After following these steps systematically, your file storage mount should succeed. If issues persist, check IBM Cloud status page for any file storage service disruptions in your region, and verify that your file storage volume status is ‘Active’ in the console.

The ‘access denied by server’ error usually means the NFS server isn’t allowing your VSI’s IP address to mount the export. IBM Cloud File Storage uses IP-based access control.

First, check if you’ve authorized your VSI’s IP address in the file storage access list. In the IBM Cloud console, go to your file storage volume, find the ‘Authorized Hosts’ section, and verify that your VSI’s private IP is listed there. If it’s not, you need to add it before the mount will work.

Thanks! I checked the authorized hosts list and my VSI’s IP (10.240.0.15) is definitely in there. I added it when I provisioned the storage. Could there be a security group rule blocking the NFS traffic even though the IP is authorized? What ports does NFS use that I should check?

For NFSv4, you primarily need TCP port 2049. Port 111 can be TCP or UDP, but TCP is more commonly used. Add both just to be safe:

  • TCP port 2049 outbound to 10.0.0.0/8
  • TCP port 111 outbound to 10.0.0.0/8
  • UDP port 111 outbound to 10.0.0.0/8

After adding these rules, try the mount command again. The changes should take effect immediately.

Added the security group rules as suggested. The mount still fails with the same error. I’m wondering if there’s something wrong with my mount command syntax. The path /IBM01SEV123456_1 looks unusual - should there be any special escaping or quoting?

Yes, security groups could definitely be the issue. NFS requires multiple ports to be open. For NFSv4 (which you’re using based on your mount command), you need:

  • Port 2049 (TCP) - main NFS port
  • Port 111 (TCP/UDP) - portmapper/rpcbind

Check both your VSI’s security group outbound rules and make sure these ports are allowed to the file storage subnet. The file storage endpoint is on IBM Cloud’s private network, so you need outbound rules to the 10.0.0.0/8 range or more specifically to the storage service network.

I reviewed the security group and it only has port 22 (SSH) and 443 (HTTPS) explicitly allowed outbound. There’s no default ‘allow all outbound’ rule. I’ll add rules for ports 2049 and 111 to the IBM Cloud private network range. Should these be TCP only or both TCP and UDP?