We recently migrated our Greengrass core device’s private key and certificate to a hardware security module (HSM) for enhanced security compliance. After the migration, the core device fails to establish connection with AWS IoT Core and shows offline status in Device Defender monitoring.
The PKCS#11 provider configuration appears correct in our greengrass config, pointing to the HSM slot and label where we stored the device certificate. We’ve verified the HSM is accessible and the certificate chain is valid.
Error: Failed to establish MQTT connection
Cause: Unable to load private key from PKCS#11 provider
Provider: /usr/lib/softhsm/libsofthsm2.so
Slot: 0x1a2b3c4d
The device was functioning normally before HSM migration using file-based keys. Device Defender shows the device as disconnected for 6 hours now, which is impacting our real-time monitoring and compliance reporting. Has anyone successfully integrated Greengrass with HSM-based key storage while maintaining Device Defender connectivity?
Thanks for the suggestions. I verified the library path and permissions - those look correct. The slot ID is properly formatted in hex. When I run pkcs11-tool, I can see the certificate and it shows as valid. The PIN is configured in an environment variable that Greengrass should have access to. Still getting the same connection failure.
One thing that often gets missed - HSM session timeouts. Some HSMs automatically close sessions after a period of inactivity, which can cause connection failures when Greengrass tries to reuse the session. Check your HSM configuration for session timeout settings and consider implementing session keepalive or automatic reconnection logic in your PKCS#11 wrapper configuration.
I’ve seen this issue before. The problem is usually in the greengrass config.yaml where you define the PKCS#11 configuration. Make sure you’re specifying both the certificate URI and the private key URI correctly. The certificate should reference the HSM slot and label, and you need to set the appropriate PIN/password for HSM access. Also, double-check that your HSM supports the key algorithms required by AWS IoT (RSA 2048 or EC P-256). Device Defender monitoring relies on the core device maintaining an active MQTT connection, so until the TLS handshake completes successfully with the HSM-backed key, you’ll see the offline status.
Check your PKCS#11 provider library path and permissions first. The Greengrass core process needs read/execute access to the HSM library. Also verify the slot ID format - some HSMs use hexadecimal format (0x…) while others use decimal. Try running a simple pkcs11-tool command to list available slots and confirm your certificate is visible.
Are you running Greengrass as a system service? If so, make sure the environment variables (especially the HSM PIN) are available to the systemd service context, not just your user session. You might need to add them to the service file under [Service] section with Environment= directives. Also check the Greengrass logs at /greengrass/v2/logs/greengrass.log for more detailed PKCS#11 errors.