In a recently installed ZS5-2 (ZFSSA) it was necessary to exchange one of the Infiniband Network Cards with a 10 Gbit Dual Port Fibre Channel Card. After the replacement there was a mismatch in the device naming of the new interfaces. On the first storage head it showed up as ixgbe2 and ixgbe3 and on the other head as ixgbe1 and ixgbe3.
Why is this fact important? The reason for this is as on top of the interfaces there are some virtual IPs configured which have to switchover to the other head in case of errors or due to maintenance. This won’t work if the names of the network devices are different between the heads.
Long time ago I have done renaming/renumbering of devices in a regular Solaris Sparc Server with modifying ‘/etc/path_to_inst’ and I was nearly quite sure this would do the trick for the ZFSSA as well. In the ZFSSA the file is on a slightly different location ‘/etc/devices/path_to_inst’.
The way to get things fixed
To get things fixed it is necessary to reinitialize the whole device configuration on each storage head of the ZFSSA.
- Login to the Storage head via ILOM console
- Check current Configuration
- Switch from CLI to Shell-Mode
- Instruct Appliance to regenerate /etc/default/path_to_inst after reboot
- Update boot environment
- Reboot
- Check configuration
- Repeat steps for second storage head
- correct datalink configuration –> mapping to the new device names
Now login to ZFSSA Ilom Console and switch to CLI:
ssh root@zfssa-ilom Hostname: zfssa-head1 -> start /SP/console zfssa-head1 console login: root
Display Network Configuration:
zfssa-head1:> configuration net devices show Devices: DEVICE UP SPEED MAC i40e0 true 1000 Mbit/s 0:10:e0:ba:e2:41 i40e1 true 1000 Mbit/s 0:10:e0:ba:e2:42 i40e2 false 0 Mbit/s 0:10:e0:ba:e2:43 i40e3 false 0 Mbit/s 0:10:e0:ba:e2:44 ibp0 true 32000 Mbit/s not available ibp1 true 32000 Mbit/s not available ibp6 true 32000 Mbit/s not available ibp7 true 32000 Mbit/s not available ixgbe2 true 10000 Mbit/s 90:e2:ba:ef:1f:22 ixgbe3 true 10000 Mbit/s 90:e2:ba:ef:1f:23
In this case the numbering of the 10GBit interfaces is wrong. It is currently ixgeb2 and ixgbe3.
As this is the only 10Gbit Network Card the names of the interfaces should be ixgbe0 and ixgbe1.
In addition check the current device mapping for the network card from “/etc/devices/path_to_inst”.
To get this piece of information switch from CLI to Shell Mode. Be careful in shell mode as it is not supported to change anything without advice from Oracle Support. If you do on your own it could break your system or could lead to problem when doing firmware upgrades in the future.
zfssa-head1:> shell Executing shell commands may invalidate your service contract. Continue? (Y/N) Y Executing raw shell; "exit" to return to appliance shell ... zfssa-head1# grep ixgbe /etc/devices/path_to_inst "/pci@0,0/pci8086,2f02@1/pci108e,7b11@0" 0 "ixgbe" "/pci@0,0/pci8086,2f02@1/pci108e,7b11@0,1" 1 "ixgbe" "/pci@0,0/pci8086,2f08@3/pci108e,7b11@0" 2 "ixgbe" "/pci@0,0/pci8086,2f08@3/pci108e,7b11@0,1" 3 "ixgbe"
To fix the name number issue with the devices, instruct the ZS5-2 to regenerate the “/etc/devices/path_to_inst”.
echo '#path_to_inst_bootstrap_1' > /etc/devices/path_to_inst
Now update the boot archive with the changed configuration and reboot the system.
bootadm update-archive Updating boot archive ... done. reboot
After reboot check the device configuration if everything is now as expected.
zfssa-head1 console login: root zfssa-head1:> configuration net devices show Devices: DEVICE UP SPEED MAC i40e0 true 1000 Mbit/s 0:10:e0:ba:e2:41 i40e1 truee 1000 Mbit/s 0:10:e0:ba:e2:42 i40e2 false 0 Mbit/s 0:10:e0:ba:e2:43 i40e3 false 0 Mbit/s 0:10:e0:ba:e2:44 ibp0 true 32000 Mbit/s not available ibp1 true 32000 Mbit/s not available ibp2 true 32000 Mbit/s not available ibp3 true 32000 Mbit/s not available ixgbe0 true 10000 Mbit/s 90:e2:ba:ef:1f:22 ixgbe1 true 10000 Mbit/s 90:e2:ba:ef:1f:23 zfssa-head1# grep ixgbe /etc/devices/path_to_inst "/pci@0,0/pci8086,2f08@3/pci108e,7b11@0" 0 "ixgbe" "/pci@0,0/pci8086,2f08@3/pci108e,7b11@0,1" 1 "ixgbe"
In case that the network device names are also incorrect on the second storage head, repeat the procedure.
Curiosities after Device Rename
After rename of the devices took place one can be faced with the following error when connecting to the Web-Frontend of the ZFSSA.
Additionally the device association between “Devices” and “Datalinks” for the interface ixgbe0 and ixgbe1 is empty.
Manually associate the device “ixgbe1” to the datalink “ixgbe1”. Apply the configuration changes and everything is fine.
Caveat
Please keep in mind that when renaming the 10Gbit Network Interfaces maybe other wrong named interfaces get corrected as well. This could temporary lead to failed interfaces and ip addresses which formerly were working correctly.
Conclusion
Now you have the approach how to fix incorrect device naming on your own. If unsure, please contact Oracle Support.