NVMe hotplugging

As i get some questions about it in recent times: With a normal SATA/SAS disk you can simply unplug a disk without warning and preparation. NVMe are different. For all practical purposes they are PCI devices. And they want to be handled as such. Unplugging a NVMe drive without using the proper procedure is pretty much like yanking out a PCI card without preparation. So when you want to remove a NVMe disk you have to use the hotplug command in Solaris. To cite the documentation:

># hotplug list –lc
Connection  State    Description
Path
------------------------------------------------------------------ 
pcie13      ENABLED  PCIe-Native /pci@7a,0/pci8086,2f08@3/pci111d,80b5@4
# hotplug poweroff pcie13
# hotplug list –lc
Connection  State    Description
Path
------------------------------------------------------------------
pcie13      PRESENT  PCIe-Native /pci@7a,0/pci8086,2f08@3/pci111d,80b5@4

On a Oracle server a “OK to remove” light will turn on. Now it should appear as empty.

># hotplug list –lc
Connection  State    Description
Path
------------------------------------------------------------------
pcie13      EMPTY    PCIe-Native /pci@7a,0/pci8086,2f08@3/pci111d,80b5@4

For replacement, wait 10 seconds after removal. Then insert the new NVMe drive. You have to enable it before using it.

>#hotplug enable pcie13
# hotplug list –lc
Connection  State    Description
Path
------------------------------------------------------------------
pcie13      Enabled  PCIe-Native /pci@7a,0/pci8086,2f08@3/pci111d,80b5@4