New Solaris features: Notifications from the Fault Management Architecture

The notification framework in Solaris is not only able to notify you of problems with services. As you may know, the fault management architecture monitors the hardware of a Solaris system. With the fmdump command you can retrieve the information. For example you get this informatoin when you unplug an USB stick with an active ZFS filesystem:

root@solaris:/home/jmoekamp# fmdump -v<br />
TIME                 UUID                                 SUNW-MSG-ID EVENT
Nov 22 12:59:48.7489 22dffa96-bf27-4f41-ab1f-dfedcdfb11e7 ZFS-8000-HC Diagnosed<br />
  100%  fault.fs.zfs.io_failure_wait
        Problem in: zfs://pool=usbstick1<br />
           Affects: zfs://pool=usbstick1<br />
               FRU: -<br />
          Location: -

Wouldn’t it be nice to receive this and similar information about the failure of components. It’s really simple to configure. However i assume , that you’ve already installed smtp-notify package with

root@solaris:/# pkg install service/fault-management/smtp-notify

To configure the notification, you just have to configure the target mail adress for the mails.

root@solaris:/# svccfg setnotify problem-diagnosed mailto:root@localhost

To check the configuration you can use the listnotify command:

root@solaris:/# svccfg listnotify problem-diagnosed
    Event: problem-diagnosed (source: svc:/system/fm/notify-params:default)
        Notification Type: smtp
            Active: true
            reply-to: root@localhost
            to: root@localhost

        Notification Type: snmp
            Active: true

        Notification Type: syslog
            Active: true

When you unplug the USB-stick with while the ZFS pool is imported, you will find a mail like this one in your mailbox:

root@solaris:/# mail
From noaccess@solaris.local Mon Nov 22 12:59:48 2010
Date: Mon, 22 Nov 2010 12:59:48 +0100 (CET)
From: No Access User <noaccess@solaris.local>
Message-Id: <201011221159.oAMBxm5o002968@solaris.local>
Subject: Fault Management Event: solaris:ZFS-8000-HC
To: root@solaris.local
Content-Length: 560

SUNW-MSG-ID: ZFS-8000-HC, TYPE: Error, VER: 1, SEVERITY: Major
EVENT-TIME: Mon Nov 22 12:59:44 CET 2010
PLATFORM: VirtualBox, CSN: 0, HOSTNAME: solaris
SOURCE: zfs-diagnosis, REV: 1.0
EVENT-ID: 22dffa96-bf27-4f41-ab1f-dfedcdfb11e7
DESC: The ZFS pool has experienced currently unrecoverable I/O
	    failures.  Refer to for more information.
AUTO-RESPONSE: No automated response will be taken.
IMPACT: Read and write I/Os cannot be serviced.
REC-ACTION: Make sure the affected devices are connected, then run
	    'zpool clear'.

? q

Neat, isn’t it?