New Solaris features: How to monitor SMF services via mail?
It’s a small feature compared to other additions to Solaris, however already i got the first questions how to configure it: the capability of the Service Management Facility to send mails when a state transitions occur.
Some theory
The capability to send notifications in case of service transitions is just a part of the capabilities of this feature. At first the feature can’t only send mails, you can send SNMP traps as well. Additionally the feature isn’t just able to send mails or SNMP traps in case of service state transitions, you can also configure the system to send mail in case of problems detected by the Fault Management Architecture at the hardware. But both is stuff for different articles.
In it’s lifetime a service goes trough a number of states. From being disabled up to up-and-running. You will find a description of the states in my tutorial about SMF. The description of the states is located almost at at the end of part 2.
The notification frameworks adds mechanisms to enable SMF to tell the outside world, when a services transitions away from one state or into a state. So number of such transitions are monitored by the notification framework:
to-uninitialized
from-uninitialized
to-maintenance
from-maintenance
to-offline
from-offline
to-disabled
from-disabled
to-online
from-online
to-degraded
from-degraded
Installation
The demon needed to send this mail isn’t in the default installation. So you have to install the package pkg install service/fault-management/smtp-notify in addition to your basic system:
root@solaris:/# pkg install service/fault-management/smtp-notify
Packages to install: 1
Create boot environment: No
Services to restart: 1
DOWNLOAD PKGS FILES XFER (MB)
Completed 1/1 4/4 0.0/0.0
PHASE ACTIONS
Install Phase 24/24
PHASE ITEMS
Package State Update Phase 1/1
Image State Update Phase 2/2
Now you can enable the daemon by enabling the service controlling it.
To force it into maintainance mode i just added a SNAFU at the beginning of the file /etc/apache2/2.2/httpd.conf and restarted the service with
svcadm restart apache-22
Let’s check the situation. The apache-22 service has transitioned into the maintainance mode:
root@solaris:/# svcs -a | grep "apache"
maintenance 6:12:58 svc:/network/http:apache22
When you check your mail you will find a mail:
root@solaris:/# mail<br />
From noaccess@solaris.local Thu Nov 18 06:12:59 2010<br />
Date: Thu, 18 Nov 2010 06:12:59 +0100 (CET)<br />
From: No Access User <noaccess@solaris.local><br />
Message-Id: <201011180512.oAI5CxVV001278@solaris.local><br />
Subject: Fault Management Event: solaris:SMF-8000-YX<br />
To: root@solaris.local<br />
Content-Length: 668
SUNW-MSG-ID: SMF-8000-YX, TYPE: defect, VER: 1, SEVERITY: major<br />
EVENT-TIME: Thu Nov 18 06:12:58 CET 2010<br />
PLATFORM: VirtualBox, CSN: 0, HOSTNAME: solaris<br />
SOURCE: software-diagnosis, REV: 0.1<br />
EVENT-ID: 345ed233-8d58-ef72-d050-c552d0c78670<br />
DESC: A service failed - a start, stop or refresh method failed.<br />
Refer to http://sun.com/msg/SMF-8000-YX for more information.<br />
AUTO-RESPONSE: The service has been placed into the maintenance state.<br />
IMPACT: svc:/network/http:apache22 is unavailable.<br />
REC-ACTION: Run 'svcs -xv svc:/network/http:apache22' to determine the generic reason why the service failed, the location of any logfiles, and a list of other services impacted.
? q
<pre></pre>
Managing notifications
When you want to know what notification are already configured for a service, you can use the listnotify subcommand of svccfg.
</code></blockquote>
Please note, that the -g option is missing. When you use the -g the configuration done in svc:/system/svc/global:default no matter what service you've declare later on in the command line. Let's check the configuration:
It's important to know, that a service has as well the global as the locally set notifications. Let's assume you've set from-maintenance at svc:/system/svc/global and you've set from-online at svc:/network/http:apache22. Then the service svc:/network/http:apache22 will have both notifications:
When you look at the source for each event, you can see, what's the source of this notification setting.
Conclusion
Due to the smtp-notify capability of the Service Management facility it's really easy to give all services managed by SMF the capability to send some kind of notification to the admin.