The ActiveAlarmTable Service
Introduced in JBoss v4.0.3, the ActiveAlarmTable service is a table
that can be used to hold the current (active) system alarms. Assuming that
the occurence of an alarm is indicated by a JMX notification, the ActiveAlarmTable
is essentially a NotificationListener based on ListenerServiceMBeanSupport
that can subscribe for any JMX notification in the system, in order to feed
the alarm table.
The main goal of providing this service is to make it easier to discover
the collective fault status of the system. If the alarm table is updated
correctly it should help answering the question "what is wrong with my system, now".
Alarm Types & Severity
Alarm is an indication of an error condition in the system. The distinguishing
factor of alarms from other system events, is that alarms have a "severity"
(or it can be that their severity is "unknown").
According to most management standards the available severities are:
NORMAL
WARNING
MINOR
MAJOR
CRITICAL
UNKNOWN
Another characteristic of alarms is that they can be stateful or stateless.
A stateless alarm indicates an instantanious error condition, e.g. a security
violation, or a deployment failure. The mbean that produces the alarm doesn't
keep track of the error condition to report its clearance (or this may be
simply impossible). The state of a stateless alarm is always NONE, while the
severity can be anything other than NORMAL.
A stateful alarm represents an error condition in the system, e.g. "memory low",
that can clear itself out. The mbean that produces the alarm can, after a while,
decide that the free memory is back to normal levels, so the alarm can be
cleared. To do that, the mbean has to keep track of the alarm status. The
state changes for stateful alarms are indicated with emitted alarm notifications
that carry the alarm states CREATED, CHANGED, or CLEARED. A CLEARED alarm
must carry a severity of type NORMAL.
A common source of confusion is that we often associate an alarm with the Notification
carrying the alarm. A notification is simply an "event in transit". In the case
of stateless alarms, the notification carries the full state of the alarm,
because every instantiation of the alarm represents a separate event. With stateful
alarms, the notification merely carries the state change of the alarm, i.e.
you can have 3 notifications referring to the same logical alarm condition.
So for example, an alarm can be first CREATED with a severity WARNING,
then CHANGED with an escallated severity of CRITICAL, and finally CLEARED
with a NORMAL severity.
How the ActiveAlarmTable interprets received notifications.
Every notification received by the ActiveAlarmTable is considered to be an alarm.
Depending on the type of notification we have three cases:
A subclass of org.jboss.monitor.alarm.AlarmNotification that carries
an alarmState of CREATED, CHANGED or CLEARED. This is considered to be
a stateful alarm. The notification source is combined with the notification
type to form an Id. Multiple notifications with the same Id are stored
as a single entry in the table, since they represent a single error condition.
A subclass of org.jboss.monitor.alarm.AlarmNotification that carries
an alarmState of NONE. This is treated as a stateless alarm and is always stored
in the table as a new entry.
Any other Notification is treated as a stateless alarm. Since its severity
is not known (which is the case with an AlarmNotification) we assume a
severity of UNKNOWN.
So it is easier for stateless alarm to fill the table, since every notification
is considered a different alarm. For example, a security manager mbean may
throw multiple notifications of type "jboss.security.violation" and each is
a different fault.
On the other hand, stateful notifications of the same type
and the same source create a single entry in the table. For example,
multiple notifications of type "jboss.alarm.memory.low" from the same instance
jboss.monitor:service=MemoryMonitor refer to the same fault. If an mbean
wants to produce multiple stateful alarms it needs to use different notification
types, e.g. to represent the failed or otherwise normal status of a pool of
3 connection we could use the types:
jboss.alarm.connection.failed.01
jboss.alarm.connection.failed.02
jboss.alarm.connection.failed.03
Acknowledgement of Alarms
The alarms are kept in the table for somebody (usually an administrator) to
see and react. An alarm can be acknowledged to indicate that it's been handled.
In that case:
A stateles alarm is removed from the table
A stateful alarm, if it has cleared itself out, it is removed from the table,
otherwsie, it is just marked as acknowledged, in its current state. Every
change in its state resets the acknowledge flag.
Configuring the ActiveAlarmTable
The following attributes can be configured:
MaxTableSize, can be used to set an upper limit to the number of
entries kept in the table in order to avoid overflow. When the table
is full, new entries are ignored. Alarm tables shouldn't have to keep more
than a few hundred entries. The default is 1000.
LogLevel, the log level to use for received notification, can be
set to NONE, default is DEBUG
SubscriptionList, can be used to subscribe for any notification
in the system.
Retrieving the active alarms
The active alarm table can be retrieved from the service in the form
of a AlarmTableNotification array. AlarmTableNotification is an
extension of AlarmNotification, that adds acknowledge info and uses
the payload (UserData) field, to store the original alarm notification.
The ActiveAlarmTable itself produces jboss.alarm.table.update notifications
whenever updated.
Feeding the table with alarms
The table can be fed with any notification in the system but there is already
a number of services that can actually produce AlarmNotification notifications:
ScriptingListener, a scripting facility that allows you to map
ordinary notifications to alarm notifications; however this is a generic
service and it can be used for other purposes.
JMXNotificationAppender, a simple log4j appender that transforms logging
events to JMX notifications, so they can feed the alarm table.
MemoryMonitor, a simple free memory monitor that produces stateful
alarm notifications.
Producing alarms from your own MBeans
If you want your mbeans to produce natively stateful/stateless AlarmNotification
notifications, that can feed the alarm table directly, you may want to
consider using the AlarmManager helper, that greatly simplifies this task
(especially when it comes to stateful notifications).
Demo
The ActiveAlarmTable is disabled by default. To enable
1. edit conf/jboss-service.xml, uncomment the entry:
... <mbean code="org.jboss.monitor.services.ActiveAlarmTable" name="jboss.monitor:service=ActiveAlarmTable"> <attribute name="SubscriptionList"> <subscription-list> <mbean name="jboss.monitor:*"> <notification type="jboss.alarm"></notification> <notification type="JBOSS_MONITOR_NOTIFICATION"></notification> </mbean> <mbean name="jboss.system:service=Logging,type=JMXNotificationAppender"></mbean> </subscription-list> </attribute> </mbean> ...
2. edit conf/log4j.xml and uncomment the JMXNotificationAppender
... <!-- Emit events as JMX notifications --> <appender name="JMX" class="org.jboss.monitor.services.JMXNotificationAppender"> <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"></errorHandler> <param name="Threshold" value="WARN"/> <param name="ObjectName" value="jboss.system:service=Logging,type=JMXNotificationAppender"/> <layout class="org.apache.log4j.PatternLayout"> <param name="ConversionPattern" value="%d %-5p [%c] %m"/> </layout> </appender> ...
Also, remember to add the JMX appender in the root category
... <root> <appender-ref ref="CONSOLE"></appender-ref> <appender-ref ref="FILE"></appender-ref> <appender-ref ref="JMX"></appender-ref> <!-- ADD HERE --> </root> ...
This appender transforms log4j messages to JMX notification so they can
be fed to the table.
3. for an example of a stateful alarm, edit deploy/monitoring-service.xml
and uncomment the MemoryMonitor:
<mbean code="org.jboss.monitor.services.MemoryMonitor" name="jboss.monitor:service=MemoryMonitor"> <attribute name="FreeMemoryWarningThreshold">95M</attribute> <attribute name="FreeMemoryCriticalThreshold">80M</attribute> </mbean>
4. just for the sake of the example, create a faulty deployment, e.g.
create an empty directory deploy/bad.sar
5. restart the server!
6. Go to the web-console http://localhost:8080/web-console/
Under the Monitoring tab, there should be a new entry, "Alarm Table". By
clicking on this you should get something similar to the picture below.
In this example we see an error thrown by the URLDeploymentScanner
when the initial deployment round was complete, pointing out bad.sar
as a failed deployment. This is a stateless alarm and it will just go away
if you Ack(nowledge) it.
Another stateful alarm may be created depending on the settings of the
MemoryMonitor service. The state of this alarm changes depending on
the free memory of the system. Acknowledging the alarm will make it
go away, if it CLEARED by the system, too.
There should be another alarm showing the stackstrace of the bad.sar
deployment, but I have already ack'ed this in the following picture:
Related:
SubscriptionList
Referenced by:
Comments