LogEncapsulator

This module provides various classes that enable one to monitor log files using the event framework included in this package. There are several classes that are part of this module, each of which are fully documented. A brief overview of the entire module is provided in subsequent paragraphs.

The primary class that users interact with is the LogEncapsulator. The LogEncapsulator implements the EventGenerator interface (its a source of Events). Each LogEncapsulator is configured with one or more files to monitor using a single set of Rules. A Rule is used to determine if a match has occurred when monitoring, and when that Event does occur, the Rule specifies the EventHandler used to process the Event (see the Event module for details on the event framework).

The EventHandler used to process a LogEncapsulator generated Event can expect that the Event's data property contains a LogMatch object. The LogMatch object simply encapsulates the line from the log file that triggered the match, as well as the match object from the re package. This package includes various EventHandlers that can be used as part of a Rule definition.

Sample use of this module follows:

#!/usr/bin/python

from time import sleep
from com.kazmier.event.Event import EventDispatcher
from com.kazmier.event.Event import ChainEventHandler
from com.kazmier.event.Event import PrintEventHandler
from com.kazmier.logwrap.LogEncapsulator import Rule
from com.kazmier.logwrap.LogEncapsulator import LogEncapsulator
from com.kazmier.logwrap.LogEncapsulator import WaitToCountEventHandler

dispatcher = EventDispatcher()
try:
    encapsulator = LogEncapsulator("Test Encapsulator")
    encapsulator.watch_file("/var/log/syslog")
    encapsulator.watch_file("/var/log/maillog")
    encapsulator.add_rule(Rule("some pattern", PrintEventHandler()))
    encapsulator.add_rule(Rule("another", ChainEventHandler(
                                            WaitToCountEventHandler(),
                                            PrintEventHandler())))
    dispatcher.error_handler(ErrorHandlerAdapter(PrintEventHandler()))
    dispatcher.add_event_generator(encapsulator)
    dispatcher.start()
    while 1:
        sleep(1)

except Exception, e:
    dispatcher.stop()
    print e

If you do not want to use the API directly, there is a tool called logwrap that will parse an XML file and generate all of the appropriate encapsulators and rules automatically.

class Rule(object):

A rule is comprised of a pattern and an event handler. The event handler is used to process events that are generated when the pattern is matched in a log file. In addition, a rule can also be 'quick', which indicates that upon a match, all further processing should stop. By default, a message from a log file can match multiple rules.

__init__(regexp, handler, quick=0):

Constructor that specifies the regular expression used to match lines, the event handler used to process events, and an optional flag indicating the quickness of the rule.

__str__():

Returns a string representation of a rule.

class LogMatch(object):

When an event is generated, indicating a log match, the event contains a LogMatch object that contains the line that caused the event to be generated, as well as the 're.match' object so that individual components of the matched can be fetched.

__init__(line, match):

Constructor that specifies the line that caused a match to occur, as well as the match object returned by the 're' module.

__str__():

Returns a string representation of the LogMatch.

class LogEncapsulator(EventGenerator):

Wraps one or more log files and generates events based on regular expressions that match log entries. In addition, if the log file(s) are moved or rotated, the original file(s) are closed, and the new file(s) are re-opened (see TailFollow for more details).

When the LogEncapsulator creates an Event, the event data is an instance of LogMatch which contains the original log line that caused the event, as well as a match object from the regular expression. This match object can then be used by the EventHandler when processing the Event.

The LogEncapsulator is designed to be used in the Event framework that accompanies this module. Specifically, the encapsulator is an instance of an EventGenerator and must be added to an EventDispatcher to process events that are generated.

__init__(name):

Constructor that specifies the name of the encapsulator.

__str__():

Returns the name of the encapsulator.

watch_file(filename, track=1):

Watch the specified file for new data that may or may not trigger an event based on the configured rules. An optional parameter called 'track' can be used to indicate if the file should be tracked in the event it is rotated (by default).

add_rule(rule):

Add the rule to the current set of rules that will be used to process new data from the various files being watched.

stop():

Stop the encapsulator and cleanly shutdown all resources currently in use such as the files that are being tracked via TailFollow.

get_events():

Gets all pending events that have occurred since the last time this method was invoked. An event is generated when a log entry matches one of the rules associated with this encapsulator.

class WaitToCountEventHandler(EventHandler):

An event handler that only succeeds when a particular count threshold has been exceeded for the same matched log message. This event handler is designed to be used in a ChainEventHandler as it doesn't provide any functionality by itself.

__init__(threshold=3, reset=1, match_on=(0,)):

Constructor that specifies the threshold (or count) that must be exceeded before the event handler returns success (which enables the chain to proceed). An optional boolean argument called 'reset' can be used to indicate that the handler should reset its count upon success (thereby resetting the trigger), otherwise, a message is generated for each subsequent event. Finally, the 'match_on' optional parameter specifies a tuple of fields that are used to match a message with a count. This enables users to key off of a part of the message when incrementing a count.

process_event(event):

Process an event and check if it has exceeded the current threshold. If the threshold has been exceeded, return a 1 to indicate success, otherwise return 0 which is used to stop event processing in the chain.

class IntervalCountEventHandler(EventHandler):

An event handler that succeeds upon receipt of the first matched log message and thereafter only when the interval has been exceeded. Note, this is different from the WaitToCount event handler which succeeds only after the threshold has been exceeded. This event handler is designed to be used in a ChainEventHandler as it doesn't provide any functionality by itself.

__init__(threshold=3, match_on=(0,)):

Constructor that specifies the threshold (or interval) that must intervene between matched log messages before success is returned (which enables the chain to proceed). The 'match_on' optional parameter specifies a tuple of fields that are used to match a message with a count. This enables users to key off of a part of the message when incrementing a count.

process_event(event):

Process an event and check if it has exceeded the current threshold. If the threshold has been exceeded, return a 1 to indicate success, otherwise return 0 which is used to stop event processing in the chain. Also return 1 upon receipt of the first event.

class WaitToFrequencyEventHandler(EventHandler):

An event handler that only succeeds when a particular frequency threshold has been exceeded for the same matched log message. This event handler is designed to be used in a ChainEventHandler as it doesn't provide any functionality by itself.

__init__(threshold=3, interval=60, reset=1, match_on=(0,)):

Constructor that specifies the threshold (or count) that must be exceeded within the specified interval before the event handler returns success (which enables the chain to proceed). An optional boolean argument called 'reset' can be used to indicate that the handler should reset its count upon success (thereby resetting the trigger), otherwise the previous events are taken into account in subsequent calculations. Finally, the 'match_on' optional parameter specifies a tuple of fields that are used to match a message with a count. This enables users to key off of a part of the message when incrementing a count.

process_event(event):

Process an event and check if it has exceeded the current threshold. If the threshold has been exceeded, return a 1 to indicate success, otherwise return 0 which is used to stop event processing in the chain.

class LimitToFrequencyEventHandler(EventHandler):

An event handler that only succeeds when a particular frequency threshold has not been exceeded for the same matched log message. This event handler is designed to be used in a ChainEventHandler as it doesn't provide any functionality by itself.

__init__(threshold=3, interval=60, reset=1, match_on=(0,)):

Constructor that specifies the threshold (or count) that must not be exceeded within the specified interval before the event handler returns success (which enables the chain to proceed). An optional boolean argument called 'reset' can be used to indicate that the handler should reset its count upon success (thereby resetting the trigger), otherwise the previous events are taken into account in subsequent calculations. Finally, the 'match_on' optional parameter specifies a tuple of fields that are used to match a message with a count. This enables users to key off of a part of the message when incrementing a count.

process_event(event):

Process an event and check if it has exceeded the current threshold. If the threshold has not been exceeded, return a 1 to indicate success, otherwise return 0 which is used to stop event processing in the chain.

class ErrorHandlerAdapter(EventHandler):

ErrorHandlerAdapter adapts the Event generated by the EventProcessor for errored events such that the data field of the Event object contains a LogMatch object. This enables users to use their existing library of event handlers written that expect LogMatch objects to be present. For example, this adapter class enables users to use the MainEventHandler and SnmpEventHandlers defined in this class.

The adapter works by introspecting the data object contained within the original event which is a tuple that contains the exception as well as the original event that caused the error. These two values are placed into a Match object in fields 1 and 2 respectively. Thus, handlers can use match.group(1) to obtain the exception, and match.group(2) to obtain the original event.

__init__(handler):

Constructor that specifies an event handler wishing to receive an adpated event that contains a LogMatch object in the Event's data field.

process_event(event):

Adapts the event to a format expected by logwrap event handlers.

class MailEventHandler(EventHandler):

MailEventHandler sends an email message that can optionally contain parts of the log message that generated the event. The subject and body strings are examined for any sequence of \n where n is a digit from 0 to 99 that represents the appropriate group of the match object that was included as part of the LogMatch Event data.

__init__(fromaddr, toaddrs, subject='', body='', smtphost='localhost'):

Constructor that specifies the sender, a list of recipients, a subject, and a body. The subject and body are interpolated for special characters that should be replaced by parts of the event.

process_event(event):

Sends an email in response to the Event.

class SnmpTrapEventHandler(EventHandler):

SnmpTrapEventHandler sends an SNMP trap that can optionally contain parts of the log message that generated the event as varbinds of the trap. Each varbind message body can contain \n where n is a digit from 0 to 99 that represents the appropriate group of the match object that was included as part of the LogMatch Event data.

__init__(hosts, community, trapoid, varbinds=()):

Constructor that specifies the a list of hosts, a community string to use for all of the hosts, the OID of the trap, as well as a sequence of varbinds. Each varbind sequence is a tuple that contains the OID, type, and value (all of which are specified as strings). For example:

trap = SnmpTrapEventHandler( \
         ("host1", "host2"),
         "public",
         "1.3.6.1.4.1.8233.111.1",
         (("1.3.6.1.4.1.8233.200.1", "OCTETSTRING", "Error"),
          ("1.3.6.1.4.1.8233.200.2", "INTEGER", 100),
          ("1.3.6.1.4.1.8233.200.3", "INTEGER", "\2")))

As an alternative, all of the arguments may be specified as strings. In which case, the following is the expected format:

trap = SnmpTrapEventHandler( \
         "host1, host2",
         "public",
         "1.3.6.1.4.1.8233.111.1",
         "1.3.6.1.4.1.8233.200.1: OCTETSTRING: Error
          1.3.6.1.4.1.8233.200.2: INTEGER: 100
          1.3.6.1.4.1.8233.200.3: INTEGER: \2")

The string formats are designed for external programmatic applications (such as an XML rule builder, or GUI front-end.

process_event(event):

Sends an SNMP Trap in response to the Event.