Getting Groovy, Processing XML, sending SNMP traps and RapidInsight

Published by Berkay Mollamustafaoglu on April 6, 2009 06:38 am under IT management, technology

A while ago Doug asked whether anyone knew a utility to read XML and send SNMP traps. I mentioned that it would be easy to do with Groovy but did not have time to give any details. I’ve been meaning to write a post about how we use Groovy in RapidInsight and thought this would be a good excercise.

First I have state once again that I’m not a developer nor play one on Youtube. I can however, put together scripts, especially if there are examples, but writing java code is not my cup of tea. Groovy makes this stuff easy enough to deal with for people like me (system integrators, admins, etc.)

In this example, we’ll read the RSS feed (which is XML) from Doug’s blog and take action if the feed title includes the word BSM in it :) Groovy includes powerful XML utility called xmlslurper that makes reading XML a breeze.

def url = "http://dougmcclure.net/blog/feed/"
def feed = new XmlSlurper().parse(url)
feed.channel.item.each {post ->
   if (post.title.toString().matches(".*BSM.*")) {
      println "found BSM post!: " + post.title + " – " post.link.toString()
   }
}

There it is, we already  have a Groovy script to retrieve the RSS feed, iterate through the posts and determine whether the title includes BSM keyword. One can take a look at how xmlSlurper works and use it to get the information she needs.

RapidInsight makes it even easier for the script developers to work with the external interfaces without having to learn the intricacies of each interface. In RapidInsight, we can create a class called say RssFeed, store data related to the RSS feed as object properties.Here is how our class may look like:

<model name="RssFeed">
   <properties>
      <property name="name" type="string">
      <property name="url" type="string">
      <property name="type" type="string">
   </property>
</property>

The class can be used to store the information related to the Rss feed as objects. We can create a UI for the user to enter this information, or simply use a script to create the instances. Creating an object in RapidInsight repository is straight forward, pass the properties as name value pairs to add operation:

def props = [:]
props.name = "Doug McClure BSM Blog"
props.url = "http://dougmcclure.net/blog/feed/"
props.type = "Rss 2.0"
RssFeed.add[props]
// I could have written this as a single line as well
// RssFeed.add[url:"http://dougmcclure.net/blog/feed/", type = "Rss 2.0"]

By storing the url, etc. as object properties, we can now have a more generic script that would work with any RSS feed. Next we can add “operations” to this class to deal with the Rss feeds, hiding the complexities of the external interface from the script developer. An operation can read the feed and return name value pairs in a map to the user.

This means the script developer does not have to understand the structure of the RSS or how to use XmlSlurper. The operation code can be modified to add support for different RSS formats (v1, v2), use ROME library to parse RSS instead of XmlSlurper, etc., scripts would continue to work as before and script developers like me would not have to learn something different. All it matters is what is passed to us, which is a List of Map (name value pairs), and this is same for all external interfaces.  (Take a look a this post to see more on how RapidInsight uses name value pairs when working with external interfaces)

The operation may be something like this:

//readFeed operation returns blog posts and links as a list of maps
def readFeed() {
   def feed = new XmlSlurper().parse(url)
   def posts = []
   feed.channel.item.each {
      posts << [title:it.title.toString(), link:it.link.toString()]         
   }
   return posts
}

Now that we have removed both the data (like url) and the external interface interaction out, our script can be simpler:

def dougsFeed = RssFeed.get("Doug McClure BSM Blog")
if (dougsFeed.title.matches(".*BSM.*))
   println "found BSM post!: " + dougsFeed.title + " – " + dougsFeed.link
}

Or we can extend it to do the same for each RssFeed object we may have defined:

// iterate through each RssFeed object
RssFeed.each { blog ->
        def feed = blog.readFeed()
        if (feed.title.matches(".*BSM.*))
                println "found BSM post!: blog.name <a href=${blog.link}>${blog.link}</a>"
        }
}

Let’s move to the second part of the requirement: taking an action like sending an SNMP trap. Groovy can use any java library and it seems like there is an open source java library for pretty much anything you can think of!
We use SNMP4J in RapidInsight. The java library can be used directly in the groovy script, however SNMP is more complicated than RSS, hence figuring out how to use the library is difficult for a non java developer like myself. Here again, RapidInsight helps by providing infrastructure to make it easier to work with SNMP and hide the complexity even further.

SnmpTrap class can be used to store common data such where (ip/port) to send the traps, which snmp version, etc. SnmpTrap class has an operation called send() that takes name value pairs and sends the trap.  Usage is very easy:

def trap = SnmpTrap.get(name:"myTrapDestination")

def props = [:]
props.enterprise = ".1.3.6.1.4.1.88888.12"
props.generic = 6
props.specific = 1
props.varbinds = []
trap.send(props)

// or in a single line
// trap.send(enterprise:".1.3.6.1.4.1.88888.12",specific:1,generic:6,varbinds:[])

If I only have a few different kind of SNMP traps to send, like up, down etc., I can move the trap definition into the operations to make the script itself even simpler.

//operations
def sendDownTrap() {
send(enterprise:".1.3.6.1.4.1.88888.12",specific:1,generic:6,varbinds:[])
}
def sendUpTrap() {
send(enterprise:".1.3.6.1.4.1.88888.12",specific:2,generic:6,varbinds:[])
}

// script:
def trap = SnmpTrap.get(name:"myTrapDestination")
trap.sendDownTrap()

The functionality described in this post is not difficult to implement in perl etc. for a good developer; there are libraries available and the whole thing can be implemented in a single script.  RapidInsight provides functionality at every step to make not only implementation but maintenance and operations support easier.  Scripts can be executed from a web based UI, scheduled to run periodically, only available to subset of authorized users, logging, notifications, etc.  instead of collection of scripts running as cron jobs that can get out of control rapidly.

Viewing 7 Comments

    • ^
    • v
    This is very useful.

    I have a client working with a "developer" who was saying how hard this was to do.

    While not a developer myself i know the principles thanks for laying them out here.

    Now I just need to find the time to write a spec to get this working so i can automate capture of Twitter WordPress related plugins etc.

    Thanks for taking the time.

    • ^
    • v
    Berkay

    Thanks for taking the time to blog this one... pushing external content into event stream data is getting hot! I've been working with an ISP who's attempting to align external-corp contextual event data with performance data to feed a performance database.

    J3ffG1ll
    J3ffG1ll.typepad.com


    • ^
    • v
    Hi Jeff, thanks for the comment, glad to hear you've found it useful.

    It makes a lot of sense, events and performance are intertwined, and ops need to be able to correlate them at least visually hence the need to align the events and performance data as you've described.

    In monitoring systems, performance exceptions are typically sent as events via snmp traps etc to the event management systems, bringing events from any source into the mgmt systems is essential but not always sufficient.
    In addition to the consolidation of events, in RapidInsight, we've taken an approach that we describe as "integration in the presentation layer" http://www.ifountain.com/blog/...
    where we combine all types of information (events, graphs, etc.) from multiple systems using a common model and user interface. This allows users to access performance reports/graphs in context of the events and vice versa.


    • ^
    • v
    I found a few good screen captures on the ifountain link you provided, do you have more? imo the presentation layer is critical - i'm curious how rapidinsight has solved the problem beyond the typical gauge/pie chart/list solution.

    Jeff
    • ^
    • v
    I fully agree that presentation layer is critical.

    It sounds like you're looking for new UI components/paradigms. I think you're better of looking at a live system rather than screenshots. You can use the live demo on our website.
    http://www.ifountain.com/rapid...

    In RI, we mostly use typical components, grids, maps, pie charts, etc. key being presentation layer showing all the relevant information from disparate systems seamlessly. Another key point is the interaction between the components in a view. Components are dynamic, interaction in one component reflects on others,etc.

    We found these are key enabler techniques to make presentation layer effective.

    There are also some examples of different UI components. you can try
    - service view tab, click on a menu in service view and select Get Event History (uses timeline to show events)
    - select get Device locations to see google maps type integration.
    - data is brought from the server as user moves around in the list views (live grid concept) which is key for the UI to scale to large systems.
    - in search tabs, refinement of the search query as the user click on the results

    • ^
    • v
    Hi Doug :)

    What did you end up doing about this anyway?

    Just to be sure, in the approach above, the only prerequisite that needs to be installed on the server is Java VM. The rest are just jar files (for groovy and snmp) and scripts that can be unzipped to any directory and executed. I understand that RapidInsight would be an overkill just for something like this, but even RI is works by simply unzipping it to a directory.
    XMLSlurper is part of groovy so it's already there, SNMP part above uses some utility classes in RI so they would need to be extracted from there to be used standalone but it's doable.
    • ^
    • v
    Funny!

    Now the cool thing would be to get this as self contained as possible so it's something very easy to install and use! Tracking down libraries, RPMs, packages, etc. is a pain for some folks these days, especially for non *nix environments.

    Doug
    BSM/ITSM Blog: http://dougmcclure.net

Trackbacks

blog comments powered by Disqus