Impact of Infrastructure Automation Tools on Monitoring

2 Replies

IT management teams in charge of monitoring the IT infrastructure (network, servers, applications, etc.) mostly have little insight into what it is that they monitoring. (obvious or shocking?)

The tools we use clearly indicate this fact. Monitoring tools all have some sort of “discovery” functionality to figure out, what is out there to monitor. More often than not, when we set discovery loose on the network using (SNMP, ICMP, etc.), and it finds out devices and network connections customers did not know that existed.

Server/application monitoring tools start their cycle by scanning ports to see which ones would respond, or by sniffing the traffic to figure out which servers are out there, which applications may be running on which servers, etc.

The process would not be much different if you were attacking the infrastructure to find a way in. (How many of you triggered security alerts when performing discovery?) We’re outsiders. In enterprise environment, we often don’t even know the owners/developers of the applications.

In monitoring field, this has been the norm for so long that it no longer bother us. It should. Of course, monitoring teams & tools don’t do this for fun. It has to be done because in most cases, there is no truth teller; no place to get this kind of information. It is not uncommon for the monitoring tools to feed data they discover to inventory tools etc.

There are efforts like CMDB projects that attempt to create a repository that provides this information to all management tools but these projects often run into organizational as well as technical obstacles, and things are getting harder by the day with the dynamism introduced by virtualization and the cloud technologies.

What if we didn’t have to do all this crap to know what’s what? What if monitoring tools could be told which applications run on which server, where that server is in the network, etc. ?

There is indeed a better way, at least for some use cases. Proliferation of infrastructure automation tools (aka configuration management tools) such as Chef (and the management APIs exposed by VMWare, etc.) have the potential to change not only for how we deploy and maintain servers and applications but also how we monitor them.

Most obvious impact is that using these tools mean monitoring tools can have a reliable source to learn about the infrastructure that should be monitored. What the role of the servers are, how they are configured, which application components run on which server, what the change history is, etc. This is a huge step forward.

When you know how things should be, it’s much easier to detect the exceptions. A significant portion of the problems happen due to changes somewhere in the infrastructure. Ability to automate changes, see the change history and roll back when needed is an invaluable. And being able to correlate the configuration changes with the monitoring data can significantly reduce troubleshooting time and hence improve availability.

Another impact is that a safe framework that enables operations folks to take actions to troubleshoot and resolve problems (combined with run book automation, workflow, wiki, etc.) may finally mean that level 1/2 support folks can do more than record and route without giving them full access to the systems (which is not feasible), reducing number of problems escalated to higher levels and increasing overall productivity.

I should state just for the record that I don’t mean that infrastructure automation tools like Chef introduce brand new technology. Opsware (now HP), BladeLogic (now BMC), ConfigureSoft (now VMWare) for server configuration management, and TrueControl (now HP via Opsware), Voyence (now EMC), AlterPoint for network configuration management have been around for some time. But confluence of factors such as success of the (Apache licensed) open source model of Chef, and increasing acceptance of cloud economics, and patterns such as availability of open APIs move Chef into the center stage. It does not take great wisdom to infer that price point such as $50/month for 20 devices will make Chef very hard to ignore. Price is indeed a feature.

Looking forward to see how infrastructure tools like Chef will evolve as they move further into the enterprise world. Monitoring folks need to pay attention.

SaaS model in IT Operations Management – is it in our future?

Leave a reply

Is delivering IT operations management as “software as a service (SaaS)” a viable option?

I think it’s a question worth contemplating for anyone involved in IT Ops. Yes cloud hysteria is everywhere, and yes a lot of what’s going on is vendors rephrasing the same products with the latest buzzwords. Nonetheless, there are also signs of a major shift, that can potentially have a major impact on IT Operations Management.

There is no doubt that IT Ops will be drastically different going forward when organizations start using more and more “cloud” services but this is not the focus of this post. What I wanted to hash out is whether SaaS is a viable model for delivering IT management itself. And even going further, whether it will become the dominant model in not so distant future.

Let’s start with a look at the current state of IT Operations Management first. Is there actually a problem that needsa solution? I’m pretty sure we all agree that there is indeed a problem.

Most organizations are stuck in the muck

Currently implementing just the base solutions take so much time and effort that only few organizations have the means and the will to proceed any further. There has been little innovation in the field and even the ideas and technologies that have been around for many years don’t get applied.

For example, let’s think of what it takes to implement and maintain an event management solution in a large network.IBM’s Netcool suite is widely accepted as the defacto standard product for event management and has a very large user base. Yet the solution has many moving parts:

Probes
Tiered Object Servers for aggregation, presentation, etc.
Bi-directional gateways for replication,
Webtop, TIP, etc. to provide web based UI
Reporter and Oracle for reporting
…

At least a dozen application processes. Just installing the right versions of the included software, avoiding compatibility issue and integrating the components is a major undertaking, let alone mastering how to develop solutions using them. This is just to consolidate events in a single repository, nothing advanced at all.

As a result of this complexity, highly skilled resources get bogged down implementing & maintaining the base solution, struggling to find the bandwidth to implement features/techniques that would truly add value: enrichment, automation, correlation, visualization, service management etc.

It is also very costly and difficult to build sophisticated solutions on top of such complicated and hard to maintain foundation. Hence organizations find it hard to show ROI, and justify any further investment. Solutions at best stagnate where they are, performing the bare minimum, and at worst they degrade in time to become eventually unusable.

So how may SaaS help ?

SaaS is not a magic bullet. If someone took the same product suites and attempted to provide as a service they would have little to no chance to succeed. The dominant products from Big 4+ vendors are quite old and not designed for the “cloud”. But a solution that is designed from ground up with new constraints and opportunities of introduced by cloud and other modern technologies may have a significant impact.

What if event management was available as a SaaS offering? An event management solution that has high availability, no scalability limitations, modern web based UI with impressive visualization capabilities, correlation using complex event processing techniques, workflow, integrated reporting etc. ? Even more, what if they also offered a development platform for others to build solutions as well, similar to SalesForces’s Force.com? Would it not change the entire landscape?

Such an offering can potentially solve majority of the problems, Ops organizations currently struggle with (the muck), freeing up their resources to move up the chain and tackle more value add projects. It would also potentially provide substanstial savings, making it quite attractive to business.

SaaS as a model has moved to mainstream. It is no longer necessary to explain to people what it is, why and how it provides value. Although it may not have been embraced by everyone in the enterprise world, there are signs that it may even becoming the preferred approach for many organizations.

And SaaS offerings have come to IT management as well. Service-now.com ITSM service is a stellar example of the power and potential of SaaS in IT management. It has already changed the ITSM landscape, forcing established players to scramble to offer their own solutions as SaaS.There are also already number of SaaS offerings in the market typically targeting SMBs. Can an event management solution for the large enterprises be far behind?

No doubt there are obstacles, both technical and organizational, that may hinder adoption of SaaS in IT Operations. Most obvious ones seem to be security concerns and integration, but are these show stoppers or just issues that need to be worked out? These concerns are valid for any application and although they are source of concern, they do not seem to hinder adoption of SaaS in other areas. Is there something that makes IT Operations Management so unique that it can be immune to the SaaS tidal wave?

One thing is for certain that if SaaS gains traction in IT Ops, our lives will never be the same! I think it is time to assess what the implications of SaaS may be and figure out what we need to do to surf the wave rather than getting swept by it.

What do you think? I would love to hear what your thoughts and compare notes…

Vmware Springsource and Hyperic: Brave new world and a lot of questions

6 Replies

So Vmware is acquiring SpringSource. Like most people, my first reaction was say what? I didn’t get it; what’s VMWare got to do with SpringSource and Java ?
I read the blog posts by Steve Herrod and Rod Johnson. They were explaining their vision at a high level as expected, and it did not help much. I was still drawing a blank. I hit twitter, read some immediate reactions and subsequent blog posts, hoping they do the hard work and spell it out for me. No such luck. Comments were so diagonally opposed to one another that I’ve concluded it wasn’t just me who was confused. @daveofdoom thought Hyperic was a key tech, Reuven Cohen concluded that, Hyperic was unneeded by VMWare and would be shut down. Confused yet?

A sneaking suspicion has come over me: We don’t understand what these guys are really up to just yet, and they are up to something groundbreaking, that may shape the future.

No help from usually insightful sources, (amazing how easily I got used to readily available quality analysis for free!) I had to do some thinking and speculation myself. I reread the announcements and started thinking.

First, why am I interested? We use many of the technologies provided by these companies in some form. iFountain is a java shop, and use VMWare for development and testing. Our product RapidInsight is built on Grails and heavily uses Groovy, and has integration with Hyperic (one of the best server/application management solutions in the market, open source or otherwise.)

What does this acquisition mean to us?
1. VMWare Springsource Java stack combination.
As mentioned in Steve Herrod’s post, it no longer makes sense for the VMs to be ignorant of the applications running on them. VMs and the applications need to be introduced to each other. If they become good friends, and start talking and sharing, all kinds of cool things can happen.
In my experience, actual hardware and operating system has lost its importance a while ago. If you develop pure java applications, JVM becomes your environment. Why do we need another layer of abstraction, especially if it is virtual? To run a java application like RapidInsight in the cloud (EC2, etc.) or on VMWare hosted image, one has to create an OS image, configure it, make it secure, maintain/administer it, etc. just to host the JVM so that java app can run.
What if instead, VMWare could run JVMs natively just like it’s running linux or windows?

JVM running natively without an OS would presumably have a lot smaller footprint, and save resources. However, more substantial savings would probably come from eliminating administrative overhead of managing the OS.
From an ISV standpoint, this combination may enable ISVs to deploy applications in VMWare JVM ready format, to be installed and running within minutes.

2. Instrumentation of the applications and VMware
Hyperic is different than other solutions in the market it’s competing. It has an agent that is installed on the host and can gather in depth information that is essential for the “conversation” between the applications and VMware.
Using Hyperic, applications can be instrumented, giving in-depth information on how the application is performing, etc. Imagine VMWare assigning another CPU and more memory to the JVM on the fly automatically based on various performance parameters! (afaik neither is currently possible without a restart of the JVM). Now that would be something!

3. Development tools – cloud integration
We use various VMWare images for development. Having an IDE that is integrated with the cloud (VMWare) resources and enable running apps directly on the cloud by push of a button from IDE sounds good to me. If VMWare/Springsource continues to invest in having great support for Groovy and Grails in Eclipse, and integrate it with the cloud, it will be a great combination.

In short, I can see VMWare becoming THE platform for us to develop our products. All of a sudden, this is not such a crazy acquisition. Hats off to everyone involved, there is some great potential here!

From traditional IT management perspective, things are a bit blurry. Springsource acquisition of Hyperic hinted that Hyperic may become more of a specialized management vendor, focusing on java application management. With the acquisition of Springsource by VMWare, this has become a stronger possibility.

With this acquisition, Hyperic (product line) is moving into dominating the future of IT management with a vertically integrated solution. Does VMWare intend to compete in the traditional IT management market with the likes of Zenoss, Groundworks, Tivoli, HP, BMC, etc. as well, or will they gradually abandon this market? What is the future of Hyperic product line, if any, apart from being a key enabling technology for VMWare’s cloud platform?

A perfect day ..

Leave a reply

Yesterday was a perfect day. When one of these days come around, it’s important to take notice.
Weather was great in Geneva. A sunny spring day, with blue skies, yet not hot. We’ve started the day with long due yard work, and by noon having made good progress we’ve decided to have a picnic by the lake. The village we live in, Mies, has a public park by the lake only minutes from our house. It’s a perfect spot for a picnic, and we had it practically for ourselves.

I watched my mom, aka “Babanne” (means Grandmother in Turkish), and my 3 year old daughter, Maya, throw rocks into the lake. The joy radiating from their faces alone was enough to make my day.
As I sat there, watching still snow covered Alps across Lake Geneva and chatting with my wife about life, I felt as happy as a man can be. We’ve realized that I was with three most important females of my life in a somewhat unique time frame in our lives. We are all healthy which is so crucial yet we often don’t realize its importance till we’re not.
My mom is aging but she is still very active. Full of energy, more so than I am. Looking at her, I sometimes wonder how is it possible for me to be so lazy Our relationship is as strong as ever.
My daughter is 3, she is as sweet and as cute as a child can be. She’s growing fast, incredibly fast, makes me look for pause button, but none is available. My wife and I contemplate whether we’ll miss these days as she hit teen years and start hating us as many of our friends warn us that it will come to be
And my wife sitting next to me, lovely as ever. As we push towards 40 (6 months away for me), it doesn’t escape from us that we’re no longer young, especially when we meet people who weren’t even born when we were in college. Yet we also know that we’re not “old” either. We’re healthy and nothing is holding us back.

So a perfect day it was. I’ve tried to hang on to it as much as I could, and pushed thoughts of what future may have in store for us as far as possible. Next sixth months will have a lot of turmoil for us, no sense in worrying about it all that in such a bright day ..

Getting Groovy, Processing XML, sending SNMP traps and RapidInsight

7 Replies

A while ago Doug asked whether anyone knew a utility to read XML and send SNMP traps. I mentioned that it would be easy to do with Groovy but did not have time to give any details. I’ve been meaning to write a post about how we use Groovy in RapidInsight and thought this would be a good excercise.

First I have state once again that I’m not a developer nor play one on Youtube. I can however, put together scripts, especially if there are examples, but writing java code is not my cup of tea. Groovy makes this stuff easy enough to deal with for people like me (system integrators, admins, etc.)

In this example, we’ll read the RSS feed (which is XML) from Doug’s blog and take action if the feed title includes the word BSM in it Groovy includes powerful XML utility called xmlslurper that makes reading XML a breeze.

[code lang="java"]
def url = "http://dougmcclure.net/blog/feed/"
def feed = new XmlSlurper().parse(url)
feed.channel.item.each {post ->
if (post.title.toString().matches(".*BSM.*")) {
println "found BSM post!: " + post.title + " - " post.link.toString()
}
}
[/code]

There it is, we already have a Groovy script to retrieve the RSS feed, iterate through the posts and determine whether the title includes BSM keyword. One can take a look at how xmlSlurper works and use it to get the information she needs.

RapidInsight makes it even easier for the script developers to work with the external interfaces without having to learn the intricacies of each interface. In RapidInsight, we can create a class called say RssFeed, store data related to the RSS feed as object properties.Here is how our class may look like:

[code lang="xml"]

[/code]

The class can be used to store the information related to the Rss feed as objects. We can create a UI for the user to enter this information, or simply use a script to create the instances. Creating an object in RapidInsight repository is straight forward, pass the properties as name value pairs to add operation:
[code lang="java"]
def props = [:]
props.name = "Doug McClure BSM Blog"
props.url = "http://dougmcclure.net/blog/feed/"
props.type = "Rss 2.0"
RssFeed.add[props]
// I could have written this as a single line as well
// RssFeed.add[url:"http://dougmcclure.net/blog/feed/", type = "Rss 2.0"]
[/code]

By storing the url, etc. as object properties, we can now have a more generic script that would work with any RSS feed. Next we can add “operations” to this class to deal with the Rss feeds, hiding the complexities of the external interface from the script developer. An operation can read the feed and return name value pairs in a map to the user.

This means the script developer does not have to understand the structure of the RSS or how to use XmlSlurper. The operation code can be modified to add support for different RSS formats (v1, v2), use ROME library to parse RSS instead of XmlSlurper, etc., scripts would continue to work as before and script developers like me would not have to learn something different. All it matters is what is passed to us, which is a List of Map (name value pairs), and this is same for all external interfaces. (Take a look a this post to see more on how RapidInsight uses name value pairs when working with external interfaces)

The operation may be something like this:
[code lang="java"]
//readFeed operation returns blog posts and links as a list of maps
def readFeed() {
def feed = new XmlSlurper().parse(url)
def posts = []
feed.channel.item.each {
posts << [title:it.title.toString(), link:it.link.toString()]
}
return posts
}
[/code]

Now that we have removed both the data (like url) and the external interface interaction out, our script can be simpler:

[code lang="java"]
def dougsFeed = RssFeed.get("Doug McClure BSM Blog")
if (dougsFeed.title.matches(".*BSM.*))
println "found BSM post!: " + dougsFeed.title + " - " + dougsFeed.link
}
[/code]
Or we can extend it to do the same for each RssFeed object we may have defined:
[code lang="java"]
// iterate through each RssFeed object
RssFeed.each { blog ->
def feed = blog.readFeed()
if (feed.title.matches(".*BSM.*))
println "found BSM post!: blog.name ${blog.link}"
}
}
[/code]

Let’s move to the second part of the requirement: taking an action like sending an SNMP trap. Groovy can use any java library and it seems like there is an open source java library for pretty much anything you can think of!
We use SNMP4J in RapidInsight. The java library can be used directly in the groovy script, however SNMP is more complicated than RSS, hence figuring out how to use the library is difficult for a non java developer like myself. Here again, RapidInsight helps by providing infrastructure to make it easier to work with SNMP and hide the complexity even further.

SnmpTrap class can be used to store common data such where (ip/port) to send the traps, which snmp version, etc. SnmpTrap class has an operation called send() that takes name value pairs and sends the trap. Usage is very easy:

[code lang="java"]
def trap = SnmpTrap.get(name:"myTrapDestination")

def props = [:]
props.enterprise = ".1.3.6.1.4.1.88888.12"
props.generic = 6
props.specific = 1
props.varbinds = []
trap.send(props)

// or in a single line
// trap.send(enterprise:".1.3.6.1.4.1.88888.12",specific:1,generic:6,varbinds:[])
[/code]

If I only have a few different kind of SNMP traps to send, like up, down etc., I can move the trap definition into the operations to make the script itself even simpler.

[code lang="java"]

//operations
def sendDownTrap() {
send(enterprise:".1.3.6.1.4.1.88888.12",specific:1,generic:6,varbinds:[])
}
def sendUpTrap() {
send(enterprise:".1.3.6.1.4.1.88888.12",specific:2,generic:6,varbinds:[])
}

// script:
def trap = SnmpTrap.get(name:"myTrapDestination")
trap.sendDownTrap()
[/code]

The functionality described in this post is not difficult to implement in perl etc. for a good developer; there are libraries available and the whole thing can be implemented in a single script. RapidInsight provides functionality at every step to make not only implementation but maintenance and operations support easier. Scripts can be executed from a web based UI, scheduled to run periodically, only available to subset of authorized users, logging, notifications, etc. instead of collection of scripts running as cron jobs that can get out of control rapidly.

Service Management and the Cloud Computing: World’s are colliding! George is getting upset!

Leave a reply

Are Service Management and Cloud Computing mating? Indeed they are, and it should not be surprising. In the previous post, I had argued that, putting the latest industrial jargon aside, service management is the essential enabling paradigm in today’s IT organizations.

Each entity in the IT organization to define the services they deliver, measure availability, performance and quality of these services, and expose this information in both human and machine consumable forms.

How does cloud computing fit in to this? Cloud computing providers are also service providers, whether these services are infrastructure or applications or anything in between. Higher level business services consist of combination of services provided by external and internal service providers.

Cloud computing providers are typically external. The services they provide are more easily recognizable, and since they are external providers, the need to monitor and report on services they provide is more apparent and acceptable. As a result, adoption of cloud computing will enforce the need for service management and service management will smoothen the path to cloud computing in the enterprise.

In the last post I’d argued that the instead of wholesale approach, letting each entity to use whatever tools they want but expose the service management information through standard, agreed on interfaces is more likely to succeed in BSM projects. Cloud computing enforces this notion of Service Management and APIs.

A typical, relatively sophisticated business service has number of components managed by different IT groups. Even when all the components (infrastructure, apps, etc.) of the service are provided by entities internal to the organization, it’s proven to be quite difficult to implement a solution that can monitor all the components and assess the impact of problems on the business service.

What happens when some or most the components rely on services provided by external providers? It becomes even more unrealistic to presume that you can monitor them all. Instead, it is more feasible to require service monitoring information to be exposed via APIs so that they can be consumed by the BSM application.

Obviously this is not a new notion, it’s software development 101, but it applies to operations as well, and to move into service management paradigm where services can be black boxes, having access to information via established APIs become essential.

Monitoring the cloud must focus on enabling management of the services provided by the cloud as part of higher level services. Traditional monitoring methods will not work. Tell the customers how their services are performing, not just with pretty charts but also through APIs so that services from the cloud can be managed like services provided internally.

Service Management BSM and APIs

4 Replies

Service management is the essential enabling paradigm for today’s IT organizations. Putting the tools and the jargon aside, every organizational unit needs to identify the following:

what are the services they deliver
who their customers are (internal or external)
who their (service) providers are
what internal/external dependencies the services have
how they can measure the availability, performance and quality of the services they provide
expose the service information both in human and machine consumable formats

Fundamentally, that’s all there is to it. The rest is all about how to get there.

“Business Service Management is an enabling technology at the intersection of business and IT alignment”. The final layer that provides the interface to business in terms that is meaningful to the business.

BSM implementers face a tough challenge. On one hand, BSM solutions are required to have a simple and abstract facade that hides complexities of the IT infrastructure and provide information to business users in their own language. Yet on the other hand, to be more than a pretty face with no brains, BSM is required to be very sophisticated under the hood in order to be able to model, discover and instrument services and the dependent IT infrastructure components.

In many implementations, modeling and instrumenting this complex environment (referred as BSM Heavy approach) forces BSM projects to become very large and expensive projects where IT silos are often forced to replace tools they are using, in addition to going through a religious conversion. As a result this approach has a low success rate.

An alternative approach is to go around the IT silos by implementing tools in parallel for small number of critical business services vertically. This approach (referred as BSM Light) attempts to avoid organizational obstacles and can be implemented in shorter time frames, making it possible to demonstrate progress quicker.

I’ve always thought that success of BSM projects depends more on how well organizational challenges are handled than how good the tools are from a technical stand point. But what type of organizational change the tools dictate is something beyond technical abilities of the tools. The tools that enable companies to implement solutions without dramatic organizational shifts as a prerequisite have a much better chance to actually achieve the organizational change.

An alternative approach recently articulated by Robin Harwani has great potential to minimize some of the organizational obstacles.

Instead of forcing IT silos to replace tools or implementing tools in parallel, BSM projects can define interfaces (APIs) for the management information to be exposed. Various IT departments can use their own tools as long as they expose the information using defined interfaces enabling BSM tools to focus on using the provided information to model business services in a higher level. ( There is another three letter acronym I’m avoiding here as the term introduces more questions than it answers).

The most important advantage of this approach is that it minimizes the required organizational change and participation/buy-in. It provides IT groups freedom to use whatever tools they see fit as long as they embrace the service management paradigm described above and expose management information through agreed upon interfaces to be used by BSM systems.

This approach can reduce the scope of the BSM project, divide the work into more manageable chunk and increase likelyhood of a successful completion.

Final thought: Change cannot be a prerequisite for the success of a project. The project teams must consider instigating the change and removing organizational resistance as a key constraint that has to be addressed. The processes, the tools, the players all have a role to play on this.

Laptop reborn with new hard drive and Windows 7

Leave a reply

I have been having problems with the Dell Inspiron 1505 laptop for a while. Hard drive has been giving signals of a failure, and OS had become so cluttered with numerous applications I have tried last 2 years. Inexplicable hard drive trashing despite 2GB RAM was really getting to nerves and slowing me down.

A reinstall of OS and essential apps would have solved some of the problems but the initial install was painful due to number of drivers that had to be added etc. The laptop comes with loads of junk/trial software and a clean install was unexpectedly difficult. I couldn’t risk any downtime during reinstall so I kept postponing it.

At the end, I ordered a new, faster (7200rpm) hard drive and decided to try Windows7 beta with it, fully expecting to revert back to XP. Happy to say Windows 7 exceeded expectations and was much better at recognizing all the hardware. In addition, with XP I had to use VMWare images for number of applications that required Windows Server 2003, since they did not work with XP. With Windows 7, they Windows 2003 compatibility mode solved this problem.

My definition of a good OS have somewhat changed throught the years. I want an OS that handles the hardware seamlessly and won’t do much more than that. Windows 7 does handle the hardware nicely. The search interface to launch apps is a winner! within minutes I got used to doing everything from going into folders to running apps without touching the mouse using the search box. It’s really well done.

Security is a pain in the ass! It’s going to take me a while to learn how to disable various automatic stuff, firewall configs, etc. I can’t imagine a non-technical user dealing with this stuff. It’s crazy complex. I can’t imagine helping someone over the phone, etc. to get software installed. There are so many things where things can go wrong in the name of security. I am an administrator and I’m still getting “you don’t have sufficient access rights” messages. WTF? If I don’t have the rights, who does, NSA?

When I started installing the applications I use, I’ve noticed that majority of them are free applications, if not open source. I have mixed feelings about this. Does this mean that desktop software market has essentially disappeared and not possible to charge for software anymore? Apparently I’m not paying for them. The least I can do is to list and thank the creators. Here is the list in no specific order:

Firefox Despite continuous annoyance with memory usage, browser of choice. I use it with delicious, firebug, greasemonkey, read it later, add-ons. Eliminated a lot of add-ons, some of them will likely make it back in time as I need them. Haven’t decided what to use for tabs yet for example.
Sun JDK So easy to take it granted. Java VM is such an invaluable technology. Thanks Sun!
OpenOffice Encore, kudos to Sun. It’s amazing that we can get this for free.
Skype I live in Skype, use it 24/7 both in professional and personal life. Who would have thought that we can have video conferencing for free.
Putty much more than an SSH client. What would I do without tunneling getting me though security hoops?
VLC Media Player It just works, doesn’t install bunch of crap on your PC.
WinSCP scp and ftp. Like putty, it’s an essential tool I would be hard pressed without it.
Digsby my current/recent multi IM/social networking tool.
Crimson Editor My text editor for quite some time. works very well. thanks!
JungleDisk I’m a paying customer using it to backup files into Amazon S3.
Mozy Backup solution for my PC, works well enough.
Synergy What a gem! Use it to control other PCs in the office from a single keyboard/mouse.
DynDns another free service/software, essential to work with others when you don’t have a static IP.
iReasoning Mib Browser these guys have great SNMP libraries. They make the mib browser available as a free tool. Works well.
Groovy my favority scripting language
Active Perl Perl is still must have in my work. Active Perl distribution made installing Perl on Windows a pleasant experience.
Picasa I still prefer desktop photo organizers. Picasa from Google is pleasant to use.
RealVNC VNC is part of the essential tool in the arsenal of a remote worker.
7zip works with different compression file formats nicely. another essential tool.
Adobe AIR, Acrobat Reader & Flash Player Adobe is yet another great tech company providing number of great technologies over the years that have become defacto must have tools.
Jing Project After using the free version for a while, I’ve become a paying customer when they offered the Pro version. Use it all the time for screenshots and screencasts.
VMware Player. Another essential tool for development, support, testing, etc. In addition, it has become a key enabler for me to create clean environments to connect to different client networks.

So there it is. Overall, it feels like I got a new laptop since everything changed and works faster. Now I need to figure out how I can save this state of the laptop so that I can come back to this clean state if necessary. Hoping that Retrospect, the backup software that came with the external USB drive will help with that.

Open and transparent (support) beats closed anytime

Leave a reply

Have you ever spend hours/days jumping through the hoops, painstakingly collecting data on a problem and find out that the others had the problem and done all this before? If only you could have had access to this hugely valuable information base, you’d save hours/days and minimize down time?

Quality of support has long been used against open source software, suggesting that enterprises need an entity to deal with that is accountable to them. Fair enough, even though the mailing lists can be more responsive than some support organizations, enterprises require more assurance. This issue is largely resolved since there are now open source companies providing support for most open source products. The fact that this is still being tossed around as a concern is annoying to say the least but c’est la vie.

What irritates me more is that how inferior proprietary vendor support systems can be due to specifically their closed nature. Most bugs/problems are experienced by multiple users/customers and problems that are difficult to isolate can take a lot of time and effort.

Support information (generated by customers) is mostly locked inside enterprise systems, and not visible/searchable by the public or even the customers. Correlation of the problems reported by the users, their troubleshooting journey, the problems encountered on the way, gotchas, etc. all stay hidden from rest of the people.

In my experience, most people prefer using self help tools and research themselves before calling support unless there is an emergency. If the customers could search/access to this information, not only the customers would save time and may be reduce down time, but the vendor would also benefit from reduced support costs since fewer support calls would come in. Even further, customers may even be able to avoid problems altogether by learning from experiences of others. So why is this information not accessible, available at least by the customers?

tools may be inadequate. The support systems may not naturally facilitate this type of sharing and collaboration
vendors may not want this information out in the open for “competitive reasons”, keep dirty laundry hidden so to speak
support information may include sensitive information that customer may not want to be public
others.. ?

Open source products by their nature are more transparent, the problems are reported and discussed out in the open in mailing lists, bug tracking systems, forums, etc. hence Google is ready to help you find the answers. Vendors -open source or otherwise- should make the effort to make the support process as transparent as possible, the cost of keeping this information locked inside isolated systems is just too high. Open source companies have an opportunity here to differentitate themselves in yet another way that directly impacts the customer experience.

Integrating IT management products, proprietary vs open source and APIs

2 Replies

I recently had an interesting discussion with a colleague on pros and cons of proprietary vs open source software and access to code vs APIs. We’ve done a lot of work on integrating our products with proprietary products in the past and recently have been working on integrating RapidInsight with open source products like OpenNMS and Hyperic, and have been comparing pros and cons of both worlds.

Integration in proprietary world is painful experience in most cases. It is hard to get a hold of the software, often not even possible. There are all kinds of restrictions on what you can and cannot do with the software that prevents high quality integration, contracts need to be signed, and small companies like us are often ignored.
It does not need to be this way. Proprietary software can have open APIs and vendors can establish a process that can be used by anyone, much like it’s been done by the web companies (Amazon, Google, etc.), but it’s rarely done this way.

8 years ago, working for a consulting company, I had given a presentation in the company about the developments in the web world, web services APIs, its impact on IT management, and systems integration. The company had significant revenue stream implementing IT management solutions by integration a set of IT management software products, usual suspects like Netcool, Remedy, Smarts, Nervecenter, Infovista, Patrol etc. and most of us were specialists on one or more of these products. I had suggested that with the spread of common/standard integration APIs (SOAP, XMLRPC), nature of integration projects would shift from point to point integration into use of the standard APIs, and generic enterprise integration tools. If so, the demand for the skillsets would also shift, requiring more software development skills than systems integration skills, in depth expertise and understanding of inner workings of the IT management products.

Needless to say, it’s an understatement to say that I was wrong! In the last 8 years, very little has changed in the IT management world. Ad-hoc, point to point integration is still the mainstream, and often the only option. Consultants do just fine with the same skills they had a decade ago and IT management projects still have very high costs, implementation (integration) costs often well exceeding the licensing costs. I’ve seen may RFP, product comparision matrixes, etc. over the years and support for standard APIs is often just a check box. Massive cost impact of not having standard APIs are mostly missed, and as long as there is no strong demand from the customers, vendors will not move.

Some of the proprietary software products do have APIs and if you can get access to the software (for us typically on a client site who wants us to integrate with) you can use the API to develop descent integration. But there are still risks for the customer if the API is not available publicly available. If the API is not embraced and widely used, the vendor has no qualms about changing it liberally from version to version, so upgrades become a much riskier proposition. So how widely the API is used is crucial for the customers. When an API reaches high level of use, organizations become more conservative in changing them and think twice before breaking more compatibility.

In the open source world, it is assumed that integration is easier since the code is available. And it is indeed easier. To start with, you don’t have to deal with the commercial/legal contract nonsense to implement the the integration. I can tell you this phase takes longer and is more frustrating than writing the code.
Since the code is open, you can look at the code, learn from it, modify it if necessary, etc. You don’t necessarily need the help from a company, etc. We’ve recently integrated RapidInsight with OpenNMS and Hyperic with only using publicly available resources, forums, mailing lists, etc.

Yet source code being open is not a replacement for a well documented API. There is no guarantee that the data structures, methods, etc. will not change from version to version. After all, an API is a software contract, a commitment to the outside world and implies stability, but internal code and data structures do not. So integrating open source products without standard APIs is carrying some of the same risks of integrating proprietary products.

The good news is that since open source products are developed transparently, one can get a heads up, adjust the integration for the new version, provide feedback during development to ensure backwards compatibility if it is possible, and even better participate in development of the standard APIs.

Rapidinsight OpenNMS integration gets data directly from the underlying postgres database and uses java methods for graphs. Fortunately, OpenNMS has a large community and evolves slowly, hence integration should be safe for some time. In the long term, implementing an API on the OpenNMS side to expose the information would be the better solution. If OpenNMS community sees value in this integration, we would certainly look at contributing at that end.

For Hyperic integration, we’ve used some groovy classes (we love groovy! ), implemented a Hyperic plugin to expose the information we need, and used some UI methods to access to the graphs. We’ve already noticed that something has changed with v4.0 and we’ll need to adjust the integration accordingly to make the integration work with the new version. The good news is they seem to be working on implementing a web services API which would make our lives so much easier!

Wouldn’t it be great if there was a site like programmable web for IT management software