Chronicles of a Wandering Mind

I recently had an interesting discussion with a colleague on pros and cons of proprietary vs open source software and access to code vs APIs. We’ve done a lot of work on integrating our products with proprietary products in the past and recently have been working on integrating RapidInsight with open source products like OpenNMS and Hyperic, and have been comparing pros and cons of both worlds.

Integration in proprietary world is painful experience in most cases. It is hard to get a hold of the software, often not even possible. There are all kinds of restrictions on what you can and cannot do with the software that prevents high quality integration, contracts need to be signed, and small companies like us are often ignored.
It does not need to be this way. Proprietary software can have open APIs and vendors can establish a process that can be used by anyone, much like it’s been done by the web companies (Amazon, Google, etc.), but it’s rarely done this way.

8 years ago, working for a consulting company, I had given a presentation in the company about the developments in the web world, web services APIs, its impact on IT management, and systems integration. The company had significant revenue stream implementing IT management solutions by integration a set of IT management software products, usual suspects like Netcool, Remedy, Smarts, Nervecenter, Infovista, Patrol etc. and most of us were specialists on one or more of these products. I had suggested that with the spread of common/standard integration APIs (SOAP, XMLRPC), nature of integration projects would shift from point to point integration into use of the standard APIs, and generic enterprise integration tools. If so, the demand for the skillsets would also shift, requiring more software development skills than systems integration skills, in depth expertise and understanding of inner workings of the IT management products.

Needless to say, it’s an understatement to say that I was wrong! In the last 8 years, very little has changed in the IT management world. Ad-hoc, point to point integration is still the mainstream, and often the only option. Consultants do just fine with the same skills they had a decade ago and IT management projects still have very high costs, implementation (integration) costs often well exceeding the licensing costs. I’ve seen may RFP, product comparision matrixes, etc. over the years and support for standard APIs is often just a check box. Massive cost impact of not having standard APIs are mostly missed, and as long as there is no strong demand from the customers, vendors will not move.

Some of the proprietary software products do have APIs and if you can get access to the software (for us typically on a client site who wants us to integrate with) you can use the API to develop descent integration. But there are still risks for the customer if the API is not available publicly available. If the API is not embraced and widely used, the vendor has no qualms about changing it liberally from version to version, so upgrades become a much riskier proposition. So how widely the API is used is crucial for the customers. When an API reaches high level of use, organizations become more conservative in changing them and think twice before breaking more compatibility.

In the open source world, it is assumed that integration is easier since the code is available. And it is indeed easier. To start with, you don’t have to deal with the commercial/legal contract nonsense to implement the the integration. I can tell you this phase takes longer and is more frustrating than writing the code.
Since the code is open, you can look at the code, learn from it, modify it if necessary, etc. You don’t necessarily need the help from a company, etc.  We’ve recently integrated RapidInsight with OpenNMS and Hyperic with only using publicly available resources, forums, mailing lists, etc.

Yet source code being open is not a replacement for a well documented API. There is no guarantee that the data structures, methods, etc. will not change from version to version. After all, an API is a software contract, a commitment to the outside world and implies stability, but internal code and data structures do not. So integrating open source products without standard APIs is carrying some of the same risks of integrating proprietary products.

The good news is that since open source products are developed transparently, one can get a heads up, adjust the integration for the new version, provide feedback during development to ensure backwards compatibility if it is possible, and even better participate in development of the standard APIs.

Rapidinsight OpenNMS integration gets data directly from the underlying postgres database and uses java methods for graphs. Fortunately, OpenNMS has a large community and evolves slowly, hence integration should be safe for some time. In the long term, implementing an API on the OpenNMS side to expose the information would be the better solution. If OpenNMS community sees value in this integration, we would certainly look at contributing at that end.

For Hyperic integration, we’ve used some groovy classes (we love groovy! ), implemented a Hyperic plugin to expose the information we need, and used some UI methods to access to the graphs. We’ve already noticed that something has changed with v4.0 and we’ll need to adjust the integration accordingly to make the integration work with the new version. The good news is they seem to be working on implementing a web services API which would make our lives so much easier!

Wouldn’t it be great if there was a site like programmable web for IT management software