IT management teams in charge of monitoring the IT infrastructure (network, servers, applications, etc.) mostly have little insight into what it is that they monitoring. (obvious or shocking?)

The tools we use clearly indicate this fact. Monitoring tools all have some sort of “discovery” functionality to figure out, what is out there to monitor. More often than not, when we set discovery loose on the network using (SNMP, ICMP, etc.), and it finds out devices and network connections customers did not know that existed.

Server/application monitoring tools start their cycle by scanning ports to see which ones would respond, or by sniffing the traffic to figure out which servers are out there, which applications may be running on which servers, etc.

The process would not be much different if you were attacking the infrastructure to find a way in. (How many of you triggered security alerts when performing discovery?) We’re outsiders. In enterprise environment, we often don’t even know the owners/developers of the applications.

In monitoring field, this has been the norm for so long that it no longer bother us. It should. Of course, monitoring teams & tools don’t do this for fun. It has to be done because in most cases, there is no truth teller; no place to get this kind of information. It is not uncommon for the monitoring tools to feed data they discover to inventory tools etc.

There are efforts like CMDB projects that attempt to create a repository that provides this information to all management tools but these projects often run into organizational as well as technical obstacles, and things are getting harder by the day with the dynamism introduced by virtualization and the cloud technologies.

What if we didn’t have to do all this crap to know what’s what? What if monitoring tools could be told which applications run on which server, where that server is in the network, etc. ?

There is indeed a better way, at least for some use cases. Proliferation of infrastructure automation tools (aka configuration management tools) such as Chef (and the management APIs exposed by VMWare, etc.) have the potential to change not only for how we deploy and maintain servers and applications but also how we monitor them.

Most obvious impact is that using these tools mean monitoring tools can have a reliable source to learn about the infrastructure that should be monitored. What the role of the servers are, how they are configured, which application components run on which server, what the change history is, etc. This is a huge step forward.

When you know how things should be, it’s much easier to detect the exceptions. A significant portion of the problems happen due to changes somewhere in the infrastructure. Ability to automate changes, see the change history and roll back when needed is an invaluable. And being able to correlate the configuration changes with the monitoring data can significantly reduce troubleshooting time and hence improve availability.

Another impact is that a safe framework that enables operations folks to take actions to troubleshoot and resolve problems (combined with run book automation, workflow, wiki, etc.) may finally mean that level 1/2 support folks can do more than record and route without giving them full access to the systems (which is not feasible), reducing number of problems escalated to higher levels and increasing overall productivity.

I should state just for the record that I don’t mean that infrastructure automation tools like Chef introduce brand new technology. Opsware (now HP), BladeLogic (now BMC), ConfigureSoft (now VMWare) for server configuration management, and TrueControl (now HP via Opsware), Voyence (now EMC), AlterPoint for network configuration management have been around for some time. But confluence of factors such as success of the (Apache licensed) open source model of Chef, and increasing acceptance of cloud economics, and patterns such as availability of open APIs move Chef into the center stage. It does not take great wisdom to infer that price point such as $50/month for 20 devices will make Chef very hard to ignore. Price is indeed a feature.

Looking forward to see how infrastructure tools like Chef will evolve as they move further into the enterprise world. Monitoring folks need to pay attention.

Is delivering IT operations management as “software as a service (SaaS)” a viable option?

I think it’s a question worth contemplating for anyone involved in IT Ops. Yes cloud hysteria is everywhere, and yes a lot of what’s going on is vendors rephrasing the same products with the latest buzzwords. Nonetheless, there are also signs of a major shift, that can potentially have a major impact on IT Operations Management.

There is no doubt that IT Ops will be drastically different going forward when organizations start using more and more “cloud” services but this is not the focus of this post. What I wanted to hash out is whether SaaS is a viable model for delivering IT management itself.  And even going further, whether it will become the dominant model in not so distant future.

Let’s start with a look at the current state of IT Operations Management first. Is there actually a problem that needsa solution? I’m pretty sure we all agree that there is indeed a problem.

Most organizations are stuck in the muck

Currently implementing just the base solutions take so much time and effort that only few organizations have the means and the will to proceed any further. There has been little innovation in the field and even the ideas and technologies that have been around for many years don’t get applied.

For example, let’s think of what it takes to implement and maintain an event management solution in a large network.IBM’s Netcool suite is widely accepted as the defacto standard product for event management and has a very large user base. Yet the solution has many moving parts:

• Probes
• Tiered Object Servers for aggregation, presentation, etc.
• Bi-directional gateways for replication,
• Webtop, TIP, etc. to provide web based UI
• Reporter and Oracle for reporting
• …

At least a dozen application processes. Just installing the right versions of the included software, avoiding compatibility issue and integrating the components is a major undertaking, let alone mastering how to develop solutions using them. This is just to consolidate events in a single repository, nothing advanced at all.

As a result of this complexity, highly skilled resources get bogged down implementing & maintaining the base solution, struggling to find the bandwidth to implement features/techniques that would truly add value: enrichment, automation, correlation, visualization, service management etc.

It is also very costly and difficult to build sophisticated solutions on top of such complicated and hard to maintain foundation. Hence organizations find it hard to show ROI, and justify any further investment. Solutions at best stagnate where they are, performing the bare minimum, and at worst they degrade in time to become eventually unusable.

So how may SaaS help ?

SaaS is not a magic bullet. If someone took the same product suites and attempted to provide as a service they would have little to no chance to succeed. The dominant products from Big 4+ vendors are quite old and not designed for the “cloud”. But a solution that is designed from ground up with new constraints and opportunities of introduced by cloud and other modern technologies may have a significant impact.

What if event management was available as a SaaS offering? An event management solution that has high availability, no scalability limitations, modern web based UI with impressive visualization capabilities, correlation using complex event processing techniques, workflow, integrated reporting etc. ? Even more, what if they also offered a development platform for others to build solutions as well, similar to SalesForces’s Force.com? Would it not change the entire landscape?

Such an offering can potentially solve majority of the problems, Ops organizations currently struggle with (the muck), freeing up their resources to move up the chain and tackle more value add projects. It would also potentially provide substanstial savings, making it quite attractive to business.

SaaS as a model has moved to mainstream. It is no longer necessary to explain to people what it is, why and how it provides value. Although it may not have been embraced by everyone in the enterprise world, there are signs that it may even becoming the preferred approach for many organizations.

And SaaS offerings have come to IT management as well. Service-now.com ITSM service is a stellar example of the power and potential of SaaS in IT management. It has already changed the ITSM landscape, forcing established players to scramble to offer their own solutions as SaaS.There are also already number of SaaS offerings in the market typically targeting SMBs. Can an event management solution for the large enterprises be far behind?

No doubt there are obstacles, both technical and organizational, that may hinder adoption of SaaS in IT Operations. Most obvious ones seem to be security concerns and integration, but are these show stoppers or just issues that need to be worked out? These concerns are valid for any application and although they are source of concern, they do not seem to hinder adoption of SaaS in other areas. Is there something that makes IT Operations Management so unique that it can be immune to the SaaS tidal wave?

One thing is for certain that if SaaS gains traction in IT Ops, our lives will never be the same! I think it is time to assess what the implications of SaaS may be and figure out what we need to do to surf the wave rather than getting swept by it.

What do you think? I would love to hear what your thoughts and compare notes…

A sneaking suspicion has come over me: We don’t understand what these guys are really up to just yet, and they are up to something groundbreaking, that may shape the future.

No help from usually insightful sources, (amazing how easily I got used to readily available quality analysis for free!) I had to do some thinking and speculation myself. I reread the announcements and started thinking.

First, why am I interested? We use many of the technologies provided by these companies in some form. iFountain is a java shop, and use VMWare for development and testing. Our product RapidInsight is built on Grails and heavily uses Groovy, and has integration with Hyperic (one of the best server/application management solutions in the market, open source or otherwise.)

What does this acquisition mean to us?
1. VMWare Springsource Java stack combination.
As mentioned in Steve Herrod’s post, it no longer makes sense for the VMs to be ignorant of the applications running on them. VMs and the applications need to be introduced to each other. If they become good friends, and start talking and sharing, all kinds of cool things can happen.
In my experience, actual hardware and operating system has lost its importance a while ago. If you develop pure java applications, JVM becomes your environment. Why do we need another layer of abstraction, especially if it is virtual? To run a java application like RapidInsight in the cloud (EC2, etc.) or on VMWare hosted image, one has to create an OS image, configure it, make it secure, maintain/administer it, etc. just to host the JVM so that java app can run.
What if instead, VMWare could run JVMs natively just like it’s running linux or windows?

JVM running natively without an OS would presumably have a lot smaller footprint, and save resources. However, more substantial savings would probably come from eliminating administrative overhead of managing the OS.
From an ISV standpoint, this combination may enable ISVs to deploy applications in VMWare JVM ready format, to be installed and running within minutes.

2. Instrumentation of the applications and VMware
Hyperic is different than other solutions in the market it’s competing. It has an agent that is installed on the host and can gather in depth information that is essential for the “conversation” between the applications and VMware.
Using Hyperic, applications can be instrumented, giving in-depth information on how the application is performing, etc. Imagine VMWare assigning another CPU and more memory to the JVM on the fly automatically based on various performance parameters! (afaik neither is currently possible without a restart of the JVM). Now that would be something!

3. Development tools – cloud integration
We use various VMWare images for development. Having an IDE that is integrated with the cloud (VMWare) resources and enable running apps directly on the cloud by push of a button from IDE sounds good to me. If VMWare/Springsource continues to invest in having great support for Groovy and Grails in Eclipse, and integrate it with the cloud, it will be a great combination.

In short, I can see VMWare becoming THE platform for us to develop our products. All of a sudden, this is not such a crazy acquisition. Hats off to everyone involved, there is some great potential here!

From traditional IT management perspective, things are a bit blurry. Springsource acquisition of Hyperic hinted that Hyperic may become more of a specialized management vendor, focusing on java application management. With the acquisition of Springsource by VMWare, this has become a stronger possibility.

With this acquisition, Hyperic (product line) is moving into dominating the future of IT management with a vertically integrated solution. Does VMWare intend to compete in the traditional IT management market with the likes of Zenoss, Groundworks, Tivoli, HP, BMC, etc. as well, or will they gradually abandon this market? What is the future of Hyperic product line, if any, apart from being a key enabling technology for VMWare’s cloud platform?

First I have state once again that I’m not a developer nor play one on Youtube. I can however, put together scripts, especially if there are examples, but writing java code is not my cup of tea. Groovy makes this stuff easy enough to deal with for people like me (system integrators, admins, etc.)

In this example, we’ll read the RSS feed (which is XML) from Doug’s blog and take action if the feed title includes the word BSM in it Groovy includes powerful XML utility called xmlslurper that makes reading XML a breeze.

There it is, we already  have a Groovy script to retrieve the RSS feed, iterate through the posts and determine whether the title includes BSM keyword. One can take a look at how xmlSlurper works and use it to get the information she needs.

RapidInsight makes it even easier for the script developers to work with the external interfaces without having to learn the intricacies of each interface. In RapidInsight, we can create a class called say RssFeed, store data related to the RSS feed as object properties.Here is how our class may look like:

The class can be used to store the information related to the Rss feed as objects. We can create a UI for the user to enter this information, or simply use a script to create the instances. Creating an object in RapidInsight repository is straight forward, pass the properties as name value pairs to add operation:

By storing the url, etc. as object properties, we can now have a more generic script that would work with any RSS feed. Next we can add “operations” to this class to deal with the Rss feeds, hiding the complexities of the external interface from the script developer. An operation can read the feed and return name value pairs in a map to the user.

This means the script developer does not have to understand the structure of the RSS or how to use XmlSlurper. The operation code can be modified to add support for different RSS formats (v1, v2), use ROME library to parse RSS instead of XmlSlurper, etc., scripts would continue to work as before and script developers like me would not have to learn something different. All it matters is what is passed to us, which is a List of Map (name value pairs), and this is same for all external interfaces.  (Take a look a this post to see more on how RapidInsight uses name value pairs when working with external interfaces)

The operation may be something like this:

Now that we have removed both the data (like url) and the external interface interaction out, our script can be simpler:

Or we can extend it to do the same for each RssFeed object we may have defined:

Let’s move to the second part of the requirement: taking an action like sending an SNMP trap. Groovy can use any java library and it seems like there is an open source java library for pretty much anything you can think of!
We use SNMP4J in RapidInsight. The java library can be used directly in the groovy script, however SNMP is more complicated than RSS, hence figuring out how to use the library is difficult for a non java developer like myself. Here again, RapidInsight helps by providing infrastructure to make it easier to work with SNMP and hide the complexity even further.

SnmpTrap class can be used to store common data such where (ip/port) to send the traps, which snmp version, etc. SnmpTrap class has an operation called send() that takes name value pairs and sends the trap.  Usage is very easy:

If I only have a few different kind of SNMP traps to send, like up, down etc., I can move the trap definition into the operations to make the script itself even simpler.

The functionality described in this post is not difficult to implement in perl etc. for a good developer; there are libraries available and the whole thing can be implemented in a single script.  RapidInsight provides functionality at every step to make not only implementation but maintenance and operations support easier.  Scripts can be executed from a web based UI, scheduled to run periodically, only available to subset of authorized users, logging, notifications, etc.  instead of collection of scripts running as cron jobs that can get out of control rapidly.

A reinstall of OS and essential apps would have solved some of the problems but the initial install was painful due to number of drivers that had to be added etc. The laptop comes with loads of junk/trial software and a clean install was unexpectedly difficult. I couldn’t risk any downtime during reinstall so I kept postponing it.

At the end, I ordered a new, faster (7200rpm) hard drive and decided to try Windows7 beta with it, fully expecting to revert back to XP. Happy to say Windows 7 exceeded expectations and was much better at recognizing all the hardware. In addition, with XP I had to use VMWare images for number of applications that required Windows Server 2003, since they did not work with XP. With Windows 7, they Windows 2003 compatibility mode solved this problem.

My definition of a good OS have somewhat changed throught the years. I want an OS that handles the hardware seamlessly and won’t do much more than that. Windows 7 does handle the hardware nicely. The search interface to launch apps is a winner! within minutes I got used to doing everything from going into folders to running apps without touching the mouse using the search box.  It’s really well done.

Security is a pain in the ass! It’s going to take me a while to learn how to disable various automatic stuff, firewall configs, etc. I can’t imagine a non-technical user dealing with this stuff. It’s crazy complex. I can’t imagine helping someone over the phone, etc. to get software installed. There are so many things where things can go wrong in the name of security.  I am an administrator and I’m still getting “you don’t have sufficient access rights” messages. WTF? If I don’t have the rights, who does, NSA?

When I started installing the applications I use, I’ve noticed that majority of them are free applications, if not open source. I have mixed feelings about this. Does this mean that desktop software market has essentially disappeared and not possible to charge for software anymore? Apparently I’m not paying for them. The least I can do is to list and thank the creators. Here is the list in no specific order:

• Firefox Despite continuous annoyance with memory usage, browser of choice. I use it with delicious, firebug, greasemonkey, read it later, add-ons. Eliminated a lot of add-ons, some of them will likely make it back in time as I need them. Haven’t decided what to use for tabs yet for example.
• Sun JDK So easy to take it granted. Java VM is such an invaluable technology. Thanks Sun!
• OpenOffice Encore, kudos to Sun. It’s amazing that we can get this for free.
• Skype I live in Skype, use it 24/7 both in professional and personal life. Who would have thought that we can have video conferencing for free.
• Putty much more than an SSH client. What would I do without tunneling getting me though security hoops?
  
• VLC Media Player It just works, doesn’t install bunch of crap on your PC.
• WinSCP scp and ftp. Like putty, it’s an essential tool I would be hard pressed without it.
• Digsby my current/recent multi IM/social networking tool.
  
• Crimson Editor My text editor for quite some time. works very well. thanks!
• JungleDisk I’m a paying customer using it to backup files into Amazon S3.
  
• Mozy Backup solution for my PC, works well enough.
• Synergy What a gem! Use it to control other PCs in the office from a single keyboard/mouse.
• DynDns another free service/software, essential to work with others when you don’t have a static IP.
  
• iReasoning Mib Browser these guys have great SNMP libraries. They make the mib browser available as a free tool. Works well.
• Groovy my favority scripting language
• Active Perl Perl is still must have in my work. Active Perl distribution made installing Perl on Windows a pleasant experience.
• Picasa I still prefer desktop photo organizers. Picasa from Google is pleasant to use.
• RealVNC VNC is part of the essential tool in the arsenal of a remote worker.
• 7zip works with different compression file formats nicely. another essential tool.
• Adobe AIR, Acrobat Reader & Flash Player Adobe is yet another great tech company providing number of great technologies over the years that have become defacto must have tools.
• Jing Project After using the free version for a while, I’ve become a paying customer when they offered the Pro version. Use it all the time for screenshots and screencasts.
• VMware Player. Another essential tool for development, support, testing, etc. In addition, it has become a key enabler for me to create clean environments to connect to different client networks.

So there it is.  Overall, it feels like I got a new laptop since everything changed and works faster. Now I need to figure out how I can save this state of the laptop so that I can come back to this clean state if necessary.  Hoping that Retrospect, the backup software that came with the external USB drive will help with that.

Quality of support has long been used against open source software, suggesting that enterprises need an entity to deal with that is accountable to them. Fair enough, even though the mailing lists can be more responsive than some support organizations, enterprises require more assurance. This issue is largely resolved since there are now open source companies providing support for most open source products. The fact that this is still being tossed around as a concern is annoying to say the least but c’est la vie.

What irritates me more is that how inferior proprietary vendor support systems can be due to specifically their closed nature. Most bugs/problems are experienced by multiple users/customers and problems that are difficult to isolate can take a lot of time and effort.

Support information (generated by customers) is mostly locked inside enterprise systems, and not visible/searchable by the public or even the customers. Correlation of the problems reported by the users, their troubleshooting journey, the problems encountered on the way, gotchas, etc. all stay hidden from rest of the people.

In my experience, most people prefer using self help tools and research themselves before calling support unless there is an emergency.  If the customers could search/access to this information, not only the customers would save time and may be reduce down time, but the vendor would also benefit from reduced support costs since fewer support calls would come in. Even further, customers may even be able to avoid problems altogether by learning from experiences of others.  So why is this information not accessible, available at least by the customers?

• tools may be inadequate. The support systems may not naturally facilitate this type of sharing and collaboration
• vendors may not want this information out in the open for “competitive reasons”, keep dirty laundry hidden so to speak
• support information may include sensitive information that customer may not want to be public
• others.. ?

Open source products by their nature are more transparent, the problems are reported and discussed out in the open in mailing lists, bug tracking systems, forums, etc.  hence Google is ready to help you find the answers.  Vendors -open source or otherwise- should make the effort to make the support process as transparent as possible, the cost of keeping this information locked inside isolated systems is just too high. Open source companies have an opportunity here to differentitate themselves in yet another way that directly impacts the customer experience.

Integration in proprietary world is painful experience in most cases. It is hard to get a hold of the software, often not even possible. There are all kinds of restrictions on what you can and cannot do with the software that prevents high quality integration, contracts need to be signed, and small companies like us are often ignored.
It does not need to be this way. Proprietary software can have open APIs and vendors can establish a process that can be used by anyone, much like it’s been done by the web companies (Amazon, Google, etc.), but it’s rarely done this way.

8 years ago, working for a consulting company, I had given a presentation in the company about the developments in the web world, web services APIs, its impact on IT management, and systems integration. The company had significant revenue stream implementing IT management solutions by integration a set of IT management software products, usual suspects like Netcool, Remedy, Smarts, Nervecenter, Infovista, Patrol etc. and most of us were specialists on one or more of these products. I had suggested that with the spread of common/standard integration APIs (SOAP, XMLRPC), nature of integration projects would shift from point to point integration into use of the standard APIs, and generic enterprise integration tools. If so, the demand for the skillsets would also shift, requiring more software development skills than systems integration skills, in depth expertise and understanding of inner workings of the IT management products.

Needless to say, it’s an understatement to say that I was wrong! In the last 8 years, very little has changed in the IT management world. Ad-hoc, point to point integration is still the mainstream, and often the only option. Consultants do just fine with the same skills they had a decade ago and IT management projects still have very high costs, implementation (integration) costs often well exceeding the licensing costs. I’ve seen may RFP, product comparision matrixes, etc. over the years and support for standard APIs is often just a check box. Massive cost impact of not having standard APIs are mostly missed, and as long as there is no strong demand from the customers, vendors will not move.

Some of the proprietary software products do have APIs and if you can get access to the software (for us typically on a client site who wants us to integrate with) you can use the API to develop descent integration. But there are still risks for the customer if the API is not available publicly available. If the API is not embraced and widely used, the vendor has no qualms about changing it liberally from version to version, so upgrades become a much riskier proposition. So how widely the API is used is crucial for the customers. When an API reaches high level of use, organizations become more conservative in changing them and think twice before breaking more compatibility.

In the open source world, it is assumed that integration is easier since the code is available. And it is indeed easier. To start with, you don’t have to deal with the commercial/legal contract nonsense to implement the the integration. I can tell you this phase takes longer and is more frustrating than writing the code.
Since the code is open, you can look at the code, learn from it, modify it if necessary, etc. You don’t necessarily need the help from a company, etc.  We’ve recently integrated RapidInsight with OpenNMS and Hyperic with only using publicly available resources, forums, mailing lists, etc.

Yet source code being open is not a replacement for a well documented API. There is no guarantee that the data structures, methods, etc. will not change from version to version. After all, an API is a software contract, a commitment to the outside world and implies stability, but internal code and data structures do not. So integrating open source products without standard APIs is carrying some of the same risks of integrating proprietary products.

The good news is that since open source products are developed transparently, one can get a heads up, adjust the integration for the new version, provide feedback during development to ensure backwards compatibility if it is possible, and even better participate in development of the standard APIs.

Rapidinsight OpenNMS integration gets data directly from the underlying postgres database and uses java methods for graphs. Fortunately, OpenNMS has a large community and evolves slowly, hence integration should be safe for some time. In the long term, implementing an API on the OpenNMS side to expose the information would be the better solution. If OpenNMS community sees value in this integration, we would certainly look at contributing at that end.

For Hyperic integration, we’ve used some groovy classes (we love groovy! ), implemented a Hyperic plugin to expose the information we need, and used some UI methods to access to the graphs. We’ve already noticed that something has changed with v4.0 and we’ll need to adjust the integration accordingly to make the integration work with the new version. The good news is they seem to be working on implementing a web services API which would make our lives so much easier!

Wouldn’t it be great if there was a site like programmable web for IT management software

Web offers many choices to people and to me RWW model offers the best return on my (time) investment. What makes RWW different?

Content with substance.
You can tell that it’s not just blurted out. I was surprised to read that the blogs play “breaking the news” game like the traditional media.  Erick Schonfeld had written:

“There is not much time for story-telling (except for weekend posts like this one). It is mostly breaking news, reporting facts and providing analysis.”

Breaking news is clearly important to a lot of people. Techcrunch provides is very useful to many, as it has millions of readers (including myself). However from my perspective, “breaking blog post” is not as valuable as well throughout, researched one. It won’t kill me if I read about a hot new startup a day late. I rather read about it with supporting information that provides context. That requires research and takes time.
Many RWW posts are like references, often contain in-depth analysis, compare & contrast different players, philosophies etc. I can imagine that it takes time to write them. But as a result, they don’t become stale as fast as”hot news” posts. I wonder how much traffic RWW gets, through google hits for older posts. I suspect it is significant.

Comments rock
I prefer reading blogs like RWW, Techcrunch, A VC a day later. Significant part of the value provided by these blogs is the comments added by the readers. Reading the posts later give me a chance to read the comments on the posts, which are as valuable(if not more) as the post itself.

I suspect the next innovation for the conversations on the web will come in the comments space. I think we desparately need better tools to manage comments (for the readers). An obvious idea is digg like system for comments, to get good comments bubble up. Amazon book reviews may be a model for this.

Practicioners beat reporters any day
RWW has number of contributors who are not primarily reporters but practitioners in the field. Personally, I think hands-on experiences of someone in the field is many times more valuable than a (no offense) reporter. It helps that RWW writers are using heavily the cutting edge web tools themselves and share their experiences.
When the subject steers outside the expertise of the regular contributers, it is best to get a guest writer to contribute rather than fumble a shallow opinion based post that does not add any value.I hope RWW will continue to have subject matter experts contribute. It is not easy to find these people and blogs like RWW play a critical role in bringing talented people to daylight for all to enjoy.

A new year resolution for me: I have to get better at being concise. In summary, a big thanks to RWW. Hope their business rewards them sufficiently for the excellent service they provide.

• package solutions were not able to cope with the size (scale and performance)
• unique requirements that not met by packaged solutions
• maintaining control over how the solution evolves
• strategic differentiation from competitors

Obvious disadvantages of the in-house build solutions is the potential high cost of ongoing develoment and maintenance, dependency on small number of people, etc. It is understandable that developing every solution in-house for a single organization is a costly proposition regardless of how big the organization is.

Like it or not, most service providers have been driving towards using packaged solutions whenever there is a viable solution in the market. Problem with the packaged solutions is the proprietary nature of them. Hard to find skill sets in proprietary products, lack of innovation by the vendors, feature set not meeting the requirements, scalability challenges to meet the needs of large organizations, and integration hurdles significantly diminish the perceived cost savings and strategic value of these solutions.

Open source solutions offer an alternative that may be best of both worlds. Organizations can can collaborate in developing solutions that meet their common requirements, giving organizations the opportunity to share the cost of development and still have control over the solution, etc. Open nature of the open source solutions make integration much easier as well.

Yet there are no significant open source projects in IT management driven by these large organizations. Most open source solutions is product of companies or handful of individuals, and typically target the SMB market. Service providers invest massive resources for in-house development, often have multiple teams with dozens of developers. Moving the development of these solutions into open source projects would offer significant benefit to these organizations. I’ve seen projects in “maintenance mode”, meaning none of the original developers are around anymore and no one knows how the solution works, and no enhancement is possible.

There is clearly a failure here, inability to collaborate, corporate barriers to share, etc. Whatever it is, it’s preventing these organizations to take advantage of what open source development model would have to offer.

So what does all this have to do with the clouds? There are significant parallels between the needs of cloud providers and traditional service providers in terms of IT management solutions. Scale, multi-tenancy, strategic role of the tools, need for flexibility, open interfaces etc. mean that it is not likely that cloud providers can use packaged solutions offered by Big 4 and others.

The good news is that these companies have open source in their DNA, and already use it heavily, typically using open source components to build internal solutions.They have not been so far collaborating to build solutions specific to clouds as open source, but it is more likely to happen than traditional service providers.

It is already safe to say open source software is the de facto choice in the public clouds. What remains to be seen is whether this meme will transfer through the firewalls into the private clouds. My guess is that it will. Simply because open source tools will be ahead in meeting the management needs in the cloud from scalability to integration. Thought leadership is important and open source tools like Hyperic is paving the way for the open source tools to play a leading role in mangement in the cloud. Will the cloud providers play a more active role in development of open source tools for the management in the clouds? It would serve them to learn from the troubles faced by traditional service providers.

Image via Wikipedia

If you’ve never read Seth Godin’s blog (or books), you should. It’s full of gems. One of his recent postings was on a subject that is very relevant for us, hence the title of this post is borrowed from his.

“That’s the choice most of us make when we launch a product or service. We can make a market or we can take share from a market.”

Software products are particularly hard to describe by their nature and it does not help that IT terminology is polluted. Even commonly used terms like network management, event management, etc. mean very different things to different people. As a result, we often face using existing products to describe new ones:

“This is just like the Gillette razor, but cheaper.”

“It is just like HP OpenView, but better/cheaper/works…”

As Seth indicates, this strategy takes advantage of the recognition established product has and makes it easier for the potential customer/user to compare & contrast.

I think this explains why HP OV (NNM) is still the yardstick new monitoring tool vendors use to compare/define their products even though HP OV has been stagnant for over a decade and not even close to be the best tool from a technical perspective.

RapidInsight is different than the alternatives in the IT management market. It is meant to be. We’ve designed it with the hindsight of having lived with the the shortcomings of traditional solutions currently in the market. But articulating this difference in the market is challenging.

So far, RapidInsight customers have been direct contacts. We knew the customers’ well, hence we could articulate what RapidInsight is, why it is better than alternatives, etc. within the context of the needs of these customers. But when you go to the broader market you don’t have the opportunity to talk to each potential customer at length, if at all. Direct access to customers is where smaller companies run into difficulties. We need a way to articulate what RapidInsight is, and why it is worth fir anyone to invest their time to learn more about it.

So we face with the choice Seth states eloquently in his post. Should we try to take market share by identifying RapidInsight with existing solution or should we try to make the market? The answer should not be that difficult. We’re way too small to make a market. We can easily say “RapidInsight is like Netcool, only better . An open source IT event management solution” It’s probably better marketing position for us. From a technical perspective, this is an inadequate description of our product. I want to talk about why focusing only on consolidating events, restricted proprietary languages, storing data in relational databases are flawed approaches and why one needs a built-in CMDB, object based data store, use of standard based dynamic scripting language , etc. Yet I have to accept that “making the market” is out of our reach, however “right” it feels.

So it helps when others join in describing why consolidated event management is not sufficient and we need more It turns out, HP has just announced a new product/release, HP Operations Manager,OMi.

The blog author (shouldn’t a blog have info on who the author is?) describes HP OMi as a “next-generation consolidated event and performance management product“. What make OMi different? It sits on top our HP’s CMDB, which means it has access to service dependency information and have access to availability and performance data. It sounds like folks in HP understand that consolidated event management alone is a positive but insufficient, and why a solution that models and consolidates all IT operations information is needed. That will be great if they can educate the market to why. May be then, we can say RapidInsight is like OMi, but only better