Chronicles of a Wandering Mind

At iFountain, we’ve embraced the open source business model since the beginning of 2008.
Since then, we’ve been working on not only moving our code but also our development practices to open source. We’ve established a separate site for open source development, ifountain.org, where everything is out in the open, source code, documents, discussions, project plans, issues, etc. We’ve also defined what we mean by “open development“  and we try to live by it.

So far, the development is still done by iFountain employees. It should be no surprise to anyone, it wasn’t to us. Building a community is not easy, and takes time. Int he foreseeable future, we don’t expect a lot of external developer help (though it would be more than welcome) but we hope that we can establish a community that would guide where the project is heading. Most IT management folks (including this one) are not software developers, hence they may not be able to contribute code, but they are the subject matter experts, have first hand knowledge of what is needed in the field, hence can help the project immensely by guiding it with suggestions, feature requests, evangelizing etc.

The importance and value of the community for a project, even one supported by a commercial entity, is well explained and understood, so no need for me to repeat it here, needless to say, we will continue to build our community as the project takes shape. Sooner the better.

Netcoolusers is a lively mailing list based community and I started following it after I got my NCC back in 2000. There has been several discussions on the list about alternative web based interfaces that take advantage of web 2.0 technologies, etc. in the past. Several people had were interested in alternatives, stating that Webtop does not meet their requirements and some had to build in-house solutions themselves. I had not mentioned RapidInsight in the list at the time, even though it was such a solution since it was a commercial product.

Today, I’ve sent an email to netcoolusers mailing list announcing the RapidInsight open source project in the list, as it is directly relevant to the Netcool community.  Email had brief summary of the motivation for the project has come from, gave some highlights and included a link to the open source site, where interested parties can learn more about the project and take a look at the demo, download the software etc.

Next think I know, I was kicked out of the mailing list by the administrator (Jim Popovitch) for “unsolicited commercial solicitation”. You can take a look at the email and judge yourself. I certainly don’t see it as such. There is well established precedence where open source projects are mentioned freely including ones by the mailing list admins. I replied to Jim’s email explaining my point of view and left it at that. As much as Netcoolusers is a “community”, it is controlled by two people afaik, and there is no mechanism for due process. They make the rules and they are the judge and the jury.

Just sigh and move on… But it didn’t end there.

Others responded to my email, asking questions, naturally unaware that I can no longer respond to their emails, as there is no indication that I got kicked out.  Then came this email from another list admin, Jacob Steinberger.  Now hold on a minute! How about distorting the facts, and spreading misinformation. Is that not against the TOS of the mailing list? No? How about just plain decency?

Jacob writes: “While the email initially looks like a great thing to help the IBMuse Netcool GUI move in a direction that we have longed for the last half of his email and Blurry’s forwarding of his private email, shows that he (and iFountain) are out to make a buck.”

The last half of my email lists some of the RapidInsight features describing why it may be off interest to users, and asks for feedback and participation. That somehow suggests that I have evil intentions to “make a buck”? Oh no, iFountain will offer support for an open source project, run for the hills! bad, bad boy!

What I described to Blurry was that we plan to follow what’s referred as the JBoss model, as it is well established in the market. The product will be available with GPL v2 license and we will offer support and professional services. Having said that, bear in mind, I did NOT even mention any of this in my email to the list.

“Trying to sell something, whether it’s a product, consulting services or support, is strictly against the TOS of INUG. Any violators of this  policy will be removed from the list.”
There is nothing about a sale of product or services or support in my email to the list. Just the announcement of availability of RapidInsight as an open source project, that’s it. Announcement of an open source project is not a sales offer. The information on our intend to offer support was in a private email to Blurry as he asked about it directly. He chose to forward the information to the list when he found out I got banned from the list.

If that’s not bad enough, he did not stop there:
“Additionally, when a potential vendor’s website (iFoutain’s) states …

Thou shall have unrestricted access to the software. The software products will be available for download from the website without barriers. The community will be able to download and start using the software right away.
… yet requires you to create an account to download software, you have to sit back and go “humm”.”

My email to the list included one link to the ifountain.org site. From there, there is a link to download the mentioned software directly, no user accounts needed. And as I mentioned above, not only that, the source code and all its revisions (subversion) are also available directly from the site without any user restrictions, as it should be for any open source project. Why all the haste to judge?

Banning someone from the list based on rules and interpretation of those rules you’ve defined yourself is bad enough. Bad mouthing someone with false information when they can no longer respond is simply wrong. I’ve emailed Jacob before I posted this to give him a chance to correct himself but have not heard back from him.

This week HP has announced to acquisition of EDS for $13.9 billion. Naturally, the acquisition of such size got fair bit of attention in financial circles, talking heads analyzing the numbers and speculating what it may mean for two companies in very broad terms.

With this acquisition, HP becomes an IT services behemoth, second largest only after IBM, and going forward, competition between IBM and HP is expected to be fierce, especially for large outsourcing deals. Obvious enough.

Yet this deal will likely to have massive impact on IT industry in general, not just outsourcing sector. Following IBM closely, as a strategy, HP has just created the second vertically integrated one stop IT shop for customers, providing everything from hardware to software to services. To appreciate the importance of this acquisition, I believe we need to consider this deal along with other acquisitions HP made recently (Mercury Interactive, Opsware, Peregrine,Trustgenix,Tower Software, SPI Dynamics, Bristol, etc.)

With these acquisitions along with already substantial Openview family, HP has a large portfolio of IT management software and now a large services organization that can deliver solutions using these tools. Naturally, HP services organization will still have to collaborate (when it has to) with 3rd parties to meet the requirements of their customers and not just push HP software, just as IBM professional services do, at least in theory.

In practice, experience suggest that it’ll become increasingly harder for other vendors to compete with internal HP products. Projects will have to justify why they need to use 3rd party products instead of internal ones and often the path of least resistance will be to use internal ones unless the customer explicitly dictates otherwise.

I think this is a major problem for any player that is not a one stop shop IT services provider. Software companies without substantial IT services organizations as well as independent IT services companies and systems integrators may find themselves cut off from significant portion of the market.

HP has a large and effective ecosystem. The mutually beneficial relationships HP has with its channel and integration partners has long played a significant role in HP’s success in the IT management market. With this move HP is no doubt jeopardizing its relationship with its ecosystem. It is not surprising however that HP thinks the risk is worth it, and the gains that will come from becoming a one stop shop will more than compensate for the loss of business it generates through the ecosystem.

Enterprises have long been working on reducing the number of suppliers and work with handful of preferred ones and the allure of one supplier that is responsible for everything seems to be too hard to resist for many.

The impact of loosing the channel may be greater than anticipated. Independent services organizations that partner with software companies acquired by IBM in the recent years have been finding themselves competing with their old partners and some of them have been making moves to alternatives. With HP adding a large services organizations, it also becomes a competitor for any services organization that may previously be a partner that sells and implements software products from HP’s or IBM’s portfolio of products.

BMC CEO was quick to move in to take advantage of the coming conflict and characterizes EDS acquisition as “declaration of war” by HP to its ecosystem. BMC also has been in an acquisition binge lately, acquiring BladeLogic, Emprisa Networks, RealOps and Proactivenet within the last year but BMC does not have a large professional services organization so it makes to position itself as the supplier of choice for professional services companies that will be alienated by the HP EDS acquisition. CA may also make a similar move having assembled a solid set of tools through acquisitions (Concord, Aprisma, etc.).

However, if HP’s strategy works out and HP software products starts to gain marketshare, it will not be surprising to see further consolidation in the market in the form or pairing between services organizations like Accenture and software companies like EMC, CA and BMC.

IT market is more and more looking like a Highlander movie. There can be only one a few.

As it is in many market segments, companies in IT management industry are looking for an open source business model that works.  The prevailing approach seems to be the hybrid model. In this model, the companies offer both open source and proprietary versions of their products and proprietary versions typically provide additional high end features that is not available in the open source versions of their products.

This business model is only available to the companies that have the IP ownership of the open source code and not an option for more traditional open source projects where IP rights are distributed among many people and there is no single holder.

The model is a win win for both the vendors and the customers provided that the open source versions of the products are viable solutions that stand on their own and not just a ramp that requires every user to upgrade to the paid versions. Customers get access to open source software that solves their problems and vendors get well established benefits of having an open source community.

Not everyone agrees with the above statement.  There is a discussion that started with cote naming the open source IT management companies “little 4″ as contrast to the proprietary “big 4″ (IBM,CA,HP,BMC), and heated up again with QClusters exit from the openQRM project, hence little 4 becoming “Little 3″ .

openQRM was not really in the same category with the other 3 in little 4 anyway, and its exit provided an opportunity to reevaluate the open source IT management companies. John Willis took a more minimalist approach, coming up with his own name, calling Hyperic and Zenoss the “mighty two“, others suggested Groundwork and OpenNMS should round up the new Little 4.

John states that Hyperic and Zenoss has a better chance to succeed in the enterprise as they are software companies with significant funding and solid infrastructure. Not surprisingly opposing view comes primarily from the OpenNMS camp (read the comments in John’s post).  Tarus is not a fan of the hybrid model. He has criticized the hybrid model (and hence Hyperic and Zenoss), claiming it to be flawed, several times in his posts and comments. He advocates the OpenNMS group’s professional services only model as the superior (and even the only viable) model and states the success of the OpenNMS project as the evidence.

The hybrid model is not an option for OpenNMS group as (unlike hyperic/zenoss) they don’t own the IP rights for the code.  OpenNMS apparently has an active developer community that contributes code (surprisingly rare as more and more open source code is developed by few individuals or companies) and OpenNMS group has one or two people who may be dedicated to development rather than services.  OpenNMS folks seem to be content with their position, but I wonder why they don’t aspire for more.

Despite being compared to Hyperic and Zenoss, OpenNMS is different product. IMHO, it is the only true (open source) “network monitoring” product available. Hyperic/Zenoss/Nagios are primarily for server monitoring with some capabilities to monitor network devices, yet still not an option for serious network monitoring.  The paste of opennms development is slow which is understandable given the fact that there is no dedicated development team. Let’s imagine for a second that OpenNMS group also used the hybrid model and external funding to staff developers to work on opennms. Could they have developed a discovery engine that auto discovers the network layers (layer 2, 3, routing, etc.)? Visual maps to represent these layers? What is OpenNMS offered this additional functionality only in the paid version but all the functionality available now was still available. What if on top of what is currently available, foss version of opennms had gotten a package that can be installed and run on windows easily 3-4 years ago? Would having the option to pay for this additional functionality be a bad thing for the community? Would removing entry barriers such as extensive documentation and easy installation packages not help increase opennms user base?

Open source companies need to continue to innovate in the business model level to find ways to fund the projects that work both for the companies and the communities. Hybrid model is what has emerged so far as one viable option. It is not perfect but it offers an alternative that is in many ways better than pure professional services model. There does not have to be “one true way”.  The alternative may be getting squeezed out of the market. Tarus likes to compare OpenNMS with HP and the likes, suggesting that OpenNMS is a viable alternative to them. It is. but it is playing the catch up game instead of leading the field. The tougher competition for opennms (and other open source projects) is not coming from the large proprietary companies like HP and IBM, but from small, agile companies like solarwinds and adventnet with nicely packaged, easy to use products with often lower prices than the open source companies.

If open source companies cannot innovate both business model and technologically, they may get squeezed by these companies in the lower end of the market and the larger proprietary companies with massive sales forces in the higher end of the market.

In the network monitoring is a commodity myth, I argued that network monitoring is far from being a commodity and on the contrary needs innovation to cope with the increasing complexity.

As cote mentioned in the comments of that post, there has been some fresh blood in the IT management industry. Several open source companies/projects are tackling the monitoring problem, which is a good thing, yet I feel we’re still missing some pieces. AFAIK, most of the monitoring solutions seem to be following existing paradigms :

• monitoring the devices (nodes) through SNMP agent
• synthetic transcations to determine the status of services running on nodes

The understanding of the network topology is missing in both paradigms. In other words, nodes are what’s being monitored. Not the network. The network topology (except layer 3) is largely unknown. This limits the effectiveness of the monitoring. Monitoring tools (or rather functionality offered by the tools) can be categorized broadly as the following:

• Polling the devices: Most common approach in IP networks. Most IP networking devices have an SNMP agent that supports at least MIBII so basic availability and performance information can be obtained. For more detailed information however, use proprietary MIBs is needed. Many IT management guys spent long hours trying to understand these MIBs, which data is where, compile them to be used by their monitoring tools, etc.
• Listening for exceptions: Not every network device has an available agent that can be polled, especially in lower layers below IP. And when available, ability to listen for information is useful as it can be more immediate. In IP networks, these are typically SNMP traps or syslog events. In others, there are often element managers that convey messages. Again, IT management folks spent countless, often frustrating hours, trying to make sense of the traps, syslog events, etc. normalizing them, translate them into human language, identifying what is important and what’s not etc.
• Listening to the pipes: It is possible to learn a lot by listening to what goes on the network. Flow tools (Netflow and its kin cFlow, J-Flow, netstream, sflow, etc.) generate end to end traffic statistics based on flow of data through the network device that support it. Another approach seems to be analyzing the traffic going through a device using a span port. Although it seems this method is popular to analyze application traffic. I don’t have a lot of personal experience with these tools so I’ll leave it to others to explain it better or correct me. From what I see these tools often require hardware distributed throughout the network to get full visibility which may be a hurdle for adoption.

IMHO, all of the approaches I’ve tried to summarize above have some shortcomings. As far as I can see, the situation may improve in two ways:

• someone may come up with a new technology, a clever way to monitor the network and identifytthe problems, may be discover & represent the network etc. IMO, this can only happen if some of the investment and attention in tools that target “business users” with sexy, shiny UIs flow back to the muck. When the payoff is so low (who wants to tackle a “commodity” problem?) significant investment is not likely.
• The power of the community is harvested to solve tedious problems once and share rather than each user struggling to solve the same problems over and over independently. There are already some examples of this splunk is attempting to create a repository of log events and what they mean. ZipTie open source project is working on solving device configuration through collaboration of vendors and customers (how come they are not a member?)

There is a lot more that can be done in the monitoring realm, if we can manage to setup the right collaboration platform (commercially, legally as well as technically) to facilitate sharing, which is sorely lacking in IT management for whatever the reasons may be.

From what I can see, ZipTie model is particularly interesting and suitable. Ability to collaborate and share is potentially a major competitive advantage for open source projects. I believe there are opportunities here for collaboration among open source projects/companies and their users/customers.

For example, in the case of discovery and representation of the network topology, how to get the topology data out of vast number of different type of devices is can be shared. If a common model can be defined to represent the topology, adapters to populate the model for each device can be developed.

In case of trap and event log processing, the knowhow of what each trap may mean, what the varbinds are can be shared. And again if a commong model can be defined to represent the traps/events, adapters to convert the traps into the common model can be developed.

I think these activities are naturally conducive to be solved through collaboration, and the life in the trenches would improve significantly if we were tackling them together instead of drowning in them alone.

There is a persistent meme in the industry that states (network) monitoring is now a commodity. This meme is so persistent that it seems it’s no longer even disputed. There are lots of different monitoring tools, many of them are open source and/or free, and they’ve been around for a long time, hence the thinking goes, monitoring is now a commodity.

It is quite puzzling to me how terribly wrong this meme is. How can we be so wrong? IMHO, network monitoring is not a commodity. Far from it. Network monitoring is still largely an unsolved problem. The tools we have to monitor the “network” are largely inadequate.

Network is a complex beast, and level of complexity is increasing by leaps and bounds as well as the criticality of it.. It has layers over layers and only limited set of people understand it all. Our monitoring of the network is mostly limited to what we understand the most: the nodes in the network. We don’t really monitor the network itself which is a complex distributed application running on these nodes.

This reminds me a famous Nasreddin Hodja folk tail where he looses his ring in the basement of his house but people find him looking for it outside, on the road. When asked why he is looking for it outside, he says that the basement is too dark, and he can’t see anything there.

It seems to me that somewhat like Hodja, we’re monitoring the nodes in the network since we can, and not monitoring the network because, well, we can’t. The problem is largely related to instrumentation. More or less standard instrumentation SNMB MIBII, etc.) to monitor the status of a device and its ports & interfaces has been available for quite some time but very little instrumentation is available to determine the network topology, and whatever is available is not standard.

Without the understanding the network topology and the role of the nodes in that topology, the value of monitoring of the nodes is quite limited. We end up collecting a lot of information that does not necessarily helps us determine what’s wrong. This is also largely the cause of the disconnect between the users and IT organizations when talking about availability reporting. IT reports on availability of the nodes in the network which does not necessarily equate to the availability of the services that run on the network.

As an alternative when the services are monitored directly, we may be able to determine whether the service is up or down, but cannot determine what the cause of problem may be by looking at the monitoring tools.

The focus in IT management market has moved up to stack so to speak to “business level” where tools which shiny user interfaces that provide “executive dashboards” are all the rage. IT departments have hell of a time justfying an investment in better monitoring tools but have easier time investing in tools that address the higher level. Ironically, the higher level tools rely on the information provided by the lower level tools such as the monitoring tools hence without solving the monitoring problem, it’s not feasible to have meaningful dashboards.

Beating up the IT organizations has become such a popular sport that no one seems to listen to what they have to say. As a result, IT management discussions increasingly risk loosing touch with reality. I confess to be jealous of cote’s blog biline “one foot in the muck, the other in the utopia” as I believe is the right philosopy to solve any problem worth solving. Network monitoring is in desparate need of innovation and attention, but that is not likely to happen if we start paying more attention to what the people in the muck are saying and kill this false meme of monitoring is a commodity

I don’t have the answer to how to solve this problem, but I think the community may well have. In the next post, I’ll lay out not what I think may be an answer but what I hope may trigger some thoughts on what can be done to tackle the problem of “network” monitoring.

John Willis asks whether ITIL still matters in the world of Amazon and Google (what I once referred as “best in class infrastructure providers“). ITIL skepticism is not new; there has been skeptics since the beginning for variety of reasons; some more valid than others. John is raising the issue from a different perspective. He stipulates that ITIL may not be required if majority of the services are provided by external giant service providers like a utility.

From my perspective, the availability of these services from the likes of Google/Amazon make ITIL and ITSM more relevant and necessary in the enterprise, not less. One of the core ideas of ITIL/ITSM is to have a service perspective and managing the dependencies of the services to the infrastructure used to provide these services. The fact that some of the infrastructure components are provided by external providers who supposedly have great availability numbers does not change the fact that enterprises still have to manage the “service”.
As I stated in a previous post the question we should be asking is how we can you end to end management of a service when the infrastructure for the service relies on combination of multiple internal and external service providers.

ITIL/ITSM offers some guidelines on how to cope this complex world. How should the enterprises troubleshoot problems? What should the service desk processes be like? Business/end users have never cared much about the availability of the servers, they care about the service. The services as perceived by the users are rarely provided in their entirety by a single provider. Most of the mission critical services have multiple components provided by different internal and external entities. What should be the operational processes to manage these services?
A typical scenario that exposes the cleavage between different silos in the enterprise is the “blame the network syndrome” where users complain about the performance of an application, and every group (silo) blames another and the network group gets stuck with proving their innocence. How do you “convince” all parties involved -the connectivity providers (LAN/WAN/Security, etc), application providers, platform (server) providers (internal or external) - to cooperate in order to resolve problems quickly? This has always been difficult, and rise of giant service providers don’t alleviate the pain. Processes are still needed, guidelines are still needed, learning from the experiences of other still needed.
This is not to deny the significance of the change in the field . No doubt the game is changing as stated by John, but the implications of these changes are not so apparent. The rise of service providers that promise 99.99% availability may mean enterprises will more and more use the services provided by these external providers, instead of keeping them internal. If that’s the case, enterprises will need to learn how to manage services that are not under their direct control. It may also mean that if they do keep them internal for whatever reason, management of these services can no longer be an afterthought as it often has been.
The game is changing and we must figure out how to adapt. Unfortunately, enterprise (IT management) users are not out on the web sharing their thoughts with each other in mass. As the web 2.0 culture infiltrates the enterprise, who knows may be the enterprise folks come out to play and we can come up with an ITIL that is developed like an open source application, out in the open with participation of hundreds. Who knows, may be OMG will be the catalyzer for wider discussions, once can only hope …