Business Process Management /Technology History

Hi guys I came across this wonderful article about BPM and how it has evolved, being working in this domain for no of years I find this article very fascinating and thought of sharing it with you guys …Enjoy


In the beginning there was workflow. To be more precise, there was person-to-person routing of scanned documents through a pre-determined process map.
FileNet was the first to use the term “workflow” in this context, back in the early 1980’s when they started up, although they were quickly joined in the marketplace by IBM and other product vendors. Most of these early workflow systems were document-focused, that is, the only purpose of the workflow was to move a scanned image of a paper document from one person to another so that they could perform some action on a different system, such as transaction data entry. This was a logical step after organizations started scanning their documents in order to preserve and share them: why not scan them before working on them, then pass them around electronically to try and improve efficiencies? Unfortunately, any direct integration between these other systems and the workflow processes was custom-built, expensive, and not very flexible.



From these early beginnings, workflow systems evolved fairly slowly during the 90’s into products that were much more functional and flexible, primarily in the area of better integration capabilities, more diverse server and client platforms, and some basic process modelling and process monitoring tools. In many cases, however, the workflow products were still very document-centric: the document scanning/management business was such a cash cow for many of these vendors that they didn’t show a lot of vision when it came to finding uses for workflow concepts outside of routing documents around an office. That set the stage for two things to happen:



Third-party add-ons to popular workflow products would proliferate, to do all the things that the workflow vendors didn’t deem important.
More nimble “pure” workflow startups (the early BPM products) would ignore the document-centric side and build highly-functional workflow products that were unable to handle an enterprise workload, but would push the envelope in terms of functionality.

While workflow was getting its start in the 1980’s, enterprise application integration, or EAI, emerged independently for system-to-system integration.
A key player in the early days (and still today) was IBM with their MQ Series messaging, which became a de facto standard for system-to-system communication in many large organizations. Many other vendors entered the market, and a variety of technical architectures for message brokers emerged, but all had the same basic goal: to automate the near-real-time exchange of data between systems, typically mainframe-based transaction processing systems or server-based relational databases.
Even the simple functionality provided by early versions of EAI could reap huge benefits in two areas:
It was no longer necessary to re-enter data in order to transfer it between systems, reducing data entry efforts and error rates.
Information could be exchanged between systems in near-real-time, rather than relying on overnight batch updates.



These early EAI systems had no human-facing functionality, since they assumed that the applications being integrated provided all required user interface. In other words, if an error occurred sending data from system A to system B, the data or other problem would be fixed in one of those two systems, not in the EAI system.
Workflow and EAI lived very separate lives during that time, although they are now seen as two ends of the same integration spectrum. Not only were they created by two different camps of vendors (with a few not-very-succesful exceptions), and based on different technologies and standards, but they were sold to two different parts of the customer organization: workflow was typically sold to business units as a departmental solution, where as EAI was sold to IT as an infrastructure component.






In the late 1990’s, as workflow vendors saw the benefits of EAI and understood how it could replace the more heavily customized integration that was typical for workflow solutions, they began adding EAI capabilities to their workflow products. Although there were exceptions, this was typically done by the workflow vendor striking an OEM agreement with an EAI vendor to allow the EAI product to be embedded seamlessly into the workflow product.










Not surprisingly, EAI vendors saw the benefits of having some human-facing steps in the system-to-system processes, and began adding rudimentary human-facing capabilities. These functions, usually built by the EAI vendor directly rather than partnering with a workflow vendor, were fairly rudimentary since they were intended just for the repair of processes that had an exception condition that could not be handled automatically. You could say that these processes were “human interrupted” rather than “human facing”.
Although we now had systems that (theoretically) spanned the full range of the integration space, these were really just workflow systems with a bit of EAI, or EAI systems with a bit of workflow, and the vendors still sold to their strong suit.


Organizations that had implemented workflow quickly realized that once the process became electronic instead of paper-driven, it wasn’t possible to monitor the process just by walking around the shop floor. Workflow monitoring and reporting were born from the need to understand where the process was — and wasn’t — working, and this was often a highly customized capability. After years of customers building their own custom reporting and monitoring tools, or using third-party analytics, the workflow products started to extend their native capabilities into this area. The early real-time monitoring grew to become a part of what we now know as business activity monitoring, or BAM. Process management and governance, both through this real-time monitoring and historical reporting and analysis, became a critical part of the process to customers, especially those implementing quality programs such as Six Sigma. The need to optimize processes pushed beyond analytics to process simulation and optimization tools.








EAI products grew in a different direction altogether. Most of the products provided some degree of reporting and analytics, and didn’t need the process governance associated with human-facing workflow. Instead, EAI vendors started to look outside the organization, and extended EAI to business-to-business integration, or B2Bi. This process collaboration allowed their customers to implement processes — still primarily system-to-system — that loosely coupled their business processes with those of their customers and other trading partners, not just flows between internal systems.


This would become one of the most significant advances for the type of BPM that we have today, allowing organizations to include both human and system participants in processes that span multiple organizations. The ROI on B2Bi can be enormous, and allows for the creation of a sort of configurable “business process firewall” as a standard process interface between an organization and its trading partners so that either can change their internal processes and data without disturbing the other.



workflow and EAI markets started to converge. Both workflow and EAI products extended to include functionality that was useful in both spheres, such as business rules engines and more advanced process modelling tools. Business rules engines were typically integrated through OEM agreements, while process modelling was usually built as an integral part of the workflow or EAI product. Workflow products beefed up their EAI capabilities, and EAI improved their workflow tools

They called it all “BPM suites”, but broke it down into five categories, with the key part of the market falling into the “pure-play” and “integration-focussed” categories:

Integration-focussed BPM manages system-to-system flows for process integration across multiple applications. At the time, products in this category were slowly emerging into the human-to-human flow applications but lagged in user sophistication and user-friendliness.
Pure-play BPM provides overarching design and connection between application-specific and integration-focussed workflows, for example, by triggering a subflow in another system and carrying the result back to the master process. These were the über-BPM products, modeling processes across the enterprise, not just subprocesses within the enterprise, and including modeling and analysis tools, business rules engines, and process simulation and optimization.
Basically, Gartner (and other analysts, quickly and slavishly followed by the vendors) put the words “workflow” and “EAI” out of fashion, and ushered in the era of referring to anything even remotely related to process as BPM. They then split the BPM space (mainly) into pure-play and integration-focussed, which corresponded roughly to the old divisions of workflow and EAI vendors. Along the way, I’m sure that they sold a boatload of analysis and consulting.

Keith Swenson has a post this week about how the term “workflow” fell out of favour during this period due to “marketing fluff”, a point with which I completely agree — after about 2001, you couldn’t use the word workflow in a conversation with an IT group without someone sneering at you for being old school. His point is that BPM has now come to be synonymous with EAI; I’m not sure that I agree with that, but I do think that BPM is one of the most misused and misunderstood terms around today.

In the years following the dot-com bust in 2000, a number of new BPM vendors came into being, mostly in the coveted pure-play space. (Funnily enough, “pure-play BPM” is now not the desirable place to be, having been replaced by the “BPM suites” space that, according to some large analysts’ research, seems to have nearly identical functionality to pure-play BPM.) In many cases, these were started by those who were bounced out of their previous positions during the bursting of the bubble, so there was a lot of experience being put to the task of starting this new generation of BPM vendors.

Every analyst, vendor and customer defines BPM differently, because the current definition of BPM is very broad, and there are many vendors jostling for position within it. EAI/ESB-type vendors call their products BPM, but the products may contain only rudimentary human-facing functionality. Workflow-type vendors, also labelling themselves as BPM, lack the necessary infrastructure for integration, and often handle automated steps poorly. Some pure integration products call themselves workflow, just to confuse things further. There’s a lot of complementary products, such as process analytics and simulation, and business rules engines: BPM vendors will either tell you that a particular capability must be part of the base BPM product (if their product has it), or should never be part of the base BPM product (if their product doesn’t have it). And now there’s the whole SOA wild card thrown into the mix.
BPM is definitely a case where the whole is greater than the sum of the parts. It’s not just workflow plus EAI plus B2Bi plus business rules, plus plus plus: it’s the near-seamless integration of all of these tools into a single suite that provides an organization with the ability to do things that they could never do before. That doesn’t mean that all the tools have to be from the same vendor, but it’s essential to deliver all of the BPM functionality in a single environment of closed-loop process improvement.

Smith and Fingar’s book Business Process Management, The Third Wave describes this “third wave” as providing the ability to create a single definition of a business process from which different views of that process can be rendered and new information systems can be built. This allows different people with different skills — business manager, business analyst, regular old user, programmer — to view and manipulate the same process in a representation suitable for them and derived from the same source. They make a great analogy with HTML, where a business user may use a high-level tool like FrontPage to view and edit HTML, whereas a developer may edit the HTML code directly, but they’re still working from the same source. Round-tripping between a business analyst’s modelling tool and a developer runtime environment is one way to do this, although it violates the “same source” in the purest sense, but we definitely have to get rid of the strictly one-way paths from business analysis to implementation that exist now in many organizations.

Furthermore, Smith and Fingar point out that in the world of BPM, the ability to change is far more prized than the ability to create in the first place, and that BPM has the potential to actually remove application development from the cycle — the “zero code” Holy Grail that gets a lot of press these days. They make an analogy with VisiCalc, which took customized data analysis out of the hands of the IT department and put it in the hands of the business users, thereby taking software development off the critical path for achieving results.

Getting back to the point of this post, what is the current state of BPM?

First of all, we have several companies from the pure-play BPM/BPM suites market: they provide excellent human-facing BPM and at least adequate integration capabilities, with some providing outstanding integration. At the Gartner BPM summit earlier this year, they listed three “major players” in this category who had revenues upwards of $100M — FileNet, Pegasystems and Global 360 — and five “up and comers” with revenues above $30M — Appian, Lombardi, Savvion, Metastorm and Ultimus — while ignoring anything smaller than that. All eight of these vendors hit into the right zones in the Gartner and Forrester charts, which means that they either have the necessary functionality or are partnered with someone to provide it.
Second, we have a couple of integration-focussed BPM vendors who have purchased pure-play BPM vendors to create the complete range of functionality. The two highest-profile examples are the TIBCO acquisition of Staffware in 2004, and the BEA acquisition of Fuego earlier this year. In both cases, there seems to be a reasonable fit, but my concern is that the human-facing BPM side is going to become weaker since the main focus of these companies is on integration.
Third, we have the large software companies that have developed (or acquired) a BPM product: IBM, Microsoft and Fujitsu all spring to mind. In many cases, such as IBM and Microsoft, their BPM products are primiarly integration-focussed without a lot of human-facing support, and likely started as a “would you like fries with that” sort of offering for customers who were already committed to their architecture. IBM’s MQ Series messaging is probably still the most commonly used piece of integration middleware in financial services, although I think that they call it (and everything else) “WebSphere” these days, and IBM rightly has it as a cornerstone of their BPM strategy. Fujitsu is the odd one out here, with what appears to be a fully-functional BPMS; unfortunately, they’ve been marketing it in stealth mode and most people are completely unaware of it: as I said in one of my posts about the Gartner BPM summit, “who knew that Fujitsu did BPM?”

We’ll continue to see most of the business functionality envelope being pushed by the vendors in the first category as they seek better ways to integrate business rules, analytics, performance management and other capabilities into BPM; in fact, the most innovation seems to be coming from the smaller vendors in this category because of the lack of baggage that I discussed in part 7.

Because of the current focus on process improvement in all organizations, I don’t think that there’s any great risk of any of the vendors that I’ve listed here going out of the BPM business any time soon. However, the integration vendors will acquire some of the smaller BPM suite vendors to round out their portfolios, and the large software companies will acquire some of everything, in a continuing Darwinian cycle.

Before you vendors start adding self-promoting comments to this post, keep in mind that this is not intended to be a comprehensive list or review of BPMS vendors, and I know that you’re all very special in your own way.

The big advantage that a new vendor has in any industry is the lack of baggage, and nowhere was this truer than in BPM: they could start designing the next generation of BPM without having to reuse their existing technology or support their installed base, because they had neither. The BPM market needed to be reinvented, and these upstart young companies were the only ones who could shake things up enough to do it. I’m not going to credit the new arrivals with all the innovation in BPM during that time, but they certainly lit a fire under the old guard. Suddenly, we had BPM calling web services, or being called as a web service, in order to speed integration (and eventually become part of the SOA ecosystem). We had BAM, or at least some half-decent process monitoring and analytics for a change. We had simulation and optimization. We had integration with third-party modelling tools. We had business rule integration.
The startup environment during that time wasn’t the best — not a lot of venture funding around for technology, the perception that this was just a rehashing of the well-established workflow market — but a few of the vendors have become successful and many others are still straggling in their wake.

Meanwhile, the established BPM vendors had a big challenge on their hands: although few of the upstarts were challenging them directly for sales, the new guys were changing the perception of what the BPM market should be, forcing the big guys to follow suit as Gartner and the other large analysts published lists of must-have features that included this new functionality. Many of the larger vendors lagged badly in implementing new features, and look a little tired these days in comparison to the shiny bright newcomers. In some very conservative industries, such as financial services and insurance where most of my clients are, this hasn’t been a problem because they’d rather pick a vendor with a longer track record and the proven ability to process hundreds of thousands of items per day. However, this is where many of the dinosaur-like CIOs are fighting the losing battle against the push for emerging technology, and eventually the new kids will prove themselves scalable and stable enough for even the most conservative industries.