Using Analytics to Modify In-Flight Processes

I recently had the pleasure of presenting the Oracle Key Note at the Butler Business Process Management & Integration symposium. The subject of the keynote was to look at how we can use business analytics to enable us modify processes already in-flight. When you consider the traditional closed loop BPM Lifecycle, as illustrated below, the emphasis has always been very much on using analytics about how processes have performed in the past, in order that we can modify the actual process definition in order to improve / optimize future versions of the process. Whilst this approach has many benefits, the challenge was how we could modify processes already in-flight. In order to achieve this we need to overcome a number of challenges; firstly we need to collect analytics in (near) real time in order to base our decisions on up to date information; secondly we need a controlled way of modifying the in-flight process based on the data. Collecting Real Time Business Analytics Business Activity Monitoring (BAM) provides us with the tool to collect the near time analytics on which we can base our decision on how we wish to modify our in-flight processes. For those of you who are not familiar with Oracle BAM, it enables the business to gain a real-time view of what’s happening with the business. To achieve this it provides the following key components: Capture Real Time Data - Within a BPEL process you can place a sensor on any activity with a process, when triggered this generates an event which is picked up by BAM in real time. Note: You can actually use events generated pretty much by any event based system (e.g. database triggers, messaging systems, etc), it’s just that BPEL makes it very easy. Analyze Processes, Trends, and Context - Next Oracle BAM is able to correlate all these events and synthesize them into meaningful data objects within it’s active data cache Interface for Business Users - The business can then build interactive real-time dashboards and proactive alerts on top of this active data to enable them to monitor the business processes. For more information on BAM see http://otn.oracle.com/bam. Modifying In-Flight Processes Once we have the real time analytics, the next challenge here is to use the data to modify the processes already in flight. Now when we talk about modifying In-Flight Processes; most people assume that this involves quickly writing a new version of the process, testing it, deploy it, and then migrating the existing in-flight processes to this new improved version. The reality is that is rarely as quick as required!!! Rather what’s really required from a business perspective is to be able to modify the flow through a process (and its sub-processes) in order to obtain the desired business outcome. There are three basic patterns here which provide a way of achieving this, these are: Modify Process Flow – This uses business rules, which can be modified externally to the process to change the flow through a process instance. Exception Management – This uses BAM to launch a process to manage an exception which is (typically) occurring across multiple processes. Dynamic Process Assembly – Here we use BAM as source of real time data against which to evaluate Business Rules and use the results to dynamically call sub processes and assemble the end to end process on the fly. Modify Process Flow With this approach we are not looking to modify the actual process, rather modify the “path” through the process. If we look at any process, then at various points the process will hit a decision point which will determine which route it should take the through the process. Rather than building these decision points directly into the process we can externalise them in a rules engine such as Oracle Business Rules or iLog JRules, as illustrated below. This then enables the business to modify the rules based on what’s currently happening within the business (as indicated through BAM) and thus get the process to take a different route. An excellent case study for this is Cattles (a UK company which lends money to the secondary market), which uses a combination of BPEL, BAM and Rules for this purpose. Exception Management BPEL already provides a comprehensive way to handle exceptions, within the context of a process. However on some occasions we want to take a more holistic view to managing exceptions. This is certainly the case where we are suddenly “hitting” a common exception across multiple processes. For example, if we have a loan flow process that’s going out to an external credit rating agency, and for some reason that credit rating check fails. Within the individual process BPEL process, we could have built in a simple re-try mechanism that simply waits a period of time before re-trying the service. However if we start having a high number of failures (e.g. we may 1000’s of these processes running at any time), rather than handle the same exception multiple times (at least the same root cause) we can use BAM to detect that we have an issue and kick-off a single process to handle that exception (as illustrated below). Dynamic Process Assembly Rather than use BAM to feed a Dashboard, we can use it as a real time data source. A BPEL process can then query this real time data and pass this to the rules engine; enabling us to evaluate rules based on real time data. At this point we could just use the result from the rules engine to dictate the flow through a process (as we did with the first example – Modify Process Flow). However (as the title implies) we can take it a step further and dynamically assemble the process. The trick here is to have multiple “sub” processes all with the same WSDL definition. The rules engine rather than returning a “decision” now returns the end-point of the sub process to call, which the main BPEL process can then call dynamically (as illustrated below). For more details on how to implement dynamic routing, see my previous blog; Using BPEL to Implement Dynamic Content Based Routing.

Writing a Recursive BPEL Process

Recently I was working with a client who wanted to implement a recursive process (i.e. one that calls itself). Now Recursion is a classic programming pattern, and in theory it should be pretty straight forward for a process to call itself. However at first sight it’s not so obvious. The issue here is that the way a BPEL Process calls out to another process is to drag a Partner Link on to your BPEL Process and then use the Service Browser to select the deployed process that you wish to call. Of course with our scenario, the first issue we hit is that we haven’t yet deployed the process as we’re still writing in it! The obvious thing to do here is create a stub process, with just the basic receive and reply operations defined and then deploy the stub. This works great, we can now select the process within the ‘Service Browser’ and finish implementing the process. However the next problem occurs when we try and re-deploy the completed process. Here the deployment fails with an error that it is unable to validate the WSDL for the Partner Link. The issue here is that as part of the deployment process we are “over-writing” the old stub version of the process, with a new version (as we want to keep the version number the same). The way BPEL PM achieves this, is to un-deploy the old version of the process, before deploying the new version. As part of the deployment, the BPEL engine validates the WSDL for all Partner Links including our recently un-deployed version of the process and as a result fails! Fortunately there is a simple work around. First create and deploy the sub process as before, then once deployed go to the BPEL Console and select the process from the dashboard. Then select the WSDL tab and click on the URL for the WSDL Location. This will open a browser containing the WSDL for the BPEL Process, save the file to your local file system (File-> Save As in IE) as .wsdl file. Now when you create your Partner Link in the process, instead of using the ‘Service Browser’ use ‘Browse WSDL files from your Local File System’ and select the WSDL file you just saved. Note: When prompted to make a local copy specify ‘yes’. From here you will be able to implement and deploy your BPEL process without any problems. Example I’ve created a simple example process, based of course on the classic Factorial example. You can download this here. Deployment Considerations The one drawback with this approach is that the WSDL file will now contain the Endpoint location for the service within in it. Thus is you were to deploy the process on a different server it would fail at run-time. So you need to modify the WSDL file at deployment time so that the endpoint reflects the hostname and port number of where the process is actually being deployed. The simplest way to do this is update your build script so that ant will automatically do this for you. Final Thought Now in theory you could have incredibly nested processes using this approach, however I would advise is bad practice and is likely to have performance implications. For example; if we ran the Factorial process to work out 50 Factorial, that would result in 50 process instances. Now if we expected to handle 1000 process running in parallel, this would result in 50,000 process instances – so the actual number of process instance could increase very dramatically. So I would recommend using this approach with caution to ensure that it doesn’t result in a dramatic increase in the overall number of process instances.