Correlation Illustrated

Correlation This post would effort to explore the meaning and need of correlating requests and the best way to do them using Oracle SOA Suite. What & Why In common business scenarios it is not possible to have a synchronous response to each of our requests. Mostly the business has to deal with vendors, who would need to undergo a detailed process to come up with a solution. However, this response may not be necessarily binding business from carrying on its other activities. In such scenarios, the business enters in to asynchronous communication with these third parties. The async process necessarily stresses that the response may come back at a later time, through callback or polling mechanism as is suitable. In case of a single instance its quite easy to match the response to request, however, when there are multiple instances the business would be wondering which request has been responded to. For example, if a travel booking agent spawns 10 requests for Frankfurt to Paris, and after a while airline responds to one of those 10 requests, it would not be safe to assume that this is a response to first request only. Hence the travel agent would tag in an ID to segregate one request for another, and would expect the airline to return this id unaltered to match the request. This id, which enables requestor to correlate the response sent by the provider, is called Correlation Id. How (in Oracle BPEL) Oracle BPEL provides us with two mechanisms through which an asynchronous callback message could map to its calling instance in the crowd. WS-Addressing Correlation Sets WS-Addressing “How does the callback maps to the correct instance out of so many waiting processes”, it’s a very common surprise to all beginners, creating first ever Asynchronous Invoke in BPEL Process Manager. The answer is, any time an asynchronous service is invoked, Oracle BPM, instills a message UID in WS-Addressing headers that float with SOAP Packets, which are used internally by the BPEL Server to correlate the flowing request/response. This is the default mechanism supported by Oracle BPEL Process Manager, which is overridden automatically when the user opts to create his own correlation set. Correlation Sets Correlation Sets enable you to correlate asynchronous messages based on message body contents. Why? The question now arises, why ever would a BPEL developer like to over-ride default WS-Addressing correlation and use content-based correlation. Well, he won’t have to; if the called service is an Oracle BPEL, or the one, which supports and retains WS-Addressing headers sent, and returns the same in response. The reasons, to use content-based correlation, could be many; some of the familiar ones that come to my mind are – 1. The asynchronous web service invoked by BPEL does not support/understand WS-Addressing headers. 2. When BPEL instance is expecting messages from a third party, for example, a file-poller. 3. When asynchronous call has multiple hops and reply is received directly from end system. For example, Request path is A > B > C > D and response comes to A directly from D, i.e. D > A. How? WE now understand and appreciate, need of correlation and the two mechanisms to achieve correlation. In WS-Addressing, we, as in developer, does not bother of how the correlation ID is created and maintained. However, in content-based correlation the responsibility of managing correlation set, defining property alias and initializing the correlation with appropriate value falls on developer. To understand how to perform these tasks, we would start by understanding in which activities do we need correlation. The Correlation is required whenever data is going out or coming in to the BPEL process, hence Correlation works only with Invoke, Receive, Pick and Reply activities. To create a correlation of our own, we need to create the Correlation Set first. A correlation set is a set of properties shared by all messages in the correlated group. The correlation set is normally instantiated on invoke or receive activity. When correlation is instantiated on receive activity it is fairly straight forward, you create a correlation set, and select initiate = “yes”. However, when you have to create correlation set on invoke, we have one addition option to understand, it’s the “pattern”. Now before we move ahead with “pattern”, lets explore what is “initiate” option. If initiate option is set to “yes” this means that we assign the value to correlation variable. When initiate option is set to “no” this means, we want to validate the value against, value stored in correlation variable. With this in mind, we would try to understand, the “pattern” option of Correlation tab in invoke activity. When initiate = “yes”, Pattern=”in” means the correlation is being instantiated with a value that is coming into the BPEL, from another service. Clearly, this invoke is then not the one which is invoking an asynchronous process, as the pattern suggests a response coming back. Pattern = “out” means the correlation is being instantiated on the message that is going out of our BPEL. Pattern = “out-in” means the correlation would be set on a variable that is part of both outgoing and incoming parameter, in this synchronous invoke. However, its nature is very much questionable. When initiate=”no” The above patterns would reflect which way the variable needs to be validated, inward, outward or both ways. Creating a Correlation Set  Correlation sets uses correlation tokens available inside messages, headers or business documents. Correlation tokens are the declarative properties of messages. A property is an element within a message identified by a query. Property aliases are the constructs to specify the queries to identify properties. To create Correlation Set click the Create icon, and insert the appropriate values. As shown in the diagram below: Here you would also need to create a Property (not shown in diagram above, its just beneath Xpath text box. This way we declared one of the property alias for instantiation, we need to create another one for validation. Select the BPEL in diagram, in the adjacent Structure window, you would see set of properties. Right Click Property Alias, and select “Create Property Alias…”. Now, select the appropriate message part, and then enter Xpath query string in the text box. You can use Ctrl+Space for Autofill. Now, select the appropriate message part, and then enter Xpath query string in the text box. You can use Ctrl+Space for Autofill. References : Antony Reynold's Correlation Antony Reynold's on Pattern and Initiate

Oracle BPEL 11g - Correlation Sets

Correlation Sets is as set of configurations that allow the process to receive messages during its execution even if the invoker doesn't have the Correlation ID necessary to indicate to the engine to which instance the message needs to be delivered. With Correlation Sets a group of business data can be used to inform the engine how to match the message received with the correct instance that is already in execution. To understand correlation sets is necessary to know some main concepts: Property: like a variable, a property stores the information that will be used to match when a new message arrives. Property Alias: is used in two situations - as a assign, to define what information needs to be stored in the property, and as a query, to match the information received in a message with the information stored in the property. Itens below is a step-by-step showing how to configure Correlation Sets: 1 - Create a new operation in the process interface, allowing it to receive a message during process execution Open the WSDL that describes BPEL service interface. Find messages and operations already defined. Create a new message and a new operation in the inbound PortType that uses the message created. You can use some data type already defined, define a new data type at the XSD file already imported by the WSDL or import any XSD to use a data type defined externally. Verify that the new operation is available at the process interface. 2 - Change the process to use the new operation during the process execution Open the BPEL process and add a new Receive activity in the middle of process execution. Before and after this Receive you may have as many activities as necessary. Point the new activity to the PartnerLink that represents the process interface. Select the operation defined previously on WSDL and create a new variable to store the message that will be received. 3 - Create and initialize the Correlation Set with business value that will be used to match the message that will be received Open the Receive activity that starts the process. Select the "Correlation" tab and click on "Create" button. Important: to initialize the Correlation Set you could use any Invoke or Receive activity before the Receive activity you will receive the message that you want to match. Define Correlation Set name and click on "Add" button. Choose the "input" field from "InputVariable". Click on "+" button right to "Property" field and define the name of the property that will store the input field value. Click "OK" once to confirm Property creation, again to confirm Property Alias creation and one more time to confirm Correlation Set creation. Change "Initiate" field to "yes". This will indicate to the engine that you want to initilize the Property with the field content defined by the XPath expression of the Property Alias. 4 - Match the incoming message Open the Receive activity that will receive the intermediate message. Select the "Correlation" tab and click on "Add" button. Select the Correlation Set previously created and click on "OK". Select the Correlation Set added and click on "Edit" button. Select the Property of the Correlation Set an click again on "Edit" button. Choose the "input" field from Intermediate Receive's variable and click "OK". Then, click "OK" again to confirm the Correlation Set change. This will create a new Property Alias that matches intermediate message value received with the value stored in the Property initialized previously. This time, leave the "Initiate" field as default ("no"), because we don't want to change the property value. We just want to match its value with the value received by the Intermediate operation message. With this configuration, BPEL engine can match business value stored in the property (initialized in the first Receive activity) with the the message value received by the Intermediate Operation and deliver the message for the correct running instance. To download this sample, click here.

BPEL - Correlation, fault handling, and compensation

The previous entries have covered the fundamentals of BPEL, providing an understanding of the activities defined and how they can be combined together using structured activities and the <link> construct. In this entry, the advanced properties of the language that are essential to the definition and execution of a business process are covered. BPEL uses correlation to match returning or known customers with a long-running business process, fault handling to recover from expected as well as unexpected faults, and compensation to "undo" already committed steps in case something goes wrong in the middle of the process or, for example, a client wishes to explicitly cancel a transaction. Introduction Now that I've covered the fundamental concepts of BPEL in the previous entries, it is time to dive into some of the more advanced topics. This entrt will provide you with an overview of message correlation, fault handling, and compensation as it applies to a BPEL environment. Message correlation Message correlation is the BPEL mechanism which allows processes to participate in stateful conversations. It can be used, for example, to match returning or known customers to long-running business processes. When a message arrives for a Web service which has been implemented using BPEL, that message must be delivered somewhere -- either to a new or an existing instance of the process. The task of determining to which conversation a message belongs, in BPEL's case the task of locating/instantiating the instance, is what message correlation is all about. In many distributed object systems, one component of the routing of a message involves examining the message for an explicit instance ID which identifies the destination. Although the routing process is similar, BPEL instances are not identified by an explicit instance field, but are instead identified by one or more sets of key data fields within the exchanged messages. For example, an order number may be used to identify a particular instance of a process within an order fulfillment system. In BPEL terms, these collections of data fields which identify the process instance are known as correlation sets. Each BPEL correlation set has a name associated with it, and is composed of WSDL-defined properties. A property is a named, typed data element which is defined within a WSDL document, and whose value is extracted from an instance of a WSDL message by applying a message-specific XPath expression. In WSDL, a propertyAlias defines each such mapping. The mappings are message specific, hence a single property can have multiple propertyAliases associated with it. For example, a WSDL document might say that property name corresponds to part username of WSDL message loginmsg and to part lastname of ordermsg. Together, the properties and propertyAliases provide BPEL authors with a way to reference a single, logical piece of information in a consistent way, even if it might appear in different forms across a set of messages. Using correlation sets To use a correlation set, a BPEL author defines the set by enumerating the properties which comprise it, and then references that set from receive, reply, invoke, or pick activities. A BPEL runtime uses the definition and references to determine the interesting pieces of information that it should examine during different points of executing the process. Each instance of the process has an instantiation of each of the correlation sets which are defined for the process. Each of these instantiations is initialized exactly once during the execution of the process instance, and is subsequently only used during comparisons involving incoming and outgoing messages. If an attempt is made to reinitialize a correlation set, or to use one which has not been initialized, then the runtime will throw a bpws:correlationViolation. Extracting values for a correlation set referenced by an activity, either for initialization or comparison purposes, involves applying the processAlias, which corresponds to the particular WSDL message being examined, for each of the properties which comprise the referenced correlation set. As receive and pick activities provide the entry points into a process, correlation sets often appear on them to enable message-to-instance routing. If a correlation set appears on a receive or pick activity, and it does not have the initiation attribute set to 'yes', then when a message arrives for that particular receive or pick, the values of the properties which comprise the correlation set are extracted from the incoming message and compared to the values stored for the same correlation set in all of the instances of that process, and the message is routed to the instance which has matching values. Similarly, correlation sets on reply and invoke activities, which deal with outbound operations, are often used to validate that outgoing messages contain data which is consistent with the data contained within specified correlation set instances. For example, if a correlation set appears on a reply activity, and it does not have the initiation attribute set to 'yes', then the values of the properties which comprise the correlation set are extracted from the outgoing message and compared to the values store for the instance's instantiated correlation set. If the values in the message which correspond to the correlation set are found to be different from those contained within the instance's correlation set, the runtime throws a bpws:correlationViolation. On all of the activities on which correlation sets may appear, the initiation attribute indicates to the runtime that the correlation set should be initialized using the values contained within the message which is input to the current activity. Correlation sets which appear within invoke activities have an additional attribute, pattern, which specifies during the execution of the invoke activity the correlation set that is to be applied. The pattern attribute may have a value of 'in', meaning that the correlation set is to be applied when the response comes back for the invocation, 'out', meaning that the correlation set is to be applied when the invocation is made, or 'both', which naturally means that it is to be applied during both phases. Multiple correlation sets, some of which are initialized and some of which are used for comparison, can appear on a single activity. The current BPEL specification does not define the semantics of locating an instance based on multiple correlation sets. Error handling and roll-back When executing a BPEL process, errors might occur either in the services being invoked or within the process itself. BPEL provides a mechanism to explicitly catch such errors and handle them by executing subroutines specified in fault handler elements. Additionally, activities that have completed might later need to be undone because they form part of a longer transaction that had to be aborted. Compensation handlers allow the creator of the process to define certain actions that should be taken to undo a unit of work and restore the data to what it was before that work was done. The handling of such situations usually affects a set of activities that are associated with each other. In BPEL, this is done by enclosing them in the scope structured activity. A scope provides the context for the activities nested within it, and it is where fault and compensation handlers are defined. Therefore, you can think of the scope as encapsulating a possibly compensatable, recoverable unit of work. The entire process provides the global scope: it contains one main activity and allows the specification of fault and compensation handlers. It is also able to define data containers and correlation sets. However, the future directions of BPEL state that all scopes will be able to do the same, and that containers defined in a scope will be visible only to activities nested within that scope. In order to understand how a scope catches or propagates a fault that originated within it, I will first recap the life cycle of a BPEL activity. An activity remains dormant, waiting to be activated by its parent activity and its incoming links. Once it gets control from its parent, and all its incoming links come in, it can evaluate its join condition at which point one of two things can happen: it either throws a joinFailure fault because the join condition was not satisfied, or it runs. Assuming the latter, after it has run successfully, it ends by evaluating and firing all its outgoing links. On the other hand, the activity might fault while it is running because it was an invocation that returned a fault, one of the built-in BPEL faults occurred while it was executing, or because it is a throw activity. In this case, it notifies the scope it belongs to. Upon receiving a fault, the scope must stop all its nested activities. One other reason can cause an activity not to run successfully; it might never receive all its links or control from its parent. For example, this will happen to an activity in the same scope as the one that faulted but that was slated to run later on, or to an activity that is a branch of a switch statement that was never taken. In both of these cases where an activity cannot successfully run and complete, it needs to send out all its outgoing links with a negative value. Handling faults BPEL includes the ability to catch and handle errors at different levels of activity nesting. A fault handler can be defined on any scope and either bound to a particular kind of fault (defined by the fault's qualified name, or its message type) or to any faults not caught by a more specific handler. A handler simply contains an activity that will run in case an error occurs. For example, it might contain a reply activity that notifies a partner that an error has occurred. As noted earlier, the first thing a scope does once it receives a fault is to stop all its nested activities. In the default case, the scope stops its activities, and then rethrows the fault to its parent and so on until the top-level process is reached. When a scope rethrows the fault, the scope itself then ends abnormally, and it sends out all its outgoing links with a negative value. However, handlers allow any scope to intercept a fault and take appropriate action. Once a fault is handled in a scope, the scope ends normally with the values of its outgoing links being evaluated as usual. This is illustrated in Figure 1. Figure 1. Handling faults In this figure, the activity faults because one of its links came in with a negative value and its join condition is a boolean and of the link values. It therefore throws the fault to its scope, and you see all the links leaving it go negative. The scope itself has no handler for the fault, so it rethrows the fault. The next scope up is the process itself. You see the links of the inner scope get negative values because it could not take care of the problem itself and so terminated abnormally. The process has a handler, takes care of the fault, and completes normally. It follows that if the handler had been on the inner scope, the links leaving it would have been evaluated regularly, and the fault would not have been thrown to the process. The invoke activity exhibits a short-cut mechanism for defining handlers directly on its definition. The behavior is effectively that it becomes wrapped in a scope that will have these specified handlers. BPEL defines a set of built-in faults that notify of errors such as assignment type mismatches, a scope forcing the termination of a nested scope, and activating a reply activity that doesn't have a matching receive activity. The BPEL engine is responsible for detecting these faults and throwing them to the appropriate scope. Compensation While a business process is running, it might be necessary to undo one of the steps that have already been successfully completed. The specification of these undo steps are defined using compensation handlers that can be defined at the scope level. Each handler contains one activity which is run when a scope needs to be compensated. The activities within such a handler must see the container data to be that which it was when the scope completed. Due to activities sharing containers and loops caused by while activities, completing scopes that are compensation-enabled must save a snapshot of the data for the handler to possibly use later. Once a scope completes successfully, its compensation handler becomes ready to run. This can happen in either of two cases: explicit or implicit compensation. Explicit compensation occurs upon the execution of a compensate activity. This activity may occur anywhere, and refers to the name of the scope that it wants compensated. Scopes that might be rolled back in this manner must therefore be named. BPEL further specifies that names of scopes must be unique within a BPEL process. When a compensate activity is reached, it runs the compensation handler on the specified scope. On the other hand, implicit compensation occurs when faults are being handled and propagated. Consider the scenario in which a scope A contains a compensatable scope B that has completed normally, but then another nested activity in A throws a fault. Implicit compensation ensures that whatever happened in scope B gets undone by running its compensation handler. Therefore, implicit compensation of a scope goes through all its nested scopes and runs their compensation handlers in reverse order of completion of those scopes.

Correlation Sets

BPEL Correlation  Where Did I Put That Process? A BPEL process is initiated and makes a call to an ERP system to raise a purchase order, generating a purchase order number.  Later that purchase order causes another system to raise an invoice and send the invoice to the BPEL process.  How does the BPEL engine know which BPEL process should receive this invoice and process it.  This is dealt with a thing called correlation. From e-mails and phone calls that I receive it appears a lot of people struggle with BPEL correlation.  It seems that the questions falls into two categories, why would I want it, and how do I do it? What is Correlation? Correlation is basicallly the process of matching an inbound message to the BPEL engine with a specific process.  Normally this matching is hidden from us.  Synchronous calls have no need of correlation because the conversation context is maintained on the stack or across a TCP connection.  Consenting BPEL processes will usually correlate messages using WS-Addressing headers to pass around magic tokens that act like the session cookies in a web application. Why Do I Need Worry About It? Well most of the time you don't!  As I mentioned before, calling another BPEL process or using a synchronous call will mean you don't have to think about it.  You do need to worry about it in the following situations amongst others. When using an asynchronous service that doesn't support WS-Addressing When receiving unsolicited messages from another system When communicating via files In these casess we need to be able to tell BPEL to look at some content of the message in order to select the correct process instance to receive the message. How Do I Get the Right Process Instance? BPEL provides a construct called a correlation set to allow for custom correlation.  A correlation set is a collection of properties used by the BPEL engine to identify the correct process to receive a message.  Each property in the correlation set may be mapped to an element in one or more message types through property aliases as shown below. Things to Remember About Correlation I see some common misunderstandings about custom correlation.  So lets knock them off now. Only the process receiving the messsage needs to worry about correlation <BR/>As long as the sending service includes sufficient information in the message to be able to correlate it with previous activities there is no need for the sender to even be aware that correlation is occuring. Correlation properties must be unique for the duration of the life of the BPEL process that set them. <BR/>Make sure that you can't have two processes working with the same correlation tokens, for example using social security numbers to correlate an expense claims process would be a bad idea if an individual could kick off two seperate instances of the process. Properties can be made up values or actual business identifiers such as purchase orders or numbers.  They don't have to be strings, they can be any reasonable XML type. A Quick Guide to Custom Correlation Enough of the theory, how does it work in practice?  Consider a process A that call a process B that calls a process C that calls a process A.  This is one of the scenarios (113) in the BPEL samples distributed with Oracle BPEL PM. So we have a cycle A->B->C->A.  Three different asynch calls. Note only process A needs correlation because only A receives more than one call. On the invoke from A to B we add a correlation in the correlation tab for the invoke using BPEL Designer.  In here we will create the correlation set and create a property to correlate the exchange.  We set this to initiate the correlation, meaning that it will associate this process with the given value. On the receive from C to A we add the same correlation set with its property as we did for the invoke from A to B.  However this time we mark the receive as not to initiate the correlation, meaning that the BPEL PM will use this to select the right process instance. We now go to the BPEL structure diagram in the BEL Designer and add the property alias.  We create two property aliases maping to appropriate elements in each message that will have the same value in message from A to B, as in the message from C to A.  Note that the elements can be different names and in different structures in the two messages, but they must contain the same value if correlation is to work. At this point BPEL designer has done almost everything.  We need to manually edit the bpel.xml file and add the following XMl fragment to each partner link that will participate in the correlation. <property name="correlation">correlationSet</property> Note that " correlationSet" is a fixed value. I have uploaded a sample of this process.  Note deploying it may be tricky due to circular dependencies.  How to deploy it is left as an exercise to the reader, but if the worst comes to the worst deploy an empty version of the process B, then deploy process A, then process C and then the real process B.

BPEL Correlation

Where Did I Put That Process? A BPEL process is initiated and makes a call to an ERP system to raise a purchase order, generating a purchase order number. Later that purchase order causes another system to raise an invoice and send the invoice to the BPEL process. How does the BPEL engine know which BPEL process should receive this invoice and process it. This is dealt with a thing called correlation. From e-mails and phone calls that I receive it appears a lot of people struggle with BPEL correlation. It seems that the questions falls into two categories, why would I want it, and how do I do it? What is Correlation? Correlation is basicallly the process of matching an inbound message to the BPEL engine with a specific process. Normally this matching is hidden from us. Synchronous calls have no need of correlation because the conversation context is maintained on the stack or across a TCP connection. Consenting BPEL processes will usually correlate messages using WS-Addressing headers to pass around magic tokens that act like the session cookies in a web application. Why Do I Need Worry About It? Well most of the time you don't! As I mentioned before, calling another BPEL process or using a synchronous call will mean you don't have to think about it. You do need to worry about it in the following situations amongst others. When using an asynchronous service that doesn't support WS-Addressing When receiving unsolicited messages from another system When communicating via files In these casess we need to be able to tell BPEL to look at some content of the message in order to select the correct process instance to receive the message. How Do I Get the Right Process Instance? BPEL provides a construct called a correlation set to allow for custom correlation. A correlation set is a collection of properties used by the BPEL engine to identify the correct process to receive a message. Each property in the correlation set may be mapped to an element in one or more message types through property aliases as shown below. Things to Remember About Correlation I see some common misunderstandings about custom correlation. So lets knock them off now. Only the process receiving the messsage needs to worry about correlation. As long as the sending service includes sufficient information in the message to be able to correlate it with previous activities there is no need for the sender to even be aware that correlation is occuring. Correlation properties must be unique for the duration of the life of the BPEL process that set them. Make sure that you can't have two processes working with the same correlation tokens, for example using social security numbers to correlate an expense claims process would be a bad idea if an individual could kick off two seperate instances of the process. Properties can be made up values or actual business identifiers such as purchase orders or numbers. They don't have to be strings, they can be any reasonable XML type. A Quick Guide to Custom Correlation Enough of the theory, how does it work in practice? Consider a process A that call a process B that calls a process C that calls a process A. This is one of the scenarios (113) in the BPEL samples distributed with Oracle BPEL PM. So we have a cycle A->B->C->A. Three different asynch calls. Note only process A needs correlation because only A receives more than one call. On the invoke from A to B we add a correlation in the correlation tab for the invoke using BPEL Designer. In here we will create the correlation set and create a property to correlate the exchange. We set this to initiate the correlation, meaning that it will associate this process with the given value. On the receive from C to A we add the same correlation set with its property as we did for the invoke from A to B. However this time we mark the receive as not to initiate the correlation, meaning that the BPEL PM will use this to select the right process instance. We now go to the BPEL structure diagram in the BEL Designer and add the property alias. We create two property aliases maping to appropriate elements in each message that will have the same value in message from A to B, as in the message from C to A. Note that the elements can be different names and in different structures in the two messages, but they must contain the same value if correlation is to work. At this point BPEL designer has done almost everything. We need to manually edit the bpel.xml file and add the following XMl fragment to each partner link that will participate in the correlation. Note that "correlationSet" is a fixed value. I have uploaded a sample of this process. Note deploying it may be tricky due to circular dependencies. How to deploy it is left as an exercise to the reader, but if the worst comes to the worst deploy an empty version of the process B, then deploy process A, then process C and then the real process B. Useful References Here are some useful references on correlation. Oracle BPEL PM Tutorial 13 : Using BPEL correlation sets – File Adapter mid-process receive Oracle BPEL PM Technical Note #006: Correlation of Adapter messages within BPEL Process IBMTutorial on BPEL4WS