Oracle SOA 11g - Important SQL queries to get the details about composites from SOAINFRA schema

Query to find the execution time of the BPEL instances:- SELECT * FROM (SELECT COMPOSITE_NAME COMPOSITENAME,A.CMPST_ID COMPOSITE_INSTANCE_ID,creation_date BEGIN_TIME,modify_date END_TIME ,(extract(day from modify_date - creation_date)*86400+ extract(hour from modify_date - creation_date)*3600+ extract(minute from modify_date - creation_date)*60+ extract(second from modify_date - creation_date)) duration_in_second,A.* FROM cube_instance A where state = 5 and CREATION_date BETWEEN TO_DATE('19/01/2012 00:58:00','dd/mm/yyyy HH24:MI:SS')AND TO_DATE('19/01/2012 23:59:59','dd/mm/yyyy HH24:MI:SS') AND COMPOSITE_NAME IN (<<Composite Name>>)) ORDER BY COMPOSITE_NAME,duration_in_second DESC Query to find the execution time of the Mediator instances:- SELECT * FROM (SELECT COMPONENT_NAME COMPONENTNAME,A.COMPOSITE_INSTANCE_ID INSTANCE_ID,created_time BEGIN_TIME,updated_time END_TIME ,(extract(day from updated_time - created_time)*86400+ extract(hour from updated_time - created_time)*3600+ extract(minute from updated_time - created_time)*60+ extract(second from updated_time - created_time)) duration_in_second,A.* FROM mediator_instance A where CREATED_TIME BETWEEN TO_DATE('30/01/2012 00:58:00','dd/mm/yyyy HH24:MI:SS')AND TO_DATE('31/01/2012 23:59:59','dd/mm/yyyy HH24:MI:SS') AND A.COMPONENT_NAME like '%<<Meditor Composite Name>>%') ORDER BY COMPONENT_NAME,duration_in_second DESC Query to find the count of Composite instances created on particular time:- BPEL instances:- select * from (select count(COMPONENT_NAME) count,COMPONENT_NAME from CUBE_INSTANCE where CREATION_DATE BETWEEN TO_DATE('30/01/2012 12:58:00','dd/mm/yyyy HH24:MI:SS')AND TO_DATE('31/01/2012 23:00:00','dd/mm/yyyy HH24:MI:SS') /*and component_state<5*/ group by COMPONENT_NAME ) order by count desc Mediator Instances:- select * from (select count(COMPONENT_NAME) count,COMPONENT_NAME from MEDIATOR_INSTANCE where CREATED_TIME BETWEEN TO_DATE('30/01/2012 12:58:00','dd/mm/yyyy HH24:MI:SS')AND TO_DATE('31/01/2012 23:00:00','dd/mm/yyyy HH24:MI:SS') group by COMPONENT_NAME ) order by count desc Getting the count of instances in different state for the BPEL Components:- select composite_name,count(*),DECODE(cube_instance.STATE, 0, 'STATE_INITIATED', 1, 'STATE_OPEN_RUNNING', 2, 'STATE_OPEN_SUSPENDED', 3, 'STATE_OPEN_FAULTED', 4, 'STATE_CLOSED_PENDING_CANCEL', 5, 'STATE_CLOSED_COMPLETED', 6, 'STATE_CLOSED_FAULTED', 7, 'STATE_CLOSED_CANCELLED', 8, 'STATE_CLOSED_ABORTED', 9, 'STATE_CLOSED_STALE', 10,'STATE_CLOSED_ROLLED_BACK','unknown') state from CUBE_INSTANCE group by state,composite_name order by composite_name; Getting the count of instances in different state for the Mediator Components:- select COMPONENT_NAME,count(*),DECODE(mediator_instance.COMPONENT_STATE, 0, 'STATE_CLOSED_COMPLETED', 2, 'STATE_OPEN_FAULTED', 4,'STATE_RECOVERY_REQUIRED', 8, 'STATE_OPEN_RUNNING', 16, 'STATE_CLOSED_STALE', 48, 'STATE_TERMINATED', 50, 'STATE_TERMINATED','unknown') STATE from mediator_instance group by component_state,COMPONENT_NAME order by COMPONENT_NAME; Querying the call back message status from dlv_message table:- Error to deliver the call back to the receive activity:- select count(*) from dlv_message where dlv_type = 2 and state=1 Callback arrived but did not get resolved:- select count(*) from dlv_message where dlv_type = 2 and state=0 Callback successfully delivered to receive:- select count(*) from dlv_message where dlv_type = 2 and state=2 Querying recoverable invoke messages :- select * from dlv_message where dlv_type = 1 and state = 0; To check for rejected messages for a specific composite: select count(*) from rejected_message where composite_dn like '%<<Composite Name>>%';

Oracle SOA 11g - Callback not reaching the calling service in a Clustered environment

In 10g, we had a scenario like shown in the below diagram. Process A Process A calls Process B and waits for the call back; after receiving the call back; The Process A calls again the same Process B and waits for the second call back. This flow is working fine in 10g but after migrating to 11g the second call back is not reaching the Process A. In the SOA log file we could able to see the below error message - an unhandled exception has been thrown in the Collaxa Cube systemr; exception reported is: "javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.1.3.v20110304-r9073): org.eclipse.persistence.exceptions.DatabaseException Internal Exception: java.sql.SQLIntegrityConstraintViolationException: ORA-00001: unique constraint (SOA_SOAINFRA.AT_PK) violated Error Code: 1 Call: INSERT INTO AUDIT_TRAIL (CIKEY, COUNT_ID, NUM_OF_EVENTS, BLOCK_USIZE, CI_PARTITION_DATE, BLOCK_CSIZE, BLOCK, LOG) VALUES (?, ?, ?, ?, ?, ?, ?, ?) bind => [851450, 5, 0, 118110, 2012-01-12 16:08:04.063, 5048, 3, [B@69157b01] Query: InsertObjectQuery(com.collaxa.cube.persistence.dto.AuditTrail@494ca627) at org.eclipse.persistence.internal.jpa.EntityManagerImpl.flush(EntityManagerImpl.java:744) After the detail analysis and the Oracle SR, the root cause is “Audit Trail caching issue is causing the process not to get to complete due to the exception while doing the second call back”. Let’s say in a Cluster with four nodes Process A first created on node 1 and carried on and called first call to Process B and dehydrates. Assume that second call hit a node other than node 1, let’s say node 4. Node 4 carries on doing the work and does the second call to Process B and dehydrates. If the second call back hits Node 1, rather than getting the latest audit trail from database, it’s using the cache and trying to update the audit trial with an id which node 4 already used. That’s throwing an exception and rolling back. For that reason the call back is not getting delivered. As a temporary solution by switching of the audit trail for Process A, it doesn’t try to insert the data into audit trail and the transaction won’t get roll back and the call has reached successfully.

Oracle SOA 11g – Skip locking for DBAdapter connection factories in clustered environments

Skip Locking is the new property introduced in 11g for the DBAdapter Connection factories. “In a distributed scenario, each polling instance will try to balance the load by not greedily attempting to process all unprocessed rows by itself. What that means is that at a time, an instance will only fetch at most MaxTransactionSize rows”. When using skip locking, if a full MaxTransactionSize rows are fetched, the next MaxTransactionSize rows can be immediately fetched continuously. This is because concurrent threads do no block each other when using skip locking. The Skip Locking offers performance benefits. However, with skip locking disabled, all threads will try to lock the same rows, and only one will succeed. Consequently, once this thread has processed MaxTransactionSize rows, it will pause until the next polling interval, to allow other threads to also lock and process rows. If we want to limit the number of records processed(MaxTransactionSize) in a particular polling interval then the SkipLocking should be disabled, the maximum throughput for each node with distributed polling enabled but uses SkipLocking disabled is: NumberOfThreads x MaxTransactionSize/PollingInterval

Persistence Properties in Oracle SOA 11g

BPEL Persistence properties are used to control, when and how a process need to be dehydrated. inMemoryOptimization This property indicates to Oracle BPEL Server that this process is a transient process and dehydration of the instance is not required. When set to true, Oracle BPEL Server keeps the instances of this process in memory only during the course of execution. This property can only be set to true for transient processes (process type does not incur any intermediate dehydration points during execution). false (default): instances are persisted completely and recorded in the dehydration store database for a synchronous BPEL process. true: Oracle BPEL Process Manager keeps instances in memory only. completionPersistPolicy This property controls if and when to persist instances. If an instance is not saved, it does not appear in Oracle BPEL Console. This property is applicable to transient BPEL processes (process type does not incur any intermediate dehydration points during execution). This property is only used when inMemoryOptimization is set to true. This parameter strongly impacts the amount of data stored in the database (in particular, the cube_instance, cube_scope, and work_item tables). It can also impact throughput. on (default): The completed instance is saved normally. deferred: The completed instance is saved, but with a different thread and in another transaction, If a server fails, some instances may not be saved. faulted: Only the faulted instances are saved. off: No instances of this process are saved. <component name="mySampleBPELComponent"> ... <property name="bpel.config.completionPersistPolicy">faulted</property> <property name="bpel.config.inMemoryOptimization">true</property> ... </component> oneWayDeliveryPolicy The oneWayDeliveryPolicy is from the Oracle 10g configuration property deliveryPersistencePolicy.The new configuration property name is bpel.config.oneWayDeliveryPolicy. This property controls database persistence of messages entering Oracle BPEL Server. Its used when we need to have a sync-type call based on a one way operation. This is mainly used when we need to make an adapter synchronous to the BPEL Process. By default, incoming requests are saved in the following delivery service database tables: dlv_message async.persist: Messages are persisted in the database. sync.cache: Messages are stored in memory. sync: Direct invocation occurs on the same thread. <component name="mySampleBPELProcess"> ... <property name="bpel.config.transaction" >required</property> <property name="bpel.config.oneWayDeliveryPolicy">sync</property> ...

Increasing the performance of EM console in Oracle SOA 11g

EM console is very slow when accessing to view the composite/component details or the instance details. We can try to improve the performance by little by following the below steps. 1. Login to the SOA 11g EM Console 2. Right Click on SOA-INFRA and go to SOA Administration->Common properties 3. Set the Data Display Options as shown below.               Select the option “Disable fetching of instance and fault count metrics. Each metric can still be  retrieved on demand”. Set the Duration Details to 1 hour (based on your requirement) as shown below. This will enable to fetch the instance and fault count metrics based on the demand also the default search criteria will display the last one hour data, this will improve the performance of the em console. Enable fmwctl discovery cache:- Logging into Enterprise Manager Fusion Middleware Control 11g (fmwctl) takes a long time.  Sometimes as long as 45-60 seconds.  This is often viewed as slow response time, performance or hanging issues, fmwctl discovery is always performed as part of login.   For installations with thousands of targets, fmwctl discovery may take 45-60 seconds. This delay is expected because EM discovery mbeans need to be invoked for every target. Solution is to cache the discovery results in the servlet context and use it for subsequent logins. This discovery result will be shared by all the fmwctl users. This will still require the entire discovery to be done at least once. Follow the metalink note 1423893.1 to enable the discovery caching. If the caching is enabled, fmwctl will use the cached discovery results and login to the em console will be fast.The default setting is "not use the cached results"

Audit Trail configurations in Oracle SOA 11G

Until 11.1.1.3, BPEL audit trails are saved to database in the same JTA transaction as the main transaction. The below are the some of the issues caused. Latency of saving audit trail to database is included as part of the overall latency of the main business transaction. When the main transaction is rolled back for whatever reason, the audit trail did not get saved, because audit trails are saved in the main transaction as well. Thus no trace of what had happened can be found on the BPEL Console (or EM Console in 11G). Since SOA 11.1.1.3, management of audit trail memory has been enhanced. The properties to configure the audit trail: auditStorePolicy Use this flag to choose the audit store strategy syncSingleWrite - would store audit data synchronously when it saves the cube instance, this is done as part of the cube instance transaction. This is the default value. And the behavior is the same as in the 10.1.3.x version. syncMultipleWrite - would store audit data synchronously using a separate local transaction. By "synchronously" it means the BPEL service engine is saving the audit trail in the same thread as the main transaction. However, it is doing it in a "separate local transaction". Because they are on the same thread, latency of saving audit trail is still included into overall latency of the main transaction. However, because they are on separate transactions, you can configure BPEL engine (using AuditFlushByteThreshold and AuditFlushEventThreshold) to flush out the audit trail from memory to database periodically regardless how long the main transaction would take. Moreover, having them on two separate transaction means the rollback of main transaction will NOT affect the audit trail transaction. That is, you will still see audit trail even if the main transaction rolls back. async - would store the audit data asynchronously using an in-memory queue and pool of audit threads. This is almost the same as "syncMultipleWrite", except that it is done not just in a separate transaction but also in a separate thread. The pro is the audit trail latency is NOT directly included in the latency of main transaction (but because they still share the computing resources and database, the latency is still indirectly related). The con is that because audit trails are being saved asynchronously, the audit trail may be out of sync from the main transaction (as the name 'async' implies). AuditFlushByteThreshold and AuditFlushEventThreshold When auditStorePolicy=syncMultipleWrite or auditStorePolicy=async, you use these two flags to control how often the engine should flush the audit events. These two properties do NOT apply to auditStorePolicy=syncSingleWrite. auditFlushByteThreshold means after adding an event to the current batch, the engine would check if current batch byte size is greater than this value or not. if yes, then it would flush the current batch. The default value is 2048000 (byte), i.e. 2MB. Similarly, auditFlushEventThreshold means this limit is reached; the engine would trigger the store call. The default value is 300 (event) Both values need to be tuned based on the application and requirements. AuditDetailThreshold This is the maximum size (in bytes) an audit trail details string can be before it is stored separately from the audit trail. The default value is 50000 (byte). This is the same property as in 10.1.3.x. AuditLevel.this is the same property in 10.1.3.x. How to Configure All the above properties can be configured via the SOA EM Console. The path is EM -> SOA ->SOA-INFRA - > SOA Administration -> BPEL Properties -> More BPEL Configuration Properties

weblogic.transaction.internal.TimedOutException:Transaction timed out after 301 seconds – Oracle SOA 11g

Frequently the BPEL instances are getting rolled back with the following exception.  The root cause of the issue is, JTA transaction getting timeout before completing the execution of the BPEL instance; the JTA transaction should be active to complete the execution successfully. [2012-04-26T05:15:45.139+00:00] [SOA1] [ERROR] [] [oracle.soa.bpel.engine] [tid: orabpel.invoke.pool-4.thread-23] [userId: <anonymous>] [ecid: 0000JRh3heOAxGoqwSuXMG1F_qb^000aP5,0:1:100080683] [APP: soa-infra] [composite_name: AM_Invoker] [component_name: AM_Invoker] [component_instance_id: 5977508] The reason was The execution of this instance "5977508" for process "AM_Invoker" is supposed to be in an active jta transaction, the current transaction status is "MARKED_ROLLBACK" . Root cause: null  [APP: soa-infra] Error while invoking bean "cube delivery": JTA transaction is not in active state.[[ The transaction became inactive when executing activity "5977508-BpInv1-BpSeq9.78-2" for instance "5,977,508", bpel engine cannot proceed further without an active transaction. please debug the invoked subsystem on why the transaction is not in active status. the transaction status is "MARKED_ROLLBACK". Message handle error.error while attempting to process the message "com.collaxa.cube.engine.dispatch.message.invoke.InvokeInstanceMessage"; the reported exception is: Transaction Rolledback.: weblogic.transaction.internal.TimedOutException: Transaction timed out after 301 seconds BEA1-2980B7B48F411EE46991 Solution: Increase the JTA time out and the BPEL EJB transaction timeout. Change the JTA transaction timeout: Log in to Oracle WebLogic Server Administration Console. In the Domain Structure, select Services > JTA Increase the JTA transaction timeout value to some higher value like 3600 Change the BPEL EJB transaction timeout settings: Log in to Oracle WebLogic Server Administration Console. In the Domain Structure, click Deployments. Expand soa-infra > EJBs. Update the following EJBs transaction Timeout value to some higher value like 1200:  BPELActivityManagerBean  BPELEngineBean  BPELInstanceManagerBean  BPELProcessManagerBean  BPELServerManagerBean Click Save. Restart Oracle WebLogic Server. There is no standard value for transaction timeout property. We need to trail out and set the value. If the component still fails with the transaction timeout error then revisit the component design and tune the same.

Proxy Configuration in SOA 11g

If we are invoking any web services and it needs to be passed through Proxy then the proxy server details should be configured on soa11g server. In SOA11g we have two ways of configuring the proxy. The first way is to configure the Server Level and the second level is Composite level. Configuration can be done using Enterprise manager/JDeveloper 1) Server Level from EM console:  Login to the em console Navigate to SOA Infrastructure->SOA Administration->Common Properties -> More SOA Infra Advanced Configuration Properties... Enter the HttpProxyHost, HttpProxyPort, HttpProxyUsername and HttpProxyPassword Click on Apply and then restart Server   2) Composite service Level from EM console: If we want to configure the configure proxy only to a particular composite the can be configured in composite Level. Navigate to the Process in EM and in the dashboard click on the Service (Adapter used to invoke the web service to which the proxy is required) under the Services and References section. Click on the properties tab and you can find the different properties as shown below where you can fill the Proxyhost, ProxyPort. 3) Setting Composite service Level from JDeveloper: Open the Composite.xml and select the service to which the proxy configuration is required. Add the binding properties oracle.webservices.proxyHost, oracle.webservices.proxyPort Click on Save.

File/FTP Adapter Threading Model – Oracle SOA 11g

The Oracle File and FTP Adapters use the following threading models: Default Threading Model Modified Threading Model Default Threading Model: In the default threading model, a poller is created for each inbound Oracle File or FTP Adapter endpoint. The poller enqueues file metadata into an in-memory queue, which is processed by a global pool of processor threads. We can set the thread count of global processor through the oracle.tip.adapter.file.numProcessorThreads property at the pc.properties file. Create pc.properties file with this property and copy it to server and then reference it in the WLS classpath in setDomainEnv.sh. The following steps highlight the functioning of the default threading model: The poller periodically looks for files in the input directory. The interval at which the poller looks for files is specified using the PollingFrequency parameter in the inbound JCA file. For each new file that the poller detects in the configured inbound directory, the poller enqueues information such as file name, file directory, modified time, and file size into an internal in-memory queue. A global pool of processor worker threads waits to process from the in-memory queue. Processor worker threads pick up files from the internal queue, and perform the following actions: Stream the file content to an appropriate translator (for example, a translator for reading text, binary, XML, or opaque data.) Publishes the XML result from the translator to the SCA infrastructure. Performs the required post processing, such as deletion or archival after the file is processed. Modified Threading Model: We can modify the default threading behavior of Oracle File and FTP Adapters. Modifying the threading model results in a modified throttling behavior of Oracle File and FTP Adapters. The following sections describe the modified threading behavior of the Oracle File and FTP Adapters: Single Threaded Model Partitioned Threaded Model Single Threaded Model: The single threaded model is a modified threaded model that enables the poller to assume the role of a processor. The poller thread processes the files in the same thread. The global pool of processor threads is not used in this model. You can define the property for a single threaded model in the inbound JCA file as follows: <activation-spec className="oracle.tip.adapter.file.inbound.FileActivationSpec">    <property../>    <property name="SingleThreadModel" value="true"/>    <property../> </activation-spec> Partitioned Threaded Model: The partitioned threaded model is a modified threaded model in which the in-memory queue is partitioned and each composite application receives its own in-memory queue. The Oracle File and FTP Adapters are enabled to create their own processor threads rather than depend on the global pool of processor worker threads for processing the enqueued files. You can define the property for a partitioned model in the inbound JCA file as follows: <activation-spec className="oracle.tip.adapter.file.inbound.FileActivationSpec">   <property../>   <property name="ThreadCount" value="4"/>   <property../> </activation-spec> In the preceding example for defining the property for a partitioned model: If the ThreadCount property is set to 0, then the threading behavior is like that of the single threaded model. If the ThreadCount property is set to -1, then the global thread pool is used, as in the default threading model. The maximum value for the ThreadCount property is 40.

Deployment plan file to replace the JCA Adapters properties – Oracle SOA 11g

We can create and edit a configuration plan file in which we can replace the following attributes and properties:  Any composite, service component, reference, service, and binding properties in the SOA composite application file (composite.xml) Attribute values for bindings (for example, the location for binding.ws) schemaLocation attribute of an import in a WSDL file location attribute of an include in a WSDL file schemaLocation attribute of an include, import, and redefine in an XSD file Any properties in JCA adapter files The configuration plan does not alter XSLT artifacts in the SOA composite application. If you want to modify any XSL, do so in the XSLT Mapper. Using a configuration plan is not useful. For example, you cannot change references in XSL using the configuration plan file. Instead, they must be changed manually in the XSLT Mapper in Oracle JDeveloper when moving to and from test, development, and production environments. Generate the Plan file: The plan file can be generated through JDeveloper or through ANT script. Generating Plan file through JDeveloper: Right Click on Composite.xml and click on generate config Plan. Change the generated plan file contents accordingly to replace the properties and attributes of the composites. Change the wsdlAndSchema section accordingly to replace the WSDL location and the schemaLocation of the WSDL’s and XSD’s    <wsdlAndSchema name="ExplorePlanFile.wsdl|FileAdapter_file.jca|FileAdapter.wsdl|xsd/AA_table.xsd|xsd/ExplorePlanFile.xsd">       <searchReplace>          <search>http://localhost:7001/</search>          <replace>http://10.15.23.24:8004/</replace>       </searchReplace>    </wsdlAndSchema> By default the wsdlAndSchema section will include the jca files to replace the JCA file properties. The default configuration as shown below is not replacing the jca properties after the deployment of the composite. <wsdlAndSchema name="FileAdapter_file.jca|FileAdapter.wsdl|xsd/AA_table.xsd|xsd/ExplorePlanFile.xsd">       <searchReplace> <search>D://Albin/Files</search>   <replace>D://Albin/Files/Sample</replace>      </searchReplace>    </wsdlAndSchema> The same should be changed as shown below to replace the JCA adapter properties  File Adapter properties: <wsdlAndSchema name="FileAdapter_file.jca">       <jca:property name="PhysicalDirectory">          <searchReplace>             <search>D://Albin/Files</search>             <replace>D://Albin/Files/Sample</replace>          </searchReplace>       </jca:property>        <jca:property name="Append">          <searchReplace>             <search>false</search>             <replace>true</replace>          </searchReplace>       </jca:property>         </wsdlAndSchema> MQ Adapter properties: <wsdlAndSchema name="ENQ_GE_EI_EAI_TRIGGER_mq.jca|OrderStatusMonitorMQAdp_mq.jca">                 <jca:property name="QueueName">                           <searchReplace>             <search>S_Order_Queue</search>             <replace>P_Order_Queue</replace>         </searchReplace>         </jca:property>    </wsdlAndSchema> The same searchReplace can be used to replace all the JCA adapters’ properties. Attaching the plan file while deploying the composite from JDeveloper:   The plan file can be attached through ant script during the deployment of the composite to the server.     <ant antfile="${oracle.home}/bin/ant-sca-deploy.xml" inheritAll="false"              target="deploy">             <property name="wl_home" value="${wl_home}"/>             <property name="oracle.home" value="${oracle.home}"/>             <property name="serverURL" value="${soa.cluster.serverURL}"/>             <property name="user" value="${soa.user}"/>             <property name="password" value="${soa.password}"/>             <property name="overwrite" value="${soa.composite.overwrite}"/>             <property name="forceDefault" value="${soa.composite.forceDefault}"/>             <property name="sarLocation" value="${deploy.sarLocation}"/>             <property name="configplan" value="${deploy.configplan}"/>         </ant>

Getting the SOAP Envelope of the request/response messages in Oracle SOA 11g

The bpelx:inputHeaderVariable and bpelx:outputHeaderVariable attributes in invoke/receive activity will help us to add or retrieve the custom SOAP headers to SOAP message in oracle SOA. If you want to see the whole SOAP message (header & body) it’s not possible out-of-the-box on the EM Console or in the logs. In order to see these SOAP conversations for a SOA composite service you can follow below steps: Open the Composite.xml Attach the management policy named oracle/log_policy to the webservice reference to which the SOAP headers needs to be monitored.  Redeploy the composite. Check the below log file to view the SOAP messages whenever the webservice call is made.$DOMAIN_HOME/domain_name/servers/mserver_name/logs/owsm/msglogging/diagnostic.log

ResourceLimitException : No resources currently available in pool to allocate to applications – Oracle SOA 11g

You could have observed the below exception frequently in the weblogic server log file, weblogic server throws the below exception when there is not enough database connections in the pool to serve the incoming requests reporting increasing the connection pool count. This will affect the overall system  performance java.sql.SQLException: Internal error: Cannot obtain XAConnection weblogic.common.resourcepool.ResourceLimitException: No resources currently available in pool SOADataSource to allocate to applications, please increase the size of the pool and retry.. at weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:577) at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:342) at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:329) at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:417) at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:324) The above exception is due to there is not enough connections available in the pool to service the requests. To identify which datasource / connection pool requires more connection do the following. Go to Weblogic Admin console Go to Environment -> Servers Select the server on which the connection pool needs to be monitored Go to Monitoring -> JDBC Customize the table to add the below two properties This page will list all the datasources/ connection pools targeted to that particular server and how many connections are waiting to get the connection. If more number of connections are waiting, that is not a good which will cause performance impact and also cause stuck threads on the server if the wait time is going to be beyond the thread stuck time (By default it's 10 mins). (This screen shot taken by reducing the max capacity to 5 for blogging purpose, unfortunately I did not captured the actual screen shot) Above screenshot shows SOADataSource state is "Overloaded" and also the current wait is 19. There are 19 resources waiting for connection . The above setup is definitely a problem which will result database calls performance issue. If the connection pool count is not increased with respect to the invoker thread count, then it will cause this kind of contention cause lot of calls to be waiting for connection. Solution to the above problem is to increase the number of connections in the connection pool. Steps: Go to WLS Console Select Services -> Datasources -> select the datasource you want to change the pool size Go to Configuration -> connection pool Change the Maximum Capacity to the count which is required for your environment.