Sunday, December 16, 2012

ResourceLimitException : No resources currently available in pool to allocate to applications – Oracle SOA 11g

You could have observed the below exception frequently in the weblogic server log file, weblogic server throws the below exception when there is not enough database connections in the pool to serve the incoming requests reporting increasing the connection pool count. This will affect the overall system  performance

java.sql.SQLException: Internal error: Cannot obtain XAConnection weblogic.common.resourcepool.ResourceLimitException: No resources currently available in pool SOADataSource to allocate to applications, please increase the size of the pool and retry..
at weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:577)
at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:342)
at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:329)
at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:417)
at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:324)


The above exception is due to there is not enough connections available in the pool to service the requests.
To identify which datasource / connection pool requires more connection do the following.
  • Go to Weblogic Admin console
  • Go to Environment -> Servers
  • Select the server on which the connection pool needs to be monitored
  • Go to Monitoring -> JDBC
  • Customize the table to add the below two properties
This page will list all the datasources/ connection pools targeted to that particular server and how many connections are waiting to get the connection. If more number of connections are waiting, that is not a good which will cause performance impact and also cause stuck threads on the server if the wait time is going to be beyond the thread stuck time (By default it's 10 mins).

(This screen shot taken by reducing the max capacity to 5 for blogging purpose, unfortunately I did not captured the actual screen shot)

Above screenshot shows SOADataSource state is "Overloaded" and also the current wait is 19. There are 19 resources waiting for connection . The above setup is definitely a problem which will result database calls performance issue.

If the connection pool count is not increased with respect to the invoker thread count, then it will cause this kind of contention cause lot of calls to be waiting for connection.


Solution to the above problem is to increase the number of connections in the connection pool.

Steps:
  • Go to WLS Console
  • Select Services -> Datasources -> select the datasource you want to change the pool size
  • Go to Configuration -> connection pool
  • Change the Maximum Capacity to the count which is required for your environment.

########

Friday, December 14, 2012

DB Adapter Message Throttling in Oracle SOA 11g

DB throttling is the mechanism to control the number of database records processed by the SOA engine in a particular interval through DB Adapter. 
Throttling also can be used to control the number of records send to the end systems. If the throttling is not defined, the end systems may flood with number of messages that will affect the functioning of the end systems. Throttling parameters should be configured based on the end systems capacity to process the incoming messages.
Until Oracle SOA 11.1.1.6.0 the message throttling will not work as we expect, all the messages matching the where condition of the polling sql are processed in the same polling interval.
As of Oracle Adapters release 11.1.1.6.0 we can set the inbound DBAdapter property RowsPerPollingInterval to control the throttling. It acts as a limit on the number of records which can be processed in one polling interval. The default value is unlimited.
The Patch 12881289 should be applied to enable this for SOA 11.1.1.5.0 and earlier versions.
The maximum rows processed per second are:
Number of active nodes in SOA cluster x NumberOfThreads x RowsPerPollingInterval / PollingInterval
MaxTransactionSize can be thought of as RowsPerDatabaseTransaction or DatabaseFetchSize that is how many records will be fetched to DB Adapter engine from the database for each transaction. It does not affect how many rows can be processed in one polling interval period.
The one exception is the following configuration:
-distributed polling checked, usesSkipLocking="false"


In this one case RowsPerPollingInterval will default to MaxTransactionSize instead of unlimited in case of RowsPerPollingInterval property is not specified in the jca file.
If RowsPerPollingInterval is set to lower than MaxTransactionSize or MaxRaiseSize, they will be effectively lowered to RowsPerPollingInterval that means the value RowsPerPollingInterval will considered for MaxTransactionSize and MaxRaiseSize.
There is no UI support to add the RowsPerPollingInterval property. We have to add the property manually to the db.jca file for the inbound polling service. Add it to the same section as the properties MaxRaiseSize, MaxTransactionSize, and PollingInterval, in any order.

########