Weblogic10.3.6 JCA connection pool framework

Have you ever seen below error message in soa11g  server logs,if answer is yes, then there is a reason to stop and think the way you activate your JCA adapter JNDI deployments

error message:
Unable to execute allocateConnection(...) on ConnectionManager. A stale Connection Factory or Connection Handle may be used
Try to re-lookup Connection Factory-xxx  from JNDI and get a new Connection Handle

Why this error:
There is a disconnect between WLS and JCA connection pool manager.
When we update JCA adapter deployment with new JNDI and redeploy the adapter instance then WLS invalidates all the existing connections and it seems WLS even doesn't know that JCA connection pool is the owner of these connections and it doesn't inform JCA pool manager that it has invalidated the connections.
Now JCA pool manager has no idea of what happened and it still maintains invalid connections in its pool.

Obviously when client requests JCA connection then JCA manager try to return the dead connection which results above error.

Solution#1

We saw this issue even in SOA11.1.1.6 and WLS 10.3.6
Try applying patch 11812911

Solution#2

This solution is about just updating weblogic-ra.xml without redeploying it

•  Update adapter JNDI from the weblogic administration console deployments
•  Save the changes
•  Go to the Deployments page
•  Check the required Adapter
•  Click on the Update button
•  Choose "Update this application in place with new deployment plan changes. (We need to specify a deployment plan for this option) Do not choose the default option: "Redeploy this application using the following deployment files"

• Click finish 

 Error stack in logs:

<Error> <oracle.soa.adapter> <JCABinding=> processname Error while performing endpoint Activation: javax.resource.spi.IllegalStateException: [Connector:199176]Unable to execute allocateConnection(...) on ConnectionManager. A stale Connection Factory or Connection Handle may be used. The connection pool associated with it has already been destroyed. Try to re-lookup Connection Factory eis/AQ/Oracle from JNDI and get a new Connection Handle.>

soa
.adapter> <JCABinding=> processname

javax.resource.spi.IllegalStateException: [Connector:199176]Unable to execute allocateConnection(...) on ConnectionManager. A stale Connection Factory or Connection Handle may be used. The connection pool associated with it has already been destroyed. Try to re-lookup Connection Factory eis/AQ/Oracle from JNDI and get a new Connection Handle.

at weblogic.connector.outbound.ConnectionManagerImpl.checkIfPoolIsValid(ConnectionManagerImpl.java:442)

at weblogic.connector.outbound.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:109)

at oracle.tip.adapter.aq.v2.jca.AQConnectionFactory.getConnection(AQConnectionFactory.java:83)

at oracle.tip.adapter.aq.v2.database.DBDequeueAgentFactory.createDBDequeueAgent(DBDequeueAgentFactory.java:43)

at oracle.tip.adapter.aq.v2.AQEndpoint.activate(AQEndpoint.java:55)

at oracle.tip.adapter.aq.v2.jca.AQResourceAdapter.endpointActivation(AQResourceAdapter.java:62)

at oracle.integration.platform.blocks.adapter.fw.impl.AdapterFrameworkImpl.endpointActivation(AdapterFrameworkImpl.java:492)

at oracle.integration.platform.blocks.adapter.fw.jca.lifecycle.JCAEndpointActivation.performEndpointActivation(JCAEndpointActivation.java:649)

Solution#3
There is another administrative task we can do to overcome this issue

Got to Deployments -> Adapter -> Control tab -> Outbound Connection Pools sub tab

Locate all entries of required JNDI (there will be JNDI instance equal to number of servers including Admin ), check those instances and click the Force Reset button. If you can’t see the Force Reset button  then you do not have the appropriate permissions.

Sometimes these workarounds might not work then do not hesitate to restart managed severs where ever adapter is targeted