XAER_NOTA start() failed The XID is not valid

When you see error something like -'XAER_NOTA start() failed The XID is not valid', then do not worry because that is something to do with JTA transaction timeout.

This issue is because the default XA transaction timeout on the XA resource, in this case the XA datasource, is insufficient, causing a timeout. By default, the XA transaction timeout for 60 seconds. You can enable "Set XA Transaction Timeout" to true for the XA datasource and specify the "XA transaction timeout" value in seconds to increase this timeout.

Solution Steps

• Log into WLS Administration Console
• Click on Services -> Data Sources and then click on data source you wanted to modify
• Click on Transaction Tab
• Click on check box next to "Set XA Transaction Timeout"
• Make sure that "XA Transaction Timeout" has a value of 0
• Save the configuration.
• Bounce SOA server 

When this 'XA Transaction Timeout"' is set to zero, the XAResource Session Timeout will be set to the global transaction timeout  which is of 600 secs
Please refer to Oracle note ID 1352715.1


Below error log may seen when this issue happens.


SOA error log:

[soa_server1] [ERROR] [] [oracle.soa.bpel.engine.dispatch] [tid: orabpel.invoke.pool-4.thread-9] [userId: ] [ecid: 097b277aaf0a827d:-

d54fd89:13c60a5bfc1:-8000-00000000001d808c,1:18696] [APP: soa-infra] database communication failure[[
java.sql.SQLException: Unexpected exception while enlisting XAConnection java.sql.SQLException: XA error: XAResource.XAER_NOTA start() failed on resource 'SOADataSource_>domain nsme<':

XAER_NOTA : The XID is not valid
oracle.jdbc.xa.OracleXAException
        at oracle.jdbc.xa.OracleXAResource.checkError(OracleXAResource.java:1616)
        at oracle.jdbc.xa.client.OracleXAResource.start(OracleXAResource.java:336)
        at weblogic.jdbc.jta.DataSource.start(DataSource.java:790)
        at weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInfo.java:1247)
        at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1180)
        at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:300)
        at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:561)
        at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:488)
        at weblogic.jdbc.jta.DataSource.enlist(DataSource.java:1673)
        at weblogic.jdbc.jta.DataSource.refreshXAConnAndEnlist(DataSource.java:1577)
        at weblogic.jdbc.wrapper.JTAConnection.getXAConn(JTAConnection.java:215)
        at weblogic.jdbc.wrapper.JTAConnection.checkConnection(JTAConnection.java:84)
        at weblogic.jdbc.wrapper.JTAConnection.checkConnection(JTAConnection.java:74)
        at weblogic.jdbc.wrapper.Connection.preInvocationHandler(Connection.java:100)
        at weblogic.jdbc.wrapper.Connection.prepareStatement(Connection.java:545)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.prepareStatement(DatabaseAccessor.java:1474)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.prepareStatement(DatabaseAccessor.java:1423)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseCall.prepareStatement(DatabaseCall.java:697)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:585)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:535)
        at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:1717)
        at org.eclipse.persistence.sessions.server.ClientSession.executeCall(ClientSession.java:253)
        at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:207)
        at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:193)
        at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.insertObject(DatasourceCallQueryMechanism.java:342)
        at org.eclipse.persistence.internal.queries.StatementQueryMechanism.insertObject(StatementQueryMechanism.java:162)
        at org.eclipse.persistence.internal.queries.StatementQueryMechanism.insertObject(StatementQueryMechanism.java:177)
        at org.eclipse.persistence.internal.queries.DatabaseQueryMechanism.insertObjectForWrite(DatabaseQueryMechanism.java:472)
        at org.eclipse.persistence.queries.InsertObjectQuery.executeCommit(InsertObjectQuery.java:80)
        at org.eclipse.persistence.queries.InsertObjectQuery.executeCommitWithChangeSet(InsertObjectQuery.java:90)
        at org.eclipse.persistence.internal.queries.DatabaseQueryMechanism.executeWriteWithChangeSet(DatabaseQueryMechanism.java:287)
        at org.eclipse.persistence.queries.WriteObjectQuery.executeDatabaseQuery(WriteObjectQuery.java:58)
        at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:844)
        at org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:743)
        at org.eclipse.persistence.queries.ObjectLevelModifyQuery.executeInUnitOfWorkObjectLevelModifyQuery(ObjectLevelModifyQuery.java:108)
        at org.eclipse.persistence.queries.ObjectLevelModifyQuery.executeInUnitOfWork(ObjectLevelModifyQuery.java:85)
        at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2871)
        at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1516)
        at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1498)

Weblogic10.3.6 JCA connection pool framework

Have you ever seen below error message in soa11g  server logs,if answer is yes, then there is a reason to stop and think the way you activate your JCA adapter JNDI deployments

error message:
Unable to execute allocateConnection(...) on ConnectionManager. A stale Connection Factory or Connection Handle may be used
Try to re-lookup Connection Factory-xxx  from JNDI and get a new Connection Handle

Why this error:
There is a disconnect between WLS and JCA connection pool manager.
When we update JCA adapter deployment with new JNDI and redeploy the adapter instance then WLS invalidates all the existing connections and it seems WLS even doesn't know that JCA connection pool is the owner of these connections and it doesn't inform JCA pool manager that it has invalidated the connections.
Now JCA pool manager has no idea of what happened and it still maintains invalid connections in its pool.

Obviously when client requests JCA connection then JCA manager try to return the dead connection which results above error.

Solution#1

We saw this issue even in SOA11.1.1.6 and WLS 10.3.6
Try applying patch 11812911

Solution#2

This solution is about just updating weblogic-ra.xml without redeploying it

•  Update adapter JNDI from the weblogic administration console deployments
•  Save the changes
•  Go to the Deployments page
•  Check the required Adapter
•  Click on the Update button
•  Choose "Update this application in place with new deployment plan changes. (We need to specify a deployment plan for this option) Do not choose the default option: "Redeploy this application using the following deployment files"

• Click finish 

 Error stack in logs:

<Error> <oracle.soa.adapter> <JCABinding=> processname Error while performing endpoint Activation: javax.resource.spi.IllegalStateException: [Connector:199176]Unable to execute allocateConnection(...) on ConnectionManager. A stale Connection Factory or Connection Handle may be used. The connection pool associated with it has already been destroyed. Try to re-lookup Connection Factory eis/AQ/Oracle from JNDI and get a new Connection Handle.>

soa
.adapter> <JCABinding=> processname

javax.resource.spi.IllegalStateException: [Connector:199176]Unable to execute allocateConnection(...) on ConnectionManager. A stale Connection Factory or Connection Handle may be used. The connection pool associated with it has already been destroyed. Try to re-lookup Connection Factory eis/AQ/Oracle from JNDI and get a new Connection Handle.

at weblogic.connector.outbound.ConnectionManagerImpl.checkIfPoolIsValid(ConnectionManagerImpl.java:442)

at weblogic.connector.outbound.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:109)

at oracle.tip.adapter.aq.v2.jca.AQConnectionFactory.getConnection(AQConnectionFactory.java:83)

at oracle.tip.adapter.aq.v2.database.DBDequeueAgentFactory.createDBDequeueAgent(DBDequeueAgentFactory.java:43)

at oracle.tip.adapter.aq.v2.AQEndpoint.activate(AQEndpoint.java:55)

at oracle.tip.adapter.aq.v2.jca.AQResourceAdapter.endpointActivation(AQResourceAdapter.java:62)

at oracle.integration.platform.blocks.adapter.fw.impl.AdapterFrameworkImpl.endpointActivation(AdapterFrameworkImpl.java:492)

at oracle.integration.platform.blocks.adapter.fw.jca.lifecycle.JCAEndpointActivation.performEndpointActivation(JCAEndpointActivation.java:649)

Solution#3
There is another administrative task we can do to overcome this issue

Got to Deployments -> Adapter -> Control tab -> Outbound Connection Pools sub tab

Locate all entries of required JNDI (there will be JNDI instance equal to number of servers including Admin ), check those instances and click the Force Reset button. If you can’t see the Force Reset button  then you do not have the appropriate permissions.

Sometimes these workarounds might not work then do not hesitate to restart managed severs where ever adapter is targeted   

Handling weblogic T3 protocol message size issues

Do not be surprised when you see an error like below in weblogic server logs.
Incoming message of size: '10000080' bytes exceeds the configured maximum of: '10000000' bytes for protocol: 't3

In nutshell, above error happens when either weblogic server is not configured to handle message of size more than 10MB or client connected to WLS could not handle more than 10MB message size.
Most of the times it is JMS message processing but need not be always JMS.

First lets understand what is T3 and how it works.

T3 is a proprietary communication protocol being used by WLS to handle data traffic between java to java applications or JVM to JVM
WLS has a mechanism to keep track of JVMs with which it is connected and always creates just one connection to the connected JVM.
These connected JVMs could be server JVM to server JVM or client JVM to server JVM.

For example, if a Java client accesses JSP page  and a JDBC connection pool on WebLogic Server, a single network connection is established between the WebLogic Server JVM and the client JVM. This eliminates the need of creating multiple connections and managing them.T3 protocol also does multiplexing data packets invisibly on the single connection

Features of T3 protocol

• High performance for various above said reasons
• Multiplexes various data sources data packets into single connection
• Minimizes the packet size

By default T3 protocol is configured to handle messages of size 10 MB, if server receives messages more than 10 MB size then it can not handle.

As per my knowledge there are three options to handles this.

Option#1(Preferable)

Try to analyze why server is receiving that huge payload, who is sending it.In general it is not advisable to handle huge messages unless really really required.
10 MB set by WLS is reasonable value and most appropriate for many environments.
so solution is try to reduce the payload size.
Before sending  huge message I suggest transform it to condensed form.
XML is verbose.So lets be very careful about it.

Solution to reduce paylaod size:
Transform from source element to target element only if source element has data in it.This has to be done for each element.This will reduce the size.

Option#2(Chunked messages)
There are some ways to send big messages in small pieces.For example in http, we can use chunked mode transfer and OSB also supports transferring messages in chunked mode.

Option#3(Increasing message size)

If option#1 and 2 does not help you then you can try out this.

1. Go to servers--->protocols--->general--->max message size--->change it to required value (It is preferable to have same size on both the admin and the soa servers but not mandatory)

2. Go to servers--->configuration--->server start--->arguments--->-Dweblogic.MaxMessageSize = same value given in previous step

3.Set weblogic.MaxMessageSize = 'required value' as system property on client side

bounce client and the servers.