OSB FTP/SFTP poller transaction Rollback

In this article we will talk about how OSB 11g FTP poller works and common known issues

When OSB proxy connects to FTP server for reading file or business service for uploading file, OSB is always FTP client and FTP server is server.

How FTP authentication happens?
FTP transport support two types of authentications and it is always one-way. FTP server authenticates OSB client but not the other way around

• Anonymous:FTP server does not expect any credentials to connect to it.OSB is connected as anonymous user
• External user:OSB has to pass specific user credentials via service account

How SFTP authentication happens?
SFTP  transport authentication happens in two-way mode.FTP server authenticates OSB client and vice versa
SFTP transport supports following authentication models

• Username-password authentication:This is easiest and quickest method.FTP server authenticates the connection/client with static user name/password combination and OSB provides login credentials via service account.OSB authenticates FTP server via known_hosts file.known hosts file is a  combination of  FTP host name,IP address and public key of FTP server and resides at DOMAIN_NAME/config/osb/transports/sftp. We need to create path if doesn't exists
• Host based authentication: This method is used when all users/services in OSB domain share same OSB server public key.FTP server authenticates OSB using public key of OSB and OSB client authenticates FTP server using known_hosts file.OSB passes its public key using service key providers.Check OSB documentation on how to create service accounts.
• Public key authentication:This method is used when each user/service in OSB domain has their own public/private key pair.Authentication process is same as Host based mechanism with only difference is that we configure service key provider with user/service specific public key

In above last two methods, public keys of FTP server and OSB needs to be exchanged and setup by admin.FTP server needs to setup to accept connections from OSB.

How FTP/SFTP poller mechanism works?
File processing is done asynchronously.
FTP poller thread is pinned to one managed node in cluster and we need to choose that managed node during deployment.

• FTP transport proxy polls inbound directory at configured intervals
• If proxy finds file then it renames file to .stage extension to make sure it will not be picked up during next polling cycle.This is required because there might be chances that file is still under delivery until next poll cycle or poll cycles are too quick.
• A JMS entry with file metadata is made in wlsb.internal.transport.task.queue.sftp queue to indicate file is waiting to be read
• There are domain wide deployed MDBs which poll on above JMS for new file poll requests
• MDB receives the request and makes a new connection to FTP server to read file content.This task happens in transactional context
• MDB delivers the file to proxy request pipe-line
• After successful delivery of file, .stage file is removed from FTP server
• Now MDB job is done and OSB proxy process the file

If for some reason file couldn't be delivered then redelivery is attempted based on configured retry count on proxy.After retry count is exhausted then file is moved to configured error directory on proxy

Common issues?

• Permission issues on FTP server
• Network/Firewall issues between OSB and FTP.
• Please be aware of that OSB doesn't support certain types of FTP servers.
• Repeatedly trying to deliver same file
• Permission issues on Error/Archive/Download directories to OSB install user

I would like to discuss one interesting issue and resolution
When you see error something like below repeatedly in OSB log then something interested happened

ExecuteThread: '54' for queue: 'weblogic.kernel.Default (self-tuning)'>
<<anonymous>> <BEA1-4344C4674547A7E28CBA> <0000K8^j7x9FS8m5srt1iX1ITzwG000003> <1384451797938> <BEA-381803> <Unable to get file :<filename.stage on attempt number : 0 :java.io.IOException: No such file

Above error comes when OSB proxy found file in FTP server,renames file to .stage and JMS task has been created.When MDB connects to FTP server then somehow .stage file not found.It might got deleted mistakenly by someone or by some automated cleanup  process running on FTP server.
Because of MDB retrieves file within transaction context, now this transaction rolls back and JMS poll task is again put back in JMS queue.
This process repeats when MDB again picks up JMS task and try to read .stage file which will never be found again transaction rolled back.This goes to infinite loop and continues forever.

If we carefully analyze the logs, we can see below MDB transaction roll backs
[ACTIVE] ExecuteThread: '43' for queue: 'weblogic.kernel.Default (self-tuning)'> <alsb-system-user> <>
<0000K8^j7x9FS8m5srt1iX1ITzwG000003> <1384451801706> <BEA-010213> <Message-Driven EJB: PolledMessageListenerMDBEJB's transaction was rolled back.
The transaction details are: Name=[EJB com.bea.wli.sb.transports.poller.listener.PolledMessageListenerMDB.onMessage(javax.jms.Message)],Xid=BEA1-4350C4674547A7E28CBA(834804898),Status=Rolled back

Only solution I could think of is delete JMS entry from wlsb.internal.transport.task.queue.sftp.
 

JCA adapters and message rejection handling

Oracle JCA adapters support error handling capabilities.In this post we will see what is message rejection and how they can be handled.

Message rejection:Any message that got error out before being posted to SCA infrastructure is called rejected message.A best example would be when we have file adapter to translate text format to XML.If there are errors in translation then message will be rejected by JCA framework.

We can handle such type of messages by using a mechanism called rejection handlers.This mechanism is supported via Fault-policies This rejection handlers works only with Synchronous process
JCA framework categorizes the errors into two types

• Retryable (which can be retried safely)
• Non-Retryable (cannot be retried)

Messages are rejected if they are non-retryable for example, translation errors
They will be retired if they can be, for example connection errors
Retryable errors are retried either indefinitely(default behaviour configured globally) or number of times equal to jca.retry.count  in composite.xml and reject message after count is exhausted.

All rejected messages are stored in SOA dehydration store table-rejected_message

Configuring message rejection handlers
Rejection handlers are defined in fault-policies.Message rejection handlers comes into picture only when messages got rejected by JCA

If we do not configure rejection handlers then default file based rejection handler will kick off and all rejected messages will be forwarded to &lt;domain_name>/rejmsgs/&lt;managed_server>

Available rejection handlers
Following handlers can be defined in fault-policies

1. JMS queue (Rejected messages are written to configured JMS queue)
2. Web service (Configured WS will be called with rejected message)
3. Custom java (Java class will be executed)
4. File (Rejected messages are written to specified folder path)

JCA Retries

When errors are retryable, then retrying happens based on either JCA global retry which is indefinite by default or JCA properties in composite.xml
Below JCA properties are used for this purpose in composite.xml
jca.retry.count
jca.retry.interval
jca.rerty.backoff
jca.retry.maxInterval

If we do not specify any JCA properties in composite.xml then global JCA retry will be used. If we specify at composite level then composite JCA properties will take precedence.

We also have an option of changing global JCA count from indefinite to finite number using system Mbean browser from em console.Follow below steps to do this

1. Right click on soa-infra then select Administration then system Mbean browser
2. Select oracle.as.soainfra.config then adapterconfig then adapter
3. Change GlobalInboundJCAretryCount to required number

Please remember this will affect all JCA retries in the domain.Keeping indefinte is also dangerous because this will cause JCA to retry indefiniely resulting one instnace for each retry
 

File naming convention sequencing strategy for FILE/Ftp Adapter

As we know, File/Ftp adapter allows us to configure sequence as part of file naming convention when we generate files.For example, if you choose PO-data_%SEQ%.xml as the FileNamingConvention, all files would be generated as PO-data_1.xml, PO-data_2.xml,..
JCA file would have property like

<property name="FileNamingConvention" value="PO-data_%SEQ%.xml"/>

As per Oracle, sequence is maintained as mentioned below 
The sequence number is being maintained in the control directory of  corresponding composite project. For each project that use the File or Ftp Adapter, a unique directory is created for book-keeping purposes. Because this control directory must be unique, the adapter uses a digest to ensure that no two control directories are the same.
The control information for project would go under FMW_HOME/user_projects/domains/soainfra/fileftp/controlFiles/[DIGEST]/outbound where the value of DIGEST would differ from one project to another.
Within this directory, there is a file control_ob.properties file where the sequence number is maintained. The sequence number is maintained in binary form and you might need a hexadecimal editor to view its content. There is another zero byte file, SEQ_nnn. This extra file is maintained as a backup.

One of the challenges faced by the adapter run time is to guard all writes to the control files so no two threads inadvertently attempt to update the control files at the same time. It does this guarding with the help of a "Mutex". The mutex is of different types:

• In-memory
• DB-based
• Coherence-based
• User-defined

There might be scenarios, particularly when the Adapter is under heavy transactional load, where the mutex is a bottleneck. The Adapter, however, enables you to change the configuration so the adapter sequence value is derived from a database sequence or a stored procedure. In such a situation, the mutex is by-passed, and the process results in improved throughput.
The simplest way to achieve improved throughput is by switching your JNDI connection factory location for the outbound JCA file to use the eis/HAFileAdapter:
eis/HAFileAdapter is available by default when we install SOA
Above connection factory can also be used to support high availability in SOA cluster in active-active mode

Check control directory is setup for specific path to keep house keeping information 

Check how HAFile or HAFtp Adapter is configured with SOA data sources to maintain sequence in soainfra database schema

With this change, the Adapter run time creates a sequence on the Oracle database(SOA dehydration store). For example, if you do a select * from user_sequences in your soa-infra schema, you see a new sequence being created with name as SEQ___ (where the GUID differs by project).
However, to use your own sequence, you must add a new property to your JCA file called SequenceName. You must create this sequence on your soainfra schema beforehand

<property name="SequenceName" value="Adapter_Seq"/>

Writing same file to multiple FTP locations using FTP Dynamic Partnerlink

When we have a requirement to send the same file to multiple FTP servers, then probably we could create, multiple FTPAdapter references, one for each FTP server. However, this is not the most optimal approach; instead, you can use the concept of "Dynamic Partner Links".

In the later approach, we could create just one FTP adapter reference partner link and have multiple invoke activities invoking same reference.

First create all required FTP JCA deployment JNDIs (one for each FTP server)  in WLS admin console
For example,
eis/FTP/FTP1
eis/FTP/FTP2
eis/FTP/FTP3

In composite, create FTP adapter reference using one of above JCA connection-factory location JNDI

Connect to FTP adapter reference using invoke activity

Using dynamic partner link concept, we can change this connection-factory location dynamically using JCA header properties in both BPEL and Mediator services. To do so, BPEL or Mediator is required to use a reserved JCA header property jca.jndi as shown in the following.

If we carefully examine above code,we can understand below things
Two invoke activities are pointing to same reference 'FTPPut' and using JCA property jca.jndi which will be assigned required FTP connection-factory location JNDI.
This will invoke different FTP servers for writing same file.

Note: jca.jndi property is not available on invoke activity properties tab.We need to add manually to .bpel source file