OSB FTP/SFTP poller transaction Rollback

In this article we will talk about how OSB 11g FTP poller works and common known issues

When OSB proxy connects to FTP server for reading file or business service for uploading file, OSB is always FTP client and FTP server is server.

How FTP authentication happens?
FTP transport support two types of authentications and it is always one-way. FTP server authenticates OSB client but not the other way around

• Anonymous:FTP server does not expect any credentials to connect to it.OSB is connected as anonymous user
• External user:OSB has to pass specific user credentials via service account

How SFTP authentication happens?
SFTP  transport authentication happens in two-way mode.FTP server authenticates OSB client and vice versa
SFTP transport supports following authentication models

• Username-password authentication:This is easiest and quickest method.FTP server authenticates the connection/client with static user name/password combination and OSB provides login credentials via service account.OSB authenticates FTP server via known_hosts file.known hosts file is a  combination of  FTP host name,IP address and public key of FTP server and resides at DOMAIN_NAME/config/osb/transports/sftp. We need to create path if doesn't exists
• Host based authentication: This method is used when all users/services in OSB domain share same OSB server public key.FTP server authenticates OSB using public key of OSB and OSB client authenticates FTP server using known_hosts file.OSB passes its public key using service key providers.Check OSB documentation on how to create service accounts.
• Public key authentication:This method is used when each user/service in OSB domain has their own public/private key pair.Authentication process is same as Host based mechanism with only difference is that we configure service key provider with user/service specific public key

In above last two methods, public keys of FTP server and OSB needs to be exchanged and setup by admin.FTP server needs to setup to accept connections from OSB.

How FTP/SFTP poller mechanism works?
File processing is done asynchronously.
FTP poller thread is pinned to one managed node in cluster and we need to choose that managed node during deployment.

• FTP transport proxy polls inbound directory at configured intervals
• If proxy finds file then it renames file to .stage extension to make sure it will not be picked up during next polling cycle.This is required because there might be chances that file is still under delivery until next poll cycle or poll cycles are too quick.
• A JMS entry with file metadata is made in wlsb.internal.transport.task.queue.sftp queue to indicate file is waiting to be read
• There are domain wide deployed MDBs which poll on above JMS for new file poll requests
• MDB receives the request and makes a new connection to FTP server to read file content.This task happens in transactional context
• MDB delivers the file to proxy request pipe-line
• After successful delivery of file, .stage file is removed from FTP server
• Now MDB job is done and OSB proxy process the file

If for some reason file couldn't be delivered then redelivery is attempted based on configured retry count on proxy.After retry count is exhausted then file is moved to configured error directory on proxy

Common issues?

• Permission issues on FTP server
• Network/Firewall issues between OSB and FTP.
• Please be aware of that OSB doesn't support certain types of FTP servers.
• Repeatedly trying to deliver same file
• Permission issues on Error/Archive/Download directories to OSB install user

I would like to discuss one interesting issue and resolution
When you see error something like below repeatedly in OSB log then something interested happened

ExecuteThread: '54' for queue: 'weblogic.kernel.Default (self-tuning)'>
<<anonymous>> <BEA1-4344C4674547A7E28CBA> <0000K8^j7x9FS8m5srt1iX1ITzwG000003> <1384451797938> <BEA-381803> <Unable to get file :<filename.stage on attempt number : 0 :java.io.IOException: No such file

Above error comes when OSB proxy found file in FTP server,renames file to .stage and JMS task has been created.When MDB connects to FTP server then somehow .stage file not found.It might got deleted mistakenly by someone or by some automated cleanup  process running on FTP server.
Because of MDB retrieves file within transaction context, now this transaction rolls back and JMS poll task is again put back in JMS queue.
This process repeats when MDB again picks up JMS task and try to read .stage file which will never be found again transaction rolled back.This goes to infinite loop and continues forever.

If we carefully analyze the logs, we can see below MDB transaction roll backs
[ACTIVE] ExecuteThread: '43' for queue: 'weblogic.kernel.Default (self-tuning)'> <alsb-system-user> <>
<0000K8^j7x9FS8m5srt1iX1ITzwG000003> <1384451801706> <BEA-010213> <Message-Driven EJB: PolledMessageListenerMDBEJB's transaction was rolled back.
The transaction details are: Name=[EJB com.bea.wli.sb.transports.poller.listener.PolledMessageListenerMDB.onMessage(javax.jms.Message)],Xid=BEA1-4350C4674547A7E28CBA(834804898),Status=Rolled back

Only solution I could think of is delete JMS entry from wlsb.internal.transport.task.queue.sftp.