Implementing Java Fault Handling for Oracle Composite

Oracle SOA suite comes with  fault handling framework which is a collection of sub modules to automate handling of faults .
For introduction of fault handling in composites, please visit Oracle's FMW Developer's Guide for Oracle SOA Suite 

Two important files in fault framework are

• fault-policies.xml
• fault-bindings.xml 

fault-policies.xml: Talks about what kind of faults it can handle and how. Fault policy file defines fault conditions and their corresponding fault recovery actions.

fault-bindings.xml: Talks about binding of fault-policies with  composite and its components or scope of fault-policies.




As part of fault-policies.xml, we can mention various types of recovery actions like retry,re-throw,human-intervention,replay,terminate and many more.Java action is one of them.


Fault-poloicies.xml
<?xml version="1.0" encoding="UTF-8" ?>
<faultPolicies xmlns="http://schemas.oracle.com/bpel/faultpolicy">
  <faultPolicy version="2.0.1"
           id="ProcessingFaults"
               xmlns:env="http://schemas.xmlsoap.org/soap/envelope/"
               xmlns:xs="http://www.w3.org/2001/XMLSchema"
               xmlns="http://schemas.oracle.com/bpel/faultpolicy"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <Conditions>
      <!-- remote fault processing starts here: -->
    <faultName xmlns:bpelx="http://schemas.oracle.com/bpel/extension"
               name="bpelx:remoteFault">
         <condition>
                <action ref="ora-retry"/>
        </condition>
    </faultName>
   
    <!-- binding fault processing starts here: -->
     <faultName xmlns:bpelx="http://schemas.oracle.com/bpel/extension"
                name="bpelx:bindingFault">       
               <condition>
                       <action ref="ora-retry"/>
               </condition>
    </faultName>

    </Conditions> 
<!-- Above two conditions indicates perform retry actions for binding and remote faults -->
    <Actions>  
    <!-- RetryInterval is in seconds. This action handles our retry requirement and in case retry also  failed then that chains to custom java action -->
      <Action id="ora-retry">
        <retry>
          <retryCount>3</retryCount>
          <retryInterval>5</retryInterval>
          <exponentialBackoff/>
          <retryFailureAction ref="ora-custom-java"/> 
<!--in case retry also failed then that chains control to java action-->
        </retry>
      </Action>            
 <Action id="ora-custom-java">
 <!-- this is user provided custom java class. This action handles notification requirement -->
      <javaAction className="com.fault.handler.JavaFaultHandler"
             defaultAction="ora-human-intervention" propertySet="notification-properties">
            <returnValue value="RETRHOW"  ref="ora-rethrow-fault"/>
<!-- If java class returns RETRHOW then action "ora-rethrow-fault" is invoked -->
            <returnValue value="MANUAL"   ref="ora-human-intervention"/>
<!-- If java class returns MANUAL then action "ora-human-intervention" is invoked -->
       </javaAction>
     </Action>
        
<!-- This action re throws fault back to BPEL Catch/CatchAll-->
        <Action id="ora-rethrow-fault">
            <rethrowFault/>
        </Action >
  <!-- Human Intervention. This action handles human intervention part-->
        <Action id="ora-human-intervention">
           <humanIntervention/>
        </Action>
     </Actions>
   &lt!-- Below properties are read from java class--> 
     <Properties>    
          <propertySet name="notification-properties">          
            <property name="email_to">team1@gmail.com</property>
            <property name="email_cc">team2@gmail.com</property>   
            <property name="commonErrorHandlingProcess">CommonErrorHandlingBPEL</property>               
            <property name="domain">default</property>
     </propertySet>
       
  </Properties>
</faultPolicy> 
</faultPolicies>


fault-bindings.xml
<?xml version="1.0" encoding="UTF-8" ?>
<faultPolicyBindings version="2.0.1"
                     xmlns="http://schemas.oracle.com/bpel/faultpolicy"
                     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <composite faultPolicy="ProcessingFaults"/> 
</faultPolicyBindings>

This bindings file binds fault recovery actions to composite level.i.e all components with in composite.xml

Now let's see how we can write a custom java class plugged into fault handling mechanism.

Co-locate both xml files with composite.xml. 




Please create a java project in Jdev and add following libraries to project class-path.

1. BPEL Runtime
2. Oracle.soa.fabric.jar





Custom Java action class
package com.fault.handler;

import com.fault.handler.Exception;
import com.collaxa.cube.engine.fp.BPELFaultRecoveryContextImpl;

import oracle.integration.platform.faultpolicy.IFaultRecoveryContext;
import oracle.integration.platform.faultpolicy.IFaultRecoveryJavaClass;
oracle.integration.platform.faulthandling.recovery.RejectedMsgRecoveryContext;
import java.util.Map;
import org.w3c.dom.Element;
import org.w3c.dom.Node;
import java.io.StringWriter;

import javax.xml.transform.OutputKeys;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerException;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;


public class JavaFaultHandler implements IFaultRecoveryJavaClass {

    public void handleRetrySuccess(IFaultRecoveryContext iFaultRecoveryContext) {
        System.out.println("Retry Success Happend...");
       // handleFault(iFaultRecoveryContext);
    }

    private String nodeToString(Node node) {
         StringWriter sw = new StringWriter();
         try {
             Transformer t = TransformerFactory.newInstance().newTransformer();
             t.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes");
             t.transform(new DOMSource(node), new StreamResult(sw));
             } catch (TransformerException te) {
             System.out.println("nodeToString Transformer Exception");
    }
    return sw.toString();
    }
   //Below method will be invoked by fault handling mechanism when there is a fault. 
public String handleFault(IFaultRecoveryContext iFaultRecoveryContext) {
       
      
        try{
           Exception excObj = new Exception();           
 
         if(iFaultRecoveryContext instanceof BPELFaultRecoveryContextImpl){
      
          // Get BPEL specific context here
             BPELFaultRecoveryContextImpl bpelCtx =
                                         (BPELFaultRecoveryContextImpl)iFaultRecoveryContext;   
             // bit of log statements for debug purpose   
                System.out.println("*** BPEL Context + getComponentInstanceId ***: " +
                bpelCtx.getComponentInstanceId());                    
                System.out.println("*** BPEL Context + getFault ***: " +
                bpelCtx.getFault());              
                System.out.println("***  BPEL Context+ getInstanceId ***: " +
                bpelCtx.getInstanceId());              
                System.out.println("***  BPEL Context + getCompositeTitle ***: " + 
                bpelCtx.getTitle());                             
                System.out.println("***  BPEL Context + FaultedProcess Revision ***:"+
                bpelCtx.getProcessDN().getRevision());              
                                                  
                try{
// This code assumes all BPELs will have a variable called-faultHandlerVariable which is defined on canonical xsd containing generic error data        
                Element root =(Element) bpelCtx.getVariableData  
  ("faultHandlerVariable", "payload", "/");                  
                Node exception =   root.getFirstChild();
                String errorDataStr   =   nodeToString(exception);
                
                //Setting business payload  data 
                 excObj.setBusinessPayload(errorDataStr);  
                } catch (Exception e){
                    //Ignore this exception; because setting this business payload data is optional for my requirement
                    ;
                }
                  
                //Setting BPEL specific error data
                excObj.setErrorCode((bpelCtx.getFault().getFaultName().toString().contains("remoteFault"))?"remoteFault":"bindingFault");
                excObj.setErrorMessage(bpelCtx.getFault().getMessage().toString());
                excObj.setErrorInstanceId(bpelCtx.getInstanceId());
                excObj.setErrorProcessId( bpelCtx.getCompositeName());             
                excObj.setErrorProcessVersion(bpelCtx.getProcessDN().getRevision());                                              
                           
                } else if (iFaultRecoveryContext instanceof RejectedMsgRecoveryContext) {
              
                //Get Rejected messages context here
                System.out.println("*** iFaultRecoveryContext is an instanceof RejectedMsgRecoveryContext ***");
                System.out.println("==========================================================");
            
            } else{
             //Get Non-BPEL context here, if error is raised from other than BPEL component
               System.out.println("*** iFaultRecoveryContext is neither instanceof BPELFaultRecoveryContextImpl nor RejectedMsgRecoveryContext***");               
      
              System.out.println("====================================");
            }                      
            
             //Setting Email error data
             excObj.steEmailTo(props.get("email_to").toString().substring(1, props.get("email_to").toString().length()-1));
             excObj.setEmailCC(props.get("email_cc").toString().substring(1, props.get("email_cc").toString().length()-1));          
             excObj.setDomainId(props.get("domain").toString().substring(1, props.get("domain").toString().length()-1));           
          
                   
 //Setting CommonStatusLogger Process invocation context data  
             excObj.setCommonEHProcessName(props.get("commonEH_process").toString().substring(1, props.get("commonEH_process").toString().length()-1));  
                       
//CommonStatusLoggerDirectBinding direct binding Invocation
Please refer to SOA direct binding API This call invokes a composite which sends email.I am passing all error related data and email notification details as an object.
             CommonStatusLoggerDirectBinding ceh =new CommonStatusLoggerDirectBinding ();
             String status = ceh.invokeCommonErrorStatusDirect(excObj);
             if (status == "S") 
                return "MANUAL";
             else               
                return "RETRHOW";      
            }
            catch(Exception e){
            System.out.println("***JavaFaultHandler Exception******\n"+ e.getMessage());
              e.printStackTrace();
              return "RETRHOW";
    
        }
    }
}
Bean Utility class 

package com.fault.handler;

import java.util.Date;
//This BEAN class represents all the common attributes any fault handler requires.
public class Exception {
    protected String errorCode;
    protected String errorMessage;
    protected Date errorTime= new Date();
    protected String domainId="default";
    protected String errorProcessId;
    protected String errorProcessVer;
    protected String errorProcessURL;
    protected Long  errorInstanceId;
    protected String fileName;
    protected String payload;
    protected String fileDate;
    protected String sourceSystem;
    protected String reference;
    protected String emailTo="team1@gmail.com";
    protected String emailCC;
    protected String errorPartnerLink;
    protected String subject="Runtime Error: ";
    protected String providerURL;
    protected String initialContext="weblogic.jndi.WLInitialContextFactory";
    protected String securityCredential;
    protected String securityPrinciple;
    protected String commonEHProcessName="CommonErrorHandlerProcess";
    protected String commonEHProcessVer="1.0";
    protected String commonEHURL;
    protected String errorPayload;
    protected String businessPayload;
    public Exception() {
        super();       
       
    }
   //Setter methods  
    public void setErrorCode  (String errorCode ){
        this.errorCode = errorCode;
    }
    public void setErrorMessage  (String errorMessage){
        this.errorMessage = errorMessage;
    }
    public void setErrorTime  (Date errorTime){
        this.errorTime = errorTime;
    }
    public void setDomainId  (String domainId){
        this.domainId = domainId;
    }
    public void setErrorProcessId  (String errorProcessId){
        this.errorProcessId = errorProcessId;
    }
    public void setErrorProcessVersion  (String errorProcessVersion){
        this.errorProcessVer = errorProcessVersion;
    }
    public void setErrorProcessURL  (String errorProcessURL){
        this.errorProcessURL = errorProcessURL;
    }
    public void setErrorInstanceId  (Long errorInstanceId){
        this.errorInstanceId = errorInstanceId;
    }
    public void setFileName  (String fileName){
        this.fileName = fileName;
    }
    public void setPayload  (String payload){
        this.payload = payload;
    }
    public void setFileDate  (String fileDate){
        this.fileDate = fileDate;
    }
    public void setSourceSystem  (String sourceSystem){
        this.sourceSystem = sourceSystem;
    }
    public void setReference  (String reference){
        this.reference = reference;
    }
    public void steEmailTo  (String emailTo){
        this.emailTo = emailTo;
    }
    public void setEmailCC  (String emailCC){
        this.emailCC = emailCC;
    }
    public void setErrorPartnerLink  (String errorPartnerLink){
        this.errorPartnerLink = errorPartnerLink;
    }
    public void setSubject  (String subject){
        this.subject = subject;
    }
    public void setProviderURL  (String providerURL){
      this.providerURL = providerURL;
    } 
    public void setInitalContext  (String initialContext){
       this.initialContext = initialContext;
    }
    public void setSecurityCredential (String securityCredential){
       this.securityCredential = securityCredential;
    }
    public void setSecurityPrinciple (String securityPrinciple){
       this.securityPrinciple = securityPrinciple;
    }
    public void setCommonEHProcessName (String commonEHProcessName){
       this.commonEHProcessName = commonEHProcessName;
    }
    public void setCommonEHProcessVer (String commonEHProcessVer){
       this.commonEHProcessVer = commonEHProcessVer;
    }
    public void setCommonEHURL (String commonEHURL){
       this.commonEHURL = commonEHURL;
    }
   
    public void setBusinessPayload (String businessPayload){
       this.businessPayload = businessPayload;
    }
   
    //Getter methods  
     public String getErrorCode  ( ){
        return  this.errorCode;
     }
     public String getErrorMessage  (){
         return this.errorMessage;
     }
     public Date getErrorTime  (){
         return this.errorTime ;
     }
     public String getDomainId  (){
        return this.domainId ;
     }
     public String getErrorProcessId  (){
        return this.errorProcessId ;
     }
     public String getErrorProcessVersion  (){
        return  this.errorProcessVer ;
     }
     public String getErrorProcessURL  (){
       return  this.errorProcessURL ;
     }
     public Long getErrorInstanceId  (){
       return  this.errorInstanceId ;
     }
     public String getFileName  (){
        return this.fileName ;
     }
     public String getPayload  (){
       return  this.payload ;
     }
     public String getFileDate  (){
       return  this.fileDate ;
     }
     public String getSourceSystem  (){
        return this.sourceSystem ;
     }
     public String getReference  (){
        return this.reference ;
     }
     public String getEmailTo  (){
         return this.emailTo ;
     }
     public String getEmailCC  (){
      return this.emailCC ;
     }
     public String getErrorPartnerLink  (){
      return this.errorPartnerLink ;
     }
    public String getSubject  (){
      return this.subject;
    }
    public String getProviderURL  (){
      return this.providerURL;
    } 
    public String getInitialContext  (){
      return this.initialContext;
    }
    public String getSecurityCredential (){
      return this.securityCredential;
    }
    public String getSecurityPrinciple (){
      return this.securityPrinciple;
    }
    public String getCommonEHProcessName (){
      return this.commonEHProcessName ;
    }
    public String getCommonEHProcessVer (){
      return this.commonEHProcessVer;
    }
    public String getCommonEHURL (){
       return this.commonEHURL;
    }
   
    public String getBusinessPayload (){
       return this.businessPayload;
    }
    public String toString(){      
   
        errorPayload =
        "<?xml version=\"1.0\" encoding=\"UTF-8\" ?>\n" +
        "<BPELException xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"\n" +
        "xsi:schemaLocation=\"http://www.example.net/BPELException CommonErrorHandlerProcessInput.xsd\"\n" +
        "                  xmlns=\"http://www.example.net/BPELException\">\n" +
        "  <ErrorCode>"+getErrorCode()+"</ErrorCode>\n" +
        "  <ErrorMessage>"+getErrorMessage()+"</ErrorMessage>\n" +
        "  <ErrorTime>"+getErrorTime()+"</ErrorTime>\n" +
        "  <DomainId>"+getDomainId()+"</DomainId>\n" +
        "  <ProcessId>"+getErrorProcessId()+"</ProcessId>\n" +
        "  <ProcessVersion>"+getErrorProcessVersion()+"</ProcessVersion>\n" +
        "  <ProcessURL>"+getErrorProcessURL()+"</ProcessURL>\n" +
        "  <InstanceId>"+getErrorInstanceId()+"</InstanceId>\n" +
        "  <FileName>"+getFileName()+"</FileName>\n" +
        "  <payload>"+getBusinessPayload()+"</payload>\n" +
        "  <FileDate>"+getFileDate()+"</FileDate>\n" +
        "  <SourceSystem>"+getSourceSystem()+"</SourceSystem>\n" +
        "  <Reference>"+getReference()+"</Reference>+
        "  <EmailTo>"+getEmailTo()+"</EmailTo>\n" +
        "  <EmailCC>"+getEmailCC()+"</EmailCC>\n" +
        "</BPELException>\n" ;
        return this.errorPayload;
    }
     
}  

Configuring Java class

• Pack above classes into jar 
• stop  WLS server
• Place them in  $ORACLE_HOME/soa/modules/oracle.soa.ext_11.1.1
• run ant from same location
• Restart Oracle WebLogic Server

Now code is ready for testing.

In a nutshell what we are trying to implement is, whenever there is either remote or binding fault in HellowWorld1 composite, then email notification should be sent out to configured users and human intervention should be waiting for someone to log in and correct payload and resubmit.    

How to test

Create HelloWorld1 and HelloWorld2 composites.HelloWorld1 invokes HelloWorld2.
Place fault-policies.xml and fault-bindings.xml into HellowWorld1 project and deploy both.Now retire HellowWorld2.Initiate HellowWorld1 




Best practices in configuring fault-policies

In this post I talk about some of the best practices we can think of while designing fault-policies.


JCA binding level retry execution within fault policy retries

If you are using retry actions on adapters with both JCA-level retries for the outbound direction and a retry action in the fault policy file for outbound failures, the JCA-level (or binding level) retries are executed within the fault policy retries.
Lets say you have configured below JCA binding properties in composite.xml

       <property name="jca.retry.count">2</property>
        <property name="jca.retry.interval">5</property>
        <property name="jca.retry.backoff">1</property>

and assume you have following retry configuration in fault policies file for reference binding component for remoteFault

      <Action id="ora-retry">
        <retry>
          <retryCount>3</retryCount>
          <retryInterval>10</retryInterval>
          <exponentialBackoff/>
          <retryFailureAction ref="ora-human-intervention"/>
        </retry>
      </Action>

Now when remote fault returned for reference binding component, following retry sequence happens

    * Fault policy retry 1:
          o  JCA retry 1 (with 5 seconds interval)
          o  JCA retry 2 (with 10 seconds interval)
    * Fault policy retry 2:
          o  JCA retry 1 (with 5 seconds interval)
          o  JCA retry 2 (with 10 seconds interval)
    * Fault policy retry 3:
          o  JCA retry 1 (with 5 seconds interval)
          o  JCA retry 2 (with 10 seconds interval)

As a reminder, if your intention is just retrying through fault-policies then do not use JCA level retries in composite.xml

Best practices in Configuring ora-retry fault policy

When you configure a fault policy to recover instances with the ora-retry action and the number of specified instance retries is exceeded, the instance is marked as open.faulted (in-flight state). The instance remains active and keeps running.
Marking instances as open.faulted ensures that no instances are lost and this also causes reattempting retry (even after retry count is over) which you do not like at this point of time.
Here we have a best practice to follow.
You can then configure another fault handling action following the ora-retry action in the fault policy file, such as the following:
•    Configure an ora-human-intervention action to manually perform instance recovery from Oracle Enterprise Manager Fusion Middleware Control Console.
•    Configure an ora-terminate action to close the instance (mark it as closed.faulted) and never retry again.
For example     

      <Action id="ora-retry">
        <retry>
          <retryCount>3</retryCount>
          <retryInterval>3</retryInterval>
          <exponentialBackoff/>
          <retryFailureAction ref="ora-human-intervention"/>
        </retry>
      </Action>

However, if you do not set an action to be performed after an ora-retry action in the fault policy file and the number of instance retries is exceeded, the instance remains marked as open.faulted, and recovery again attempts to handle the instance.
For example, if no action is defined in the following fault policy file after ora-retry:

<Action id="ora-retry">
       <retry>
          <retryCount>2</retryCount>
          <retryInterval>2</retryInterval>
          <exponentialBackoff/>
       </retry>
  </Action>
The following actions are performed:
•    The invoke activity is attempted (using the above-mentioned fault policy code to handle the fault).
•    Two retries are attempted at increasing intervals (after two seconds, then after four seconds).
•    If all retry attempts fail, the following actions are performed:
o    A detailed fault error message is logged in the audit trail
o    The instance is marked as open.faulted (in-flight state)
o    The instance is picked up and the invoke activity is re-attempted
o    Re throws fault to system fault handler
•    Recovery may also fail. In that case, the invoke activity is re-executed. Additional audit messages are logged.

Message rejection handlers

The messages that error out before being posted to the service infrastructure are referred to as rejected messages. For example, the Oracle File Adapter selects a file having data in text format and tries to translate it to XML format (using NXSD). If there is any error in the translation, this message is rejected and will not be posted to the target composite.   

Rejected messages are stored in the database (in the rejected_message table) by default. If you do not configure a message rejection handler then default rejection message handler will handle  it which stores them on the file system. This handler stores the payload and properties of the message on the file system at a predefined location in WLS_HOME. Currently, the Oracle SOA suite does not provide the capability to resubmit rejected messages; consequently it is your responsibility to take care of the resubmission.If we do not have one then rejected messages may go unnoticed which is not good practice .I recommend to have one to handle and resubmit them.

JCA adapters and fault-policies

In nutshell standard practices for JCA EH are:

Error type	Best practise
Inbound  Retryable	Use JCA level retries in composite.xml
Inbound  Non-retryable	Use fault-policies message rejection handlers
Outbound Retryable	Use JCA level retries in composite.xml
Outbound Non-retryable	Use fault-policies

Manual recovery of faulted instances from em console

Fault policy framework provides a way for manual intervention in case of faults (System faults/business faults).

We can use this feature to change the input payload of a invoke activity and resubmit the request.

In this discussion we see how we can configure fault-policies/ bindings to achieve this.

We can perform following fault recovery actions from Oracle Enterprise Manager Fusion Middleware Control

• Retry the activity
  
• Modify a variable (All global variables)
  
• Continue the instance (mark the activity as a success)
  
• Rethrow the exception
  
• Abort the instance
  
• Throw a replay scope exception
  

 As a best practice it is always good to retry faulted invoke activity before we move to manual recovery.

Below fault-policy talks about above strategy

fault-policies.xml

<?xml version="1.0" encoding="UTF-8" ?>
<faultPolicies xmlns="http://schemas.oracle.com/bpel/faultpolicy">
  <faultPolicy version="2.0.1"
           id="ProcessingFaults"
               xmlns:env="http://schemas.xmlsoap.org/soap/envelope/"
               xmlns:xs="http://www.w3.org/2001/XMLSchema"
               xmlns="http://schemas.oracle.com/bpel/faultpolicy"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <Conditions>
      <!-- remote fault processing here: -->
    <faultName xmlns:bpelx="http://schemas.oracle.com/bpel/extension"
               name="bpelx:remoteFault">
         <condition>
                <action ref="ora-retry"/>
        </condition>
      <condition>
                <action ref="ora-human-intervention"/>
      </condition>
    </faultName>
   
    <!-- binding fault processing here: -->
    <<aultName xmlns:bpelx="http://schemas.oracle.com/bpel/extension"
                name="bpelx:bindingFault">       
            <condition>
                 <action ref="ora-human-intervention"/>
           </condition>
    </faultName>
    </Conditions>
     <Actions>
    <!-- RetryInterval is in seconds -->
      <Action id="ora-retry">
        <retry>
          <retryCount>3</retryCount>
          <retryInterval>5</retryInterval>
          <exponentialBackoff/>
        </retry>
      </Action>
       <!-- Human Intervention -->
      <Action id="ora-human-intervention">
        <humanIntervention/>
      </Action>
    </Actions>   
  </faultPolicy>
</faultPolicies>

fault-bindings.xml

This file binds the policies in fault-policies xml file with composite application level.
<?xml version="1.0" encoding="UTF-8" ?>
<faultPolicyBindings version="2.0.1"
                     xmlns="http://schemas.oracle.com/bpel/faultpolicy"
                     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <composite faultPolicy="ProcessingFaults"/>
</faultPolicyBindings>

Place policies and bindings files in same location as composite.xml
Steps for recovery from EM console:

• Log in  to em console and expand farm and SOA navigation trees on left pane
• Click on required composite on left pane
• Click on instanceId which is in running sate
• Click on recovery 
• Select the Recoverable record, in the bottom page segment choose required recovery action from drop down and click Recover button

This will recover instance form fault state.
In next post, I would like to talk about few best practices in configuring fault policies.