Oracle SOA 12c: BPEL callback processing is taking much longer than expected for Flow N activity

In one of my bpel project when I was using parallel Flow N activity, It is expected that BPEL will process callbacks from asynchronous invocations within a couple of seconds but under load it's found that there is an unexpected delay of around a minute. Sometime process got timed out.

Then we found out that When using Parallel Flow "flowN", in a clustered environment and one of the nodes is executing a particular BPEL instance, it acquires a DB lock for that instance and no other nodes can execute the same BPEL instance at the same time.  

This results in the messages (including callbacks) on the remaining nodes being rescheduled and delayed for a configured amount of time.

Solution:

Apply following JVM option and carefully reduce its value: 

-Dbpel.dispatcher.contention.rescheduleDelay=[Number less that 60] (Default is 60 sec) 

IMPORTANT NOTE: Reducing the rescheduleDelay value could create a DB contention so it's advisable to carefully tune it.

These settings solved our issue.