Changing an Async BPEL to Sync (Calling WS from pl/sql)

This week I got an interesting task of calling a BPEL from PL/SQL. It seemed much easier than I initially thought. There are utilities available to do it and I got the references here and here. Initially the code as mentioned here didn't work. Then marC's blog code worked fine. One caveat to this whole exercise was this web service invocation from pl/sql only works for synchronous services. As for async services the client needs have the capability to receive a callback soap call.

Initially my bpel service was a async one and the pl/sql code though was able to invoke it was not able to recieve the callback. So I thought of changing it to synchronous. What followed was an interesting discovery. So firstly some basics, In bpel what makes a partnerlink async or sync is the role definition. For async you would see both partner role and my role, however for sync you will see only my role(for inbound) and partner role (for outbound).

I tried to change my async bpel to sync, by just removing the call back porttype/operations/bindings from wsdl, added a reply activity, changed the initiate operation to process and added a output message. From wsdl/bpel perspective everything was fine, but It didn't work. I was getting in bpel-console - Failed to get the WSDL operation definition of "process" in portType


After some googling, It seems there is a problem, after changing the operation name or async to sync, refreshing the wsdl, it might not work, the solution is to deploy a new version 2.0 - It worked for me. Then just deploy the 2.0 as 1.0 (In Jdev we can do this) - everything gets right.

Plain Vanilla BPEL #2 (Adapters)

I noted some of the day1 bpel issues here, there are also some interesting gotchas on the adapter side. Adapters are a big area covering various issues over file/ftp, database/aq adapters, JMS and Apps Adapters.

ftp adapter - I had used an outbound ftp adapter w/o much issues, inbound ftp adapters in cluster environments have to be made singleton, refer to this

db adapter - db adapter one can do 4 types of things 1) call stored proc 2)execute Insert/Update, Delete, Select operations 3) poll database table 4) execute custom sql

While using select operation parameters can be passed using #parametername to the generated SQL. While using custom sql XSD is automatically generated as per the SQL.

aq adapter - aqs can be created of type multi-consumers, in such cases consumer name has to be used while creating adapters to send/receive messages.

I will add more as we go along.

update on May/28/09
In the mean time a lot of adapter issues have been found, so that needs a mention here. Firstly FileAdapter/FtpAdapter, we had some good challenge around getting append to work for FtpAdapter. The format of the file was based on nxsd. We tried a bunch of things around making sure the transformation is correct to nsxd, making sure ftpadapter configuration (messagecount=1, unique filename, opaque schema) are correct, but append just didn’t work. After further debugging it turned out the ftp server didn’t support append in that particular directory. So a good learning was to test the ftp server configurations by doing simple ftp operations like put/get/app etc. from the prompt and then test from the ftpadapter.

One other learning was the schema used in ftp adpater can be opaque or xsd. If it’s opaque, we need to send a base64 encoded string. And the following is code snippet to do that, make sure your MXL is converted to string by using getContentAsString()
String inString = (String)getVariableData("XMLString");
sun.misc.BASE64Encoder encoder = new sun.misc.BASE64Encoder();
String encodedStr=encoder.encode(inString.getBytes())

JMSAdpater, We had to connect weblogic JMSAdapter from OC4J and this blog had all that we need. And it worked. A good learning was we were using weblogic.jms.Connectionfactory which is a default connectionfactory, so we don’t see it in the admin console, so making the right configurations on oc4j-ra.xml and putting the right entries while creating the JMS Adapter wizard are the key points.

an erlier post on aq-jms on Weblogic.

Another day#1 issue is JCA port configuration in the adapter wsdl configuration is to use mcf (connection details as created by Jdev) or the JNDI url created in admin console. If you have both how will it behave? best thing we found is to use the one in server and not use mcf at all.


Update on Aug/19/09
DBAdapter if the mapping order is different in toplink mapping and XSLT, it throws cannot insert null error, the solution to that was to use detectomission to false in the dbadpater wsdl.

On another note, its possible to change the JNDI for outbound adapters, by using similar mechanism as per dynamic partner links, for dynamic partnerlinks here is a link.

On DBAdaper pooling, there is a distributedpolling setting for it to work in clusters.

FTPAdapter, one catch is the directory specified is relative to the user default directly after login.


update 1/7/2010
While creating nxsd for a file with header/detail structure, the file had lot of field length element separated by space, and using the wizard the lengths had to be specified, I noticed while using the ruler, you should not put the last position which is the length of the line or else it throws error. good reference here

Plain Vanilla BPEL

This month I had to work on couple of bpel processes and it reminded me last yr when I was working on my first bpel. Anyone starting on bpel development goes through a series of similar hick-ups to get a comfort feel on the subject.

A plain vanilla bpel is one, which takes an input xml in one format and sends another xml in a different format. So there are 3 areas of challenge that we learn about bpel on day1 namely 1)xsd of the messages 2)transformation 3)adapter details

Each area has some unique challenges, XSD needs some good understanding of XML namespaces, creating elements and types, importing/extending xsd and wsdls etc. Generally biggest challenge in bpel lies in XSLT, particularly if you have complex mapping requirements. JDev's XSL Mapper is a good starting point, It helps you visually verify both the XSDs and do some preliminary mappings, how ever if you have to use some if-else kind of logic, the mapper doesn't work, so we need to fallback on the source view and do hand coding of XSLT. XSLT is a functional language and it doesn't follow the Java coding principles, and you will first be surprised at how you cannot do many things that you easily do in Java. For example, you cannot change a variable value in XSLT, it’s like a constant. Then there would be many learning around XPATH expressions, how to pass parameters to XSLT, single quote, double quote issues etc.

Also many XSLT methods like ora:getInstanceId() or ora:getFaultString() doesn’t work in XSLT, so they have to be assigned separately to the resulting xml.

One best practice is to ‘test the XSLT outside of bpel’ and then use in bpel to avoid the endless loops of build/deploy/test the bpel. On the Console the XSLT error will also not be clear, it will give a FOTY0001 kind of code, and we have to check the actual error in server log.

One other best practice is to use addAuditTrailEntry or System.out.println to verify the variable values in bpel runtime.

I will cover the adapter issues that we face on day1 in another blog.

update7/2/10
good article on using the position() as a variable while selecting something in XSLT here

SOA Suite Skill Map

While working on SOA Suite various areas of technology criss-cross each other, Its sometimes overwhelming to look at issues in a structured manner. From a Reference Architecture perspective, things are clear, they stack up as follow

1. User Interaction Layer (Portal, Composite Applications, BAM)
2. Connectivity Layer (Security, ESB, OWSM)
3. Integration Layer (SOA Suite, BPM, B2B, ODI)
4. Infrastructure Layer (Weblogic Suite, Coherence)

Also the cross-cutting Layers of IDE, management and Governance

Coming back to SOA Suite - I divide it into four major buckets
-BPEL
-ESB
-Adapters
-App Server (OC4J or Weblogic)

Here is the list of items that would come under each category, I have split each category/area into Development and Administration parts - This can help to map issues to one of these areas/items and so in troubleshooting. I plan to keep adding to the list, basically very high-level taxonomy that in turn will have lot of details in the development/administration guides.

updates on 7/22/09
Two other aspects of SOA Suite are increasingly being used and require skills on are
-BAM
-B2B

Error Handling in BPEL #2

As I noted about BPEL Error handling here, SOA Suite 10.1.3.3 higher has a Error handling framework in form of fault policies, which is very useful to set up automatic recovery requirements in bpel.

Basically a fault policy says what to do (action) for which fault (condition). And it can be attached to a partnerlink, port type, process or domain. Attaching a fault policy to partnerlink, porttype or process can be done by including it in the bpel.xml. Domain level fault policy attaching can be done in fault-bindings.xml under each domain in $BPELHOME/bpel/domains/"domain-name"/config

Any error happening while invoking a partnerlink, will get captured by this framework and based on the fault policy, action will be taken. The pre-defined actions are retry, rethrow, human-intervention etc.

If we use rethrow as action, the fault will go back to bpel and will be handled by any catch/catch all blocks.

By default, the error handling framework doesn't do anything, as out of box, no policy is configured.

please check this for more information

Update on 6/Apr:
to refresh fault policy changes, server restart is not required, we can use the following link http://host:port/BPELConsole/domain_name/doReloadFaultPolicy.jsp. Also while creating a faultpolicy file, dont forget to put the correct Id name in the faultPolicy tag same as in bpel.xml

Update on 30/Apr
While showcasing faultpolicy to a customer, it became clear that businessfaults couldn’t be handled by fault policies, only technical faults. Which is quite a disadvantage, however its possible to handle business faults by converting the partnerlinks to separate bpel processes and then throwing the businessfault. Also for customizing fault-policy to do requirements like notification or calling another bpel also seems to be a challenge. Can we get the ora-java in the actions available in bpel console? need to check that.

AQ-JMS on SOA Suite10.1.3.4 on WebLogic9.2

There are 3 options in SOA Suite on Weblogic for messaging
-AQ Adapter
-AQ with JMS
-JMS Providers
--OC4J JMS Provider (only on OC4J)
--Weblogic JMS Provider
--Any other 3rd party JMS provider

Aq is the native queuing mechanism in Oracle, which uses database to create queue tables and queues/topics.

Other JMS Providers primarily use a in-memory/file based approach to create queues and topics.

To create Aq/JMS adapters on SOA Suite on Weblogic, here are the steps -

AQ/JMS basically means AQ queues with JMS message type. AQ/JMS supports SYS.AQ$_JMS_TEXT_MESSAGE, SYS.AQ$_JMS_BYTES_MESSAGE, SYS.AQ$_JMS_STREAM_MESSAG, SYS.AQ$_JMS_MAP_MESSAGE, SYS.AQ$_JMS_MESSAGE.

For the sqls to create the queue tables/ queues here is a reference

Next what we realized AQ/JMS is not supported on weblogic yet, so we had to use a custom code created by Robert Patrick

The code is also available here

Basically - we need to create a Weblogic startup class AQJMSStartupClass.jar, and use one property file, couple of user credential files, and a couple of jars (aqapi13.jar and ojdbc14.jar), we followed the readme file and created the required files.

Next we created a weblogic startup class with the necessary parameters, and restarted the soa server, and it basically used the startup class to create the AQJMS JNDI entries for QueueConnectionFactories, Queue etc.

Then we created a JMS Adapter entry in weblogc-ra.xml to use the AQJMS_QueueConnectionFactory. Used this JMS Adapter JNDI name in the bpel wsdl.

This just worked great, we have seen some AQ related errors in log, which I will cover in next post.

update on Aug/19/09
The AQ realted error like 'Cannot delist resource when transaction state is committed' got fixed after applying MLR#8

Error Handling in BPEL

Error handling is always a key core service on BPEL and there are always confusions around how to handle it,

There are various aspects to error handling in BPEL/ESB
1. esb errors
2. bpel partner link errors
3. bpel non-partnerlink errors

The simple answer is catch and catch-all

catch is for system defined faults like bindingfault, remotefault, selectionfailure etc. There are 12 of them that we can see in the Fault explorer. Once we catch them, we create a variable of type RuntimeFaultMessage from RuntimeFault.wsdl (under SOA_HOME\bpel\system\xmllib)


catchall is for everything else, and we can get the error String by ora:getFaultAsString()

The best practice to handle these faults in catch/catch-all blocks is to do the following steps

1. create a custom fault message, based on all the values we need to capture for example processid, instanceid etc.

2. send the fault message to a custom error handler, which basically can log the error, send it to worklist or send an email

3. send a reply back to client about the error message, this would need the wsdl to have provisions for sending fault data as part of response or fault

4. Terminate

Alternatives we can also use ThrowFault in step#3 if the wsdl has provisions for faults. However I found that while using throwFault step#2 gets rolled back, so adding a checkpoint() resolves it.

SOA Suite10.1.3.3 onwards also has the built in fault policy to handle errors. Basically whenever partnerlink errors happen, bpel checks with fault policy to decide what to do. In the fault policy we can configure different actiontypes like retry, rethrowfault etc.

I will cover fault policies and error handling frameworks in more detail in another post.

Clusters Vs Grids

Recently I worked on a Coherence data grid and then to set up a SOA cluster on Weblogic. That made me wonder what is the difference between Grid and Clusters. So couple of goggling and there are many links that explain it. However here it is in my own understanding.

Grids: Grid computing would be something to do with to optimize the resource usage like CPU, memory or IO. When we built a Coherence Data grid, It’s basically a cache server to cache the data so that we can avoid the expensive database trip. So we reduce the IO and make the application faster. And these cache servers can be started many in number across machines, they all talk to each other using multicast communication. Which is actually the same technology used in the Weblogic clusters to keep the managed servers in sync. Similarly other grids optimize CPU usage, which would otherwise be unused and so waste of money to keep and maintain them. Server Virtualization is a Grid solution.

Clusters: on the other hand basically provide scalability, so that the application can support more a more user requests, also with high availability due to failover support in cluster aware managed servers. Clusters are an extension to distributed computing and it lets scale the application dynamically meaning add more managed servers anytime you have more demand. Similar failover and dynamically adding more servers is supported in Coherence Grids as well. Coherence Cache servers also partitions the data across multiple servers thus providing more scalability.

Interesting thread on weblogic clusters here
Interetsting discussion on scalability here


So basically the underline technology like multicast communication could be same, and the results like performance (response time), scalability (throughput) could be same, however Grids and Clusters differ in the basic goals they work on. It would not be wrong to say Grids will help you scale up while Clusters will help to Scale out.

SOA Suite 10.1.3.4 on Weblogic HA installation

Recently we did SOA Suite 10.1.3.4 on Weblogic simple HA installation following the enterprise deployment guide. This went quite smoothly without much issues. However we had to redo it once, as one key thing doing this installation is - the directory structure of SOA_HOME and BEA_HOME has to be same in both the nodes.

We were doing this remotely so, to start with we got the vnc access to the both the boxes, also make sure 8001, 9700, 9701, 9702 ports are open for you.

steps are quite straight forward
1. set up RAC DB, we went for a single DB, make sure all DB initialization parameters are set
2. install SOA Suite 10.1.3.1, patch 10.1.3.4 basic install, remember that if you try to access EM after these steps, it will not come up as ascontrol webapp by default is not enabled, so you can do that using this
3. install weblogic, then run the weblogic scripts to configure the domain, clusters, nodes, datasources, and deployapps
couple of things here to note are - make sure to deploy the datasources to cluster, if you get error in any scripts related to unable to get lock you can follow this. Also start SOA Server from Admin console and remmeber we dont need to use opmnstart/stop here.
4. after setting up apphost2 as per the document, we were successfully able to bring up all the consoles, we did one deviation form document for esb_dt setup, instead of setting 9700 in esb_parameter table we set 9702
5. next was webhost and loadbalancer setup

From a deployment perspective, we are deploying our bpels to both the nodes and using roundrobin for load balancing to both the nodes.

We will be exploring more to validate the setup and failover scenarios.

Update on 2/mar/09:
We faced a major issue when we installed one more SOA/HA/Weblogic Environment and by mistake we used the same multicast ip/port. We had to reinstall both the setups the earlier one and the new one again. We got some multicast errors while starting server2, which is
Error: Cluster: BEA-000110 Multicast socket receive error: java.io.EOFException
java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readLong(DataInputStream.java:380)
at java.io.ObjectInputStream$BlockDataInputStream.readLong(ObjectInputStream.java:2744)
at java.io.ObjectInputStream.readLong(ObjectInputStream.java:941)
at weblogic.cluster.HeartbeatMessage.readExternal(HeartbeatMessage.java:55)
Truncated. see log file for complete stacktrace


So one has to be very careful with multicast ip/port being unique with every install at the install time itself, changing it later on doesn't help.

updated on 13/mar/09

While our weblogic cluster was working fine, we found some issues while accessing one of the bpel consoles as 404 - Not Found. This made us look into BPEL Clustering on top of Weblogic clustering. The SOA-HA Faq document is very useful to understand this. Basically the BPEL nodes talks to each other to sync processes, any changes in bpel.xml, adapter state etc. And it uses the jGroups configuration to do this. collaxa-config xml has to be updated with 4 changes as per the EDG. The jGroup changes are little tricky. Basically jgroups-protocol xml file has to have a unique multicast ip address (while using UDP not TCP) and if the servers have multiple ips then bind_to_all_interfaces can be made false and bind_addr can be added with the one ip. While our jGroup settings are fine and bpel nodes seem to propagate new processes without restart, we still have a 404 issue, which seems to be something to do with the network ips.

BPEL-Weblogic ClassLoader issue for Coherence

I had to deploy some existing code developed on SOA Suite on OC4J to SOA Suite on Weblogic, It was a good struggle for me, so here it goes -

The code basically is a bpel calling a java web service using wsif. And the java service uses the caching software coherence to read/write to an in memory cache.

1. First of all the java web service in Jdev didn't get deployed to weblogic - It gave me - java.lang.IllegalStateException: could not find schema type named {http}//weblogic/types/}getHelloResponse" (edited the actual type).

After some googling, it seems there r interoperability issues - so I had to use weblogic ant tasks to create/deploy the web service. Though the ant tasks like wsdlctask etc. were available my ant script didn’t run as it didn’t get xmlbean libraries, which I too couldn’t find in weblogic/lib. So my next attempt was to try develop this in workshop, which I am still trying as workshop didn't like my localhost weblogic server SOA domain, created as part of the SOA Suite on Weblogic installation scripts.

2. Since my java web service attempts were not getting anywhere, I decided to package the Java class (which would be called by WSIF anyway) in the bpel itself. That went pretty well, my wsif binding started working after a few issues with the namespace in my schema, bam code in bpel (which I removed later as it was throwing JCE Error asking to put new jars in jre/lib/ext at deployment). I couldn't find a jaxb ant task to replace the schemac, so I had to go with schemac to generate the java binding classes form the wsdl/schemas. Here too I faced compilation issues with the classes generated, so I had to edit the schema to remove the (maxOccurs="100000") entries. So finally when my bpel was ready with everything else, I had to test the coherence part.

3. The coherence part was the real hurdle to cross, by packaging the coherence.jar in my bpel output jar, I couldn't start the CacheFactory - It was complaining - Caused by: (Operation failed!; nested exception is: (Wrapped: Failed to load the factory) java.lang.reflect.InvocationTargetException - Wrapped: Failed to load configuration resource: coherence-cache-config.xml) java.io.IOException: Configuration is missing: "coherence-cache-config.xml", loader=null. So I tried putting the coherence.jar in the weblogic server classpath, and then it was working fine. I was able to create the CacheFactory and just then when I thought its done, the put/get to cache started throwing - (Wrapped) java.io.IOException: readObject failed: java.lang.ClassNotFoundException: com.schemas.Customer.CustomerQueryResultSet
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method).

Finally with the help of Coherence support, I learnt that I need to pass the classloader to CacheFactory.getCache() as the Classloaders get different when we go out of bpel domain to the coherence code and vice-versa. After passing the classloader, the whole thing worked fine.
sample code below -
NamedCache cache;
Thread thread = Thread.currentThread();
ClassLoader loaderPrev = thread.getContextClassLoader();


try
{
thread.setContextClassLoader(com.tangosol.net.NamedCache.class.getClassLoader());
cache = CacheFactory.getCache(sName);
}
finally
{
Thread.currentThread().setContextClassLoader(loaderPrev);
}

Oracle Middleware

Oracle Products are primarily of 3 types
-Database
-Packaged Applications
-Middleware

Oracle Middleware products are categorized as

-SOA/BPM
I have already written about SOA/BPM line of products here
-Web2.0
Under this Enterprise Content management, portal servers from both the oracle and bea products.
-Application Grid
This has basically the app server (weblogic, oc4j) and the various grid technology products like Coherence, web cache etc.
-Identity management
-BI/EPM

Each category has lot of products for any kind of customer requirements and application patterns.

2009++

Recently I worked on a POC to call coherence java code from SOA Suite using WSIF binding, while trying to google up on the subject, found an interesting article, on 'real-time SOA'. This term 'real-time SOA' kind of hit me and I started googling up more, and hit upon many articles in sys-con on cloud computing. After Going through couple of you-tube videos on cloud computing and looking at the list 100s of companies working on related technologies like SaaS, Virtulaization etc. - I am amazed what the future holds.

I just completed 10yrs in IT as a developer in the outsourcing industry, and as I understand this decade (98-08) is what can be branded as web1.0, soa1.0, cloud computing1.0 - all 1.0 - and the next decade is going to be 2.0. We are already seeing it - be it SOA/BPM or Web2.0/AJAX, the 2.0 is starting to unfold.

After Listening to Bill Coleman here - I am right now just sleepless thinking how exciting the future looks.

My Oracle SOA Experience log -

From Apr/08

Installation of SOA Suite 10.1.3.3
Installation of AIA FP2.0.1
MDM PIP Development
Upgrading SOA Suite to 10.1.3.4 on Weblogic9.2

BPA suite installation
OSB installation and POC
Oracle Coherence

All installations on Windows laptop.

SOA Suite HA installation on Weblogic9.2/Linux

Oracle SOA Offerings -

Oracle SOA Products are packaged with multiple related products as Suites like..

1. SOA Suite
2. BPA Suite
3. EDA Suite
4. ODI Suite
5. BPM Suite

SOA Suite has
-BPEL Process Manager
This is the primary product used for orchestration of services. This is BPEL standards compliant product with support for management, high-availability and hot-plugability.
-ESB
There are 2 ESBs in Oracle product offerings, OSB (this is the earlier Aqualogic service bus) and OESB (Oracle ESB). OSB is for more standalone enterprise-wide connectivity requirements and OESB can be used a project based mediation requirements.
-Rules
-human workflow
-OWSM
Oracle Webservice Manager is the Gateway product, which provides security to the webservices hosted on BPEL or ESB.

SOA Suite runs on Weblogic and OC4J.


BPA Suite
-Business Process Architect/ Publisher / Simulator / Repository

BPM suite
-Oracle BPM (this was earlier Aqualogic BPM)
-Oracle BAM

EDA Suite
-Oracle CEP (earlier Aqualogic Event Server)

SOA Governance products
-Enterprise Repository
-SOA Management pack

ODI Suite
-Oracle Data Integrator
ODI is for bulk data load requirements.
-Oracle Data Quality and Profiling
-Oracle Coherence
-Oracle Adapters

Based on the customer applications requirements, the right product mix should be used to realize the enterprise goals.

Patterns

I love this word 'Patterns', It kind of summarizes everything that we learn in IT world. We have seen so many categorizations of patterns over last 10-15 yrs -
Design Patterns (GoF , Gang of four)
J2EE Patterns
EJB Patterns
SOA Patterns
Architecture Patterns
Integration Patterns
Application Patterns

In 2003, my friend Satya and I used to write a tutorial on GOF Patterns, Its still available here

AIA

AIA - Oracle's SOA Architecture for PIP development.
AIA - application integration architecture
SOA - Service oriented architecture
PIP - Process Integration packs

In the journey from EAI to SOA, somewhere people started using BPEL for building integration flows. XML, XSLT, Adapters are the common technologies, whats new is WebServices or more specifically WSDLs.

In the Oracle eco system, there are many packaged Applications like Ebusiness suite, Siebel, Jd Edwards , Peoplesoft etc. And It's a common requirement to have integration flows between them in various customer implementations. So in Oracle we offer PIPs build over SOA middleware using AIA architecture.

10

I just completed 10 yrs in Indian IT industry starting as a developer and growing to Architect roles. Most of my experience was in the J2EE application development space. Last 2 years I have been working in the SOA space. Started with learning about SOA Methodology, Maturity Models, Technology Standards and Governance. looked at IBM Stack of products and now working using Oracle/BEA SOA stack products for last 6 months.

Oracle/BEA Stack of SOA products looks very exciting and I feel very passionate about its roadmap and also hoping doing exciting work in this space.

I am planning to write my experience in this blog from day to day development issues to architectural insights.

SOA blog

SOA - Service Oriented Architecture

SOA is an IT architecture around creating services for the business processes for maximum reuse, faster time-to-market and cost reductions. Web Services would be the key technology for the service implementation. Web Services are based on communication using standards like SOAP, WSDL, UDDI and many others.