SOA 11g FOD (Fusion Order Demo)

FOD or Fusion Order Demo is the platform to learn any new Oracle product, I had used it while learning 10gR3, also AIA2.0, and now for 11g. I downloaded the FOD from here, the packaged deployment scripts were great and it just went through besides few hick-ups, tips here

However I couldn't execute the Order-booking demo end to end (some or other errors crop up oracle.jbo.SQLStmtException: JBO-27122: SQL error during statement preparation, or Caused By: javax.xml.soap.SOAPException: Message send failed: Connection refused), I could run the composite and played around em to learn the essential new concepts.

So from 10gR3(weblogic-based), 11g is much different and many important changes, let me try summarize some

1. Earlier we were creating 3 schema (orabpel, oraesb, oraowsm), now it’s more - mds, soainfra, bam and user messaging. MDS is the new thing, which is the metadata storage for all composites.

2. The terminology has changed, In one soa_server there are 4 service engines bpel, mediator (earlier esb), workflow and rules. The same server also has b2b, owsm. The bam server is separate and that’s new Java version.

3. The em is completely new, its shows all the composites, there instances, traces, component traces etc. I couldn't find test screens for components, its there at composite level. Admin stuff by right clicking on soa_infra.

4. Of course the whole idea of composite is new, and a composite has 3 parts service, components and References. So a lot of new config files, deployment fils etc. Complete new composite editor in JDev11g.

5. New frameworks around EDN (even driven networks), SDO (Service Data Objects), Spring etc. to make coding a different experience.

More details as and when I work on them, for now 11g looks just great!

SOA 11g install

finally I got some time to look at 11g and the first set of challenges were to 1) set Jdev 11g and 2) to install SOA 11g on Linux

1) For Jdev we need 11.1.1.2.0 above to be able to see SOA projects, After installing JDev the SOA composite editor can be installed using -> help -> check for updates

2) Installing 11g on Linux is covered in couple of blogs one here, official site here, broadly 5 steps
a) installing DB - 11.1.0.7 (linux_11gR1_database_1013.zip and patch 6890831)
b) using RCU for SOA schemas (ofm_rcu_linux_11.1.1.2.0_disk1_1of1.zip)
c) installing weblogic 10.3.2.0 (ope111130_wls1032_linux32.bin) using bundled Sun JDK
d) installing SOA 11g (ofm_soa_generic_11.1.1.2.0_disk1_1of1.zip)
e) configuring SOA for Weblogic

Good thing is this install is much easier that 1gR4 on weblogic9.2 as all configurations are wizard based.

For DB install all the steps are covered here - make sure all steps r followed also as in patch 6890831. For additional linux packages yum can be used. While dbconsole stop/start there will be an OC4J Configuration errror, the solutions is here

For weblogic/soa set up create a Middleware directory, chowner to the oracle user created as part of db install, as OUI will not work with root user.

While starting weblogic admin server if you see following error - your weblogic version is not proper, use 10.3.2.0

Failed to initialize the application 'em' due to error java.lang.NoSuchMethodError: weblogic.application.ApplicationLifecycleEvent.getDeploymentOperation()Lweblogic/application/DeploymentOperationType;.java.lang.NoSuchMethodError: weblogic.application.ApplicationLifecycleEvent.getDeploymentOperation()Lweblogic/application/DeploymentOperationType

While starting managed serevrs if you see errors like -
BEA-000386 Server subsystem failed. Reason: java.lang.NullPointerExceptionjava.lang.NullPointerException at weblogic.management.bootstrap.BootStrap.find(BootStrap.java:473)
or
BEA-000286 Failed to invoke startup class "JPS Startup Class", java.lang.ClassNotFoundException: oracle.security.jps.wls.JpsWlsStartupClassjava.lang.ClassNotFoundException: oracle.security.jps.wls.JpsWlsStartupClass at java.lang.Class.forName0(Native Method)

start the servers as root user and best way to start the managed servers is from the admin console, and for that start nodemanager, also make sure to modify nodemanger.properties under wlserver_10.3/common/nodemanager/ with StartScriptEnabled to true.

Next is to understand the big-bang changes in 11g and appreciate the value it has to offer.

bpel tuning, best practices and troubleshooting

Last week we went to production on SOA10.1.3.4 on Weblogic9.2, 3 node cluster, with lot of interfaces, and It was a very good exercise to prepare the servers, here are some references, if you read through all of them then you can avoid most of the issues ;)

1. tuning
check the apache-weblogic plugin parameters here
Oracle 10g tuning guide here (however lot of info here doesn't apply to weblogic)
tuning bpel quartz scheduler mentioned here
weblogic tuning guide here
linux parameters here (some good benchmark data here)
JVM parameters here (JVM Proxy settings here)
dehydation db tuning already covered as part of bpel tuning, however make sure DBAs tune processes, sessions, sga space, tablespace properly

2. best practices
if you have heavy batch jobs make sure you understand optSoapShortcut in cluster scenario, pls check this
metalink IDs for esb-bpel communication optimizations (780822.1 (esb-bpel) 726490.1 (bpel-esb))
bpel performance best practices here
esb performance best practices here , one more ppt, one more
a must read on bpel-adapter Integration here and optimizations metalink id [565944.1 and 730580.1]

3. troubleshooting
gcviewer to monitor gc memory issues
threaddump can be created on weblogic console, also can be created by kill -QUIT to monitor socket, thread issues
ORA-600 Errors Metalink ID [754336.1,18485.1, 333338.1 and 460244.1]
Cannot deserialize DOM element" Error When Running a BPEL Process [ID 559261.1]
FOTY0001: TYPE Error Using ORA:PARSEESCAPEDXML() and "&" in Input [ID 861637.1]
ORABPEL-11802 Error Calling Stored Procedure Using a Database Adapter [ID 887736.1]
BPEL waits well after expiry time set by wait or retry by fault policy[ID: 561172.1]

Careers in SOA

As the erstwhile EAI jobs become SOA jobs, it’s interesting to see how many dimensions are there to this job and how to enhance one's skill to be on top of it.

Technical (Type 1-4) - It is also possible that one person might have to be involved in all these types (also to be occupied through-out the project life-cycle), so one has to prepare for it. Also some of the work is product specific roles. By the way, learning the technology might not guarantee success, as they say attitude is the key :)

Work-Type1- This goes to the folks good in linux/unix, mostly the unix admin or dba - they can help in installing all the software (linux, database, app servers, jdk, etc.), RAC install, DR setup, backup activities, code migration, system upgrades, applying patches. These people can grow to be Infrastructure/Data Architects or Cloud Architects :)

Work-Type2- This goes to folks good in JVM, App Server, SOA Server, product architecture, call them 'Specialists' - they can configure the SOA/App servers, clusters, tune them as required, monitor system health, read log file, create SRs, finding patches, system sizing, create solutions for a product stack etc. These people can grow to be Technical/Solution Architects.

Work-Type3- This goes to the folks good in Java, XSLT, JMS, EJBs, transactions, SQL, PL/SQL, BPEL API, Adapters, they are the Developers - (Here I am not including the skills which are more primary in a E2.0 space including JSF, Spring, Hibernate, Struts etc., however a Developer may have to work on some of these if the project demands) . This group can grow to be Application/Solution Architects.

Work-Type4- Lets say this is the group of Architects, who take care governance aspects of the organization, including documentations as per Enterprise standards, coming up with the physical/logical architectures, enterprise security, hardware provisioning, roadmaps, big picture etc. This group is the Enterprise Architects.

Work-Type5- This is the group of project managers, sponsors who take care of planning, budgeting/financials, status tracking, business users reporting etc. This group is the Managers/Leaders of organizations.

I have not included the testing, business analyst, training roles in these categories. Also assumed all role types know SOA and its principles.

Email CSV File As Attachment

I had to create a bpel, which gets data from couple of db tables, and create a csv file and send it as email attachment. I bumped into couple of interesting challenges, luckily google is always there to help :), let me capture some of the lessons learnt.

1. creating a csv file in append mode
2. setting/getting filename in the fileadapter
3. handling multiple records, nillable fields in nxsd
4. interesting xslt, xpath functions
5. sending email with attachment


Creating a csv file using file adapter was quite straightforward, you have to get a sample csv file that you want to crate and give that to the fileadapter wizard to create an nxsd (native xsd). more on it creating csv file here reading csv file here

As the file was created in append mode (how-to here), the file name has to be unique, it had to be set by creating a fileadapter header variable and passing it in invoke. If the file name is created by fileadapter, its name can be obtained as prescribed in technotes 10.1.3.3 by adding an output of header type to the write operation.

I had to write a header to the file, and fileadpater supports multiple record types, however all these record types need a condition Value for each record type to follow, which was not possible to define for data w/o starting with any fixed value, so I had to abandon the idea of using records, I printed the header separately as all my fields were string type. So the nxsd can be tweaked to change datatypes or making fields as optional (nillable=true). more here


While working thru transformations came across very effective xslt functions like translate, which replaces any specific character in the string with any other value. function create-delimited-string creates a delimited string out of a particular node in a repeating XML structure.

Finally sending the file as email attachment, bpel sample sendEMailWithAttchment clearly shows how to do it, however I faced two challenges. One was if the ora:readFile cannot find the file it will throw XSLT error (here ), which is difficult to debug. And some reason my text/html data was overwriting the csv data, for which I changed the order or data setting, setting csv first and then text/html.

update-Nov10

I could not use the translate method to replace newline characters, the sql replace(col,chr(10),null) helped there.

Also if you get javax.xml.xpath.XPathExpressionException: FOTY0001: type error for nothing wrong in XSLT, its because you edited the XSLT even before bpel loaded the parts in the Xform activity, so always wait till the parts load before clicking edit for the XSLT.

Update Dec-7

In order to make the directory to which we write the file as dynamic, we can edit the Outboundheader xsd to add 'directory' there after fileName. And send te directory name as a preference to the bpel at runtime.

transaction=participate

The first thing to note is global transactions will only work among sync processes, so an async bpel/esb cannot participate in a global transaction, so the way to make an async process sync is set delievryPersistPolicy to off.immediate.

Now partnerlinks in sync processes participate in global tx in 2 ways
1. set transaction=participate at the partnerlink (child BPEL or esb call) level
2. set transaction=participate at global level under in bpel.xml (It seems default in 10.1.3.4) - This is for all adapter calls (DBAdapter, AQ, JMS etc.) to participate in global tx

Now for the global tx to work, you have to use datasource with global tx enabled and using an XA driver(may not be true XA).

So when there is error (bindingfault) from dbadapter calls, it will mark the tx for rollback, but it will actually not rollback.

What we have seen is you have to set handleTopLevelFault=false in the configurations/bpel.xml so that rollback works. The other approach is to throw rollback exception in catch-all block.

The following seems to be best-way for global tx handling (10.1.3.4 on weblogic 9.2)
1. XA driver with Global tx enabled
2. in catch-all throwing rollback exception

This is rolling back the db inserts, as well dehydrating the bpel.

As bpel dehydration is happening, it seems it’s happening in a separate thread/tx (need to check if its a supported feature in 10.1.3.4). However we also saw the instance getting into recover(invoke) while using handleTopLevelFault=false.

a good discussion in OTN here and here
some good bpel tx details here and here

update on 23/10
10.1.3.4 does support audit in a seprate thread in async mode as per release notes.

ORABPEL-11825Attempt to use an unsupported database platform.

If you get this error -

WSIF JCA Execute of operation 'T' failed due to: Attempt to use an unsupported database platform.Database platform is not supported: oracle.toplink.platform.database.DatabasePlatform; nested exception is: ORABPEL-11825Attempt to use an unsupported database platform.

Then DBAdpater configuration needs set the PlatformClassName, sample values

• oracle.toplink.platform.database.Oracle9Platform
• oracle.toplink.platform.database.DB2Platform
• oracle.toplink.platform.database.SQLServerPlatform

you can also take this value from the MCF property settings.