SOA composite deployment coherence issue

While deploying a composite to a soa cluster - the deployment was stuck for more than 20 mnts. To make it worse when it was cancelled/retried it corrupted the MDS causing soa-infra to fail while restart.

Clearly the logs showed STUCK THREAD

<[STUCK] ExecuteThread: '56' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "602" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 602461 ms
[
POST /soa-infra/deployer HTTP/1.1
Connection: TE
TE: trailers, deflate, gzip, compress
User-Agent: Oracle HTTPClient Version 10h
Accept-Encoding: gzip, x-gzip, compress, x-compress
ECID-Context: 
Authorization: Basic amNoZW42Ol8xYW1BZG1pbg==
Content-type: application/octet-stream
Content-Length: 69483

]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-701 "[STUCK] ExecuteThread: '56' for queue: 'weblogic.kernel.Default (self-tuning)'" {
    -- Waiting for notification on: java.util.HashMap@4343a522[fat lock]
    java.lang.Object.wait(Object.java:???)
    oracle.integration.platform.blocks.deploy.CoherenceCompositeDeploymentCoordinatorImpl.submitRequestAndWaitForCompletion(CoherenceCompositeDeploymentCoordinatorImpl.java:352)
    oracle.integration.platform.blocks.deploy.CoherenceCompositeDeploymentCoordinatorImpl.coordinateCompositeRedeploy(CoherenceCompositeDeploymentCoordinatorImpl.java:255)
    oracle.integration.platform.blocks.deploy.servlet.BaseDeployProcessor.overwriteExistingComposite(BaseDeployProcessor.java:487)
    oracle.integration.platform.blocks.deploy.servlet.BaseDeployProcessor.deploySARs(BaseDeployProcessor.java:298)
    ^-- Holding lock: java.lang.Object@73823526[thin lock]


The soa-infra error

weblogic.application.ModuleException: [HTTP:101216]Servlet: "FabricInit" failed to preload on startup in Web application: "/soa-infra".
oracle.fabric.common.FabricException: Error in getting XML input stream: oramds:/deployed-composites/AccountBS_rev1.0/composite.xml: oracle.mds.exception.MDSException: MDS-00054: The file to be loaded oramds:/deployed-composites/AccountBS_rev1.0/composite.xml does not exist.

In case of soa-Infra error this blog has steps on how to recover

The deployment STUCK THREAD issue points to coherence related issues, there are many useful troubleshooting documents on oracle.support

General Coherence Network Troubleshooting And Configuration Advice (Doc ID 1389045.1)

Coherence and SOA Suite Integration Recommendations (Doc ID 1557370.1)

Troubleshooting Tips for Coherence - Oracle Service Oriented Architecture (SOA) Suite Integration Issues (Doc ID 1388786.1)

"oracle.integration.platform.blocks.deploy.CoherenceCompositeDeploymentCoordinatorImpl.submitRequestAndWaitForCompletion" Error and Slow Response While Accessing Composites In EM Console (Doc ID 1437883.1)

SOA 11g Composite Deployment Results in Stuck Thread Error: <[STUCK] ExecuteThread - Unable to Deploy the Composites in a Cluster (Doc ID 1086654.1)

SOA 11g Health Check: Verify Consistency of Coherence wka and wka.port Configuration (Doc ID 1578203.1)

SOA 11g: How Many Nodes are Required to be Specified as Coherence WKA Members in a SOA/OSB Cluster? (Doc ID 1511706.1)

Stuck Threads during SOA Cluster Deployment (Doc ID 1564586.1)

IpMontor Failed To Verify The Reachability Of Senior Member (Doc ID 1530288.1)

OSB-SOA-OSB zig zag pattern

Recently we ran into performance problems with some of our services following a OSB-SOA-OSB pattern, most of the services follow a OSB-SOA pattern, but for some where the tuxedo transport is used we ran into this pattern as OSB has the tuxedo transport unlike SOA suite.

As we all know OSB is stateless and SOA is generally not stateless and needs a DB, services following a OSB+SOA pattern, also need a consistent logging solution. These custom logging solutions are generally JMS based asynchronous solutions where a listener picks up the log messages and write to db.

There are multiple challenges in a OSB-SOA-OSB-SOA pattern

1) if OSB and SOA domains are separate - then there will be network hops while calling OSB to SOA and other way round - we can use the soa direct (t3) based communication but it has it's own challenges

some of the challenges are
a) if you are using t3 - you cannot use load balancer url - so have to be careful to give all managed server node urls (and test that load balancing is happening)
b) you cannot set a time out on these calls (mostly the JTA timeout takes effect?)
c) there could be additional complications if you use owsm policy at endpoints
d) transaction behavior could also be a challenge

2) having a consistent logging solution, as it is better to do logging as near to the source as possible, we should deploy a common solution for both OSB and SOA, so you would have to set up JMS queues in both OSB and SOA and deploy the code in both environments

3) The complete solution might not scale well - as the threading models for OSB and SOA are very different, and when you have a OSB to SOA to OSB and multiple of such calls - some thread deadlock situations will not be surprising


what was our problem?

we faced a huge latency in our response time, it turned out we were using the publish activity in OSB to do our logging, however publish is not really asynchronous if you publish to a proxy service

this blog helped us to confirm this.

We were publishing to a proxy service which publish to a JMS based business service - if the JMS configurations are not correct - this whole thread waits for a few seconds.

The other problem we had is the tuning of OSB and SOA, There are lot of material available on tuning - however few critical learning are

1) OSB - using workmanagers is absolute must to avoid any problematic areas going out of proportion and bringing down the node, work manager will help contain any spike that a service might have and consuming resourced causing others to starve

how many max threads to assign, assigning work managers to all services or only to some services, assigning to only proxy services or both proxy and business services - these are again areas to tune and test.


2) SOA - if audit is made 'off' at soa-infra level at least a 1 sec drop in response time was noticed, soa db connections, use of Gridlink datasource are also important tuning parameters.

some interesting db queries to check soa table space size here and time taken by soa components here

In summary, we managed to improve response time and throughput but OSB-SOA-OSB patterns will always have a bit overhead compared to only OSB option or only SOA option.

My recommendation would be to have OSB+SOA in same domain/node and then leverage soa direct transport to co-locate as much processing as possible, that would be a much faster option, I believe with 12c such domain topology might become more popular.

The Hadoop Puzzle

"Big data is at the foundation of all the megatrends happening today" - when I saw this here - It made perfect sense, with all the different applications of the hadoop technology. I try to explore a few trends in this blog.

hadoop core

At the heart of hadoop technology there are 2 parts

• The distributed file system or HDFS
• The data processing part using MR (Map-Reduce) on HDFS

And the overall master-worker clustering technology that makes everything work 

where does hadoop fits in the new world?

Typically we have applications in the pattern of OLTP + OLAP

OLTP is the Transaction part of it based on RDBMS

OLAP is the DW+Analytics part of it based on also RDBMS Or Advanced MPP databases such as Teradata

so the typical flow of data is

OLTP DB --> (ETL) --> DW --> Analytics and Reporting

When you put hadoop in this flow, you get at least 3 types of new flows

1.     OLTP DB, Other sources --> (ETL) --> Hadoop --> DW --> Analytics and Reporting

2.     OLTP DB, Other sources --> (ETL) --> Hadoop --> No SQL DB --> Online Applications

3.     OLTP DB, Other sources --> (ETL) --> Hadoop --> real-time Analytics

This can also be explained as

1.     batch processing

2.     online processing

3.     real-time processing

batch processing

This is the classic application of MR (Map-Reduce) to processing HDFS data, MR code is written in Java. Pig Latin is a language developed by yahoo to generate MR code.

Hive is a SQL like database on top of HDFS also uses MR.

Mahout is a machine learning library on top of HDFS also uses MR.

This application basically fills in the gap in existing technologies to process large datasets.

online processing

This is a scenario where a No SQL database such as HBase, Cassandra is used. No SQL databases are distributed databases unlike RDBMS hence supporting infinite scale. HBase is based on HDFS but doesn’t use MR, Cassandra can be standalone or based on HDFS. Such a data store can be used as a backend of a web applciation. web log analysis can based on such an architecture.

This application basically fills in the gap in existing technologies to store large datasets for faster online access.

real-time processing

This is the in-memory option of faster data processing using products such Spark, Storm on top of HDFS, obviously they dont use MR. Spark can be based on cassandra as well without using HDFS.

This is the most exciting application of hadoop as it really can enable many new application styles and trends.

In summary, many permutation and combination of hadoop core to create many kind of applications and mega trends.

(my apologies for not expanding a lot of acronyms and not providing links, please google on any word if you found interesting - also please leave me a comment if any questions)

few more OSB stories

Recently we finally went-live with a OSB service which on avg, takes 11 secs (it does a lot internally) with 100K+ transactions per day. Two problems we had during this effort


1. StackOverflow Error

There were quite a few Satckoverflow errors, which turned out to be because of a x-path query below

declare function xf:escape-for-regex( $arg as xs:string? )  as xs:string {

      

   replace($arg, '(\.|\[|\]|\\|\||\-|\^|\$|\?|\*|\+|\{|\}|\(|\))','\\$1')

} ;

This function was used to retrieve a certain error message from a complex structured error response, however with some type of error responses, this went into a infinite loop  causing stackoverflow errors.

What was interesting is, weblogic server survived these errors quite gracefully without causing too much trouble with 100+ such errors happening every 10 mnts.

What's even interesting is we didn't see this error in dev environments, so it was a bit inconclusive if the error was due to a certain resource constraint or purely due to the function.

2. Runtime-configuration

One thing I struggled with OSB is to set some global parameters to be set during run time (BPEL has support for runtime global parameters), customization files don't allow any custom properties other than supported by it. Only option was to use some kind of property file and read it through Java call outs. One option we used was to use Xquery files to define these properties. But it was not possible to change these during deployment or runtime. Other option is to keep it in database table and modify through some kind of script or screen.

How I built my first mobile app

When I thought of building an app the first question was what platform to use php, java, node.js etc.. I wanted a mobile-first approach, and for mobile I picked the hybrid app approach, which is a web-app built using html/css/java script however can run as a native app.

And for backend I picked java ee as this the platform I am most comfortable with, I picked NetBeans as my ide, and the bundled Glassfish 3 container

So the steps for creating the app went something as below
Create database model and tables in mysql (mysql workbench is of great help)
used JPA to create entities (netbeans 'create entity wizard' automated the whole thing)
used REST - Jersey to create the rest services (netbeans 'create rest from entity wizard' automated the whole thing)

Creating and testing the backend was fun, I created a session bean to create my business methods using multiple JPA Entities and the calling the session bean methods from the REST service operations. Tested the rest services from a Chrome app called 'Advanced REST Client'

Then it was time for front end, netbeans helps create html5 projects with seeded Angular JS, I used that, this video helped to get a context of Angular JS, Once I built a basic UI and the Angular JS controller to call the rest services, It was time to test locally.

The first issue I got was
XMLHttpRequest cannot load Origin http://localhost:8383 is not allowed by Access-Control-Allow-Origin

This was a Big issue for me, lot of material is available on fixing CORS, however finally I got an excellent solution here , by just using this CORS filter, I didn’t had to change anything in backend all GET, POST started working fine.

Now was the time to deploy the app, I deployed my backend app on cloudbees, I really liked cloudbees run@cloud service, after installing the SDK, by just 2 commands my app was up and running

bees app:deploy -a avijeetd/myapp -t glassfish3 D:\NetBeansProjects\myapp\dist\myapp.war

bees app:bind -db myapp -a myapp -as myapp

cloudbees provide free DB (upto 5MB) and free deployment of one application.

For the mobile app, I used phonegap, phonegap build service, can convert the HTML/CSS/JS application to native applications, such as apk (android application package) file. I used github to host the source code, again free service for public repository.

So by using all these free could services, I was finally happy to see my app on an android phone. Now time to add more features to the app such as security (authentication), multi-user support in data base etc. then maybe I can tell you to download the app from the market place :-)

2014 - year of re-learning

Anyone more than 10yrs in 'Application development and Integration' world must be grappling with the fact that there are a lot of new technologies to learn and may be not sure which direction to pursue in 2014 - here is my guide, 4 roads to travel in 2014.

Before that, what has really changed, when I saw this post, It was clear that the application we built 10yrs back has been slowly making way for the new applications. One thing that has not changed is still the classification of Layers in applications which are Client-side/UI, Server-side/Business Logic/Service/Integration and the Data. There was one more aspect we didn't care much 10 yrs back (now it's known as devops).

10 yrs back all the Client-side and Server-side was combined as one application written in one language such as Java using multiple frameworks. Now what has happened is all these layers have separated and grown to be independent.

• Client-side has grown to be a Thick-client again. great intro
• Server-side has grown to be more reactive. great intro
• And the Data has grown to be a 'polyglot persistence' and just not RDBMS. great intro

So as each tiers have become more and more specialized, time to give special attention to each of them. Here are my 4 roads to travel in 2014,

1. Client-side/UI (Mobile first Road)

• Responsive web-app based on HTML/CSS/JavaScript
• Native app based on HTML/CSS/JavaScript - phonegap/cordova
• Native app based on native technology - iOS, Andriod SDK
• Java 8 > Internet of Things great intro

2. Server-side (Scalability to Elasticity Road)

• reactive applications / async / non-blocking io
• API / Smart integration / cloud integration / social integration /GEO integration
• 100% model driven application development (as Pega puts it)

3. Data (Big Data Road)

• NoSQL
• In-memory Data grid
• Big Data Analytics (Hadoop / R)

4. DevOps (Automation Road)

• puppet
• hudson/jenkins
• maven/python

Lot of roads to travel, thankfully all the roads lead to the same place called 'the new application', happy navigating in 2014.

DBAdapter - Merge operation

If you use merge operation for updating records using DBAdapter, make sure property DetectOmission is set to false as by default it's true. Or else values coming as Null will not get udpated in database records - causing updates buggy.

  property name="DetectOmissions" value="false"/ 


update:dec'03

There was another interesting learning on DBAdapter, when we had to insert a record to 5 related tables. All the tables were related by foreign keys. DBAdapter wizard lets you pick your tables, define their relationship, pick database sequences for primary keys. And all this internally uses Eclipselink JPA - looked pretty good.

OSB - MQ transport - reply to queue manger name / queue name

Recently we had a good (rather hard) experience with consuming a legacy application service in OSB. The legacy application exposes a MQ based interface. It receives the request message in one queue and reads the reply to queue manager and reply to queue names from the message header and then puts the response in that specified queue manager/queue which is on their side.

If you have worked with MQs, setting up local queues pointing them to remote queues with correct queue manager names etc. can be quite tricky. We wasted many weeks to debug an issue where the remote response queue manager host name was not correct, coordinating with another team, doing it across 2 different environments was a big pain.

More painful was the request/response never worked for months, after quite a bit of heartburn, It was found that while we were setting the reply to response queue manager/queue names in the request message header, it was getting overwritten by OSB with the local queue manger/queue names.

It turned out if you are using mq-transport in request-response mode (which is the synchronous mode) - the response queue manager/queue name has to be same as the request queue manager/queue name if not OSB will overwrite it. This however is not a problem if the message pattern is one-way (i.e only request).

please see more details about 'reply to queue manager' and 'reply to queue' transport header properties here

So to solve the problem we introduced an intermediate queue in between our request queue and remote request queue. The first BS (business service) uses the mq transport in request/response mode and pushes the request to an intermediate queue. Another proxy service listening on to this intermediate queue picks up the message and pushes it to the real request queue with reply to queue manager/queue names in the transport headers as required by the legacy application.

There were some apprehension about correlation id, message id etc, however this worked well, and we were able to move forward.

Framework for picking a web-framework

If you have built database-oriented web applications in the past and wondering now which is the best framework to build going ahead, the below classification might help. I tried creating a decision tree for some of the most popular frameworks.

As you can see in one extreme it's the  Java based standard (JEE) frameworks and at the other extreme it's the non Java,  non sharing (no servlet session) completely new frameworks and in the middle is the Spring MVC and Grails.

A very good presentation in favor of Play here and slides  here

A good comparison of JVM based web app frameworks here, winner is Grails, comparison matrix here

Whatever you pick between the HTML/Javascript and your backend/middleware, it’s important to know how these frameworks differ at a fundamental level which can affect scalability, performance, developer productivity, cost etc.

It's important to note that these frameworks are just not presentation frameworks, they extend to use an ORM framework (mostly hibernate) and have their own build tool, ide etc. Each one of them pretty much extends to a platform and will support cloud deployment.


Interesting video here on why twitter is moving from Ruby on Rails to JVM.

Now time for the new debate how much client side MVC and how much Server side MVC? / REST API?

OSB11g

Finally I got my first project on OSB11g into production, We had used OSB11g before, however primarily for service-virtualization, routing, light transformation, security etc. This project is special as OSB11g was the only platform for all service implementation as well as composition. It's kind of conflicting to build services on OSB11g using DBAdapters instead of using JEE components such as JPA or any other OR mapping product. I had a discussion on this here

Unfortunately I didn't get to code, however I tried a bit to do code-reviews, couple of findings on the IDE part here -

• If you want to only read OSB code - still you have to install OSB as the OSB IDE plugin for OEPE is bundled in the OSB install - hopefully 12c will remove this issue
• Proxy message flows are pretty unreadable at first sight, what helped was - use 'Collapse All' to see the highest level stages and then expand
• Overall Proxy message flows are divided with multiple stages (with multiple pipeline pairs) ending with a Route Node
• It's very important to understand Service callout, Publish and Route Node - also the related thread behavior - great article here 
• As we implemented a very complex orchestration in OSB - one proxy service had almost 100s of activities calling around 30 services - with error handling code - It is very important that the code is readable and maintainable - so it's important to use the Notes feature to add documentation 

There were many challenges in the design aspects as we were building all the application logic in OSB services -

• how to design the PS-LPS structure - in order to improve code modularity we tried using as many LPS, however there was a debate on how to expose these LPSs if there is a need to expose them as services (by using Operation branching in one PS or the LPS by having its own PS)
• How to design the CRUD operations - as separate DBAdapter services or having one service with routing to all the business services example here
• Handling transactions and concurrency - we needed to update multiple records in a database table. Though OSB supports transactions, handling concurrency was a big challenge. We wanted to implement  optimistic locking - however the nature of fine-grained services made the interfaces difficult to implement the feature 
• We decided to implement all the stateless business logic in Java, which was primarily to iterate through multiple hash-maps and do matching kind of logic. From OSB these Java methods were called using Java Callout. XML Beans was used for the data binding between OSB and Java. This worked very well, however there was one challenge to make this Java code compiled as part of OSB code - we end up using Jar files as part of OSB code
• As all developers were doing parallel development of the same service (each developer doing some LPS) - there were challenges in building the code without errors initially

Couple of other issues -

• One issue that I will remember always from this implementation was about File transport. OSB needs File/FTP transports to have the staging directory in a shared storage (or network file system) so that all nodes in cluster can access it. As the polling configuration in proxy service is active-passive however the actual processing is always active-active. So if you don't have NFS or SAN in your cluster environment - you cannot use this transport. more here. Similar requirement for using HAFileAdapter or HAFtpAdapter.
• Also when we were trying using SFTP transport - the requirement was to use public-key based authentication, for which we needed to create a Service Key provider - for which unfortunately we didn't get any clear steps to follow. Which was easier for SFTP Adapter where the private key is provided as part of the adapter configurations.
• Had a issue with MQ transport around setting the correct encoding - UTF-8 - multiple properties on mq connector, transport header, MQRFH2 header properties etc. following these 2 steps resolved the issue
• Set the Character Set to 1208 in Transport Header activity.
• Copy All headers property in the Transport header activity, here the outbound context will be populated to queue

Is your SOA project unable to create WOW effect!

Are your customers not so impressed with the results of SOA projects and is the reason the projects take too long.

SOA/BPM came as an alternative for the only reason that it provides higher developer productivity and business agility (because for reusability). Is that promise met?

Even today all SOA projects (any small to medium package or feature) takes somewhere 4 to 6 months. This is where the WOW effect goes out.

The reasons are well known, recently a number of discussions on Industrializing SOA (or Factory model for SOA) cover that, can be found here and here

Like memory leaks, time leaks in the SOA project. In one we see 'Out of memory' and the other 'Out of budget'. Ask your customers how much time gets wasted because of these issues not well managed.

•        Development Approach
•   Agile / Iterative
•        Deployment automation
• Maven/Hudson Vs ANT
•  Continuous Integration
•  Configuration plans
•  Dependency between components
•      Testing automation
•  Regression testing / unit test scripts
•   Test coverage
•   Way to manage test data sets
•  Integration testing
•  Error Handling / monitoring
•        Version control
•    Binary management
•    Use of MDS
•    Virtualization for dev and test environments
•   Automation scripts (devops)
•  stubbing out system dependencies

 It's time to move the pre-industrial SOA customers to industrialization

What’s your favorite App Server?

If you are like me, you have worked all your life around a JVM + App server. From JDK1.2 to now Java7 And from J2EE to now JEE6. While we started as the Sun loyalists, somewhere in 2004/2005 - Spring became more popular, clearly killing Entity Beans also to some extent EJBs? Fast forward 2013 now as it stands, many competing stacks with JEE footprints

 1. JEE (Full profile) - Weblogic12c, Glassfish3.0, JBOSS AS7.1/EA6.0P, WebSpehere8.0
 2. JEE (WebProfile) - ApacheTomEE4.0 more here
 3. Servlet Containers - Jetty, tc server

To see the difference between JEE Web-profile and Full profile please see towards end of this page here (main difference are JMS, Web service technologies, JCA, Full EJB etc not included in Web profile)

 Now based on these Java platforms we have a number of PaaS coming up

1. Cloudfoundry from vmware based on tc server
2. Oracle Java cloud  based on Weblogic
3. OpenShift (JBoss cloud) based on JBoss EAP (or the new name Wildfly)

 (Not to forget Apache, MuleSoft and other open source platforms supported on AWS like IaaS)

Fight for Java PaaS is getting quite hot with the 3 major app server based PaaS providers try to win market share. Who will emerge winner? Your views?

Random Thoughts - 4

I have been thinking about multiple application types, patterns trying to label all the styles of applications we see in our world. Multiple triggers for these thoughts 1) This whitepaper by mulesoft on Evolution of SOA 2) Three Gartner magic quadrants related to application infrastructure here 3) arrival of Java EE 6, more info here
And in general people wondering, Is SOA dead? Is BPM worth the investment? should we go back to Java and not BPEL?
Here are two application maps I put together, please navigate and comment what kind of application you are building and what kind you want to build in future? (Also please mention if I missed any application styles)

That is a Technology view of applications, however it is important to know where the application fits in the overall business process of the Company. Here is another view showing applications based on business processes. There are many more business processes and applications, the picture is showing some of them.

So the questions one could ask for every project are -
1) Which business process does this application will be part of? e.g. customer booking process through customer service agents in a logistics company
2) What business objectives and benefits (competitive advantage?) does the application address/bring? e.g. carrying most profitable cargo, increased revenue
3) What technology platform will the application be built on? e.g. Oracle Fusion Middleware
4) What application pattern or style does this application has? e.g. SOA style application

Product Overlap or Choice

In the Oracle middleware product stack often there are products which seem to overlap in functionality causing confusion for customers/consultants to pick one over the other. Oracle explains it as a matter of choice that customers have to select the right product for their need. Over time things are getting clearer and this confusion is settling down. Below are my views on how these products should be seen and positioned.

BPM Suite 11g Vs SOA Suite 11g

There has been lot of discussions on this, a lot of good blogs are available on google, what makes it more interesting is BPM Suite and SOA Suite share a common SCA runtime platform on top of Weblogic. BPM Suite is based on BPMN notation and SOA Suite on BPEL. Now without getting into the controversy of what is BPM and what is SOA, one thing to remember is BPM Suite license costs are higher. So it’s much more likely that customer has SOA suite or it’s easier to justify the license cost as it has BPEL, human-workflow, business rules, b2b, Adapters and BAM. It has all the tools you would need to solve your integration problems, customizing workflow requirements for existing oracle apps. So SOA Suite along with OSB can be a complete middleware platform (mid-tier) for the enterprise. BPM Suite on the other hand is better fit to build top-down business process applications in cases where a packaged application is not suitable or available.

OSB (Oracle Service Bus) 11g Vs SOA Suite 11g

This is not much of selecting one over other issue as most customers have both the products, the challenge comes as to what to implement where, how much in OSB and how much in SOA Suite. What integration patterns fit one over the other? So the basic difference between OSB and SOA Suite is that OSB is much more light-weight unlike SOA Suite, SOA Suite because has a lot of engines running in it needs a much higher capacity, tuning and better administration. So for synchronous (req/resp) integration patterns with simple compositions OSB would be sufficient. It’s important to build a logging, auditing, monitoring framework with UI around OSB so as to complement what Enterprise Manager provides for BPEL. BPEL should be used where there is a case for orchestration/complex compositions, human-workflow and long-running processes/async. A lot of customers have implemented a lot more in BPEL which seems like an anti-pattern however with better capacity and administration it should not be a problem.

OSR (Oracle Service Registry) and OER (Oracle Enterprise Repository)

These two products under SOA Governance Suite cause lot of confusion, most customer end up not using either. As Governance is a political area, most customers remain in a policy-paralysis state without knowing how to answer the questions such as 1) how to identify new services? 2) what policies and process should be in place to build/deploy the services 3) ownership of services and how to reduce TCO (Total cost of ownership)


Now OSR and OER are both metadata repositories, they keep service metadata to improve visibility and control. The main difference is what kind of metadata they keep and what kind of analytics you can run on them. OSR is simply a UDDI compliant Registry (oem agreement with HP Systinet). It’s used for programmatic, dynamic discovery of services and provides control over service provisioning. OER on the other hand is a complete Asset management product used during the plan, design, and build phase of the service lifecycle (automate approval process) as a single source of truth for service and composite application development. It provides analytics with 20+ reports such as Investment ROI, Proactive Impact Analysis, value of service & asset portfolio.



There is another product OWSM (Oracle WebService Management) which causes some confusion. These confusions are primarily from a roadmap perspective as to how they will be positioned in future. OWSM is the policy-based security product for SOA Suite. There is another product called OEG (Oracle Enterprise Gateway -oem agreement with Vordel) which is similar to IBM Datapower product. OWSM is part of SOA Suite and OSB both and is the strategic product for all last-mile security implementations.



One more area of confusion is around ODI (Oracle Data Integrator) product, as to when to use ODI and when to use SOA Suite. This is more of an ETL vs SOA debate. In fact ODI uses a methodology called ELT and not ETL. This product is interesting with the cloud integrations on rise. Currently ODI is positioned to do primarily bulk Data Load kind of use cases however it supports web services based integrations.


I will cover the Weblogic suite products in another blog post.

SOA 11g

I finally got my first SOA11g project few months back, unfortunately I am not getting to do development as much I would like to, and that’s why my blogging frequency has also gone down. :-(

Anycase, here are some my experiences with SOA11g.

1) Customer had SOA11.1.1.2 (PS1), and now the latest version is 11.1.1.6(PS5). It's not too back that 11gPS1 was released but now there is no support for it. So first thing which bewilders me is the new versions that keep coming and customers not able to upgrade to them as quickly as we developers would like. Not to forget the exact version of JDev and the plugins.

2) We continued our work on PS1, and we hit a wall with one bug.

We were passing a custom variable in soap header along with wsse security header. Apparently there was a bug #13070260 - which was throwing java.lang.ArrayIndexOutOfBoundsException

As we could not get a back-port patch, we had to look for work-around and luckily there was one, as found here, to supress the wsa headers we used this flag oracle.soa.addressing.response.enabled=false in the composite.

3) We had used a Mediator-BPEL-Mediator pattern, which looked elegant but the repetitive transformations in Mediator and BPEL were bit of a pain, particularly when we could not see the payload information for mediator in enterprise manager (may be another bug). Also the interesting property passThroughHeader that has to be set in composite for the mediator to pass the soap headers forward.

4) We were using OWSM to secure the composites and that was having a few hick ups in setting up the jks keys, understanding the polices etc.

5)We used Business Rules in our composites, and that was quite a challenge to start using properly, as we had lot of bucketsets and everytime we re-imported the schemas, had to redo the decision tables. One interesting finding was, while rules execution in one of the rules it was always going to to default flow - so by using the conflict resoultion as Override instead of RunAfter or RunBefore solved the issue.

update25/Apr: If the schemas for rules has changed, instead of re-importing everything, re-importing only the changed element helps.


I also had my first OSB11g experience in this project - and that is for another blog.

more on AIA2.4/10g

Last week while deploying some integration flows on AIA2.4/SOA10g/Weblogic, had two interesting errors -

1) While deploying the ESB (EBS in AIA) got the error -

A Service or ServiceGroup of name "XXXEBSV1" already exists under parent AIASystem

Found a good blog on this here, which talks about an ESB bug - and the solution to that is
Either you delete the esb before redeploying, or if you are doing it from a script - we can deploy a esb project with no routing and having the same GUIDs for all the services. Deploying the ESB with No routing firth and they deploying the ESB with routing will resolve the issue.


2) The second issue was with correlation sets - we were running a huge loop across EBF -> ESBV1 -> Provider BPEL -> ResponseESBV1 -> EBF

And it was important that the correlation sets work properly for the loop to continue. So we had 3 points in the EBF where callbacks were to be received. We created 3 correlation sets in each scope invoke with yes and receive with no, associated with property with type normalized string, for property alias we picked the EBM_ID, It needs 2 entries in the wsdl one for requestEBM and one for Response EBM. On the Provider BPEL part also we created correlation set on the first received activity (with yes) and then at the invoke ResponseEBS (with no) one.

AIA2.4FP installation on SOA10.1.3.4/Weblogic9.2

After a long hiatus, I am back with a useful blog. I had to install AIA2.4 (AIA2.3 downloads were not available on edelivery) on SOA10.1.3.4 on the WebLogic9.2. I tried to avoid doing it as much as possible, given there were no documentation on this, other than a few OTN forum entries and few other useful links here and here , but finally after 3 days, it was successful.

this blog already pointed to issues with this install and the metalink note#822831.1

To start with - had to first set up SOA10.1.3.4 on Weblogic9.2. This is a pretty well established process, however I faced some challenges because of Windows7

First issue was the 10g installer would not even run complaining SWAP file size issues, I commented the SWAP size checks following this link

The second issue was a big issue, the installer kept failing at the OWSM step (Unable to locate a valid OC4J URI), the error was opmn would not stop properly, due to IP6 support in Windows7. Solution found from here and here

Basically I had to edit the opmn.xml file under opmn/config to change the stop-paramters as suggested in metalinknote#1206667.1. The other important step is to uncheck the IPv6 in the network properties.

After that, things were smooth barring a few "Error occurred during initialization of VM", "OPATCH_JAVA_ERROR=OPatch Exception while trying" errors. For VM issue, closing some applications helped. For opatch issues, running the patch inside opatch directory once resolved the issues.

So after installing
Oracle XE (increase process count)
irca.bat
SOA 10.1.3.1.0 with Advance option
Patch set 7272722 for 10.1.3.4
Weblogic9.2
Patch 7337034
Patch 7490612 For weblogic SOA Domain creation
MLR #9 p8589928_101340_GENERIC
No AIA patches required
AIA installer only Software Copy mode

(for all these ensure your directory names doesn't have spaces or else you might get "Unzip: No matching wildcard found for.." )

Now was the time for AIAFP2.4 AIA 2.4 Installer V17393-01

AIAFP install needs lot of prep work, I did the below
1)increased JTA time out to 300
2)setup AQJMS Startup class (getting the aqjms_v2.0.zip was a challenge, its no longer available on OTN, luckily I had it from a previous attempt)
here only create the startup class, no need to add as a weblogic startup class
3) setting up ADF libararies, downloaded adfinstaller10.1.3.4 and followed the instruction to copy the libararies to weblogic instance
run setupadf.cmd, copy the classpath and add to the weblogic startup classpath
4)copy the esbbootstrap.jar under the JVM_HOME/jre/lib/ext

After this was ready to run AIAFP, first thing was to setup the property files
1)set up the aiaenv.bat under AIA_HOME/bin
2)deploy.properties under AIA_HOME/config
3)FPwlInstall.proprties and server.properties under AIA_HOME/Infrastructure/install/wlscripts

Templates for these property files are available under AIA_HOME/install/templates

next was to run the install scripts under AIA_HOME/install/wlscripts

If you are lucky the following line should do it all...
ant -buildfile FPWLInstall.xml

But that doesn't happen often, so I had following error
Servlet: "AIAValidationSystemAPIServiceSoapHttpPort" failed to preload on startup in Web application

It turned out, this is a known bug AIA 2.4 INSTALLATION ERROR ON WEBLOGIC [ID 1059780.1], this patch didn't work easily, so I just modified the AIAApplicationWL.EAR to change the ValidationSystemAPIService.xml under ValidationSystemAPIService.war

So if your build fails in between, It's better to run the other sections separately, The ant scripts are well organized, you can run the DeployCAVS, DeployServices targets.

That's it, then you can see the beautiful screen of http://localhost:9700/AIA :-)

update26/July
update the weblogic start arguments with -Daia.home={your AIA Home}, this is required otherwise the aia core code will not work and you will see FileNotFound for AIAConfigurationProperties.xml

Random Thoughts -3

Last few months working on few pre-sales proposals, seeing couple of trends/patterns in many of them. Many of the projects customers are planning to start have a cloud piece. Many of the customers are using SFDC (Salesforce.com) or planning to move particularly in media, hospitality industries. Subscription based cost is driving a lot of technology choices. Customers are becoming very cost sensitive and want a very flexible costing model and execution models, where they can pay as they use in buy, build or cloud choices.

From solution perspective, It becomes new challenges around how to integrate with cloud, how to handle security, how to handle RASP (Performance in particular).

Other than the cloud trend, the other trend is the regular debate of orchestration vs choreography. So for every industry or cross-industry problem there are many packaged applications, now these applications also provide regular supporting peripheral/plumbing functionality. For example, an Order-Management App will have its own inbuilt workflow, will have OOB UI to manage the process flow or administration. Now the debate is whether to use the App's such in-built features or to use an esb/bpel layer to unify these functionalities centrally across applications. That's the choreography vs orchestration debate, ESB for stateless choreography and BPEL/BPM for stateful orchestration.

Either way positioning an ESB is easy, with all the security and visibility story around services. Positioning the BPM story is a challenge, customers in hurry resort to point-to-point integrations and some even don't go for commercial esbs. That's another trend I saw open-source ESBs the entire story around JBI, Mule, Fuse, Spring, OSGi etc.

Another trend I see is customers are more interested in data/dw, bi/reporting, mdm etc. And different compliance requirements are driving lot of strategic projects in this area.

My next interest is to find architecture patterns, how soa/bpm/e2.0 can be used both strategically and tactically, particularly oracle soa.

Random Thoughts - 2

My random thoughts series continues in the new year as well, good time to look at Middleware Tech in general and SOA11g in particular. One specific step in this regard was my SOA Certification 1Z0-451, I took this in a hurry to avoid the pain of reading endless pdfs. Luckily I passed the test with an average score of 73%. The test did throw many new things about SOA11g which I discovered after reviewing the questions in the test. Few of them are below, otherwise the test is a 'developer-certification' and very broadly spread across all the products (BPEL, Mediator, HWF, Rules, OWSM etc)

new features
1. Mediator now supports a new validation called schematron validation
2. Rules have the new feature of decision tables (bucketset is a new term)
3. Rules have lot of new features functions, links etc.
4. Fault-management framework has new features (at-least location of the files)
5. OWSM is complete new territory
6. SDO / Entity variable - brand new features
7. Mediator externalizing routing rules to Business rules
8. SpringContext integration support
9. Lot of new features in Mediator
10. EDN as configurable publish-subscribe infrastructure

My next focus area is to deep-dive SOA11g and find best-practices/integration patterns. Stay tuned.