2014 - year of re-learning

Anyone more than 10yrs in 'Application development and Integration' world must be grappling with the fact that there are a lot of new technologies to learn and may be not sure which direction to pursue in 2014 - here is my guide, 4 roads to travel in 2014.

Before that, what has really changed, when I saw this post, It was clear that the application we built 10yrs back has been slowly making way for the new applications. One thing that has not changed is still the classification of Layers in applications which are Client-side/UI, Server-side/Business Logic/Service/Integration and the Data. There was one more aspect we didn't care much 10 yrs back (now it's known as devops).

10 yrs back all the Client-side and Server-side was combined as one application written in one language such as Java using multiple frameworks. Now what has happened is all these layers have separated and grown to be independent.

• Client-side has grown to be a Thick-client again. great intro
• Server-side has grown to be more reactive. great intro
• And the Data has grown to be a 'polyglot persistence' and just not RDBMS. great intro

So as each tiers have become more and more specialized, time to give special attention to each of them. Here are my 4 roads to travel in 2014,

1. Client-side/UI (Mobile first Road)

• Responsive web-app based on HTML/CSS/JavaScript
• Native app based on HTML/CSS/JavaScript - phonegap/cordova
• Native app based on native technology - iOS, Andriod SDK
• Java 8 > Internet of Things great intro

2. Server-side (Scalability to Elasticity Road)

• reactive applications / async / non-blocking io
• API / Smart integration / cloud integration / social integration /GEO integration
• 100% model driven application development (as Pega puts it)

3. Data (Big Data Road)

• NoSQL
• In-memory Data grid
• Big Data Analytics (Hadoop / R)

4. DevOps (Automation Road)

• puppet
• hudson/jenkins
• maven/python

Lot of roads to travel, thankfully all the roads lead to the same place called 'the new application', happy navigating in 2014.

DBAdapter - Merge operation

If you use merge operation for updating records using DBAdapter, make sure property DetectOmission is set to false as by default it's true. Or else values coming as Null will not get udpated in database records - causing updates buggy.

  property name="DetectOmissions" value="false"/ 


update:dec'03

There was another interesting learning on DBAdapter, when we had to insert a record to 5 related tables. All the tables were related by foreign keys. DBAdapter wizard lets you pick your tables, define their relationship, pick database sequences for primary keys. And all this internally uses Eclipselink JPA - looked pretty good.

OSB - MQ transport - reply to queue manger name / queue name

Recently we had a good (rather hard) experience with consuming a legacy application service in OSB. The legacy application exposes a MQ based interface. It receives the request message in one queue and reads the reply to queue manager and reply to queue names from the message header and then puts the response in that specified queue manager/queue which is on their side.

If you have worked with MQs, setting up local queues pointing them to remote queues with correct queue manager names etc. can be quite tricky. We wasted many weeks to debug an issue where the remote response queue manager host name was not correct, coordinating with another team, doing it across 2 different environments was a big pain.

More painful was the request/response never worked for months, after quite a bit of heartburn, It was found that while we were setting the reply to response queue manager/queue names in the request message header, it was getting overwritten by OSB with the local queue manger/queue names.

It turned out if you are using mq-transport in request-response mode (which is the synchronous mode) - the response queue manager/queue name has to be same as the request queue manager/queue name if not OSB will overwrite it. This however is not a problem if the message pattern is one-way (i.e only request).

please see more details about 'reply to queue manager' and 'reply to queue' transport header properties here

So to solve the problem we introduced an intermediate queue in between our request queue and remote request queue. The first BS (business service) uses the mq transport in request/response mode and pushes the request to an intermediate queue. Another proxy service listening on to this intermediate queue picks up the message and pushes it to the real request queue with reply to queue manager/queue names in the transport headers as required by the legacy application.

There were some apprehension about correlation id, message id etc, however this worked well, and we were able to move forward.

Framework for picking a web-framework

If you have built database-oriented web applications in the past and wondering now which is the best framework to build going ahead, the below classification might help. I tried creating a decision tree for some of the most popular frameworks.

As you can see in one extreme it's the  Java based standard (JEE) frameworks and at the other extreme it's the non Java,  non sharing (no servlet session) completely new frameworks and in the middle is the Spring MVC and Grails.

A very good presentation in favor of Play here and slides  here

A good comparison of JVM based web app frameworks here, winner is Grails, comparison matrix here

Whatever you pick between the HTML/Javascript and your backend/middleware, it’s important to know how these frameworks differ at a fundamental level which can affect scalability, performance, developer productivity, cost etc.

It's important to note that these frameworks are just not presentation frameworks, they extend to use an ORM framework (mostly hibernate) and have their own build tool, ide etc. Each one of them pretty much extends to a platform and will support cloud deployment.


Interesting video here on why twitter is moving from Ruby on Rails to JVM.

Now time for the new debate how much client side MVC and how much Server side MVC? / REST API?

OSB11g

Finally I got my first project on OSB11g into production, We had used OSB11g before, however primarily for service-virtualization, routing, light transformation, security etc. This project is special as OSB11g was the only platform for all service implementation as well as composition. It's kind of conflicting to build services on OSB11g using DBAdapters instead of using JEE components such as JPA or any other OR mapping product. I had a discussion on this here

Unfortunately I didn't get to code, however I tried a bit to do code-reviews, couple of findings on the IDE part here -

• If you want to only read OSB code - still you have to install OSB as the OSB IDE plugin for OEPE is bundled in the OSB install - hopefully 12c will remove this issue
• Proxy message flows are pretty unreadable at first sight, what helped was - use 'Collapse All' to see the highest level stages and then expand
• Overall Proxy message flows are divided with multiple stages (with multiple pipeline pairs) ending with a Route Node
• It's very important to understand Service callout, Publish and Route Node - also the related thread behavior - great article here 
• As we implemented a very complex orchestration in OSB - one proxy service had almost 100s of activities calling around 30 services - with error handling code - It is very important that the code is readable and maintainable - so it's important to use the Notes feature to add documentation 

There were many challenges in the design aspects as we were building all the application logic in OSB services -

• how to design the PS-LPS structure - in order to improve code modularity we tried using as many LPS, however there was a debate on how to expose these LPSs if there is a need to expose them as services (by using Operation branching in one PS or the LPS by having its own PS)
• How to design the CRUD operations - as separate DBAdapter services or having one service with routing to all the business services example here
• Handling transactions and concurrency - we needed to update multiple records in a database table. Though OSB supports transactions, handling concurrency was a big challenge. We wanted to implement  optimistic locking - however the nature of fine-grained services made the interfaces difficult to implement the feature 
• We decided to implement all the stateless business logic in Java, which was primarily to iterate through multiple hash-maps and do matching kind of logic. From OSB these Java methods were called using Java Callout. XML Beans was used for the data binding between OSB and Java. This worked very well, however there was one challenge to make this Java code compiled as part of OSB code - we end up using Jar files as part of OSB code
• As all developers were doing parallel development of the same service (each developer doing some LPS) - there were challenges in building the code without errors initially

Couple of other issues -

• One issue that I will remember always from this implementation was about File transport. OSB needs File/FTP transports to have the staging directory in a shared storage (or network file system) so that all nodes in cluster can access it. As the polling configuration in proxy service is active-passive however the actual processing is always active-active. So if you don't have NFS or SAN in your cluster environment - you cannot use this transport. more here. Similar requirement for using HAFileAdapter or HAFtpAdapter.
• Also when we were trying using SFTP transport - the requirement was to use public-key based authentication, for which we needed to create a Service Key provider - for which unfortunately we didn't get any clear steps to follow. Which was easier for SFTP Adapter where the private key is provided as part of the adapter configurations.
• Had a issue with MQ transport around setting the correct encoding - UTF-8 - multiple properties on mq connector, transport header, MQRFH2 header properties etc. following these 2 steps resolved the issue
• Set the Character Set to 1208 in Transport Header activity.
• Copy All headers property in the Transport header activity, here the outbound context will be populated to queue

Is your SOA project unable to create WOW effect!

Are your customers not so impressed with the results of SOA projects and is the reason the projects take too long.

SOA/BPM came as an alternative for the only reason that it provides higher developer productivity and business agility (because for reusability). Is that promise met?

Even today all SOA projects (any small to medium package or feature) takes somewhere 4 to 6 months. This is where the WOW effect goes out.

The reasons are well known, recently a number of discussions on Industrializing SOA (or Factory model for SOA) cover that, can be found here and here

Like memory leaks, time leaks in the SOA project. In one we see 'Out of memory' and the other 'Out of budget'. Ask your customers how much time gets wasted because of these issues not well managed.

•        Development Approach
•   Agile / Iterative
•        Deployment automation
• Maven/Hudson Vs ANT
•  Continuous Integration
•  Configuration plans
•  Dependency between components
•      Testing automation
•  Regression testing / unit test scripts
•   Test coverage
•   Way to manage test data sets
•  Integration testing
•  Error Handling / monitoring
•        Version control
•    Binary management
•    Use of MDS
•    Virtualization for dev and test environments
•   Automation scripts (devops)
•  stubbing out system dependencies

 It's time to move the pre-industrial SOA customers to industrialization

What’s your favorite App Server?

If you are like me, you have worked all your life around a JVM + App server. From JDK1.2 to now Java7 And from J2EE to now JEE6. While we started as the Sun loyalists, somewhere in 2004/2005 - Spring became more popular, clearly killing Entity Beans also to some extent EJBs? Fast forward 2013 now as it stands, many competing stacks with JEE footprints

 1. JEE (Full profile) - Weblogic12c, Glassfish3.0, JBOSS AS7.1/EA6.0P, WebSpehere8.0
 2. JEE (WebProfile) - ApacheTomEE4.0 more here
 3. Servlet Containers - Jetty, tc server

To see the difference between JEE Web-profile and Full profile please see towards end of this page here (main difference are JMS, Web service technologies, JCA, Full EJB etc not included in Web profile)

 Now based on these Java platforms we have a number of PaaS coming up

1. Cloudfoundry from vmware based on tc server
2. Oracle Java cloud  based on Weblogic
3. OpenShift (JBoss cloud) based on JBoss EAP (or the new name Wildfly)

 (Not to forget Apache, MuleSoft and other open source platforms supported on AWS like IaaS)

Fight for Java PaaS is getting quite hot with the 3 major app server based PaaS providers try to win market share. Who will emerge winner? Your views?

Random Thoughts - 4

I have been thinking about multiple application types, patterns trying to label all the styles of applications we see in our world. Multiple triggers for these thoughts 1) This whitepaper by mulesoft on Evolution of SOA 2) Three Gartner magic quadrants related to application infrastructure here 3) arrival of Java EE 6, more info here
And in general people wondering, Is SOA dead? Is BPM worth the investment? should we go back to Java and not BPEL?
Here are two application maps I put together, please navigate and comment what kind of application you are building and what kind you want to build in future? (Also please mention if I missed any application styles)

That is a Technology view of applications, however it is important to know where the application fits in the overall business process of the Company. Here is another view showing applications based on business processes. There are many more business processes and applications, the picture is showing some of them.

So the questions one could ask for every project are -
1) Which business process does this application will be part of? e.g. customer booking process through customer service agents in a logistics company
2) What business objectives and benefits (competitive advantage?) does the application address/bring? e.g. carrying most profitable cargo, increased revenue
3) What technology platform will the application be built on? e.g. Oracle Fusion Middleware
4) What application pattern or style does this application has? e.g. SOA style application