Building Effective Enterprise Distributed Software Systems

data-driven communication vs behavior-driven communication

by Roger Voss, Aug. 26, 2006

Seeds of Doubt

Way back in the '90s, while working at Microsoft, I built middle-ware systems using C++ and DCOM. At one point I redid some of a system with MSMQ messaging and had an ah-ha experience. MSMQ is not the most robust MOM feature-wise, but it was enough for me to see that things can go better with a pure data-centric, asynchronous messaging approach vs. behavior-centric, distributed object remote interfaces (at the time I was becoming very disenchanted with DCOM especially, but also COM in general).

A Home-Brew App Server Points The Way

A few years later on I build a custom-designed, messaging-based, middle-ware application server written in Java during a couple of months of being laid off. (The insurance company I was involved with then decided to cut back on its experimentation with Internet innovations as a cost control measure as due to 9-11 aftermath. I continued to do some R&D, though, in hopes of landing back with the original entrepreneurs of the endeavor.)

The system I built during that brief sabbatical was designed to hook into a user's web browser interactions with the server-side and mirror everything to an agent in a call center - a co-browse implementation. It was essentially a specialized web proxy server intended to reside on the data center side sitting in front of conventional web sites. Design-wise it was sort of a JEE app server in microcosm, though anchored around bi-directional messaging (as opposed to EJBs sporting remotely accessible interfaces). At the time I had had no involvement with J2EE and was still relatively new to Java. It was later that I came to see some of the parallels in what I built vs. J2EE. The similarity was thematic, of course, as certainly J2EE was vastly more intricate and comprehensive.

So I landed in a totally different industry, yet a distributed application project I got assigned to needed the event-driven capabilities of fully symmetric bi-directional messaging and a middle-tier application server. The system I had built was entirely suitable and so I incorporated it with modest adaptation. The application went into production and enjoyed splendid success.

Becoming Enterprise Robust

Now my company had grand long term ambitions for this initial foray. The middle-tier app server would need to accommodate an ever-growing array of applications - some closely related, some distinct. Event-driven distributed design would be a constant theme in all cases. It would also need to support clustering for high availability. Extensive capabilities for monitoring would soon become essential so all these various apps could be managed. Ease of configuration and deployment would be significant concerns as well as this software would be deployed into multiple customer sites. In short, it needed to become a true enterprise class software system.

I was aware of the EJB 2.0/2.1 spec which incorporated Message Driven Beans for enabling easy use of JMS. In looking at J2EE I directly homed in on JMS - from my point of view messaging was the star attraction. After some months of experimentation and product evaluation, the next release of my software system replaced my home-brew app server with a J2EE app server and an enterprise-capable JMS MOM - setting the stage for several years of successful organic growth of said system that's still going strong. I finally got the chance to build the kind of distributed enterprise software that was just a gleam in the eye back in my Microsoft days. (Incidentally, I had a brief diversion of attempting to build a somewhat similar system using ASP.NET. That experience is detailed in this blog post: ASP.NET vs J2EE)

My uptake of J2EE, in retrospect, was rather unorthodox. I used MDBs near exclusively and then also incorporated JMX mbean services to do special things like device control or a UDP-based JMS auto-discovery service (mbean services gave me the latitude to do my own thread pools and device level interactions). I stayed away completely from session beans (I couldn't believe that remoting a component interface could be more convoluted than DCOM). The servlet engine was mostly used as a JMX console viewer, as otherwise I seldom had any use for servlets. Having co-designed and participated in a couple of ORM implementations while at Microsoft, I naturally gravitated to using the Hibernate ORM and consequently shunned J2EE entity beans (they just made no sense - why would anyone want to try to do so-called object persistence that way?). When I needed some state to span my high-availability, load-balanced cluster, I used a distributed JCache implementation. More lately I've incorporated the use of the Spring Framework for enabling test-driven development of my MDBs and I also use AspectJ for certain purposes.

The Secret Sauce

It is interesting that during a period of time when J2EE was gradually falling into disrepute I was enjoying great success using it as a most valuable fundamental substrate. Yet I was beginning to discern that my methodology of how to do enterprise J2EE-based distributed applications was instrumental in explaining the dichotomy.

My approach was easy to understand, easy to design with, relatively easy to implement, worked robustly when deployed, and remained easy to evolve over time as SOA was an inherent characteristic. I was therefore well pleased with J2EE while others were being frustrated with the complexity of their approach - which was all beginning to look like anti-patterns. J2EE was certainly an uber spec, but it just wasn't panning out in practice for a lot of folks.

Dueling Distributed Communication Techniques Over Lunch

Rather recently I was invited to lunch with some engineers that work for a prominent software vendor. The topic, ostensibly, was to be enterprise application servers. As preface to this meeting I wrote a whiter paper where I set the stage with a short discussion of distributed architecture philosophy. Knowing that what I was going to say would be somewhat controversial, I opted to not mince any words:

"Let me get right to it by lobbing some grenades: I recognize two arch evils in the software universe – synchronous RPC and remoted interfaces of distributed objects."

and

"The network is neither deterministic nor reliable - so get over it."

as well as

"Alas for distributed computing systems there’s no possibility of applying a magic bullet solution that waves away the problem of interface evolution management if remoted interfaces are pervasively how things are built."

and then especially this:

"Seemingly without a clue, though, the major vendors of the computing industry continue to purvey distributed computing communications standards that are by now nth generational attempts at the old saw of the remoted interface. News flash! Details at 11: An interface method invocation that is packaged up as an asynchronous message send underneath the hood does not assuage the intrinsic brittleness problem of remoted interfaces."

Well, sure enough the bulk of lunch-time conversation ended up revolving around this matter of sending data-only messages versus calling distributed object interfaces. I was put on the spot to justify why I'd want to continue with the supposedly "messy business" of writing code that sends and receives messages when I could just call interface methods - which is so naturally intuitive, of course. Yes, I concede that up front calling a remote interface can seem like the simplest approach, but in the bigger picture it has a much higher cost and in multiple ways.

Well, this lunch encounter illustrated to me just how vested the big software venders are in their technology API stacks. Here it is a perfect time to take stock of the lessons learned over the last half decade and make course corrections. That isn't likely to happen, though. The philosophy of architecture I espouse tends to undercut these cool, shiny technology APIs that the vendors' marketing campaigns are busy enticing you developers and architects to buy into. I guess they figure all the money has been wrung out of MOM and these days its become commoditized and open sourced. So is time to herd everyone on to the next great thing that they can lock corporate IT into. Plus the money has already been spent to devise the shiny new technology API stacks and needs to be recouped, or at least justified.

Ted Neward and Effective Enterprise Java

I came into Java enterprise software development in an oblique, unorthodox fashion. Yet imagine my surprise (just a matter of days after the above mentioned lunch meeting) to discover that my personal experiences and architectural conclusions are exactly echoed in Ted Neward's book Effective Enterprise Java (written in the style of the Scott Meyers Effective C++ book - Scott had a hand in helping Ted with his Effective book effort).

I have engaged in email dialog with Ted and sort of know him through that medium, but I've never attended any of his conference sessions or read his books. Basically my exposure to Ted were his articles on C# 3.0 LINQ and a podcast episode on the same subject. Yet while browsing Java books in Barnes and Nobel I decided to pick up this book of his and right off stumbled on to material that...well, I could have penned myself as it coincided precisely with my own architectural rules of thumb.

I became fascinated that Ted and I had arrived at some identical conclusions independently of one another based on our mutual real world experiences of building enterprise software systems. I was doubly fascinated because of this recent lunch meeting experience where it was evident I was coming off as a heretic of sorts to my audience. Naturally I began to take some satisfaction (and relief) in seeing that I wasn't the only one that had come to hold such questionable beliefs.

Excerpts From Ted's Book

Ted's book is written as a collection of 75 items that expound on principles and offer sage advice that are of the sort that tend to come from battle tested experience. I've selected three here that in particular underpin my own enterprise distributed software endeavors:

• Item 19: Prefer data-driven communication over behavior-driven communication
  (is referring to messaging vs. distributed object remote interface programming)
  Three reasons: evolution, intermediaries, and recoverable operations.


• Item 20: Avoid waiting for remote service request to respond
  Is better to pervasively design network interactions to be asynchronous in nature.


• Item 18: Prefer context-complete communications styles
  The OOP-centric design approach of remote interfaces tends to be more chatty on the network than sending context-complete document messages.

Of course Ted's book expands on these items for several pages each. Also a number of his other items mutually reinforce these.

A Quick Slide Show That Touches The Salient Points

I recently incorporated the above precepts into a company internal presentation. I've since made a version of my PowerPoint slide show that has corporate-specific material removed:

Data-Driven Communication vs. Behavior-Driven Communication

These principles for building distributed software work very well - that is, they are most effective. From the outset of my uptake of J2EE they have significantly constrained what I consider as worthwhile in an application server.

What is a touch ironic is that Ted aims his book at a Java programming audience that is very likely steeped in J2EE - which tends to imply that they're used to the Session Bean and its paradigm. That is the EJB that exposes a remote interface via the RPC of RMI, CORBA, or web services. The Message Driven Bean was added midlife to J2EE evolution (EJB 2.0 spec). Ted, like many others, has found much to like in JMS and MDBs (otherwise he tends to take J2EE to the woodshed). Even Bruce Tate, before he abandoned Java for Ruby on Rails, wrote positively of JMS and MDBs in the Bitter EJB book.

Yes, Ted's book was published back in 2005 and I'm admittedly rather late to discover it. Yet my recent lunch encounter tends to indicate that the advice it espouses is still as relevant as ever given the agenda trajectories of the big software vendors of our industry. As always, you'll have a basic choice to make: succumb to the vendor marketing noise (likely to be echoed by your upper management and certainly peer pressure), or else think things through for yourself and choose to adopt pragmatic approaches that work very well over the long haul - resulting in successful software systems that are actually pleasing to live with.

Conclusion

Turmoil is perhaps the best word to describe the state of affairs in the Java middle-tier software world. J2EE 1.4 continues on based primarily on existing inertia. JEE 5 is still yet to register an impact - good or bad - in terms of working production systems. Spring Framework uptake continues and in the meantime a revolt arises evidently with the intention of overthrowing all things Java (Ruby et al). Yet in the midst of all this I've been building perfectly fine J2EE distributed software according to my own particular fashion. The secret - ignore the crowd and marketing machine hype - trust in what you know from hard learned lessons. I learned ten years ago that distributed objects don't really work well. The fundamental problems have never been addressed. Happily there exist a much better approach anyway so it doesn't matter.

References

Effective Enterprise Java
Addison-Wesley Professional
Author: Ted Neward

Tim Bray on Rails, REST, XML, Java, and More
Tim Bray, one of the inventors of XML and current Director of Web Technologies for Sun Microsystems, is interviewed while attending Canada on Rails conference. There is a delightful treatment in here of the WS-Death Star stack.

Bitter EJB
Manning Publications
Authors: Bruce Tate, Mike Clark, Bob Lee, Patrick Linskey

Don't Make me Eat the Elephant Again
Author: Bruce Tate

synchronous RPC considered harmful
Yes, yes, a play on Edsger Dijkstra's "goto considered harmful", but of course is right on!
Author: Roger Voss

decoupling request/response
Counter intuitive? Perhaps at first glance
Author: Roger Voss

Building Effective Enterprise Distributed Software Systems
My article is cross-published over on JavaLobby in the Spotlight section. Follow this link if you want to check out the discussion going on over there.