there are no good data entry forms systems

A few months ago I was fed up with the technologies available for data entry forms as I surveyed the forms landscape. Of the various choices of how to implement forms applications, they all suffer drawbacks of one sort or another.

First of all let me state my bias: I currently build distributed applications that are used on internal enterprise LAN/WANs. I don't build systems that are accessed over the Internet by arbitrary external customers (I've also done that kind of stuff before too but in a manner completely different than what everybody else is doing. More on that later.)

The systems I currently build are J2EE for middle-tier and C# .NET rich client apps. I use Tibco EMS for JMS messaging to weave everything together - I stay away from RPC and HTTP as much as possible. If I need to stream a megabyte or so between nodes then might use HTTP to do it - otherwise stick with JMS messaging. It's essential to have an event driven distributed application architecture for my problem domain. I use XML currently for message formats, described in XSD. JAXB is used on the Java side and XsdObjGen is used on the .NET side.

This all works fine for my rich client apps. These client apps are not form intensive themselves - they tend to host systems that are, though, and also do things like interact with PBX and streaming video.

But I'm now considering how to address the problem of a forms hosting system itself - one that may need to push 200 to 300 potentially different forms out to end-users for various data processing and report generation. (The current form application technology my company uses is being retired by the vendor and so we need to switch to something current and more or less mainstream.)

So some folks say we should adopt HTML forms and one of the various MVC solutions for managing the presentation layer (our business logic will stay encapsulated where it is so we're already by default looking for an MVC approach).

We'd like to have the business analyst use high level GUI tools to define forms, data validation, and perhaps business rules. Hopefully the forms presentation code could thereby get generated without much need for procedural coding of the visual rendering aspect of the form. Hopefully such tools spit out mostly declarative descriptions of both appearance and data entry validation rules. There's also a consideration to perhaps tie our new system in with BPM. Some BPM systems have their own GUI business analyst tools. Some of the open source BPM tools just use text file descriptions created in a text editor.

Now as to HTML forms and HTTP protocol - I have very definite issues with that mix of technology.

With HTML the presentation is page based (ala the HTTP protocol). But I may want to update new entries into a UI widget by pushing the updates directly to the relevant widget. I don't want users to have to suffer entire page refreshes of a form just because some of the data of the form they're working with is being updated on an event notification basis from elsewhere in the distributed system. Okay, so DHTML can be used to graphically modify an HTML form widget, but there's still the issue of how the update is pushed to the widget. (The updates need to be pushed from the middle-tier side to the clients. Naturally JMS topics work great for this.)

I don't care for the flat list of scalar values that HTML forms use to return submitted form data. I'd prefer to submit the data as a structured XML document (so can be unmarshaled into an object graph at the middle-tier - where may use Hibernate to persist, etc.).

Sometimes there will be fields (or widgets) on a form to where an action or event must be fired across the distributed application based on a user interaction with the field. Such actions will not entail submitting the form data for processing. (Again, JMS messaging is more suitable than HTTP here for firing such discreet events.)

Other things I don't like about HTML is having to use JavaScript and a single threaded programming model for the form's client-side runtime execution. The rich client forms I build in .NET constantly make use of multi-threading and background JMS message processing. Yet if I start incorporating a Java applet with the form in order to get decent multi-threading and JMS messaging, then just why am I bothering to use HTML/JavaScript and HTTP to begin with?

I don't like the synchronous nature of HTTP protocol. If I submit data from an HTML form for processing, as a user I have wait for the middle-tier to complete processing and return a result on the HTTP PUT or GET operation of the data submittal. It's very silly to tie up a client in a mode waiting for that result to come back from a busy server. In contrast, my rich clients are able to submit data as an asynchronous JMS message which is placed into a queue. My rich client apps are not modally locked into the expectation of a synchronous return result. Instead any result from middle-tier processing can simply be returned asynchronously on an event notification basis.

Then there's the issue of transparency of failover for when a middle-tier cluster node fails. Tibco messaging enables my clients to transparently fail to a backup JMS server. It's all dealt with by software in the Tibco JMS implementation. Achieving transparent failover for HTTP-based web clients requires more elaborate techniques. Because JMS is highly decoupled by nature, the clients are unaware if a J2EE processing node goes down. Tibco takes care of load balancing to the J2EE cluster nodes. If a node goes down, the survivors simply process yet more messages. So I ask myself why invite the headaches of HTML forms and their HTTP protocol when Tibco EMS solves transparent failover quite nicely and easily?

In the past I've dealt with all these myriad shortcomings of HTML forms by coupling them with a Java applet that does messaging to a proxy server that in turn does HTTP to a web server. In that particular case I was using the messaging system to mirror form interactions to other users that were co-browsing in a common browse session. The interactions of one user are seen by all other participants in the co-browse session. The specialized co-browse proxy server and the web server both exist at the middle-tier and are in a common cluster.

I still need something, though, that is like XForms to where it is possible to return form data as a structured XML document. I don't want to be limited to having to use an XForms compatible runtime because I haven't liked any of the XForms capable systems that I've looked at so far. It's all experimental. There is yet nothing mainstream in XForms. And there will definitely need to be GUI tools for this XForms stuff - a person wouldn't want to try and code all that verbose XForms XML gibberish in a text editor. It's only possible to comprehend the simplest forms that are expressed using XForms (coupled to XHTML or SVG for presentation).

And no matter what manner of declarative form language I wind up with, I'll likely have to use Java applet code coupled to DHTML to do discreet operations on specific form widgets. The Java applet code interacts with the messaging system and then interacts with appropriate DHTML widget. Yeah, I'll probably have to hack the vendor's form technology in order to accomplish this in the way I need it to work.

You know, when it comes to forms technology, I feel like I live in the dark ages. Nobody has come up with a decent system for enterprise forms development purposes. The vendors should just take their HTML/HTTP forms garbage and cram it up their browsers. It's like they all imagine that we're all out there just creating Internet sites and the lame limitations of HTTP pull technology that go along with that gig.

Okay, somebody will suggest using Citrix Metaframe. I've ran these kind of systems. It works great in a lot of respects. I actually love Metaframe as a product and have great admiration for it. It can open new doors for the kind of commerce that can be done over the Internet. A client app form technology implementation can be completely arbitrary using Metaframe - so long as the forms engine runs on Windows. You certainly get the superb capability of a supervisor being able to shadow in on the form entry session of an employee. None-the-less, Citrix Metaframe still doesn't address what to use for the form technology itself.