The Wave of New Tools and Paradigms
Three significant trends will continue to impact developers substantially throughout 2004 and 2005. First, Open Source software development tools are not only becoming main stream but also are evolving their licensing terms to define a new rival standard to EULA-based commercial software. This will impact profoundly how software will be developed, priced and supported. Open Source software is rapidly becoming a major player, particularly in software development tools. And commercial vendors are already adjusting to that new reality.
Second, due to the complexity of current n-tier design and deployment, development software tools are splitting out into three new development paradigms: i)Backend Services, ii)Presentation Services, and iii)Interprocess Control. This split out is necessitated by the fact that the major IDEs either cannot deliver solutions or do so inefficiently for the new requirements: information at your fingertips, anytime, anywhere on any device. These requirements place conflicting needs on systems. Dynamic, runtime adjustable programs that are also highly secure and reliable. Or thin client apps that get most of theirstarting and ongoing instructions from a central server but that must be able to run offline and and then return to the central fold.. Fortunately, these new specialized tools can still be used effectively with the big IDE-Integrated Development Environs (think Microsoft Visual Studio, IBM Websphere Studio, Borland JBuilder, etc) because the specialized tools deliberately back integrate to the IDEs or have XML, Web Services, or other integrating connections. Also the role of the IDEs remains intact for a substantial swath of development tasks and roles. But don't be fooled, there is also a growing percentage of IT projects that are best handled by the new development services.
The third major trend is that a huge wave of new software will be appearing over the next year as recession and technically delayed software plus the new specialized toolkits reach the market in a huge tsunami of new development tools. For example, the major vendors will have big improvements of their already giant IDEs: BEA Weblogic Server, Borland JBuilder, IBM WebSphere , Oracle's JDeveloper, Sun's Java Studio among others can't fail to have major updates given the new J2SE 1.5 with its syntax improvements and J2EE 1.4 with its new extensions as well. Likewise, Microsoft's Visual Studio has already seen a major update with the release of BizTalk 2004 whose routines have been largely incorporated into the Studio. In addition, Whidbey or Visual Studio 2005 will launch in the first half of next year with further .NET based implementations.
As well there is a wave of new software coming from the Open Source communities. Already, MySQL has turned heads with its new cluster capable database. PostrgeSQL has new replication and .NET interfaces. PHP by early summer and Perl within the year will have significant new versions of their very popular web scripting languages. And NetBeans (beta is looking awfully good) and Eclipse(has a big lead in popular plugins) will be duking it out for fairest of Open Source Java IDEs with major upgrades. Ditto for category leading JBoss in J2EE servers. And this list just skims the surface of what is happening in Open Source tools.
Finally, the previously mentioned specialized tools are emerging on the market. This line up of new tools follows the MVC pattern. They include tools for M-Model tasks associated primarily with Backend Services such as MDD-Model Driven Development tools or SOA-Service Oriented Architecture tools. Next there are tools for V-Viewer tasks associated with Presentation Services such as RIA-Rich Internet Applications, PSI-Portal Server Interfaces or new Smart Graphics. Finally there are the C-Control tasks represented by InterProcess Control services lead by development tools for BPM-Business Process Management, EAI-Enterprise Application Integration and ESB-Enterprise Service Bus. These new tools help meet the new requirements of distributed processing.
In essence the problem boils down to this: delivering anytime, anywhere, and on any device information at your fingertips is simply not possible with the existing giant IDEs and 2-tier architectures. Neither CORBA nor Java nor .NET as delivered in the major IDEs is up to the task.. Some tools have to be supplemented by one or more of the specialized development tiers and associated servers. The classic example is the adding of a Presentation server to beyond current thin client development. The big IDEs can handle thin clients. Java IDEs use JSP to dynamically generate final server pages while Visual Studio has ASP.NET doing the same. But now end users want both thick and thin services; online, offline and rich media operations from the same application which the big IDEs are hard pressed to deliver. So it is up to the new Presentation Servers like Macromedia Flex or Plumtree Suite to provide more robust interfaces. But it is important to note that the big IDEs still can be used to setup backend and/or collaboration services because the new Presentation Servers will have the hooks and connectors to integrate withthe big IDEs.
The Problems of The Big IDEs
Some developers don't like big IDEs from the get go they prefer to work with Visual Notepad and do their own makes and debugging. And that is fine as long as they can remain productive. But the pace of change and complexity of systems find that big IDEs can contribute positive benefits by automating tedious and/or repetitive design, coding, and debug tasks. And most of the big comprehensive IDEs can deliver either standalone desktop .exes or 2nd generation client server applications.
These second generation client server application are nothing to sniff at - they take advantage of the Web interface, use application servers for business rules and smart load balancing and help guarantee application integrity with underlying database systems. There is even accommodation for asynchronous processing of some tasks so suppliers and customers can be integrated to some extent in to a larger system flow. Finally, many IDEs provide developer productivity aids as simple as color coding and code completion to automatic testing with refactoring hints plus build and project conveniences like Ant to sophisticated forward and reverse engineering of code snippets to drag and drop visual design.
It is almost a case of what more could you ask for ? But end users are asking for more especially now that computing has gone mobile and connected through WiFi and high bandwidth network connections. They want output to adapt to many devices: tablets, PDAs, mobile phones, smart kiosks, etc. And they want the devices to work whether connected or offline or occasionally connected. This puts enormous demands on developers. They now have to master input/output onto many and diverse devices. Also developers cannot rely on a server to be able to control and manage the session as ASP or JSP do because the device maybe offline, yet the user expects to get some work done with cached data. This puts pressure not just on Presentation Services but also on the Backend Services.
Finally, many developers and system architects argue that systems cannot afford to be developed in isolation. All the problems of isolated islands of information, silo applications and EAI-Enterprise Application Integration can be traced to a)lack of basic interoperability standards for data and programming and b)lack of planning for cross system interactions. The big IDEs are great for single projects but weaker at integrating into legacy apps and weaker still at adapting to and setting up systems with multi-step workflows or business processes that step across organizational or system boundaries.
We shall look at the new MVC-tier of tools and Servers that extend development into new paradigms. For example, almost all of the MVC tools utilize existing technologies in new fashion so developers are, in most cases, not confronted with learning a new language or methodology. Second, almost all of the MVC tools bend over backwards to back integrate into existing tools and legacy systems. Not just XML or Web Services but broad interoperability, open and defacto standards like Adobe PDF, Apple Quicktime, Macromedia SWF, W3C XMI, CORBA, EDI plus cross platform functionality are prevailing. So we start our tour of new tools with the M-Model=Backend Servers.
New Backend Server Tools
Developers have had a love-hate relationship with automated programming toolkits that expedite development by generating application code as in CASE-Computer Aided Software Engineering tools of the late 80's and early 90's or the 4GL-4th Generation Language tools of the 70's and 80's. Well they are back and primarily because backend development has become both complex and full of lots of mind-numbingly tedious work to do. Plus now there is the need to deliver code in both on-the-fly and expedited fashion as computing pervades all aspects of business. Finally, the concept of dynamic generation of code from templates has taken root with 2-tier presentation systems like ASP, JSP, and PHP using code-generation extensively. For example, displayed web pages are dynamically programmed or generated back at the server. So now developers are more conceptually disposed to solving problems with generated software.
Backend Services are threefold:
To meet these challenges, SOA-Service Oriented Architecture helps to solve some of these backend problems because it is a framework for defining any functions or services using a standards-based descriptor language which provide consistent APIs or methods/interfaces used to call business processes. . As well SOA describes processes and interfaces that are platform independent: this targets the anytime,anywhere and any device client seeking services that are OS and programming language agnostic. In effect SOA seeks to create platform independent, reusable components. For skeptics, backend services sound like the Holy Grail of software.
But in fact as basic backend services are evolving so too are methods of provisioning services for them. Because systems are operating over many tiers, there are constant data mapping and binding tasks plus other rote operations that are conducive to automation. As well the advance of object design methods with strong pattern templates and refactoring methodologies plus XML exchange and mappings puts both services and auto-generation systems on a much more solid base playing into the strengths of the new MDD-Model Driven Development and SOA-Service Oriented Architectures. Finally there is not just the appeal but the realization that using a constantly updated design model to drive auto-generation(MDD) or refactoring of large subsystems(SOA) is within practical reach. See the Backend Services references on MDD and SOA below.
The Current Presentation Challenges
Presentation Services present an even more daunting challenge for developers. The
Web interface has gradually won a preeminent role in development. However,
the Web interface has a number of problems:
of these problems are absolute show-stoppers for the Web interface. And
some problems are being addressed by current software. For example, ASP/ASP.NET,
JSP/JSF(Java Server Faces)+Struts, Cold Fusion and other markup languages
start to address issues 2, 4, 5, and 6. But it is the new generation
Applications and Portal Servers that effectively take on the Web Interface
issues. The second generation of Presentation Services
In essence, rich clients are a reaction against too thin clients. Typically, RIAs and Portals bring back to the client-side such operations as validations, simple business logic, UI control and event handling, even database interactions. The portal client also addresses Web interface issues. Notably 6)because they are designed to be multi-views but also 4) with a broader range of component support and 8)by running on a wide range of PC clients.
However, backend connectivity can create issues for 7) offline operation and 6) multi-view or portal operations since those environs become more challenging for the backend interface. In effect the client can become too big and unwieldly again. Also as the client becomes thicker to allow for these stand-alone and/or multi-views, security and performance start to become more problematic. Security is an issue because dynamic code creation and offline/online transitions create points of exposure. Performance can suffer because as the client becomes fatter and transactions longer or reconciliation takes longer there is slowing of overall transaction rates.
split of development into specialized Presentation Services appears to
be a win-win situation. Because most RIA-Rich Internet Applications and
Servers and their associated studios a)provide easy to use visual designers
speeding development while b)including strong connections to backend services
to integrate well with existing systems and development tools. And from
the programmer's point of view, most vendors are using a blend of existing
technologies and supplying complete, developer editions for free download
so getting up to speed on some of the new technologies is easier than ever
before. In sum look for Presentation Services to be split off from the
big IDEs until those tools can offer comparable online/offline, anywhere,
anytime on any device user interfaces.
InterProcess Control System Problems
InterProcess Control Systems have long been a part of software development. Message Queuing Middleware, Peer-to-Peer systems, Multi-user Agenting, Workflow Management Systems are a small sampling in this arena. There have been many good systems from a variety of vendors: Staffware, Lotus Notes, Jini, Groove, etc. But many of these software tools have been held hapless to contend against two major software disabilities. First, in computing there has been a very low level of data interoperability. For example, it has taken about 45 years of development to arrive at an adequate and universal character coding scheme, Unicode. In the 1980's, technologies and data interchange processes such as SGML, SQL, EDI and various ETL schemes emerged but also fell victim to proprietary extensions and simple non-standards compliance - close but no cigar. Only with the launch of the Web has the overwhelming need for a standard data interchange mechanism allowed XML to resist the centripetal forces of proprietary extensions and become a nearly universal and robust standard.
Finally, Web Services with its standard XML packaging has been able to supply the last magic ingredient a standard mechanism for making InterProcess Service Calls between systems on the same or different platforms. Not only that to package the request message but also sophisticated conditional response messages as well. Finally, messages and requests for service can be synchronous (I am waiting for your immediate reply) or asynchronous (signal when you are ready to serve). Actually IT community had at least two previous standards for InterProcess Service Calls but some major software players refused to embrace them.
Services changes all that. W3C, OASIS, WSI and others are now at work defining
the authentication, encryption, identity, integrity, security and other
process standards needed to consolidate the strong base put in place with
SOAP, UDDI, WSDL. This will put InterProcess Control solidly back into
Management, EAI-Enterprise Application Integration, P2P-Peer-to-Peer processing,
and Workflow Management Systems on a much more solid foundation. InterProcess
Control has earned a name - the Enterprise Service Bus or ESB (see Figure
below). The question is not "will ESB-like services happen?" (both Gartner
and IDC see major adoption by 2005 year end). The question is how and what
InterProcess Control services will be provided.
And the industry has responded with rapid developments in the past 2-3 years in all of above areas as eliminating island of information and application silos has stayed at the top of IT executives priorities. In addition, business motivators for InterProcess Control now include government compliance decrees such as Sarbanes-Oxley, SEC's rule 17a-4 on Financial Institution retention of info and client transactions or HIPAA employee health insurance compliance and other process-verification regulations such as NASDAQ 3010 and NYSE 342. As well there are organization's own process improvements as envisioned in Six Sigma and other process methodologies. Table 3 below lists some of the major players and the type of InterProcess Control system they are selling.
Forrester is reporting that BPM is doing well despite the economy just emerging from a down turn. Trial of new BPM is up by 300% over 2002 with technology and telecom firms leading the way. So expect to see the trend in system development to follow project management. Five years ago project management tools were used primarily on single, large projects; now they are used on many projects and contribute their results into portfolio of all project results. These data are then used by the project portfolio programs to not only rate the projects but also help to decide the sharing and movement of people and other resources between projects. BPM, EAI, ESB are starting (along with SOA) to foster that same global view of system applications and the running assets they represent so these components can be shared across the enterprise.
Why the Tsunami of Tools
It should now be fairly obvious why there will be a wave of tools which will be coming to market through 2004-2005. First it is the usual rate of change in software development exaggerated by the 3 year slowdown in the economy. But don't underestimate the effect of finally having open standards for data interchange and lightweight persistence with XML plus a highly interoperable programming language in Java. This has tripped off a lot of innovation in novel solutions to the tough problems of n-tier, heterogeneous, distributed processing. Also, don't discount the influence of Open Source. As we have seen Open Source is not only contributing to the new specialized Server categories but also is helping to curb proprietary extravagances. Finally, for developers it is a mixed bag. Yes, the new MVC Servers may mean smearing an application over 3-5 tiers with all the complexity that entails. But the good news is that much of the technology used is well integrated with the novel application of old familiars. So this may be more a story of not the new but the rediscovery of Paradigms Lost.
Jacques Surveyer is a consultant & photographer, see the latter at thephotofinishes.com
usability practices a good analysis of what makes a rich interface
richer - Boxesandarrows
Open Source Links
Open Source licensing Terms List contained at Open Source Institute - OSI
InterProcess Control Links
New vision of EAI/BAM/BPM-
Overview by Yankee Group for InterSystems Ensemble - Yankee
-MDA FAQ from OMG