Write for OTN
Earn money and promote your technical skills by writing a technical article for Oracle Technology Network.
Learn more
Stay Connected
OTN Architect Community
OTN ArchBeat Blog Facebook Twitter YouTube Podcast Icon

A Case for Oracle Fusion Middleware

by Lucas Jellema

An in-depth look at the interaction of people, processes, and technologies in the transition to a service-oriented architecture.

August 2013

Downloads
download-icon13-1Oracle Fusion Middleware

Author's Note

This article presents a profile of a fictitious organization, NOPERU. The story of NOPERU as told in this article is actually a collage of the events at some dozen organizations that I have been involved with over the past few years. None of these organizations sport all the characteristics of NOPERU - but all of them have gone through or are going through a similar transition as described here and all aspects of this article were taken from real life at one or usually many of these organizations.

Background

NOPERU (National Organization for Permits for Emissions and Resource Usage) is a public organization that continues to transform in terms of its business, organization and technology. Changing business requirements; new interaction channels; and increasing demands for more flexibility, faster throughput and lower costs drive these transformations, while technological evolution and new architecture patterns enable the change. NOPERU chose Oracle Fusion Middleware as the technology platform to implement the new architecture and required applications.

This article takes a close look at NOPERU's journey from its origins in the early 1990s as a largely paper-based entity with regional databases and client-server Oracle Forms applications. Its upcoming business objectives are introduced: what is required of the organization and what the higher goals behind these requirements are. The architecture roadmap is described at a high level as well as drilled down to a service oriented design. Based on the architecture roadmap and the business requirements and NOPERU went through a technology selection to determine the technology stack with which the future would be realized in terms of IT.

The article discusses that selection and details the projects subsequently planned (and executed to date). The new architecture and technology as well as the introduction of an Agile development method have had substantial consequences for the IT organization, the processes and individual staff members. The approach NOPERU has adopted with regard to the people and the organization is portrayed. Finally, the article discusses many conclusions that NOPERU has drawn that may benefit itself and other organizations.

Introducing NOPERU

NOPERU is a national organization charged with issuing permits for excessive emissions (i.e., carbon dioxide) and disproportionate usage of such resources as energy or water. Anyone-whether a commercial enterprise, government agency or private person--who emits or consumes more than what is considered "fair usage" requires such a permit. When someone builds an outdoor heated swimming pool, for example, or open-air terrace heating, such a permit needs to be obtained. When a company installs new, energy-intensive equipment, such as water boilers or deep freezers, it too needs to get a NOPERU permit. Government-sponsored projects at every level that involve consumption of large quantities of fresh water or production of high volumes of emissions must turn to NOPERU for a permit. Without the required license, any interested party can get a court to immediately put a stop to the disputed activity.

The typical process an applicant will go through at NOPERU looks something like this:

  1. File the permit application on the proper paper forms with supporting documents attached. Submit it to the regional branch that has jurisdiction as well as the appropriate sector.
  2. NOPERU responds to the submission with a receipt confirmation and an invoice for the submission fee.
  3. Once the fee has been paid, the real processing begins.
  4. Different specialists at NOPERU - financial, environmental, legal, etc. - are involved with judging the application. Frequently, NOPERU will require additional information from the applicant.
  5. Sometimes an on-site inspection will be conducted.
  6. A hearing may be part of the procedure.
  7. NOPERU publishes the ruling, which falls in one of three categories: the permit may be denied altogether, it may be awarded unconditionally, or it may be granted under specific conditions (which may include payment of a special tax).
  8. The ruling is sent to the applicant and is published in local newspapers. All permits are held in a public register that interested parties can come to visit in one of the NOPERU offices. The entire process can take from three weeks for very simple, run-of-the-mill activities, up to two years for complex, large-scale endeavors.

NOPERU was set up in the early 1990s. It still has five regional branches, each of which cater to the three sectors of citizens, corporations, and public bodies. Originally, all interactions with NOPERU took place through traditional channels: telephone, paper-based mail, fax and kiosks. NOPERU is available for synchronous interaction (immediate reaction) only during office hours. Mail and fax can be sent at any time, of course, but are processed only during office hours.

Computer applications supporting internal business processes were developed in those early days using then state-of-the-art client/server technology: Oracle Forms running against an Oracle 7 database. The internal application developed for the corporate sector was cloned to form the basis for applications submitted by public bodies and citizens. Over the years, the differences between the three applications have increased - making for three virtually unrelated systems today. Further, each application has its own database and, because the applications run locally in each regional branch, there are 15 databases in total and 15 application instances, making administrative actions and application upgrades cumbersome, expensive and risky.

NOPERU does no logical data management. An applicant is registered anew for each new application. Large corporations that operate nationwide and file hundreds of applications annually will exist hundreds of times in the databases of each of the five regions. Even individual citizens who have applied for a few permits over the years with the same office will be represented in the regional database by multiple records--records that are not kept in synch. Furthermore, NOPERU has no policy for information lifecycle management. Records exist forever. As a result, the databases grow, carrying around an increasing bulk of effectively dead data.

The original client/server applications are still in use today, almost unchanged from their initial incarnation except for the fact that they are now Forms 9i browser-based applications. These applications are very much data-oriented - basically providing a window on all the data and support for querying and manipulating that data. The application is not intuitive, and the learning curve for new staff is steep.

Worse still, the applications neither support nor enforce NOPERU's business processes, which exist largely in the heads of the people involved. The process is monitored by tracking the whereabouts of the paper file containing the documents associated with a specific application: the process enters a new stage when that file is handed over to the next actor. Further, the way permit applications are processed differs across NOPERU's five branches and, to be honest, it even slightly varies from person to person in the same office. Keeping track of processes, focusing on efficiency, and making sure deadlines are met are, mildly put, real challenges.

Business Objectives

The previous section suggests a number of areas where improvements could be made in the way IT is organized at NOPERU in order to achieve higher service quality, faster and more consistent process execution, more productive staff, lower costs in business and IT and happier faces all around.

NOPERU management, in part of their own accord and partly guided by political instructions and promptings from vocal citizens and engaged business partners, have designed a program for the future: Go Forward (2010-2018). Here are the most important driving factors in this program:

  • Speed. Permit applications must be processed much more quickly: simple, complete applications within three business days, and more complex applications within 6 weeks.
  • Accessibility. Modern interaction channels must be introduced to allow 24/7 access to submission and tracking of permit applications. Among these channels are to be a B2B service, a web portal and a mobile application. A voice response system allowing for basic interaction outside of office hours should be investigated.
  • Paperless. In 2018, NOPERU should be paperless. Both the forms and the supporting documents required for an application should be submitted in electronic format, which should dramatically reduce the workload on the inbound side of the NOPERU operation and shrink the lead time from the moment the application is submitted to the moment a staff member can actually start working on it . Furthermore, having multiple staff members work on the same application at the same time will finally be standard procedure. To cater to the professional intermediaries who manage permit application and all associated interactions with NOPERU on behalf of large corporations, NOPERU will make a portal available that makes it easy to submit and track large numbers of applications on behalf of multiple clients.
  • Usability. The learning curve for new staff should be much shorter than today. NOPERU's workforce is fairly flexible, with a high turnover in several roles. A lot of money can be saved if new staff is productive in a much shorter period of time and make fewer mistakes caused by user interface unfriendliness.

Management at NOPERU also requires more insight. It wants to be aware of delays in process execution, bottlenecks and other process inefficiencies, and wants to be able to continuously improve business processes without long lead times, massive development effort or high risks. They have heard the phrase "embrace change" at some Agile seminar - and they like it. They desire the flexibility and short time to market promised by the Agile evangelist.

Efficiency. Even though NOPERU actually makes money for the central government, it still must reduce operational costs by at least 20%. Downsizing the workforce will almost certainly be a major component of meeting such a demand. If permit applicants submit all information in electronic format and if all information about an application's progress is available on line, a significant proportion of the current workload will disappear. Self-service through portals and mobile applications might seem to be all about customer satisfaction, but it also works miracles in terms of cost savings.

Simpler applications with short learning curves and requiring less business understanding and process expertise should allow NOPERU to work with temporary workers, creating a flexible layer that can shrink and grow with the demand for permits. NOPERU's management also hopes that part of the ruling on applications can be automated - for example, by using business rules engines that capture logic currently applied by human actors.

IT Objectives

NOPERU's CIO - guided by a team of architects - has made sure that a number of specific IT objectives have been included in the Go Forward program's design. She wants to ensure that the vital role IT plays in NOPERU's operation is recognized and that a clear statement is laid down with regard to IT that serves as the starting point of the IT roadmap.

An enterprise architecture design serves as the foundation for all future projects. The enterprise architecture will be translated into modern IT architecture patterns and subsequently into the design of applications and infrastructure.

The architecture and the technology selected must allow for flexibility: changing functionality should be possible in a simple, cheap, fast and risk-free manner. NOPERU wants to go agile, both in business and in IT. Furthermore, the evolution of the systems and the transitions to new systems must occur while the shop stays open. NOPERU cannot afford to close down, especially once 24/7 online channels (web, B2B, mobile) have been introduced.

NOPERU´s IT must strike a fine balance between being up-to-date and being of proven reliability. It should:

  • Be based on current industry standards that are open and promote interaction and reuse
  • Use only proven concepts and components
  • Appeal to staff and clients
  • Allow NOPERU to find the right resources to aid in the design, development, and administration of infrastructures and systems
  • UUse software that has a large community around it and the backing of one or more large commercial parties. (NOPERU has learned this lesson the hard way.)

NOPERU intends to work with a small number of strategic vendors that have a broad product portfolio, a clear roadmap, the ability and intention to keep evolving, and a willingness to cooperate and (ideally) take responsibility for NOPERU´s success with their products. Slideware won't do: the potential of the vendor's products must be demonstrated through customer references.

NOPERU is opening up to the outside world. In the past, all interactions beyond the perimeter of its physical sites were on paper, through fax or by telephone. The only online interaction was email and a read-only website based on a database in the DMZ that was refreshed once a day from a file dump. Now, however, synchronous interactions with the NOPERU enterprise systems will have to be supported in the B2B exchange, for the mobile apps, and for a much more interactive real-time web application. Security has to be at the heart of these initiatives; because it is imperative that information be made available only to authorized parties, any party dealing with NOPERU must be identified securely. The integrity of the data will also play a more important role: automated processes do not have the same capacity as people to account for inconsistencies or simple typos. Furthermore, because the data from internal systems will be published directly to portal and B2B channels, without human checks and filters, its quality must be better than it is today.

Part of the plan is to reduce IT expenditure by consolidating onto a central infrastructure with a single database and a single application instance. This should also help with the quality of data by reducing data duplication, replication and the potential for inconsistency. It will also lower hardware costs,software license expenses, and payroll . In the initial stages of the program´s execution, however, IT spending is likely to increase because of the many projects that will have to be carried out in order to achieve the objectives. The intended consolidation of all IT infrastructure and all data into a single instance means, on the one hand, a relief for the security officer: fewer sites, administrators and environments to worry about. However, this consolidation also means that availability - which in the 24/7 world NOPERU is about to enter is essential - becomes a much harder challenge. The consolidated systems, logically a single instance, form the critical factor in virtually all of NOPERU's activities. It is a single point of failure - at least logically. Part of the IT roadmap is taking measures to safeguard the availability of all IT components (e.g., through clustering and fail-over).

Architecture Roadmap

The architecture team at NOPERU leads the way in terms of technology evolution. This team has drawn up the high-level IT architecture, selected and fine-tuned the architecture patterns that are to be applied, and worked with developers to design the reference architecture. This reference architecture provides guidelines on how to make use of the patterns when designing system components and how to make use of the selected technology to implement these patterns. It also provides a common vocabulary with which to discuss implementations.

Partly based on these architecture designs, the strategic technology and vendor selection is made; it is after all imperative that the vendor and its product portfolio is capable of implementing the architecture as designed by the NOPERU team.

The legacy at NOPERU involved a classic case of application silos: stand-alone units consisting of a database, business logic and a user interface. Each application is implemented through its own silo, using distinct and sometimes very proprietary technologies and maintained by inward-facing, somewhat self-absorbed teams. Flexibly sharing resources between these teams is neither common nor easy. Technical integration between the silos for exchanging and ideally truly sharing data is also hard to achieve; frequently, files are used to export and import data in an asynchronous batch process that may involve manual actions. Data replication is common, as is the human task of typing to re-enter data--just because data may already exist in one silo, that does not mean it is accessible to another. And because data exchange is not readily available, manually keying in that same information is frequently the easy way out.

Breaking up the silos is an absolute requirement in the new architecture. Lasagna-style is on the menu: a layered architecture with clear responsibilities assigned to each layer, and with well-defined interfaces describing the interactions between these layers.

jellema-case-for-FMW-fig01
Figure 1: The transformation from silos to layered architecture

No single team or application is owner of what essentially is (and always should have been) treated as enterprise data that can be used in many different processes. Teams and projects are not masters of their own destiny: the decisions they make about technology, application layout and implementation patterns must fit into the enterprise landscape.

The layers identified in this architecture:

  • User Interface and Programmatic Interface Layer - interaction with the outside world through human-oriented and system-oriented interfaces.
  • Business Layer - common interface to data and business logic, reusable across user and programmatic interfaces
  • Data Layer -the persistency and integrity of all enterprise information assets, including documents and other unstructured data

Identifying these layers will help to establish a clear separation of concerns at NOPERU:

  • Each layer has its own role, using its own design patterns and designated technology.
  • The implementation of each layer is encapsulated to other layers.
  • Layers can invoke only the next lower layer.
  • Layers are unaware of any other layers, except for that directly underneath them.
  • Communication is always started at the top, flowing downwards.

(Note: events are used complement this rule and to make it possible to publish information that overhead layers may need to be aware of.)

Each layer should have clear interfaces defined that describe the interaction that it supports. Part of this interface description is the definition of the operations that can be invoked in the layer, the input these operations require and the output they return - including exceptions that can be thrown - as well as a description of side effects of the call (such as emails being sent, products being shipped or data being persisted). Non-functional aspects of operations should also be described; these include availability (opening hours), costs, authorization and other security aspects, response time and accepted volumes.

Without strict guidelines to actually enforce this, the architecture team still hopes and expects that within each layer the teams will use the same or at least open technologies, shared skills, common design patterns and a largely shared infrastructure. The technology selection will be in line with this desire.

One of the key decisions made in the early stages of the architecture design was the adoption of many service-oriented architecture (SOA) principles, including decoupling (loose coupling at the very least), abstraction and encapsulation, reusability, and location virtualization. Applying these principles and implementing layered architecture will help realize crucial business requirements: flexibility, short time to market, efficiency through reuse, and risk reduction.

The Triangle

A simple illustration-an inverted triangle-played an increasingly important role in discussions about NOPERU's IT future:

jellema-case-for-FMW-fig02
Figure 2: Layers

This triangle visualizes the distinction between the layers in terms of their reuse potential and generic nature vs. their multi-channel support and level of customization.

The Data Layer is characterized by centralization and (logical) consolidation. Data assets have a single source of truth. This layer has a very high reuse potential and a generic, enterprise-wide structure. As a core enterprise resource, it has high requirements in quality, integrity, availability and confidentiality. The rate of change at this level is fairly low, at least at the meta-level. Data is not removed very frequently and the data structures evolve even more slowly.

The Interface Layer that exposes interfaces to users and applications is different. It sports a large variety of user interfaces and devices, catering to very specific channels, consumers and user roles - allowing customization and personalization to meet special requirements. The average life time of components in this layer is fairly short - especially compared to the data layer - and the rate of change is much higher.

The Business Layer - the man in the middle - is also in the middle in terms of reusability and rate of change. It offers services that are aimed at reuse by various interface components. This layer brings together assets from the data layer; implements business logic; validates, processes and enriches data; and runs processes. The rate of change is higher than at the Data Layer, as is the functional variety. Compared to the User and Application Interface Layer, however, the Data Layer evolves much more slowly, is much more focused on reuse, and caters for far fewer specific, one-off needs.

Most business requirements are expected to find the majority of the required IT effort in the top layer, a sizable portion still in the middle layer and very little effort - because of reuse - in the bottom layer. The triangle therefore also represents the work ratio in many development projects and even suggests a team composition.

The transition NOPERU is undergoing is now characterized at a very abstract level by this illustration:

jellema-case-for-FMW-fig03
Figure 3: Transition

Domains of Data

The enterprise architecture has identified five relatively independent domains within NOPERU. Data in each domain, and services exposing that data, are to be controlled by domain experts. There are to be no direct relationships between components in different domains. At any time a domain could be re-implemented using a commercial off-the-shelf offering, such as a SaaS CRM system or a third-party Expertise application.

The five domains identified by the architects are:

jellema-case-for-FMW-fig04
Figure 4: The five domains

NOPERU and each domain individually have a key asset: data. Most interactions with and across domains will involve data, making essential a common understanding of, and language for interacting about, that data.

The architects therefore have launched an initiative to create an Enterprise Data Model,.which describes all business objects that are meaningful to NOPERU's operations, including their properties and relationships. Common terminology as well as lists of reference values that are to be used to set the value of certain properties are defined in a standardized way and made available throughout the organization. Note: the model is defined at the business layer and may not necessarily be fully aligned with the database structures and other technical assets inside each of the domains. The Enterprise Data Model (EDM) is the common business language across all of NOPERU, stretching beyond IT. All interactions between the Business Layer and the Data Layer will be in terms of the EDM. Operations inside the Data Layer will use existing idiom and structures for quite some time to come, which is both unavoidable and perfectly acceptable.

Service-Oriented Architecture

The decision to make service-oriented architecture the leading architecture principle has a number of consequences. Middleware is still fairly new at NOPERU. Teams used to be organized around applications - around the silos discussed above. The Business Layer at the heart of the layered architecture will be the new focal point for all teams. This layer is made up of a number of different types of services that bring together all data from the Data Layer - structured and unstructured, across databases, document repositories, LDAP instances and mail servers - and expose access to this data in a standardized, unified way. Moreover, these services make business logic available to applications - both user interfaces and programmatic channels.

Part of the foundation of the Business Layer is the Enterprise Data Model (or "Canonical Data Model") and, more specifically, an XML representation of that model, expressed in terms of a heavily annotated XSD (XML Schema Definition). All data structures handled by the services - both input and output - are defined using business objects in this canonical XSD. The five data domains are recognizable through the namespace structure used in the XSD definitions.

The architecture team came up with a service classification scheme that helps organize the services as well as the teams working on those services. Following this scheme, the Business Layer is subdivided into these types of services:

jellema-case-for-FMW-fig05
Figure 5: Business layer services
  • Elementary Services: provide atomic functionality within a domain; their reuse potential is high, their added value usually low
  • Composite Services: combine two or more elementary services into business functions with higher added value; composite services come in two flavors:
    • within a domain
    • across domains (typically introducing the need for either global transactions or transaction compensation)
  • Process Services: asynchronous and longer running (up to minutes, hours or even days) and usually contain state while running Presentation Services: not meant for general reuse; instead, they cater to specific needs of an application or channel - either a user interface or a programmatic interface - and speak the specific language of that application
  • Utility Services: generic, domain-independent, highly reusable services, frequently of an almost infrastructural nature (e.g., logging, sending emails, value translation, geocoding, etc.)

Service design and implementation guidelines are created per service category. Governance, ownership, testing and many other aspects of the services also depend on the type of service.

Note: except for Presentation Services, all services are expressed in terms of the canonical data model. They are all recorded in a central service catalog where potential consumers will find information about the service, including functionality, contract, non-functional aspects and status. At NOPERU this service catalog is a simple Wiki that references the live Web Services Description Language (WSDL) and XSD specifications of services.

Event-Driven Architecture

For the architecture layers it was stated that a layer cannot invoke a higher layer - or even be aware of it. This does not mean, however, that a lower level layer or service will never have something to tell that could be of interest to a higher level service; it means that information must be communicated in a way other than a direct call. To address this challenge, the NOPERU architects have adopted elements from Event-Driven Architecture (EDA). Events are used as the very decoupled vehicle to convey information without any direct dependencies between the source and the recipient(s) of the information.

In addition to defining the canonical data model and identifying the services, NOPERU analysts are working on discovering "business events"--conditions or situations that may arise somewhere in NOPERU's daily operations that are potentially of interest to other parties. Events of interest to NOPERU can of course also take place in the outside world: these, too, classify as business event.

A business event is described by a name or type, a timestamp and a payload - data that clarifies what the event entails. Examples of business events at NOPERU include: Application for Permit is withdrawn, Payment has been received, Ruling on application has been made, Corporation has filed for bankruptcy, Deadline has expired in some business process, Business rule has been changed. The reference architecture at NOPERU describes an event-handling infrastructure - a generic facility that is available to all application components and services alike.

jellema-case-for-FMW-fig06
Figure 6: Reference Architecture

Anyone can publish a business event to this event handler, provided the event type is predefined and the payload has the predefined structure. After publishing the event, the publisher is not involved in any way with the delivery of the event and does not even know if the event is consumed by any party at all. Any component at any layer in the architecture can subscribe to selected types of business events. The event handler will push any published event to all subscribers to the type of this published event. Consumers of the event will receive the event with its payload and can do with it as they see fit. They are not aware of the component that published the event.

The perfect decoupling achieved through the events makes it extremely simple to add consumers or introduce new publishers of a particular type of event. Removing subscribers to an event is another zero-impact procedure, as is losing one or more publishers of events.

Through events, elementary services and even components in the Data Layer can tell their story to composite or process services or even to user interface components - without ever knowing about them. Interaction takes place, but in an entirely decoupled fashion.

Technology (Vendor) Selection

The business objectives and the derived layered architecture, as well as the more detailed architecture principles, result in a clear image of the technology components NOPERU requires. NOPERU will have to acquire from vendors a number of technology products in order to implement important elements of the solution

jellema-case-for-FMW-fig07
Figure 7: Required Technology Components

NOPERU has stated from the outset that it does not want to buy into a single integrated suite of products just because it is a single integrated suite. It wants the best product available in each category, and requires that all products be open, standards-based and easily integrated.

A number of additional assessment criteria were determined. Because IT is not NOPERU's core activity, it wants to use proven technology and products, backed by verifiable references. Products must have a clear strategy as well as a strong community and an abundance of resources--qualified IT professionals, books, internet forums and blogs, training material, etc.-and be of strategic importance to their vendors. The vendors themselves (or open source projects) should be stable and future proof. Finally, the learning curve for a product should be clear and justifiable, given NOPERU's existing workforce. NOPERU verifies analyst reviews and takes them into consideration.

While it is a substantial operation, NOPERU's resources are still limited, and it does not want to have a large variety of technologies and platforms that require different skills. It wants to focus on a small number of major industry platforms - hardware, O/S, virtualization, database and middleware - to prevent a nightmare for the administrators. For that same reason, all products selected should provide sufficient options for monitoring and configuration and should allow for automated build and deployment.

Of course, the cost of the software is an important part of the decision. Here, NOPERU will look at a number of aspects. What is the license fee per pricing unit - user, CPU core, hour of usage - and what is the estimated number of pricing units? What is the yearly support fee and what other pricing elements play a role? What discounts can be negotiated? How long are the licenses valid? What alternative constructions are available - such as a subscription fee? What assurances about software behavior is the vendor prepared to make? What SLAs will it enter? Is the vendor prepared to take some form of responsibility for the successful implementation of the software - possibly a no-cure/no-pay construction or a fee that is partly based on return on investment?

Selection Approach

NOPERU had to make two inseparable selections: vendors and products.

It published a Request for Information that focused on a checklist with features, technical characteristics, implementation requirements, training and order of magnitude pricing. NOPERU received responses from vendors of niche products, specialists in very specific areas, as well as reactions from vendors of suites of products that covered wide ranges of products. It also invited a number of parties to represent open source products.

In parallel with the RFI process, NOPERU gathered information from peer organizations - largely governmental - about product and vendor experiences. NOPERU also consulted market analysts, both on-line and in person.

The information received at this stage of the process was screened and evaluated based on the criteria listed above. This resulted in a short list of both products and vendors. The vendors on this short list were invited for the next stage: Request for Proposal.

In this stage, vendors were asked to present a plan for how their product(s) could best be used by NOPERU - including the infrastructure topology, the licenses, the migration of systems and transition of processes and staff. The proposal needed to cover the overall price, as well as any alternative compensation proposals (such as deferred payment, result- based payment, subscription-based fee and usage-based fees).

Each vendor had to present relevant customer references that NOPERU could contact and visit. It also had to lay down the product roadmap and longer-term vision and strategy. NOPERU wanted to be convinced that the products presented were indeed future proof. By this time, NOPERU had decided to take a slightly different approach, and defined several technology clusters that it wanted to select in somewhat separate stages:

  • Hardware - no immediate investment; virtualization using VMware was the short-term way forward; investigations begun into a multi-site, disaster-proof, very high- availability data center layout.
  • Database - consolidation on Oracle Database 11gR2 RAC (which means upgrading some databases and replacing some MySQL and SQL Server databases); the release of Oracle Database 12c is regarded with interest but no short-term adoption plan.
  • Mobile - no strategic selection is made for now; given the volatility in the mobile market and the fact that mobile applications are on the very outside of the enterprise with little impact and a short lifespan, it was deemed unnecessary to make a strategic selection for mobile technology.
  • Internal user interface applications - for quite some time to come, some Oracle Forms applications will be used and maintained; one application has been migrated to (or rather rebuilt in) Oracle ADF 11g. NOPERU evaluated its experiences with ADF and decided that, for now, it has no reason to select a different technology. It did of course verify the strategic importance of ADF for Oracle, the strength of the community, the roadmap and vision and the pricing, and found them satisfactory. The availability of external resources is somewhat worrying, although the realization that ADF is a Java EE framework that any Java Web Developer can quickly embrace by and large took care of that worry.

For the following clusters, a separate RFP is conducted:

  • " Service-Oriented Middleware - NOPERU needs products to implement an enterprise service bus, service orchestration, event-driven architecture and technology adapters; these products have to work together well and ideally use the same platform.
  • " Process and Human Workflow Management - in this cluster, NOPERU identified the need for decision rules (i.e., business rules), business process orchestration, human tasks and workflow management and business activity monitoring (BAM); these products have to be able to closely work together as well as work with the service oriented middleware.
  • " Enterprise Content Management - one of the business objectives in NOPERU's Go Forward program is the complete eradication of paper; working with digital documents is crucial for the organization. It requires products to store, search and publish, convert, tag and track, archive and protect digital documents in various formats. These products should of course fit in the service-oriented architecture that NOPERU will establish.
  • " Portals and external user interface applications - because of the decoupling achieved in the layered architecture and the use of services, the dependencies between portals and other external interfaces on the one hand and the products in the business layer on the other are minimal; that means that a decision on the products used for these portals can be made independently of the other product selections.
  • " Identity and Access Management - NOPERU is opening up its enterprise applications to users outside the organization; this new situation calls for a new approach to identity management, the implementation of authentication, and authorization based on identity; NOPERU uses Microsoft Active Directory for its internal staff and is not keen on abandoning that platform (as it is integrated into overall office automation). It wants to introduce products that will handle identities and authorization for external users. It also requires products that handle encryption, digital signatures and other security techniques that are to applied to certain services.

Selections

Most of the products in the clusters Service Oriented Middleware and Process and Human Workflow Management that made it onto the short list were based on the Java EE platform. For this reason, and because the technology for the internal applications was set as ADF 11g, another Java EE based technology, and good old Oracle Forms - also running on the Java EE platform - NOPERU decided to choose Java EE as the core platform for all its middleware. It also selected Oracle WebLogic 11g as the Java EE application server of choice. Another application server could be considered only when NOPERU selected a best of breed product with superior functionality that was unable to run on WebLogic. Note that NOPERU very specifically kept open the possibility to support a different platform in the User and Application Interface layer. Some internal politics may have been part of that decision; there was some resistance against going Java all the way, from some of the .Net-oriented teams.

The product selection for enterprise service bus and service orchestration--the latter quickly translated to Business Process Execution Language (BPEL)--evaluated Microsoft BizTalk and various TIBCO products, as well as some open source offerings, and then decided on Oracle SOA Suite and Oracle Service Bus. This combination also brought in the required technology adapters and support for event handling through the SOA Suite Event Delivery Network, as well as support for Java Message Service (JMS) and Advanced Queuing (AQ).

SOA Suite's support for Decision (Business) Rules and Human Workflow, as well as the strong integration with Oracle BPM Suite at both design time and run time, weighed strongly in favor of the latter when NOPERU selected a product for process orchestration and human workflow. The SOA Suite license includes most of the required functionality with rich (enough) functionality and a track record of many years. The BPM Suite adds support for true Business Process Model and Notation (BPMN) process modeling and execution, and comes with a range of run-time tools that help design, track and monitor, manage and collaborate on process instances. NOPERU ended up selecting BPM Suite because of its best of breed quality and the huge bonus of perfect integration with SOA Suite. The rapid evolution of the BPM Suite demonstrated to NOPERU the vendor's dedication to the product, and the projected usage of both SOA Suite and BPM within Oracle's Fusion Applications was proof of the strategic importance of these products to Oracle.

The product selection for content management did not go very smoothly at all. Because the subject was largely unfamiliar to NOPERU, they hired an external consultant-who was quickly let go when he was discovered to be biased. Next, the definition of "content" was hotly contested; some thought it meant the static content of websites, while others were thinking about all documents - or even all unstructured data - passing through the organization. Even the naming of the Oracle product - WebCenter Content - made some eyebrows go up; it sounded like that static website content thing that they had been able to get off the table only after much debate. Only the reassurance that WebCenter Content was in fact the Universal Content Manager restored peace and quiet. In the end, it was decided to give WebCenter Content a go - not because it was such a clear winner but because of its easy integration into the Oracle Fusion Middleware Platform and the perceived lower risk and smaller effort resulting from that integration.

Here are the products selected by NOPERU to this point:

jellema-case-for-FMW-fig08
Figure 8: Technology Selections

The choice of the portal technology remains to be made; for the time being, existing .Net and Sharepoint teams continue development work with their proven technology, connecting to the services offered from the Business Layer. Because of the standardization and interoperability of web services, it turns out to be simple to combine Microsoft Portal technology with Fusion Middleware-based web services.

The decision regarding Identity and Access Management and Security proved difficult. The components under this label must provide:

Authentication - establishing beyond any doubt the identity of the entity accessing an external NOPERU application (user interface of web service)
  • Authorization - determining which roles are enabled in the current session, given the identity of the entity accessing the application end point
  • Resource protection - ensuring that secure resources are accessed only in sessions with required roles enabled
  • Identity administration - creation and management of digital identities, including passwords and/or secure tokens
  • Identity persistence from component to component (including single sign-on)
  • Special measures such as encryption/decryption, digital signatures, non-repudiation, etc.

Various vendors offer Identity Management (IdM) components, each of which can be integrated to work with Fusion Middleware. The selection process for the IdM components turns out to be challenging. Executing a complete proof of concepts that covers all IdM requirements is not easy. IdM expertise is not widespread and it is proving hard for NOPERU as well as for the vendors to make the required experts available. The going is a little slow.

The decision is made more complex by the licensing conditions regarding all potential users of NOPERU's systems: virtually the whole country could access the portal in order to apply for a permit. NOPERU is negotiating with various vendors of IdM products on how to handle that particular situation in licensing terms. NOPERU is not prepared to acquire a license for each potential user - but some vendors are demanding exactly that. Instead, it wants to pay for the maximum number of concurrent users, or perhaps the maximum number of distinct users, in a three-month period. Alternatively, NOPERU can decide to sign a deal based on CPU cores. No final selection has been made yet.

Fortunately, the Oracle Platform Security Services (OPSS) in the WebLogic Platform insulate applications running on the platform—such as ADF applications, SOA Suite Composite applications and Service Bus services—from the actual IdM technology. OPSS processes requests, negotiates with whatever IdM solution is selected to determine the identity and the roles for the session, and makes this information available to applications running in WebLogic.

jellema-case-for-FMW-fig09
Figure 9: Oracle Platform Security Services

Any development work being done right now will not be impacted by a later decision on the specific products for identity management and authentication. This gives NOPERU some time to find an IDM solution and a satisfactory deal for it.

After the selections and the associated contracts were approved by the board of directors, things moved forward; and after the infrastructure was prepared, the software was actually installed and staff members educated. The business objectives that had lived on paper for several years and the architecture design that decorated many a whiteboard were on the verge of becoming reality.

Software Development Process, People and Organization

NOPERU knew where it wanted to be - at the end of 2018. It was also quite clear that it would not go there in one big step. It was imperative that an approach be devised that would take NOPERU's people, systems and processes to the target situation in small, controlled steps, with continuously added business value, and while keeping the shop open. The approach would also have to deal with the ambitions, anxieties, prerogatives and capacities of staff members.

There was no way NOPERU was going to be able to adopt the many new technologies and the new architecture principles all by itself in a timely and low risk fashion. The organization hired a number of experienced consultants and instructed them carefully on two tasks: delivery of the required software artifacts and the transfer of knowledge, best practices and enthusiasm to NOPERU's employees. These external experts with proven track records were hired to instruct, educate, coach and support internal resources just as much as they were brought in to design and build new software and platform components.

The expected rate of change, the number of business objectives and the desired flexibility, as well as the need to reduce risk, costs and time to market, led NOPERU to reassess the way it organized software development. Before 2011, NOPERU had a strict separation between operations and development, and also between the creation of new functionality and the maintenance of existing features. Projects were started for the former and the line organization dealt with the latter. Another clear demarcation existed between business demand and IT delivery. Once the requirements were laid down, a proposal and budget estimate would be prepared by the IT department and an agreement would be reached on what, when and for how much. This agreement was actually laid down in a contract that was signed by both departments. As soon as the signatures were in, a project would be initiated and the team would be on its way. Involvement from the business was next to nothing, until the start of the acceptance test phase. New business insights could be introduced only in a very formal way. There was no early feedback from end users or business owners on designs or prototype implementations, nor could suggestions from designers or developers for slightly different yet far cheaper implementations be put before the business. NOPERU's was a strict waterfall approach - one that frequently ended in tears.

Around 2010, word of "agile" software development began to attract the attention of NOPERU management circles. The principles of agile development include close, daily cooperation between business people and developers that should result in customer satisfaction through the rapid delivery of useful software and the business's ability to feed changing requirements, even late in development. "Embrace change" is the motto for all involved in agile projects: Be prepared for changes, and be sure that when they come, you can handle them.

Working software, rather than stacks of paper and formal design, is the foundation of agile-based projects: it is delivered frequently (in weeks rather than months) and is the principal measure of progress.

According to the agile model, face-to-face conversation is the best form of communication and projects are built around motivated individuals, who should be trusted. Self-organizing teams should execute the projects. Continuous attention to technical excellence and good design is paramount, as is simplicity-the art of maximizing the amount of work not done.

In short, an agile approach—and Scrum is the framework for agile selected at NOPERU—brings business and the IT development team much closer together. The joint focus shifts from formal contracts and procedures to the creation of business value, and from lengthy design sessions and the creation of generic technical solutions to concrete decisions and pragmatic implementations. The teams no longer work on designing and implementing today what the customer six months ago thought she was going to be needing in eight months. Instead, it works today on what the customer said yesterday she needed to have created by the end of a two-week sprint. (Note: in Scrum, "sprints" are the mini-projects that teams go through every two, three or four weeks.) And instead of wasting many hours on creating a feature exactly according to specifications, the teams will confer with the business owner about smarter implementations that provide similar functionality in perhaps slightly different ways that are far easier to realize.

Even though pressure is not incredibly high, the team must constantly focus on the current sprint with clear, tangible, short-term objectives. And because the focus is on demonstrating complete user stories with end-to-end functionality, rather than on individual tasks that by themselves do not contribute any business value, team members tend to work closely together. Only when the story is completely done, from design to implementation and including test and code reviews, will the team have something to show for the sprint. When this is the definition of "done," the team tends to really collaborate in order to finish the story, bringing various disciplines much closer together.

A NOPERU taskforce attended seminars, visited with early adopters of an agile approach and eventually started a small pilot project. Despite initial cynicism, this pilot was hugely successful. The final findings presented to the board of directors were very positive and the taskforce's suggestion to introduce agile development through the adoption of Scrum was approved by the board. In early 2011, the very next project for new business functionality was organized in an agile manner. Soon after, other projects followed suit.

Agile at NOPERU

NOPERU created Scrum teams of up to nine people. Each team had the skills to create end-to-end slices across all layers and technologies in the architecture and could develop user interfaces, services and database components; each included an analyst and tester. Each team was essentially self-governed: a Scrum master organized sessions and arranged equipment and resources, but did not act as a team lead. Administrators did not join a team, but closely collaborated with the teams; we will discuss this collaboration a little further on under the title DevOps

.

The business owners of all concurrent projects together created a backlog of "epics" - high- level areas of new or changed functionality. Very rough estimates were set on each epic in terms of total duration, business priority (based on value and urgency driven by obligations and regulations) and dependencies. The epics were subsequently turned into user stories that were thought to fit in a single sprint. Based on the ordering of the epics on the overall backlog and the user stories defined for the epics, the sprint backlog was composed. Next, the stories on the sprint backlog were refined and assigned to the Scrum teams.

NOPERU decided to work in sprints of two weeks, about the shortest you can go. This means that every two weeks the teams execute a mini-project that starts with estimating the user stories, defining all tasks and deciding on the "commitment" – the set of user stories from the back log that the team is committing to in the sprint. For each of these stories, the team promises to complete all tasks and the story as a whole - according to the definition of "done."

Sprint planning is captured on the Scrum board. At NOPERU it is a wall-sized piece of paper on which a matrix is created, with columns labeled To Do, In Progress, Review and Done. The rows represent the user stories that are part of the sprint. Each identified task is turned into a post-it note. All post-its start in the To Do column and should have reached the Done column by the end of the sprint.
jellema-case-for-FMW-fig10
Figure 10: Scrum Board

The post-it note describes in minimal detail what a task entails. It also indicates the estimated hours needed to complete the task. Once work has started on the task, its post-it should be moved to the next column with the name of the team member working on the task indicated on the note.

The shared focus of the team is the completion of all user stories on the board, starting with the top one and gradually working the way down. Ideally, every team member is capable of performing every task on the board, thus allowing people to pick the highest ranking tasks (top left on the board). In practice, skills and expertise are not distributed equally across the team and not everyone on the team is up to each of the tasks. However, the common objective of completing user stories (instead of a focus on the completion of tasks) means that team members frequently take on tasks that are not necessarily within their core competence - but that do have the highest priority for the team. Collaboration, pair-programming, frequent communication and knowledge sharing is very much part of successful Scrumming. The daily stand-up—a 15-minute meeting to realign the team, share experiences and impediments, monitor everyone's progress and evaluate the team's progress with regard to the sprint goals—is an important way of keeping the team on track.

The sprint concludes with the sprint demo, the goal towards which the team has been working. In this session, the team presents the sprint deliverables to the product owner. Ideally this means a demonstration of software components, and can also involve outcomes from a design activity or an R&D task, or decisions as a result of discussions.

The business must take on substantial responsibility in order to gain agility and continuous delivery. Rather than stating fairly vague requirements and sending them down the waterfall or specifying the functionality in great theoretical detail before abandoning them in the hands of the development team, the business product owner is constantly involved. She balances budget, time and functionality in collaboration with the team; continuously prioritizes user stories; and uses the outcomes of recent sprints to determine how to proceed in those to come. No detailed requirements are drafted in advance. Business value, as embodied by the product owner, is the constant driving force. Seen from a slightly different angle: the product owner has the opportunity to monitor the work of the development team at any time in terms of clear, concrete deliverables and to adjust the course almost daily. No big gap can arise between the business desires and the development team's interpretation. Immediate feedback from the team to the product owner can help identify better solutions and quicker implementations. These short feedback cycles also prove strongly motivating to the development teams - much more so than a faraway deadline, vague business objectives and detailed yet sometimes apparently silly, inconsistent or incomprehensible requirements of the recent past.

For the Scrum approach to work well, the product owner must be available, engaged and authorized to make decisions about priorities and functional design choices. This role can make or break the software development endeavor.

Agile and Architecture

The architects at NOPERU have somewhat mixed feelings about the agile approach. They can see the value in having business and IT staff working closely together, and the wasted opportunities of the past in terms of delivering software no longer in line with business needs at the time of release are fresh in their minds. However, one of the advantages of the classic waterfall approach is that there is plenty of time of time to create an architecture plan for whatever the project is going to do, as well as ample room for quality assurance from an architecture perspective on software design and implementation. As long as the agreed-upon deliverables were created, the project team could also ensure it stayed within the architectural boundaries.

In the agile way of doing things, there is so much focus on business value (short-term functional gains) that there is a serious risk of glossing over mid- and long-term aspects, such as maintainability, consistency, technology upgradability and non-functional components (e.g., security, stability and performance). It is easy for a team to think: "If it works for the demo to the product owner, then that is good enough." The architects feared that in the push to do the right things, the concept of doing things the right way would suffer.

And so they did a number of things to prevent this. First they translated the higher level architecture designs and conventions into tangible guidelines for designers and developers. Then they worked with all Scrum teams to ensure that following these guidelines became part of the definition of "Done" - meaning that deliverables could be considered done only when verification against the guidelines was successful. Furthermore, the architects were instrumental in organizing the "guilds" - virtual teams around specific roles and technology with members from all Scrum teams, as explained below. The guilds are an important arena for discussing the right way of doing things and creating mutual understanding. The guild members also do periodic cross-team reviews on artifacts created by their peers.

The notion of technical debt was made very explicit. Technical debt arises when, for the sake of short-term functionality (business value), sacrifices are made in terms of technical quality and adherence to architectural guidelines. This debt should clearly be identified and recorded. It is the team's responsibility to deal with this debt as quickly as possible. The product owners and the teams were given instructions on making sure that reducing such debt would be incorporated in the product back log. Architects are closely involved in monitoring the technical debt in each team. They can intervene when the debt grows too high and mediate between product owner and team to ensure the debt is quickly resolved.

NOPERU also took a big step by stating that each team would be responsible not only for creating new artifacts to support new projects but also for maintaining any software currently in production. Any shortcut taken by the team today would also be their responsibility in the future in terms of dealing with problems and maintaining the code. Any unresolved technical debt would continue to haunt the team itself.

Tools

The short release cycles and frequent, near continuous delivery can be realized only with automated build and deployment procedures that, ideally, include testing. These procedures for building and rolling out software deliverables should be coordinated and monitored.

jellema-case-for-FMW-fig11
Figure 11: Agile vs. Waterfall

Something similar applies to the environments in which the artifacts are deployed - the combination of platform and frameworks that make up the infrastructure in which the developed software runs. The environment consists both of standard software from 3rd party vendors and the NOPERU-specific configuration of this software. Controlling the creation, rollout and updates of these environments requires automated execution and control in order to achieve an agile and managed delivery process.

NOPERU selected a wide assortment of tools, most of them from open source projects, to aid with software engineering. Among these are tools that help with a variety of tasks, most in the area of DevOps.

For orchestrating and monitoring the overall build process, NOPERU selected Hudson and stuck to it when the Jenkins fork occurred. The build actions themselves are largely done using Maven, in conjunction with which Artifactory is used as a repository for code artifacts. For controlled deployment, a tool called DeployIt has been acquired (this product is not open source). Rolling out entire environments has been tackled with Puppet.

For various types of testing activities, the following tools were selected:
  • Web Services: SoapUI (functional), LoadUI (load and stress)
  • Java and ADF Business Components: jUnit (function and load)
  • li style="padding-bottom:10px;">Web applications: JMeter (functional and primarily load); Selenium (functional); Oracle Application Testing Suite (under evaluation for both functional and load testing)

In addition, some of the built-in testing features of the SOA Suite are used. Some attempts have been made to create unit tests for web components and composite services through the use of mock objects. EasyMock (for Java) and SoapUI with Jetty (for mock web services) were used for this.

Management of tickets (issues, requirements, enhancement requests) is done using JIRA. This tool is also used for coordinating the Backlog (Product and Team Sprint) and the Scrum-board.

Source code control is done using Subversion; although some investigation has been done into migrating to Git.

A number of tools are used for code quality control, mostly for Java code and not yet taking Service Bus, SOA Suite and BPM artifacts into account: Sonar (for integrating the findings from various other QA tools ), Checkstyle (for verifying adherence to coding conventions), PMD (spotting bad practices) and FindBugs (for finding potential bugs).

MediaWiki has been set up to allow teams and guilds to collaborate and share; it's used for checklists, how-to documents, configuration instructions, the catalog of services, records of design considerations and motivated decisions. Also on the Wiki: architecture guidelines, coding conventions and descriptions of DTAP environments, along with URLs for the consoles and associated tools for Oracle Service Bus, Oracle SOA Suite, Oracle WebCenter Content, and Oracle WebLogic.

Collaboration between NOPERU employees, who work in many different locations, is essential. Employees use Microsoft Lync for Instant Messaging (IM) as well as Skype to facilitate this collaboration.

Organization and Roles

The distinction between projects doing initial development and the maintenance teams doing corrective and adaptive management should be removed. With very short release cycles, the need for dedicated teams with special focus on maintenance working against a higher frequency release cycle than the projects teams largely disappeared. It seemed a waste to spread expertise in technology and application thinly across development and maintenance. And it is considered a good idea that developers and teams take responsibility for evolving - which includes correcting - their own deliverables.

At the same time, the notion of project teams was dropped. Development teams would be created to focus on certain areas of functionality, including development skills on a subset of all technology in use at NOPERU. These teams could be enlisted for user stories (features) defined by several business projects. They also were mandated to spend 20-30% of their time on corrective and adaptive maintenance. Up to 10% of the time was free to use on "small effort, big gain" activities: business requests that are not really part of a formal user story or project back log but that can add substantial business value with very little risk and effort. Given the short release cycles, this flexibility easily creates happy faces among end users and developers.

Most IT staff at NOPERU participate in a Scrum team. That is where their daily work and direct responsibility is. As mentioned above, NOPERU has also founded guilds - virtual teams across all scrum teams around common expertise. There are guilds, for example, for ADF, testing, the Middleware Platform, WebCenter Content, service-oriented design, BPM and for Oracle Service Bus and SOA Suite. These teams organize meetings, maintain a common wiki with standards and guidelines, naming conventions, samples, configuration instructions for tools, how to's, and links to relevant internet resources. They monitor evolution of their focus technology from the current vendor as well as in the industry. The guild master also acts as coach, helping junior members build their knowledge. Members of the guilds are recruited from all Scrum teams: anyone working with a particular technology or in a specific role participates in the corresponding guild. In addition to sharing information and knowledge, the guilds play an important role in retaining consistency across the Scrum teams in the way they use the technology.

Consequences per Role

While NOPERU's new software development approach impacts almost everyone involved, it has particular consequences for testers and administrators.

Testers, for example, used to have their stage in the waterfall approach; after plenty of preparation, based on detailed design documents, they would torture the software artifacts in order to verify fitness. Testing was typically done in isolation; contact with the development team occurred only via the issue tracking system. Testing focused almost exclusively on the user interface, by and large ignoring the internal components. Testing 2.0 in a Scrum way of working is very different. Testers are embedded in the Scrum teams, and testing is done as part of every sprint. Design documents are high-level, if at all available. Testing is geared towards user expectations and interface designs and involves a large chunk of non-user interface services, such as web services and database APIs. The use of tooling for automated (regression) testing is rapidly increasing. Communication and collaboration between testers and other team members (such as analysts and developers) grows from next to nothing to pretty intense. It is not uncommon for testers to engage in other activities in the early stages of a sprint, and for analysts and developers to pick up testing tasks near the end of a sprint. Feedback on software quality arrives more quickly than in the old way of working. Yet it's important to note that testing may not be as rigorous as it used to be, although the gradual build-up of automatic test-sets eventually brings testing to a fairly high level.

S

imilarly drastic are the changes for the administrators. For many years, administrators either did hardware and systems administration or database administration, with their activities decoupled from development projects and geared only toward production systems. The introduction of virtualization means that new skills were required and new options became available for quick deployment of new environments, for example, or easy rollback of an environment to a previous snapshot. Planning machines no longer meant the same thing as planning the hardware.

The next major wave of innovation consisted of introducing middleware: the application server as the core element and then several engines running on the application server, each performing different functions that required administrative effort in terms of installation, configuration and monitoring. Expertise on WebLogic Server as well as administration for the Service Bus, the SOA Suite, WebCenter Content, and BPM Suite had to be built up.

Platform administration and infrastructure administration were identified as two individual and complementary disciplines, and some steps are being taken to organize and offer the platform and infrastructure as private cloud services. Additionally, investigations have started into using external cloud facilities for such purposes such as load and stress testing.

DevOps

The lines between development and operations are getting a little bit blurred at NOPERU. One reason is the agile development approach, whose frequent delivery from development to test, acceptance test and (at least in theory) even production, requires very close collaboration between developers and administrators. Another reason is the importance of the platform - the combination of database and application server with middleware containers - in modern architectures. Developers and platform specialists must meet with regard to the application server, the service and SOA Suite, discussing caching strategies, fault handling, security requirements and other non-functional aspects that require functional knowledge to configure.

Because the typical stages that development teams go through with the software they develop are the same stages that the platform and underlying infrastructure should go through--design, build and test--it makes sense that platform specialists be engaged early on. Typically, the administrators in the infrastructure departments who take care of daily operations are the experts on installing, configuring and managing the platform - regardless of whether the platform is used at run time (production environment) or during the development and testing phases.

Figure 12 shows the layers of NOPERU's stack: Application, Platform and Infrastructure. Each of these categories will go through the preparation stages - design, build and test - and will end in the operational stage where monitoring is performed and modifications (configuration, patching, rollout of new software components, restarting of components, etc.) are made.

jellema-case-for-FMW-fig12
Figure 12: Layers in the stack

The graphic above visualizes how the team doing application development should work closely together with the team doing platform "development." At the same time, when issues arise with the application in the production environment, the corrective actions should probably be taken by the team that did the original application development, not by a separate maintenance team.

NOPERU is still working hard at closing the gap—which endures for historical reasons—between administrators of the database and middleware platform on the one hand and the developers and architects on the other. A strong motivation for really starting to collaborate arises from their mutual dependency: no developer can call herself successful if the software components she has developed do not actually make it to the production environment, and no administrator can claim success if the platform is not really offering functional business value to the end users. The rapid evolution of software in the agile approach adopted by NOPERU makes this very clear to all parties involved.

Figure 13 shows the technologies and tools that both developers and administrators have to deal with or can use to perform their jobs.

jellema-case-for-FMW-fig13
Figure 13: Technologies and Tools

Roadmap and Projects

Everyone at NOPERU seems to agree on the vision for 2018. The destination the organization is heading for is clear--but the exact roadmap is less so. In what order should the envisioned changes occur? How much can be chewed off? And when? Most importantly, what are the business priorities?

Because business goals, IT priorities (centralization and consolidation, working according to the new architecture and applying the new technology) and the desires of political decision makers do not coincide, some compromises will have to be made in the first few projects executed under the Go Forward program. Step by step, that ideal will be realized, and a pragmatic approach is required in the meantime from all IT staff involved.

B2B for the Corporate Sector

Some of NOPERU's largest corporate clients, those applying for hundreds of permits per year across their many lines of business, have requested an electronic gateway for their interactions. These corporations had been integrating with their business partners and execute B2B with suppliers, vendors, the tax office and other governmental agencies, and considered their inefficient paper-based dealings with NOPERU as a relic of the past.

NOPERU's stakeholders agreed that an automated interface was necessary, and this business requirement was seized upon as a great opportunity to start realizing part of the IT roadmap. The B2B gateway fit right in with the layered architecture and service-oriented approach. This was the first stepping stone for architects and developers alike. The Oracle Service Bus and the SOA Suite were brought in to develop the required web service and expose them in an appropriate manner with regard to security, availability, throughput and response time. A central database was set up but, because the regional instances were not yet going away, a replication mechanism was established between the central database and these regional satellites.The initial message volume was fairly small and the first six months after the initial launch turned into almost a prolonged pilot period with just two corporate clients. Later, the number of participating corporations quickly increased, as did the message volume.

This first Go Forward initiative was quite successful and relatively painless. The fact that it did not replace existing systems but only complemented them was probably part of the success. The very careful testing period and the close participation of the architecture team, as well as the involvement of several quite experienced external SOA consultants, also contributed significantly.

A major challenge was presented by the 24/7 availability demands on the B2B gateway. Until the advent of this portal, the internal systems at NOPERU needed to be available only during office hours, which left plenty of time for batch jobs, system maintenance and software upgrades. Even unplanned downtime stayed within the walls of NOPERU. But the portal heralds an era of continuous availability; at this stage it applies only to the central database and the middleware platform but, before too long, other systems will be subjected to the same regime.

Digital Documents

Each of the five NOPERU locations has a basement filled with decades worth of paper archives--and the flow continues all the time. And before the paper trail associated with a permit application reaches the archives, it winds its way through the offices - transferred from desk to desk and sometimes getting lost. One of NOPERU's largest challenges as well as opportunities for serious money-saving lies in the transition from paper to digital documents. Apart from freeing up the basement, digital documents can be transferred more easily, shared in parallel, tracked at any time, published to external parties and be interpreted and processed automatically. Permit applicants can make their associated documents available to NOPERU in an electronic format - drastically reducing the workload for NOPERU itself.

The business initiative for Digital Documents was started in 2011. It targeted a single central content server that would be exposed through an internal web application used from regional offices. Content can be uploaded through the web interface as well as through web services; any document can be retrieved through the same channels. Adding mobile support is high on the wish list, but has not yet been implemented. Along with the IT side of things - quite a challenge in its own right - this project also had to bring the organization up to speed with this paperless way of working and all the new tasks and procedures that it entailed. Given that the execution of the core business process - handling a paper-based application for a permit - was largely driven by the paper file weaving its way through the office, another way of organizing the process was needed in the transition period before Business Process Management (BPM) was launched.

New UI for Internal Staff

Although it wasn't high on the political agenda, NOPERU needed to do something about its internal applications. For example, the Citizens sector was still working with the Forms 4.5 character-based application that resulted from the Forms 3.0 application originally inaugurated in 1993. Even though this application still allowed the sector to handle permit applications, it did so in its own special way. Long-time users who knew all shortcut keys by heart, were intimately familiar with the underlying business process, and basically had worked with the application for many years were even quite productive. However, the learning curve for new users was so spectacularly long that getting temp workers to help deal with peak workloads was impossible. And finding new employees willing to enter a job with such an outdated computer system was becoming harder as well. Maintaining the system was costly and cumbersome due to the undocumented code and the five-location upgrade process. For the moment, plenty of Forms developers were still around; however, it was uncertain for how long that would last. Finally, the application was heavily data-oriented ("CRUD style") and did not embody the business process in any way. Moving towards a user interface that was more focused on task and process and less on underlying data had been a long term desire at NOPERU.

In mid-2011 a project was launched to rebuild the Permit Application for the Citizens sector. In the longer run, this newly developed application would be the platform for all three sectors. Initially - with a scheduled launch in late 2012 - only the staff in the Citizen sector would use the application. The application was to have a single instance, centrally deployed and managed and accessed from browsers. It was also to have a single, central database into which all data from the five regions was to be consolidated.

The application was developed using ADF 11g against an Oracle Database 11gR2 database. The team consisted of several experienced ADF consultants brought in from outside, along with five NOPERU developers. The latter had a lot of Oracle Forms and Database development experience and had received two weeks of ADF introduction training prior to commencing on the project. Over the course of the project, every NOPERU staff member had a personal coach assigned to ensure knowledge transfer took place, and the internal developers built up their ADF development skills.

The Permit Application (PA) system was rolled out in the first half of 2013. By May, all five locations were live on PA and the local Forms instances were switched off. Some performance wrinkles had to be ironed out, but in early July, this first application was supporting several hundred concurrent users.

Upgrade to Forms 11g

The Forms applications used for the core permit handling process by the Corporate and Government sectors have been upgraded to Forms 11g. These applications also run on WebLogic Server 11g, the same core platform used for all other middleware components. In addition, some benefits are reaped from Forms 11g functionality, including support for Advanced Queuing and the use of Database Proxy Users. This upgrade from Forms 4.5 to 11g took a relatively short time. The result is not especially pleasing to the eye compared with modern Forms 11g applications, but it will do for this transitional period. (Note: for now, the regions will have local instances of the Forms 11g application running against their local database.)

As a next step, a consolidation can be made into a central database that will then use Virtual Private Database policies to logically partition the data per region. The local Forms 11g instances can then be replaced with a central Forms 11g instance running against the consolidated database.

Portal for Corporate Clients

The corporations are served through a portal - a user interface in addition to the programmatic interface provided through the B2B gateway. Through this user interface, corporations are able to submit and track permit applications and communicate with NOPERU staff.

The portal was developed using Microsoft SharePoint against presentation services that expose functionality from composite and elementary services in a SharePoint developer-friendly way. The corporate portal reuses many of the services created when the B2B gateway was created. The central database, which has mutual replication set up with the regional databases, is also leveraged for this portal.

T

he decoupled approach in design and development of software supporting new business requirements has helped tremendously with the creation of the portal. The portal designers and SharePoint developers indicated their service requirements based on the UI designs. Subsequently a mapping was designed to existing and new composite and elementary services. The interfaces of all services and operations were designed by the designated teams and when clarity and agreement were achieved, both teams went their merry ways to implement the components.

Mobile App for Internal Managers

At this moment, another initiative is under way: NOPERU managers responsible for the bulk processing of permit applications have requested a means to get better and more up-to-date insight in the operations of their teams. Meeting the formal deadlines for each step in the permit procedure is important; in fact, there could be legal consequences. Speeding up the process and, at the very least, quickly identifying stuck processes is becoming more important.

The expectation is that when BPM has been introduced, it will render a lot of this operational insight more or less out of the box. For now, the database developers will create a number of views that expose status information about permit applications. These views in turn are exposed in the form of services that are published as RESTful, JSON-based services by a presentation service.

The mobile app—native iOS for the iPad—is developed by an external company. This company has been given the definition of the RESTful services and the JSON payloads and is now working off-site in a highly decoupled way.

Future

In the near future, NOPERU plans to initiate several new projects:

Self-Service Web Site For Citizens with 24/7 Availability

A lot of aggravation for citizens would be removed if they could check on the progress of their permit applications themselves, rather than having to contact NOPERU during office hours. And so, just like the Corporate sector has opened a portal, so will the Citizens sector. Any citizen will be able to log in to this portal using the appropriate national ID. . New permits can be requested in the portal and the status of the running requests can be checked. NOPERU can ask citizens to provide additional information or upload additional documents, and citizens can also ask questions of NOPERU.

C

itizens can provide through the portal all the information associated with a permit request that is today sent in on paper that must be scanned and entered manually. This will obviously be another substantial reduction in workload for NOPERU staff. At the same time, such self-service functionality will be perceived by users as a great boon that allows them to start a permit application in their own time and from the comfort of whatever location they fancy.

It is still to be decided which technology will be used for creating the Citizen portal. It will reuse the services already created for the Corporate portal.

Business Process Approach through BPM

As stated, NOPERU wants to adopt Business Process Management. It wants to get a better grip on the business processes, optimize and streamline them, improve implementation and automate more aspects. When the IT systems are process-driven, NOPERU expects to be able to quickly change process flow and bring new staff quickly up to speed, because the systems will guide the employees through the process. This process approach is expected to provide operational insight to management and alerts when thresholds and deadlines are in danger of being surpassed.

Standard Application for CRM

No one knows when (or if) it will happen, but the option of replacing the custom-built applications for managing client details with a commercial off-the-shelf CRM application such as Oracle Siebel is always just around the corner. The architects give strict instructions about keeping the CRM domain completely separate and encapsulated - because one day it may have to be replaced. When that happens, all services exposed from the CRM domain will then have to use the CRM application as their implementation. Any consumer of a CRM service should not have to feel any impact from such a replacement, provided that all CRM service interfaces can be implemented using the APIs in the CRM application.

Conclusions

NOPERU has traveled part of the road mapped in Go Forward (2010-2018). The first few of many projects for transitioning existing systems or introducing new business functionality through IT have been executed or are in progress. New technology has been introduced, as has a fundamentally changed way of approaching software development. While the road is bumpy, successes are far more abundant than failures. Reluctance was pretty widespread when the first steps with service orientation and layered architecture were taken, but now the mood has changed. Many IT staff members are convinced of the approach in general and their ability to play a part in it. Enthusiasm is growing along with self-confidence.

The way NOPERU is perceived by its own staff, its counterparts within the government, partners and clients has improved substantially over the past few years. The modernization of applications appeals to all involved and an increase in quality of service and information has been reported across the board. A recent survey finds shorter waiting times, fewer mistakes and a better experience. An internal evaluation yielded similar outcomes, with a higher satisfaction and a much higher score for the standard survey question "Do you intend to work at NOPERU one year from now?"

A

mong the very first steps in the Go Forward program has been the consolidation of data into a single database and the centralization of all infrastructure in a single data center (which has two physical sites for fail-over reasons). All administration activities have been centralized as well. This has turned out to be much more efficient: fewer administrators are needed and those who are needed can specialize. Instead of having to manage small pieces of everything—from network and storage all the way up to operating system and database—they now have responsibility for a much larger piece of one special area. External help is needed much less frequently because of this specialization. That in turn means costs are down and times to resolution are shorter.

Data consolidation is proceeding in several steps. First, the data was stored in a single database, logically partitioned by region and by sector. Gradually, data is becoming truly shared, reducing duplication and resulting inconsistencies. This is not just a technical process; it also requires putting pressure on business representatives who are somewhat wary of sharing their data.

The transition from Oracle Classic (Forms, CRUD-style Data, Database-oriented, SQL and PL/SQL, Discoverer, Designer) and a waterfall approach to the new world of agile and service-orientation (along with Java and web user interfaces, Mobile, and BPM) is overwhelming for NOPERU's staff. Guidance, reassurance, explanations, and almost spiritual support are absolutely necessary to motivate and enable staff in almost every role. For a long period, constant attention is required to engage people who are suspicious about what is happening. Communication is essential if NOPERU's rapid evolution is to succeed.

All roles—from business owner to administrator—need to be involved with the initial transition and with every new initiative involving software development.

"Embrace change" is the mantra of the agile approach to software development. But that remains a challenge for many of the parties involved. Even though it is acknowledged rationally that change is part of the game, changes in direction and planning are still frequently experienced as disruptive. Involving the members of the Scrum teams early on whenever such changes occur is important in order to preserve their engagement and motivation.

The Scrum methodology has quickly taken hold at NOPERU. The initial skepticism was quickly overcome for most and some of the early skeptics are now true (sometimes fanatical) Scrum believers. The commitment of team members to the team, and of teams to business objectives, has increased dramatically. The constant focus on near deadlines and the rapid feedback from the business representative turn out to be very motivating for the teams, and the quick feedback from the teams to the business also helps the latter to adjust requirements. IT staff are usually pretty smart, if somewhat overly-focused, and their input is used to improve plans and designs.

Close involvement and support for agile development from the business owner is essential to the success of the effort. That increased level of support on the part of the business owner is indicative of the impact to all roles involved in the transistion to this new way of working. Administrators and testers in particular had to adapt to radically new ways of thinking. For all its success with Scrum, NOPERU still struggles with the tension between the short-term focus of the sprints and the longer-term architecture objectives. "Technical debt" (as discussed above) should be settled before too long - and all parties agree on that in principle; however, it proves a constant struggle to actually claim part of the team's capacity from functional business value to longer-term software quality and maintainability.

With all the focus on middleware, it is useful to remember that the database and SQL & PL/SQL skills are still incredibly important to the overall success of the NOPERU application architecture. Performance, scalability, integrity can benefit from good (or suffer from lousy) usage of database capabilities.

It has proven extremely helpful for NOPERU to bring in experienced consultants: it helps prevent costly (and time consuming) mistakes that would also cause frustration and undermine confidence in what is already a fragile environment, and also helps NOPERU select and apply proven best practices. Experienced outsiders also take care of knowledge transfer in a hands-on situation, not just in a classroom.

Initially, outsiders were brought in to show how things should be done, then to collaborate with internal staff on equal footing, and finally to act as part-time coaches and quality assurers when the internal staff take over for real. The long-term objective is to build up the capabilities of the internal staff; in the short term, experienced outsiders can help boost productivity.

Service-oriented Architecture

Even though the service orientation yields substantial returns in the long run—through reuse and agile development on top of existing services—initial investments are required to put the architecture and the infrastructure in place. Some of the cost precede the gain. Embracing the layered architecture and doing this SOA thing is a strategic decision that requires some steadfastness.

The implementation of a service is not relevant to its consumers. This means among other things that legacy applications could very well be embedded in the new architecture of Go Forward if they can be wrapped inside standards-compliant interfaces that are mediated from by adapters on top of those legacy applications. Encapsulation makes it possible to then replace the legacy implementation of the service—along with its adapter—with a custom implementation using the technology of choice or perhaps with a COTS product.

The decoupled architecture approach also allows for specialized teams working on isolated, clearly identified areas within the system landscape; it also means that not every developer needs to acquire skills for all tools and technologies involved (at once).

The layered architecture and the decoupled software design make it possible to outsource chunks of works to external partners. For example, once all service interfaces have been agreed upon, another company can build a website or mobile application on top of these interfaces. Rapid development and flexible software engineering capacity – especially when a cloud-based development environment is used - are at the fingertips of the IT manager.

Governance—the approach to managing services and related assets (such as the canonical model) through their lifecycle—is critical, especially in the mid to long range. It also includes identifying new services and changes to existing ones, and cataloguing services to make them findable and ultimately reusable. QA on design and implementation to ensure consistency and adherence to the standards specified as part of the governance initiative is another aspect to organize.

Looking Forward

In the near future, NOPERU will continue executing the Go Forward program. BPM is a major new focus; not only does it ensure proper execution of procedures and provide management information about current affairs, it also helps optimize and partially automate the processes. It is hoped that new staff will require less training when they are guided through the processes. And some of the actions will be turned into self-service activities to be executed by external parties that are actors in the orchestrated processes.

Attracting new IT staff and building up internal expertise to reduce the dependency on external consultants is very important to NOPERU, which is actively approaching local specialists, trying to hire them away from their current employers. For its current staff, NOPERU is organizing development programs that will help them build up skills around the newly introduced technologies and methodologies. Staff can attend international conferences and workshops in order to meet with peers and learn the latest information. Renowned speakers are invited to conduct sessions at NOPERU itself - for internal staff as well as invitees that NOPERU hopes to hire. The opportunity to play a key role in this major technology evolution has already attracted a number of new Oracle Fusion Middleware specialists to the ranks of NOPERU's IT department.

In the slightly longer term, NOPERU hopes to do more with the data that it is sitting on, which may be interesting for external parties. NOPERU also hopes to learn more itself about peak volumes and trends in order to streamline its processes and prepare its staff.

The development of mobile apps has just started. Opening up multiple channels in earnest is among the near-future plans of NOPERU. It expects to leverage the layered architecture and the reusable services to be able to quickly roll out new ways to interact.

Finally, major developments within the national government will also impact NOPERU. All citizens and corporations will have a national digital identity that is to be used to interact with any public agency, including NOPERU. Basic information about these parties will be managed centrally in a designated national organization. Agencies will no longer be allowed to record such information in their own systems but must leverage that centrally held data. Another central agency will take care of collecting money from any party having financial dealings with any government organization. This means NOPERU will have to interact with this new agency. Note that these initiatives are very similar to what happens in service-oriented architecture: individual business functions are isolated, wrapped in well-defined interfaces that are appropriate for reuse, and have encapsulated implementations.

Final Note

Your organization is unlikely to be the spitting image of NOPERU. Yet at the same time chances are that it has substantial overlap with the starting situation, the business objectives and/or the technological transformations NOPERU is going through. The approach and steps as well as the conclusions in this article likely apply to your situation, too.

About the Author

Lucas Jellema has been active in IT (and with Oracle) since 1994. An Oracle ACE Director specializing in Oracle Fusion Middleware, Lucas is a consultant, trainer, and instructor in diverse areas including Oracle Database (SQL & PLSQL), Service Oriented Architecture, ADF, and Java. He is the author of the Oracle SOA Suite 11g Handbook, and a frequent presenter at JavaOne, Oracle Open World, ODTUG Kaleidoscope, Devoxx, OBUG and other conferences.
Connect with Lucas Jellema Oracle ACE Director