Leveraging opportunities for asynchronous processing to improve system response and scalability.
Oracle SOA Suite
Picture the following scene at your favorite restaurant. The dinner party is seated at one of the tables and the people sitting around the table are conducting a lively conversation with sparkling interaction. Then, conversation at the table dies when the waiter is taking an order. Even when the waiter has written down all requests and takes his leave to go and fulfill the request, everyone continues to be quiet until the waiter returns with the orders. There's no interaction at all while the wait goes on.
The waiter meanwhile goes to the kitchen to deliver the order-and he keeps hanging around, waiting for the order to be prepared, instead of waiting on another dinner party. Only when the waiter has delivered the order from the kitchen to the guests at the table is he released to start waiting on another table. And only when the order arrives at the table will conversation among the guests resume.
That would be a somewhat strange restaurant, soon to be no longer your favorite. From an economical perspective, it is also horrible: tying waiters up with one table until they have served the request. I do not know of any but perhaps the fanciest restaurants that work like this.
Yet in many parts of today's application architectures, this highly synchronous model is very much in use for all questions, even those that are really more one-way messages than two-way questions. It applies, for example, to most HTTP requests from browser to server, to Java web tier to Java business tier and, of course, to Java Database Connectivity (JDBC) requests from the middle tier (Figure 2).
As a consequence of the synchronous approach, many threads and associated resources are frozen while some other tier is processing a message. This approach of "ask a question then hold your breath until the response comes in" is neither smart nor necessary. Not smart because of the resource waste and reduced responsiveness of the application, and not necessary because of some recent technological developments around off-loading workloads, asynchronous processing requests and push-style communication. This article describes how, by leveraging opportunities for asynchronous processing, at least the perceived responsiveness of our systems is likely to improve. Frequently, the actual scalability is also enhanced by making much better use of the parallel processing power of today's CPUs and virtual machines and through throttling peak loads by deferring some of the work.
In real life, when you tell something to a person, you don't mind waiting for answer (beyond some form of acknowledgement that your message was heard and understood) if and when:
However, if these considerations do not apply, you want to continue with your life as soon as possible after saying your piece or asking your question. In the world of multi-tier applications, very similar considerations apply. Replace "you" in these bullets with "the browser" (talking to the middle tier) or the middle tier (talking to the database), or an application component (talking to a web service), and you have a checklist that will help you decide when interactions must be synchronous and when perhaps they could be processed asynchronously.
In the case of fire-and-forget interaction (a one-way communication) in which the party that started the conversation is not interested in an answer-and don't we all have family members that fit that bill-the request is made synchronously and, ideally, the resulting work is taken off-line or handed to another thread as soon as possible. It is therefore useful to take a look at the mechanisms we have at our disposal for handing off jobs in each of the tiers our Fusion Middleware applications typically make use of.
XMLHTTPRequest object, the call is asynchronous (AJAX). This means that the thread that issued the call can continue processing while the HTTP response has not yet been received. When the response arrives at the browser, the requesting thread is interrupted and the designated call back function is invoked by the browser. When no call back function is specified, an arriving response goes ignored. We have thus created a one-way conversation, not by eliminating the answer (which would have been the more elegant approach but is not in line with how HTTP works) but because we did not listen to the answer.
We assume a Java middle tier; after all, this column is on Fusion Middleware. Requests to the Java middle tier will usually arrive over HTTP, although of course Remote Method Invocation (RMI) and more indirect and inherently asynchronous channels (such as email or file/FTP) are also an option.
Figure 4 illustrates many mechanisms available on the Java, and especially the Java EE, platform for offloading work from one thread to another, thereby releasing the initial thread to immediately return a response to the invoking party. That gives the impression of speed: very fast processing seems to have taken place or at least the wait time for the consumer minimized.
All work performed by a thread is done synchronously; the initiator of the call waits for the thread to complete. For example, in Figure 2 the HTTP client caused a Java thread to start executing a servlet to handle the request. Until this thread completes the HTTP response, the thread is locked and the HTTP client is waiting. Completion of the thread can be accelerated in several ways:
Executor: Using the concurrency facilities in Java SE (and, starting with Java EE 7, also in Java Enterprise Edition), a task can be handed to an executor that runs independently of the thread that executed it. Various executor implementations may make use of the multiple cores available to the JVM and can be Java EE container-managed.
(Quartz) Scheduler: A scheduler can be used to take on responsibility for executing a task; the task execution is done in the background, on a different thread from the requesting one. Several scheduler libraries are available, of which Quartz is probably the best known.
EJB Asynchronous Methods: Introduced in EJB 3.1 (Java EE 6), Enterprise JavaBeans can have methods that process invocations asynchronously. This means, in practice, that when the method is invoked, the container steps in and invokes the method in a different thread based on use of the @Asynchronous annotation. The interaction is fire-and-forget as far as the calling thread is concerned.
Java Batch: Very recently, Java EE 7 introduced the Java Batch API, which the latest generation of Java EE containers should implement. Through this facility, a standardized framework to schedule and run predefined jobs is guaranteed to be available in every Java EE 7 container.
One-Way Web Service Call: When the work a thread has to perform can be invoked through a one-way Web Service call-one that immediately returns an HTTP-200 response without any payload-then the thread can trigger the job execution by calling the Web Service and quickly returning.
Work Manager: Before the release of the Java EE Concurrency Utilities in Java EE 7 under JSR-236, several Java EE containers offered a Work Manager API, an abstraction for dispatching and monitoring asynchronous work.
JMS Queue or Topic: A very robust mechanism to decouple threads on the Java EE platform is JMS (the Java Message Service). Java threads running inside the Java EE container can acquire access to a JMS Queue (single consumer) or Topic (potentially multiple consumers) and publish a message. This operation is fire-and-forget. Once published, the message is the responsibility of the Java EE container. The container routes the message to the subscribed listeners, typically Message-Driven Beans that will be invoked in a dedicated processing thread.
Grid: Grids come in various shapes, including compute grids that can be engaged for processing potentially huge jobs. The help of a grid can be enlisted in several ways, including JMX, REST and other remote communication protocols. One approach to be mentioned is the Coherence Write-Behind scenario, in which data changes are captured fail-safe in the in-memory grid and pushed to the persistent data store (typically a relational database management system) at a later point in time.
The third tier in our discussion of getting rid of the work that has to be done is the database tier. Figure 5 shows several entry points into the database, some synchronous and some asynchronous. It also visualizes some mechanisms inside the database that allow one session to communicate to another (background) session and have that background session execute the work.
The usual channel from Java middle tier threads into the database is through JDBC. From a connection pool, the middle tier thread has been allocated a dedicated database connection with a dedicated database session-the database equivalent of a Java thread. Any statement issued over this JDBC Database Connection is executed synchronously, blocking the connection and the issuing Java thread. If we manage to transfer the work load from the database session associated with the JDBC connection to some other session, then we have achieved our goal of asynchronously processing the work.
The database contains a job scheduler. Through either dbms_job or dbms_scheduler, PL/SQL units can create and schedule jobs, typically PL/SQL programs. These jobs are executed by background processes. Initiating a job is a prominent way of offloading work from a database session.
Pipes are non-transactional, in-memory communication channels that can also be used to transfer messages (and units of work) from one session to another. Using the supplied package dbms_pipe, text messages can be written to a pipe. Any database session can read from that pipe and start processing whatever message was read.
A global application context is somewhat similar to the pipe, in that it is an in-memory area that can be written to from PL/SQL not bound by the scope of transactions. Unlike a pipe, whose contents are removed upon read, the application context will retain its value until it gets explicitly removed. Note that no events or triggers can be associated with an application context and the update of its contents. That means a form of background polling would be required to actually start processing a message.
Another asynchronous interaction channel is formed by Advanced Queues (AQ). One session can publish a message that represents a unit of work on a queue; this completes the session's involvement. Asynchronously, a message handler can be invoked in a background job to process the message. Note: a WebLogic JMS Queue or Topic can be backed by an AQ. This means that publishing a message on a JMS Queue results in a message being published on the underlying AQ queue. And that means that a PL/SQL message handler associated with that queue could start asynchronously processing the message from the JMS publisher, opening up yet another asynchronous Java => (JMS => AQ =>) PL/SQL channel.
Of course, inserting a record to a regular database table is another way of transferring information from one session to another. Note, however, that an insert trigger will fire in the same session that executed the DML statement, which means that any work done by the trigger is done synchronously, blocking the thread that issued the JDBC call. A polling job could keep an eye on the table, register new records and act upon them.
As an aside: the Oracle Database supports something called "asynchronous commit"; this is a type of commit (introduced in release 10gR2) that does not wait for redo to have been completely written to disk (to the online redo log). A normal commit forces a disk I/O and does not return until the commit is complete. You will save some time if you instruct the database to do an asynchronous commit and not wait until the commit is guaranteed. You do run a risk, of course: if the database crashes before some buffered redo is written to the logs, or a file I/O problem prevents the log writer from writing buffered redo to a non-multiplexed log, then the commit is lost.
The Oracle Database is capable of accepting resources through FTP and WebDAV (in the XML DB Repository). Such a transfer of a resource ends with the creation of the resource in the database. As a result of the resource creation, events are published. In reaction to these events, background jobs can start some action, such as processing the resource.
Another asynchronous communication channel into the database is through email. Even though the database does not have native mechanisms to consume email, it is fairly simple to create a Java Stored Procedure that will do exactly that. Such a procedure could be scheduled to periodically check a mailbox and, for example, put every email's contents in a table or a queue.
One special type of job carried out by the database scheduler is file watching. The scheduler can be instructed to watch specific directories for new files. Whenever a file arrives in the watched directory, a PL/SQL program is started in background job.
Web Services published from the Service Bus are invoked in a synchronous manner. The Service Bus itself does not do asynchronous interactions. It can do one-way communication, where after accepting the request message, it will issue an HTTP 200 response code and conclude the conversation. If we want the service to return a proper SOAP/XML message based on some initial processing and validation, we can use a common approach to have part of a synchronous interaction turned into a deferred, asynchronous process. Figure 6, below, illustrates this approach.
A Proxy Service is exposed on the Service Bus. This service has a synchronous interaction style, accepting and returning a SOAP message. This proxy service does very limited processing-typically, some validation of the request message to ensure the asynchronous processing is likely to be successful. After successful validation, the message is published to a JMS Queue. This completes the work of the synchronous section of the chain. A SOAP response is returned to the invoker.
A second, internal proxy service uses an inbound JMS Adapter to listen to the JMS Queue. When a message arrives, this second proxy service picks up where the first one left of, and starts doing the actual processing-such as synchronously invoking one (or even more) business services.
When using a Service Component Architecture Composite, the options increase, largely because of the capabilities of the Business Process Execution Language (BPEL) engine. For example: a BPEL process can do some work after receiving the request, then send a response and continue processing.
The processing performed by the BPEL process after the initial response was sent to the client can be long running. It can involve calls to many business services-for example, using adapters to integrate with database or third-party applications. Some human tasks can be part of the process. BPEL is well equipped to handle asynchronous interactions, such as with humans through the human workflow service, and also with external asynchronous services. The BPEL engine handles the life cycle of the process instance (the BPEL equivalent of a Java HTTP session or a database session). The instance is dormant when no work can be done and responses or timers are awaited, and is revived when a callback delivering an asynchronous response arrives or a timer fires. As a final step, when the result of the work should somehow be made known to the erstwhile client, the BPEL process can invoke a component that is capable of routing the results in a way that the client can find them.
Back in our restaurant, the staff has found a way to break through the stifling procedures used before. Now the guests at a table can attract the attention of a waiter, tell him their wishes and then continue with their conversation as the waiter—asynchronously—makes his way to the kitchen. The waiter in turn informs the kitchen staff about the order and then goes his merry way, probably to some other table where they require his assistance. The cooks are asynchronously dealing with the order. The restaurant is more lively and more efficient as a result.
Now, however, we have to deal with the second stage: get the prepared orders from the kitchen to the guests who ordered them. Some simple rules apply (just like in a multi-tier architecture where the lower tier does not communicate to the higher level tiers):
The challenges for the restaurant staff include finding a way for the kitchen staff to draw the attention of the waiters, helping the waiters find out for which table an order delivered by the kitchen crew is destined and, finally, ensuring that the waiter is summoned to a table when he has the food for that table.
Two mechanisms are in use between kitchen crew and waiting staff. The kitchen will ring a bell whenever they have completed an order. The sound of the bell does not, of course, call out to any specific waiter, nor does it indicate for which table the kitchen has prepared an order. The kitchen staff also places all prepared orders in a heated or chilled holding area that the waiters can check whenever they are idling nearby. On each tray, the cooks place a numbered token, corresponding with the order jotted down by the waiter, that bears the table number.
The rule that waiters can approach a table only when invited by the guests to do so proves cumbersome when it comes to speedily delivering food to the tables. Waiters have invented little tricks, such trying to regularly seek eye contact with the guests at a table, or even suggesting to the guests that they should as that the waiter come to the table "as soon as the food is available." Based on that standing invitation, the waiters could go to the table at any time.
Lately, the restaurant started to employ messenger boys-very cheap labor-who constantly move around between waiters and guests to deliver messages back and forth. Whenever a waiter has a dish for a table, he sends one of the messengers to inform the table and have them formally request his presence with the food. This also means that any waiter can serve any table, instead of only the waiter originally attending the table.
When we turn our attention back to our multi-tier architecture, where we have jumped through several hoops in order to have the work performed asynchronously, we now face similar challenges when it comes to getting results back to the requesting party. By offloading the work, we severed the direct, synchronous link that makes it easy (through blocking) to return a response. It turns out that we can use solutions similar to those employed by the restaurant squad for communicating back from the processing tiers and threads to the calling party.
The challenge is two-fold: how to alert a higher level tier in the first place, and how to access the correct party [instance, session, thread, client].
The database is typically the end of the line in interactions. So how can we make it the starting point of the asynchronous response?
From within the database we can take some initiatives, similar to ringing a bell from the kitchen to draw the attention of the waiters in the restaurant. Using the Query Result Change Notification feature, which involves Java Objects registering themselves as listener with the Oracle JDBC Driver, we can have the database notify the middle tier, which results in a thread running based on that signal. Using the utl_http package or a Java Stored Procedure, we can initiate an HTTP call from within the database; this call can be handled by a servlet in the Java middle tier. The utl_tcp package allows lower-level communication with external TCP/IP-based servers. Java applications can easily be made to act as partner in a TCP/IP communication, opening up a two-way channel between database and Java middle tier.
As discussed before, a JMS Queue on the WebLogic platform can use a database Advanced Queue as its underlying store. As a result, messages published to the AQ at the database level are effectively published to the JMS Queue or Topic at middle tier level as well. Alternatively, Java Stored Procedures can be used in the database, along with an impressive collection of Java libraries, to publish to a JMS Queue. Listeners to such a queue are triggered on an independent Java thread by the JVM.
The outcomes of the background jobs in the database can be reported in emails, files written to the file system, or resources stored in the XML DB repository. However, it is up to the Java middle tier to come along and collect these results. This and all other approaches can be equated with the cold and hot holding areas where the cooks put all completed dishes, ready for shipment. From within the database, the results from the asynchronously processed workloads can be made available after having been prepared, but just like the waiters have to come and check out the holding area, a polling thread in the middle tier has to check with one of the holding areas used by the database. These can range from the above mentioned mail box, file system directory and XML DB repository folder, to a plain database table or any encapsulated structure exposed through a stored procedure.
The middle tier can be apprised of completed work in several other ways, such as an asynchronous web service response (which is really just an incoming web service call) or a WebSocket message received by a WebSocket endpoint. One way of dealing with the web socket message is to forward it to a JMS queue.
The waiters in the restaurant need to be able to associate orders prepared by the kitchen with the table that requested them. Similarly, the middle tier threads need to be able to determine from the results received or retrieved from the database what original requesting client, thread or session they are associated with. The middle tier also needs to have hooks into the client tier.
Until recently, the communication between client (browser) and middle tier (Java EE application server) was entirely was on HTTP. Since HTTP is essentially a stateless request/response model with the client in control, the middle tier cannot start a conversation to inform the client of anything, very much like the waiters who could approach a table only when explicitly invited to do so. Working around this limitation, the long poll was invented; it's essentially a request from the browser that the middle tier holds under consideration until it actually has information to send to the client. This long-running request ties up resources on both the server and the client, but it does give the impression of the server pushing information to the client. Instead of a long (or a streaming) poll request, the client can also periodically (e.g., every five seconds) submit a request to the middle tier that will typically result in an empty response but sometimes may contain a useful payload.
The WebSockets could be said to be the HTML5 equivalent of the messenger boys that were introduced in the restaurant to run around between the stately, expensive waiters and the dinner parties.
Before the middle tier can start feeding information to the client using the mechanisms described here, the results from asynchronous processing need to be communicated from the Java thread that possesses the information to the WebSocket endpoint as well as the session-specific listener objects. An application-scoped event handler could be set up that is reached from any thread with processing results as a singleton, using CDI Events or through a JMS Queue. This end point holds a list of all channels and for each channel an indication of the associated client-similar to the list of table numbers used in the restaurant.
The event handler can also reference to listener objects in HTTP sessions. When these objects are alive (e.g., because the session is involved in a long poll or in server-side event emitting), they can be notified immediately and subsequently handle the information passed in. Otherwise, the event handler can keep the information for the session until such time as it comes along inquiring after news.
Running a restaurant efficiently turns out to be challenge that bears a lot of similarity to architecting a multi-tier application. Having others do the work for you-in order to asynchronously do the heavy lifting without blocking the requesting party-is one of the premises of restaurants as well as applications. As a restaurant guest, you have the waiter work on your behalf and he in turn has a whole kitchen crew to do the work while he takes care of other guests.
Handing over work-either in the browser, the middle tier or the database-requires an asynchronous model with multiple parallel threads and a way to communicate from one thread to another. This article demonstrated many such mechanisms that help break the synchronous chain by enlisting the help of such another thread. The client-side HTML5 Web Worker in addition to the traditional (AJAX)
XMLHTTPRequest is an example of such a mechanism; so is the Java EE Concurrency Utilities on the middle tier along with the Asynchronous EJB methods and JMS for off-loading from a thread. In the Oracle database tier we have identified jobs as well as Advanced Queues and pipes, in addition to files and emails.
Note that when, eventually, a response is still required-with a report on the result of the processing that was performed in the back-some form of identification of the original requestor should travel with the request and eventually with the asynchronous response.
Sending the asynchronous response back up the tiers is not trivial. Just like the kitchen crew may not enter the restaurant proper, the database cannot step up to the Java middle tier to make itself known. Through polling strategies, the middle tier can find out what results the database has to offer. Additionally, several mechanisms allow the database to alert the database, including the AQ-JMS bridge, Query Result Change Notification and HTTP communication. In turn, the Java middle tier can leverage the HTML5 mechanism's Server Side Events and WebSockets to push results to the browser.
Fault handling is one aspect not discussed in this article that changes with an asynchronous model. With completely synchronous interactions, faults occur in real time with respect to the client. They need to be handled or they blow up in the face of the end consumer. (Which, if not gracious, is at least clear.) With asynchronous processing, exceptions take place in the background and will typically not be visible to the requesting party. Some form of fault handling is required, such as administrator alerts or automated recovery, in order to not let the request simply vanish into the void.
Using an appropriate selection of the mechanisms introduced in this article, we can quickly take off-line the work associated with a request, returning a very quick (but largely meaningless) response to the client. When the processing is complete and a more meaningful response has been produced, we can use push-channels to asynchronously deliver the response to the client.
Lucas Jellema has been active in IT (and with Oracle) since 1994. An Oracle ACE Director specializing in Oracle Fusion Middleware, Lucas is a consultant, trainer, and instructor in diverse areas including Oracle Database (SQL & PLSQL), Service Oriented Architecture, ADF, and Java. He is the author of the Oracle SOA Suite 11g Handbook, and a frequent presenter at JavaOne, Oracle Open World, ODTUG Kaleidoscope, Devoxx, OBUG and other conferences.