Building a Newsfeed Portlet for My Oracle

Using Java, XML, XSL, and Oracle9iAS Portal

Overview

Oracle9iAS Portal provides a framework and infrastructure for combining various diverse distributed information sources into easy-to-access portals, such as My Oracle. An portal page displays output in distinct regions called portlets. A basic portlet displays data, and may provide links to drill down for details or launch an application. Advanced portlets add multiple pages, navigation links, and other features to enhance the end-user's experience. My Oracle includes basic and advanced portlets that provide personalized entry points to online applications and information.

Note: In the Oracle9iAS Portal architecture, the portal never talks to a portlet directly. Instead, it communicates via a software entity called a provider that represents and manages one or more portlets. From a developer's perspective, a portlet provider is a Java class or PL/SQL package that exposes a data source or application to Oracle9iAS Portal. For more information about portals, portlets, and providers, see the Developer's Overview on OTN.

Some of the most popular portlets on My Oracle display news headlines from outside sources including CNET, CNN, and the New York Times. The figure at right shows the high-level architecture of the CNN newsfeed portlet. The flow of information and events is as follows.

1. Oracle's newsfeed engine (a Java application) sends an HTTP request to CNN's Web server, which returns an XML document containing news headlines and URLs. (Click here to display a sample XML document.)
2. The newsfeed engine gets the XML document and applies an XSL stylesheet to generate HTML. (Click here to display a sample of the generated HTML.) Then the newfeed engine stores the HTML in an Oracle database managed by the oracle.com content management system (CMS).
3. The CMS pushes the HTML into a file system that stores oracle.com content outside the Oracle firewall.
4. The newsfeed portlet provider funnels the HTML file to the instance of Oracle9iAS Portal that manages My Oracle.
5. Oracle9iAS Portal renders the portlet within the portal page.

This architecture was designed to meet requirements of the oracle.com CMS, which stores and manages Web site assets in a database behind the firewall, then writes HTML and other files into a file system outside the firewall (for more information about the oracle.com CMS, see Managing Web Content). Portals with simpler requirements can use Oracle9iAS Portal URL Services, which takes the URL of an application or Web page, parses the content, and creates a portlet. Very little programming is required to use URL Services.

Implementation Notes

The newsfeed portlet was implemented in three phases:

• Building the newsfeed engine.
• Defining the newsfeed portlet provider.
• Registering the provider with Oracle9iAS Portal.

Building the Newsfeed Engine

The newsfeed engine is a Java application that gets an XML document from the data source, converts it to HTML, and writes the HTML to a file. This is by far the most complex component of the newsfeed portlet, although, as noted above, it is not required by Oracle9iAS Portal. The engine performs these main tasks:

• Requesting the XML document
• Generating HTML
• Storing HTML in CMS

The newsfeed engine manages newsfeeds from several sources, not just CNN. It stores information about each source in the database, and at runtime uses a custom queue to manage jobs as newsfeed documents are requested and received from various sources.

Here's a sample of newsfeed data for CNN (URLs have been obfuscated).

The newsfeed engine uses a custom class named XML2HTMLObj to store and manipulate this newsfeed data. For example, when the engine starts, it queries the database and creates an instance of XML2HTMLObj for each news source in the result set. Then it sets attributes of the XML2HTMLObj instance and adds it to the newsfeed job queue for processing, as shown in the following code listing.

public class DBManager 
{
...
  public Collection readAll() throws SQLException 
    {
        ConnectionObject connObj = getConnection();
        if (connObj == null) 
            throw new SQLException("Connection unavailable.");
        Statement stmt = null;
        ResultSet rs = null;
        ArrayList list = new ArrayList();
        Connection con = connObj.getConnection();
        String sql_read = 
             "SELECT P_NAME, XML, XSL, OUTPUT, " +
                "XML_PROXY, XSL_PROXY, ISACTIVE, LASTCHANGE, " +
                "UPDATE_INTERVAL, EXTERN, XML_USERNAME, " + 
                "XML_PASSWORD, EMAIL, STATUS " + 
             "FROM XML2HTML WHERE PARENT_PNAME IS NULL" +
             "ORDER BY P_NAME";

        try 
        {
            stmt = con.createStatement();
            rs = stmt.executeQuery(sql_read);      
            while(rs.next()) 
            {
                if(rs.getBoolean(7)) 
                {
                    String pName = rs.getString(1);
                    String xml = rs.getString(2);
                    String xsl = rs.getString(3);
                    String output = rs.getString(4);
                    ...
                    XML2HTMLObj data;
                    ...
                    data.setPName(pName);
                    data.setXmlSourceName(xml);
                    data.setXslSourceName(xsl);
                    data.setOutputSourceName(output);
                    ...
                    list.add(data);
                }
        } 
        catch(SQLException sqle) 
        { ... }
     return list;
   }

...
}

As shown previously, each XML2HTMLObj instance includes an attribute that specifies an update interval. For example, the update interval for the CNN newsfeed is 30 minutes. Each time the update interval expires, the newsfeed engine sends an HTTP request to the news source server, and the server responds by returning an XML document. This functionality is implemented using Java thread management methods, specifically java.lang.Object.wait() and java.lang.Object.notify().

public class XML2HTMLTransformer extends Thread {
  ...
  private class SchedulerExecuter extends Thread
  {
    ...
    public void run()
    {
      while (true)
      {
        XML2HTMLObj job = null;
          synchronized (mExecQueueLock)
          {
            while( (mExecQueue.size() <= 0 || 
              (job = (XML2HTMLObj) mExecQueue.remove(0)) == null) 
                                                      && runFlag )
              {
                try
                {

                  // Wait for XML2HTMLTransformer to 
                  // call Object.notify to signal that
                  // a job is ready for processing
                  mExecQueueLock.wait();
                 } catch(InterruptedException ie) {}
              }
          }
          ...
          // Continue processing after notification.
          try
          {
            XSLProcessor processor = new XSLProcessor();
            // Get XML document and apply XSL sytlesheet.
            job.processXSL(processor);
            ...
          }
          catch(Exception e)
          { ... }
          // insert back into scheduler
          XML2HTMLTransformer.this.addJob(job);
          ...

The SchedulerExecuter waits until the XML2HTMLTransformer calls Object.notify, then executes the following code to retrieve an XML document, apply an XSL stylesheet to generate HTML, and call a CMS stored procedure to write the results to the CMS database. The resulting HTML file always has the same name; in other words, the new version replaces the old one in the file system. This makes it easy to define a portlet provider, as described in the next section.

    public void processXSL(XSLProcessor processor) 
        throws SAXException, XSLException, MalformedURLException, 
               IOException, SQLException 
    {
        // write to memory first
        ByteArrayOutputStream tmpOut = new ByteArrayOutputStream();
        ConnectionObject connObj = null;
        CallableStatement cstmt = null;
        try 
        {
            processor.processXSL(getXSLStylesheet(), 

                          (XMLDocument)getXMLSource(), tmpOut);
            TextUtils tu = new TextUtils();
            String s = new String(tmpOut.toByteArray(), "UTF-8");
            tu.setOriginalText(s);
            ...
            // Write to CMS database using CMS stored procedure
            connObj = DBManager.getConnection();
            if (connObj == null)
                throw new SQLException("Connection unavailable.");
            Connection conn = connObj.getConnection();
            String callStoredProcedure = 
                      "{CALL CMS_API.SAVE_PAGE(?, ?, ?, ?)}";
            cstmt = conn.prepareCall(callStoredProcedure);
            cstmt.setString(1, this.