The Joy of Overriding


by Daniel Amadei

Overriding system property names in Oracle Coherence is a very simple yet powerful approach to letting multiple nodes exist in a single JVM without conflicts

Published April 2012

Oracle Coherence provides a flexible way to change the system property names used during configuration - which prevents problems that can occur when, for example, you deploy multiple applications pertaining to different clusters to the same Oracle WebLogic Server instance.

Suppose you have multiple applications deployed to the same Oracle WebLogic Server instance. Such applications are part of different Coherence clusters that have nothing to do with each other. If you pack the operational configuration file inside the EAR of each application, you will have to find a way to override the IP address and port to which each cluster member binds itself. (If you had a single application, this process would be quite straightforward: Just add a system property to override the IP address or host name and another one to override the port, and you would be done.)

For example:

  • tangosol.coherence.localhost is the default system property name for overriding the IP address or host name to which the Coherence node will bind itself.
  • tangosol.coherence.localport is the default system property name for overriding the local port to which the Coherence node will bind itself.

If you had multiple applications running in the same JVM but pertaining to different clusters, what would you do?(Coherence nodes are “classloader scoped”, so, depending on the classloader loading Coherence classes, you may have a single node per JVM, one node per EAR, or one node per WAR. It all depends on how you point to coherence.jar: from the application server classloader, EAR, or WAR; that will determinate how many cluster nodes you may have on a single JVM. ) That’s when having the option to override system property names shines.

The Problem

Let’s explore this problem in practice. Suppose you have two applications, AppA and AppB. AppA is a member of a Coherence cluster named myCluster and has the following Coherence operational configuration:

<?xml version='1.0'?>
<coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config"
  xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config coherence-operational-config.xsd">

  <cluster-config>
    <member-identity>
      <cluster-name>myCluster</cluster-name>
    </member-identity>

    <unicast-listener>
      <well-known-addresses>
        <socket-address id="1">
          <address>10.10.10.10</address>
          <port>7777</port>
        </socket-address>
        <socket-address id="2">
          <address>10.10.10.10</address>
          <port>8888</port>
        </socket-address>
        <socket-address id="3">
          <address>10.10.10.10</address>
          <port>9999</port>
        </socket-address>
      </well-known-addresses>
      <address system-property="tangosol.coherence.localhost">10.10.10.10</address>
      <port system-property="tangosol.coherence.localport">8888</port>
      <port-auto-adjust system-property="tangosol.coherence.localport.adjust">true</port-auto-adjust>
    </unicast-listener>
  </cluster-config>
</coherence>

AppB is another application, which is a member of a different Coherence cluster named myClusterB, and has the following operational configuration:

<?xml version='1.0'?>
<coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config"
  xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config coherence-operational-config.xsd">

  <cluster-config>
    <member-identity>
      <cluster-name>myClusterB</cluster-name>
    </member-identity>

    <unicast-listener>
      <well-known-addresses>
        <socket-address id="1">
          <address>10.10.10.10</address>
          <port>5555</port>
        </socket-address>
        <socket-address id="2">
          <address>10.10.10.10</address>
          <port>4444</port>
        </socket-address>
        <socket-address id="3">
          <address>10.10.10.10</address>
          <port>3333</port>
        </socket-address>
      </well-known-addresses>
      <address system-property="tangosol.coherence.localhost">10.10.10.10</address>
      <port system-property="tangosol.coherence.localport">5555</port>
      <port-auto-adjust system-property="tangosol.coherence.localport.adjust">true</port-auto-adjust>
    </unicast-listener>
  </cluster-config>
</coherence>

When we start an Oracle WebLogic Server instance containing both applications, it starts fine as the port for each node is different. However, if we were running multiple server instances with the same operational configuration file in the same machine - or if both configurations were using the same port - we would have to override the port and hostname to avoid conflicts.

Similarly, suppose we have two WebLogic Server instances running on the same box. Each instance hosts AppA and AppB. It would be useful to override the host name and/or port via system properties, instead of specifying four different XML files (one for each Coherence node running in this box), correct? Let’s try that and see what happens.

If we use Oracle Enterprise Pack for Eclipse (OEPE) to start our WebLogic Server instance, we can specify the JAVA_OPTIONS environment variable as shown in Figure 1 . To access this screen:

  1. Double click the server name in OEPE
  2. Click the Open launch configuration link
  3. Access the Environment tab.

 override-coherence-f1

Figure 1 Specifying the system properties via the OEPE IDE 

(Outside of OEPE, we would specify these properties in the scripts used to start WebLogic Server instances.)

After we specify the property, when starting the server, the Coherence node for AppA starts fine, as it has port 9999 in its list of well-known-addresses, or WKA. (According to Oracle Coherence documentation, the definition of WKA is: "The Well Known Addresses feature is a mechanism that allows cluster members to discover and join a cluster using unicast instead of multicast. WKA is most often used when multicast networking is undesirable or unavailable in an environment or when an environment is not properly configured to support multicast. All cluster multicast communication is disabled if WKA is enabled.") The other node running on the same JVM, however, is not so lucky. As both rely on the same system property name, both Coherence nodes take it into account and override its ports to the same one! AppB Coherence operational configuration does not have port 9999 in its list of well known addresses, so we are in trouble. (Even if it had such port specified in its WKA list, we would be in the same or even worst trouble, as you could end up with nodes from different clusters trying to communicate).

As seen below, AppB node is trying to start but there’s no known server running. Also, we can see the port used by the server was shifted to 10001 as 9999 was already in use by AppA node, which is running in a totally different cluster. 

2011-12-18 21:08:42.648/48.473 Oracle Coherence GE 3.7.1.1 <Warning> (thread=Cluster, member=n/a): This Member(Id=0, Timestamp=2011-12-18 21:08:12.166, Address=10.10.10.10:10001, MachineId=43245, Location=site:,machine:DAMADEI-BR,process:5484, Role=WeblogicServer) has been attempting to join the cluster using WKA list [/10.10.10.10:3333, /10.10.10.10:4444, /10.10.10.10:5555] for 30 seconds without success; this could indicate a mis-configured WKA, or it may simply be the result of a busy cluster or active failover. 

The Solution

A possible solution to this problem, maybe the best one, is to leverage your ability to change the name of the system properties that Coherence uses to override XML configuration. Unlike many products, which use the system property names in a fixed manner, Coherence allows you to change the system property names to anything you like. For example, by specifying the following element inside Coherence’s operational configuration XML: 

<address system-property="appA.coherence.localhost">10.10.10.10</address>

...we are doing two things: first, we’re specifying the IP address (or hostname) to which Coherence should bind; second, via the system-property attribute specified we are changing the system property name Coherence will try to find to override this value.

Doing that, we can override the system property names to override the IP/Host name and port to which Coherence will bind, and we will be able to specify different properties for each application running on the same JVM.

So, if we specify the following XML for AppA - take a close look at the system-property attribute on the elements address and port - we override the system property names on them:

<?xml version='1.0'?>
<coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config"
  xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config coherence-operational-config.xsd">

  <cluster-config>
    <member-identity>
      <cluster-name>myCluster</cluster-name>
    </member-identity>

    <unicast-listener>
      <well-known-addresses>
        <socket-address id="1">
          <address>10.10.10.10</address>
          <port>7777</port>
        </socket-address>
        <socket-address id="2">
          <address>10.10.10.10</address>
          <port>8888</port>
        </socket-address>
        <socket-address id="3">
          <address>10.10.10.10</address>
          <port>9999</port>
        </socket-address>
      </well-known-addresses>
      <address system-property="appA.coherence.localhost">10.10.10.10</address>
      <port system-property="appA.coherence.localport">8888</port>
      <port-auto-adjust system-property="appA.coherence.localport.adjust">true</port-auto-adjust>
    </unicast-listener>
  </cluster-config>
</coherence>

If we specify the system property –DappA.coherence.localport=9999, this node will bind to port 9999 instead of 8888 as specified in the operational configuration XML. Specifying tangosol.coherence.localport will have no effect for this Coherence node. (It will, obviously, affect other nodes that are using the default system property names running on the same JVM.) We have overridden it!

As we have only two nodes running on the same JVM, it’s a good idea to override the configuration for AppB too, as seen below:

<?xml version='1.0'?>
<coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config"
  xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config coherence-operational-config.xsd">

  <cluster-config>
    <member-identity>
      <cluster-name>myClusterB</cluster-name>
    </member-identity>

    <unicast-listener>
      <well-known-addresses>
        <socket-address id="1">
          <address>10.10.10.10</address>
          <port>5555</port>
        </socket-address>
        <socket-address id="2">
          <address>10.10.10.10</address>
          <port>4444</port>
        </socket-address>
        <socket-address id="3">
          <address>10.10.10.10</address>
          <port>3333</port>
        </socket-address>
      </well-known-addresses>
      <address system-property="appB.coherence.localhost">10.10.10.10</address>
      <port system-property="appB.coherence.localport">5555</port>
      <port-auto-adjust system-property="appB.coherence.localport.adjust">true</port-auto-adjust>
    </unicast-listener>
  </cluster-config>
</coherence>

Now, starting the server specifying properties as shown in Figure 2 will cause the node on AppB to start on port 3333 and node on AppA to start on port 9999.

 override-coherence-f2

Figure 2 Specifying different properties for Coherence nodes on AppA and B

We can confirm that by looking at the startup log: 

Stanza from AppA node log:

ThisMember=Member(Id=1, Timestamp=2012-01-03 22:19:52.987, Address=10.10.10.10:9999, MachineId=43245, Location=site:,machine:DAMADEI-BR,process:7008, Role=WeblogicServer)

Stanza from AppB node log on the same JVM:

ThisMember=Member(Id=1, Timestamp=2012-01-03 22:19:58.267, Address=10.10.10.10:3333, MachineId=43245, Location=site:,machine:DAMADEI-BR,process:7008, Role=WeblogicServer)

Conclusion

The opportunity to override system property names is a very simple yet powerful approach to letting multiple nodes exist in a single JVM without conflicts, allowing you to avoid coding multiple XML configuration files while still letting you have a fine grained control over Coherence setup for each node.


Daniel Amadei is a principal consultant for Oracle Consulting in Brazil, specializing in SOA and integration technologies. He has been working with SOA for the last years and Java technologies since 1999. Among other certifications, he is certified as an Oracle SOA Foundation Practitioner, Oracle SOA Architect Certified Expert and Sun Certified Enterprise Architect (SCEA).