Configuring and using Coherence Persistence

1. If required, download and install the Java JDK 1.8.
   Refer to the prior table for the location of the download for your operating system.  Refer to the JDK installation instructions for your operating system.
2. Open a command prompt or terminal as required by your operating system.
3. Coherence is provided as a zipped Java jar file. Unzip using a command similar to those below unpack Coherence:
   

   $ cd c:\downloads
   $ unzip fmw_12.2.1.0.0_coherence_Disk1_1of1.zip
   Archive:  fmw_12.2.1.0.0_coherence_Disk1_1of1.zip
   inflating: fmw_12.2.1.0.0_coherence.jar
   $                                  

4. Install coherence using a command similar to:
   

   $ %JAVA_HOME%\bin\java -jar fmw_12.2.1.0.0_coherence.jar

   Note: For the remainder of this tutorial Windows 7 style commands will be shown.
   Linux and OSX users will need to modify these commands to match OS requirements.
   In certain instances both Windows and Linux style commands will be shown.
5. In the Welcome dialog click Next.
6. In the Auto Updates dialog click Next.
7. In the Installation Location dialog enter a fully qualified path to install Coherence into and click Next.
   

   

   
   For the remainder of this tutorial will refer to this location as COHERENCE_HOME.
   Consider using an environment variable set to the coherence directory for ease of use. For example:
   

   $ set COHERENCE_HOME=c:\fmw\coherence12.2.1\coherence

8. In the Installation Type dialog select Coherence with Examples and click Next.
9. In the Prerequisite Checks dialog ignore any warnings and click Next.
10. In the Installation Summary dialog click Install. Additionally you may save the response file for later review.
11. The Installation Progress dialog will display and show each stage or the installation. Click Next to view a log of the install or Finish to complete the install and exit the installer.

In this section of the tutorial we will install the Coherence JVisualVM Plugin.

1. At the command line start JVisualVM using a command similar to:
   

   $ jvisualvm –J-Dcom.oracle.coherence.jvisualvm.refreshtime=10

   Note: The –J-Dcom.oracle.coherence.jvisualvm.refreshtime=10 property changes the data refresh interval from the default 30 seconds to 10 seconds and limits wait time when JVisualVM needs to refresh a panel or pane.
2. Select Tools> Plugins and then the Available Plugins tab,
3. Select the VisualVM-MBeans plugin.
4. Click Install. The Plugin installer wizard will display.
5. Click Next.  The installer will download and install the latest version of the plugin. 
6. When complete click Finish.
7. Select  the Downloaded tab.
8. Click Add Plugins... and navigate COHERENCE_HOME/coherence/plugins/jvisualvm directory to and select the coherence-jvisualvm.nbm file.
9. Click Install. The Plugin installer wizard will display.
10. Click Next.
11. Read and then accept the terms of the licensing agreement and click Install
12. If presented with a validation warning click Continue.
13. The plugin will be installed. Click Finish when complete.
14. Close the Plugins dialog. The Coherence JVisualVM plug in is now installed.
15. Exit JVisualVM.

These next steps all require the use of COHERENCE_HOME and JAVA_HOME variables.
As a convenience a sample script is provide here. This script is shown in step 2 below and used to set up the appropriate values for the Coherence and Java home variables. Use of this script is optional and is not shown through the remainder of this tutorial.

If using the provided Windows script update the script for the COHERENCE_HOME and JAVA_HOME locations in your environment.

1. Open four command windows, two will be used for Coherence cache servers, one for JVisualVM and one for CohQL commands.
2. Using two of the terminals, start two Coherence cache-servers using a commands similar to (Windows and Linux examples given):

   $ setup_env.cmd
   $ cd %COHERENCE_HOME%
   $ bin\cache-server.cmd -jmx                                    

   Linux/Unix users use a command similar to:

   $ cd $COHERENCE_HOME
   $ bin\cache-server.sh -jmx

   Note: The -jmx argument adds several parameters to the cache server in support of JMX system management.
3. In a third command window, start the Coherence Query tool (CohQL) using a command similar to:
   

   $ bin\query.cmd 

4. Insert data into a cache using commands similar to:
   

   CohQL> insert into 'test' key(1) value('value1')
   CohQL> insert into 'test' key(2) value('value2')
   . . . 
   CohQL> insert into 'test' key(10) value('value10')

5. verify that the data was added using a select command.
   

   CohQL>  select key(),value() from 'test'

   The command should return results similar to:
   

   Results
   [1, "value1"]
   ... 
   [10, "value10"]

6. In a command window start JVisualVM using a command similar to:
   

   $ %JAVA_HOME%\bin\jvisualvm –J-Dcom.oracle.coherence.jvisualvm.refreshtime=10

7. In the JVisualVM Applications pane select one of the DefaultCacheServer instances by clicking its name.

   

   The application's information will display in the right hand pane.
8. In right hand pane select the Oracle Coherence > Caches tab.
   Notice that the cache shows the entries previously added using CohQL.
   

   

9. In right hand pane select the Oracle Coherence > Persistence tab.

   

In the previous section we populated a cache using the Coherence query tool. In this section we will

1. In right hand pane select the Oracle Coherence > Persistence tab.
2. From the Persistence tab, select the PartionedCache Service, right click and select Create snapshot.
   

   

3. In the input dialog, enter the name "snapshot.test" and click OK.
   In the Please Confirm Operation dialog, click Yes.
   In the informational Result dialog, click OK.
   Note: To preserve data consistency Coherence will pause while the snapshot is written.
4. Select the PartionedCache Service, right click and select List snapshots. Click OK.  The newly created snapshot should be listed.
   

   

5. Return to the terminal window running the Coherence Query tool and delete the data in the test cache using a command similar to:
   

   CohQL>  delete from 'test'

6. Confirm that the cache is now empty using a command similar to:
   

   CohQL>  select count() from 'test'

   Which should show a result:
   

   Results
   0
   CohQL>

7. Return to JVisualVM and in the Persistence tab, Right click the Partitioned Cache Service and select Recover Snapshot.
8. From the drop down list select the previously created snapshot and click OK. In the confirmation dialog click Yes. In the results dialog click OK.
9. Return to the Query terminal and display the restored data using a command similar to:
   

   CohQL>  select count() from 'test'

   Which should show a result similar to:
   

   Results
   10
   CohQL>

10. Exit the Coherence Query too using the byecommand.
    

    CohQL> bye
    $

11. Using [ctrl][c] or the corner X close the terminal windows, shut down both instance of Coherence, and exit JVisualVM.
    Consider keeping these windows open for reuse in later tutorial sections.  However the instances themselves should be shut down as described above.

You have now successfully used the default Coherence persistence mechanisms to create and restore simple snapshots. In the next section you will explore using Coherence Active Persistence.

There are two aspects to setting up active persistence:

• Defining the persistence-environment in the operational override file
• Associating a persistence-environment with a service

1. Ensure that all cache servers from the prior section are shutdown.
   Also ensure that JVisualVM and the Coherence query tool are shut down
2. Open a terminal window and change directory to the COHERENCE_HOME\examples\java directory

   $ cd %COHERENCE_HOME% 
   $ cd examples\java

3. Examine the override and cache configuration files:
1. Using a command similar to:
   

   $ type resource\persistence\tangosol-coherence-override.xml

   Search for the persistence-environments element
   Notice that element is wrapped in comments and resembles:
   <!--
   <persistence-environments>
   . . .
   </persistence-environments>
--> 
2. Using a command similar to:
   

   $ type resource\persistence\examples-cache-config.xml

   Search for the persistenceelement for the ExamplePartionedPofScheme.
   Notice that the examples cache configuration uses the default-active persistence definition
 
   

   

4. Build the examples using commands similar to:
   

   $ bin\build.cmd contacts
   $ bin\build.cmd persistence
   

   Wait for each command to complete then proceed to the next step.
5. Start a cache server instance using a command similar to:

   $ cd %COHERENCE_HOME%\examples\java
   $ bin\run-cache-server.cmd persistence

   Remember to ensure that the COHERENCE_HOME and JAVA_HOME variables are correctly set in each window.
6. Open a second terminal window and start a second cache server instance.

   $ cd %COHERENCE_HOME%\examples\java 
   $ bin\run-cache-server.cmd persistence

7. Open a third window and start JVisualVM.
   
8. Select either of the DefaultCacheServer instances and in the right pane select Oracle Coherence and then the Persistence subtab.
   Note: that topmost pane was expanded to show all services.
9. Notice that the persistence mode of the Partitioned cache services is shown as active.
   

   

10. Start the Coherence Query tool using a command similar to:
    

    $ cd %COHERENCE_HOME%\examples\java 
    $ bin\run.cmd persistence cohql

11. Insert data into a cache using commands similar to:

    CohQL> insert into 'test' key(1) value('value1')
    . . . 
    CohQL> insert into 'test' key(10) value('value10')

12. Return to JVisualVM and examine the examine the Oracle Coherence > Caches tab. Note that the cache now contains entries.
    Examining the Oracle Coherence > Persistence > Storage and Latency Details tab will also show an increase in Total Active Space Used.

1. Kill all running cache servers and exit JVisualVM.
   Under Windows use the [x] in each terminal window. 
   Under Linux use a command similar to ps augx | grep DefaultCacheServer and then capture the process ids and use kill -9 %pid to kill each process. 
2. Open two terminal window and start two cache server instances.

   $ cd %COHERENCE_HOME%\examples\java 
   $ bin\run-cache-server.cmd persistence

3. As each cache server starts you may notice messages indicating recovery is in progress.
   Statements such as those shown below represent Coherence recovering the cache:
   2016-08-05 12:29:31.983/37.098 Oracle Coherence GE 12.2.1.0.0 (thread=DistributedCache:PartitionedPofCache2, member=1): Recovering 257 partitions
. . .
2016-08-05 12:30:47.278/112.372 Oracle Coherence GE 12.2.1.0.0 (thread=DistributedCache:PartitionedPofCache2, member=1): Recovered PartitionSet{0..256}
from active persistence
. . .
4. Restart JVisualVM, connect to a default cache server and examine the cache.
   Notice that the original objects have been restored.
5.  Optionally, create and recover a snapshot of an active persistence cache using JVisualVM:
   
   1. Within JVisualVM select a DefaultCacheServer instance.
   2. Open the Oracle Coherence > Caches tab and note the name of the Service associated with the test cache, for example PartitionedPofCache.
   3. Open the Oracle Coherence> Persistence tab.
   4. Select the matching Partitioned cache row.
   5. Right click and select Create snapshot. Name and otherwise complete the snapshot.
   6. Using Coherence Query tool, drop all entries in a cache.
      For instance delete from 'test', where testrepresents the name of the originally created cache.
   7. Use a command such as select count() from 'test' to verify the cache contains no data.
   8. Returning to JVisualVM and using Recover snapshot to restore the snapshot.
   9. Use a CohQL command such as select key(),value() from 'test' to show that the cache data was reloaded.
6. Leave both cache servers running but exit JVisualVM and the CohQL tool.

Notes:

• Persistent data is stored under the directory COHERENCE_HOME/examples/java/persistence-data in four sub-directories.
  • active – Contains active persistence data.
  • snapshots – Contains snapshot data.
  • trash –Contains incomplete persistence data.
  • archives – Contains archiving snapshots (covered in the next section)
  • All of the above directories can be configured in the operational override file tangosol-coherence-override.xml. The archive directory must be a shared directory but the others may be local or shared.
  • In this tutorial section we are using the default-active environment and relying on setting the coherence.distributed.persistence.base.dir property (in function_set_env or set_env.cmd) to a specific directory rather than defaulting to users home
  • There are special considerations on multi-machine environment using local active directories.
    Covered in a later section on recover quorums.

To populate the Contacts cache with entries:

1. Open two terminal window and start two Coherence server using commands similar to:
   

   $ cd %COHERENCE_HOME%\examples\java 
   $ bin\run-cache-server.cmd persistence

2. Open a new terminal window and populate the cache with data using commands similar to:
   

   $ cd %COHERENCE_HOME%\examples\java 
   $ bin\run.cmd persistence populate

   The populate operation will add 10,000 entries to the contacts cache
3. Start JVisualVM and select one of the DefaultCacheServer instances
   
4. In the right pane select the Oracle Coherence > Caches tab.
5. Examine the service/Cache name instances, one of which should show a 10,000 entries.

   

Create and validate a snapshot:

1. In JVisualVM navigate to the Persistence subtab.
2. Select the cache with the larger active space used.
   Note:You may need to expand the services portion of the Caches tab.
3. Right click and select Create Snapshot. Complete the wizard to finish creating the snapshot.
   Alternatively you could use the create snapshot command in Coherence Query tool. See the Coherence Query Tool documentation for more information,
4. Start the Coherence Query tool using a command similar to:
   

   $ cd %COHERENCE_HOME%\examples\java 
   $ bin\run.cmd persistence cohql

5. Using a command similar to the one shown below, list all snapshots.  Note that two snapshots are listed.
   

   CohQL> list snapshots
   Results
   "PartitionedPofCache": ["contacts", "test-snapshot"]
   "PartitionedPofCache2": []      

6. Validate the snapshot using a command similar to:

   CohQL> validate snapshot 'contacts' 'PartitionedPofCache'
   Validating snapshot directory 'c:\. . .\contacts'
   Results
   "Success"

7. Perform a verbose validation using a command similar to

   CohQL> validate snapshot 'contacts' 'PartitionedPofCache' verbose
   Validating snapshot directory 'c:\. . . \contacts'
   Results
   Attribute                    Value
   ---------------------------- ---------------------------------------------------
   Partition Count              257
   Directory                    c:\. . .\PartitionedPofCache\contacts
   Storage Format               BDB
   Storage Version              0
   Implementation Version       0
   Number of Partitions Present 257
   Is Complete?                 true
   Is Archived Snapshot?        false
   Service Version              14
   Statistics                   contacts
                                Size=10000, Bytes=1627109, Indexes=0, Triggers=0, Listeners=0, Locks=0
   test                         Size=10, Bytes=180, Indexes=0, Triggers=0, Listeners=0, Locks=0
   

An archived snapshot is a full copy of all snapshot parts, from all members, stored in a central location and in a storage-independent format.  To archive and validate a snapshot:

1. Using JVisualVM archive the snapshot by:
   1. Selecting the Persistence subtab
   2. Select the cache with the larger active space used.
      Note:You may need to expand the services portion of the Caches tab.
   3. Right click and select Archive Snapshot.
      Select the contacts snapshot.
      Complete the wizard to finalize the archive process.Complete the wizard to finish creating the snapshot.
      Alternatively, you could use the archive snapshot command in the Coherence Query Tool.
      For example:
      

      CohQL> archive snapshot 'contacts' 'PartitionedPofCache'
      Are you sure you want to archive a snapshot called 'contacts' for service 'Parti
      tionedPofCache'? (y/n): y
      Archiving snapshot 'contacts' for service 'PartitionedPofCache'
      Results
      "Success"

2. Snapshots are contained in the directory defined by the <snapshot-archivers> element of the tangosol-coherence-override.xml file. Which for this tutorial is COHERENCE_HOME/examples/java under the persistence-data/archives directory.
   Examine the COHERENCE_HOME/examples/java/persistence-data/archives directory to find snapshot data.
   Snapshot data is typically broken down into individual files based on the number of partitions specified for a cache which defaults to 257.  How many snapshot files appear to exist for the contacts snapshot?
   

In this section we will use the Coherence Query Tool to work with snapshots. All the operations shown can also be completed within JVisualVM.   Note: When retrieving an archived snapshot, the local snapshot should first be removed.
You may also use the OVERWRITE option of the RETRIEVE SNAPSHOT command to to replace an in existing snapshot.

To retrieve and recover a snapshot:

1. Return to or open the Coherence Query Tool in a terminal window.
   Use command similar to those below if you need to start an instance of the Coherence Query Tool.

   CohQL>  remove snapshot 'contacts' 'PartitionedPofCache' 
   

   When asked for confirmation answer 'y'.
2. Delete the the existing snapshot
   

   CohQL> retrieve archived snapshot 'contacts' 'PartitionedPofCache' 
   

   When asked for confirmation answer 'y'.
3. Drop all results in the contacts cache:
   

   CohQL> delete from 'contacts'
   Results
   
   CohQL> select count() from 'contacts'
   Results
   0
   

4. Recover the snapshot:
   

     CohQL> recover snapshot 'contacts' 'PartitionedPofCache' 

   When prompted enter 'y' to confirm.
   Note if the recover snapshot command returns the error Error: (Wrapped: Unable to complete operation recoverSnapshot for service PartitionedPofCache) sleep interrupted the operation will continue in the background.
   Use the command list services to show the status of the PartitionedPofCache service.  When the service shows idle continue with the instructions below. 
   
5. Examine the content of the cache to verify the recover operation restored data:
   

   CohQL> select count() from 'contacts'
   Results
   100000
   

6. Leave all processes running as they will be used in the next section.

The PersistenceCoordinator MBean is a Coherence “Responsibility MBean”. What this means there will only ever be a single instance of this MBean. The singleton instance is responsible for coordinating the Persistence operations within the cluster.

To find the Persistence Coordinator MBean:

1. If required, Start  JVisualVM and select one of the running DefaultCacheServer instances.
2. In the right pane select the MBeans tab.
   Note: If the MBeans tab is not displayed review the section on installing the Coherence and MBeans JVisualVM plugins.
   Note:Only one of the DefaultCacheServer instance is configured for JMX. If you see the error
   Data not available because JMX connection to the JMX agent could not be established continue with the other instance.
   
3. Navigate to Coherence > Persistence > PartionedPofCache > PersistenceCoordinator.

   

4. Examine the attributes of the Persistence Coordinator MBean which include:
   

   Field	Description
   CoordinatorID	The Member Id of the current coordinator
   Idle	Coordinator is idle flag
   Operation Status	The current operation being executed, or idle if no operation in progress.
   Snapshots	A string array of know snapshots for this service.

5. Select the Operations tab. The operation tab shows the operations that are callable against the PersistenceCoordinator MBean. These are primitive operations that both the Coherence Query tool and the Coherence JVisualVM plug execute.

   

6. Using the createSnapshot operation, create a new snapshot. Enter a name, for example new snapshot and click the createSnapshot button:

   

7. Return to the Coherence Query tool, or use the list snapshots operation in the JVisualVM Persistence tab to list all known snapshots.
   Does the newly created snapshot exist?
   

   CohQL> list snapshots
   Results
   "PartitionedPofCache": ["contacts", "new-snapshot"]
   "PartitionedPofCache2": ["test"]
   

JMX Clients can subscribe to Persistence JMX notifications to obtain information about operations that are executed for a particular service. Notifications for operations exist for start, stop, and a variety of failures. In this section we will examine JMX notifications using JVisualVM. 

To enable JMX notifications:

1. Start or return to JVisualVM and select a DefaultCacheServer instance. You may reuse an existing instance if one is already open.
2. In the MBeans browser tab navigate to Coherence > Persistence > PartionedPofCache > PersistenceCoordinator.
3. Select the Notifications tab.
   Note that the Collect Notification checkbox is checked by default.   When checked, Oracle Coherence will subscribe to notifications and calculate operation duration based on begin/end times.
   Note that you may need to expand the window to see the Collect Notifications check box and Clear Notifications button.
   

   

4. Click the Subscribe button. (you may need to scroll to the bottom of the window).
5. Perform several operation on a snapshot. For example, remove, recreate, archive the contacts snapshot:
   

   CohQL> list snapshots
   CohQL> remove snapshot 'contacts' 'PartitionedPofCache'
   Are you sure you want to remove snapshot called 'contacts' for service 'PartitionedPofCache'? (y/n): y
   CohQL> create snapshot 'contacts' 'PartitionedPofCache' 
   ...
   CohQL> remove archived snapshot 'contacts' 'PartitionedPofCache'
   ...
   CohQL> archive snapshot 'contacts' 'PartitionedPofCache'
   ...
   CohQL>
   

6. Return to the Notifications tab and examine the result of the prior operations.  You should see something resembling that below:
   

   

   
   Note:Snapshot operations typically generate two sets of notifications: The begin/end pairing signifies the beginning and end of the entire snapshot operation and is used to calculate operation duration.
7. Select the Oracle Coherence > Persistence tab.
   You can clear existing notifications, or right click the table and save as a CVS file if desired.
   

The following section reuses the existing Coherence instances and you can leave them running.

A notifications example is provided which details how to programmatically determine the durations of operations.
To run this notifications example:

1. Open a terminal window and start the notifications example using commands similar to:
   

   $ cd %COHERENCE_HOME%\examples\java 
   $ bin\run.cmd persistence notifications

   Which will produce output resembling:
   

   redirecting Coherence logging to persistence.log...
   . . . 
   ------Getting MBeanServer...------
   ------Retrieving Registry...------
   ------Registering listener for PartitionedPofCache------
   ------Registering listener for PartitionedPofCache2------
   ------Waiting for notifications. CTRL-C to interrupt.------
   

2. Return to the Coherence Query tool and using commands similar to those shown perform some snapshot operations.
   Note:Confirmation requests removed for clarity.
   

   CohQL> list snapshots
   CohQL> remove snapshot 'contacts' 'PartitionedPofCache'
   CohQL> create snapshot 'contacts' 'PartitionedPofCache' 
   CohQL> remove archived snapshot 'contacts' 'PartitionedPofCache'
   CohQL> archive snapshot 'contacts' 'PartitionedPofCache'
   CohQL> 
   

3. Return to the terminal where the notifications application is running. 
   Examine the output which should show notifications being generated for each of the operations, one each for begin and end. 
   

   Thu Aug 11 10:36:52 EDT 2016 : PartitionedPofCache (remove.snapshot.begin) Removing snapshot "contacts"
   Thu Aug 11 10:36:57 EDT 2016 : PartitionedPofCache (remove.snapshot.end)   Successfully removed snapshot "contacts" (Duration=5143ms)
   Thu Aug 11 10:37:08 EDT 2016 : PartitionedPofCache (create.snapshot.begin) Building snapshot "contacts"
   Thu Aug 11 10:37:50 EDT 2016 : PartitionedPofCache (create.snapshot.end)   Successfully created snapshot "contacts" (Duration=41933ms)
   Thu Aug 11 10:39:40 EDT 2016 : PartitionedPofCache (remove.archived.snapshot.begin) Removing archived snapshot "contacts"
   Thu Aug 11 10:39:41 EDT 2016 : PartitionedPofCache (remove.archived.snapshot.end)   Successfully removed archived snapshot "contacts" (Duration=539ms)
   Thu Aug 11 10:39:51 EDT 2016 : PartitionedPofCache (archive.snapshot.begin) Archiving snapshot "contacts"
   Thu Aug 11 10:39:58 EDT 2016 : PartitionedPofCache (archive.snapshot.end)   Successfully archived snapshot "contacts" (Duration=6776ms)

4. Return to the terminal window where the notifications application example is running and stop it using [ctrl][c].

Learners interested in how the notifications examine registers for notifications are directed to the NotificationWatcher.javaexample found in: COHERENCE_HOME\examples\java\com\tangosol\examples\persistence\NotificationWatcher.java.

The information displayed on the Persistence tab in JVisualVM is obtained from a number of sources including:

• PersitenceCoordinator MBean
• ServiceMBean values aggregated for all Coherence instances

1. Return to or restart JVisualVM
2. Select a DefaultCacheServer instance.
3. In the right pane  select the MBeans tab.
   Reminder: Only one of the DefaultCacheServer instance is configured for JMX. If you see the error
   Data not available because JMX connection to the JMX agent could not be established continue with the other instance.
   
4. Navigate to the Coherence > Service > PartitionedPofCache and select instance 1"
   

   

   
   
5. In the right pane examine the attributes tab of the Service MBean. Persistence attribute are prefixed with Persistence. Not all are shown.
   Given the image, can you answer determine the persistence mode for this instance and service combination?
   

   

   
   All the attributes are explained in the Coherence documentation, several outlined below, the complete reference for Oracle Coherence MBeans can be found here. 
   • PersistenceActiveSpaceUsed – The total space used be active persistence for this node
   • PersistenceEnvironment – the details of the active and snapshot directories
   • PersistenceLatencyAverage – the average latency added by Persistence
   • PersistenceLatecyMax – the maximum latency added by Persistence
   • PersistenceSnapshotArchiver – the snapshot archiver used by the service
6. Shutdown any cache servers, Coherence Query tool instances and JVisualVM

The recover-quorum controls the number of members that must be present before recovery can begin. This is important for a number of reasons:

1. When using ‘active’ mode, there must be enough service members running to be able to recover the data without running out of memory.
2. If the recovery-hosts parameter is not set correctly then there may be partitions that are not recovered because not all persisted data is not available. The recovery-hosts parameter is shown below:

   <partitioned-quorum-policy-scheme>
      <recover-quorum>2 </recover-quorum>
     <recovery-hosts></recovery-hosts>
    </partitioned-quorum-policy-scheme>
   

Since only a single machine is running all instances the recover-hosts element is left empty.

To configure Recover Quorum:

1. Ensure that all Coherence instances, Coherence Query Tool instances and JVisualVM are shut down.
   
2. Using notepad or gedit or a similar editor, update the file COHERENCE_HOME/examples/java/resource/persistence/examples-cache-config.xmland un-comment out the following lines: (Note: There is two entries, ExamplesPartitionedPofScheme and ExamplesPartitionedPofScheme2)
   
   

   $ cd %COHERENCE_HOME%\examples\java
   $ notepad resource\persistence\examples-cache-config.xml 

   Which will resemble:
   

   . . . 
    <distributed-scheme>
       <scheme-name>ExamplesPartitionedPofScheme</scheme-name>
       . . . 
         </persistence>
         <!--
           <partitioned-quorum-policy-scheme>
              <recover-quorum>2</recover-quorum>
           </partitioned-quorum-policy-scheme>
         --> 
       <autostart>true</autostart>
       . . .                                         
   </distributed-scheme> 

3. In a new terminal window or a window previously opened start a single Coherence instance.
   Note:Consider redirecting the output of the run-cache-server.cmd command to a file to more easily search for specific output.
   

   $ cd %COHERENCE_HOME%\examples\java 
   $ bin\run-cache-server.cmd persistence

4. Examine the output of the window searching for disallow messages.
   For example:
   

   2016-08-12 10:57:00.534/19.401 Oracle Coherence GE 12.2.1.0.0 <warning> (thread=DistributedCache:PartitionedPofCache, member=1): Action recover disallowed; all-disallowed-actions: recover(2)
   2016-08-12 10:57:00.544/19.408 Oracle Coherence GE 12.2.1.0.0 <warning> (thread=DistributedCache:PartitionedPofCache2, member=1): Action recover disallowed; all-disallowed-actions: recover(2)
   

   Assuming the run command was redirected to quorum.log under Window use a search command similar to:
   

   $ type quorum.log | findstr /C:"disallow"

   Under Linux use a command similar to:
   

   $ grep disallow quorum.log

5. Open a second terminal window and start a second instance of Coherence
   

   $ cd %COHERENCE_HOME%\examples\java 
   $ bin\run-cache-server.cmd persistence

6. Examine the output of either instance and you will see messages indicating recovery in progress.
   

   2016-08-12 11:05:39.531/46.461 Oracle Coherence GE 12.2.1.0.0  (thread=DistributedCache:PartitionedPofCache, member=2): Recovering 128 partitions
   2016-08-12 11:05:44.255/51.184 Oracle Coherence GE 12.2.1.0.0  (thread=DistributedCache:PartitionedPofCache2, member=2): Recovering 128 partitions

7. When recover is complete an all actions allowed message will be generated similar to:
   

   2016-08-12 11:05:12.935/511.789 Oracle Coherence GE 12.2.1.0.0 <Warning> (thread=DistributedCache:PartitionedPofCache, member=1): All actions allowed
   2016-08-12 11:05:13.600/512.453 Oracle Coherence GE 12.2.1.0.0 <Warning> (thread=DistributedCache:PartitionedPofCache2, member=1): All actions allowed
   

The notification example can be used to watch for quorum based recovery notifications. In order to be notified of such operations, the notification recipient must be registered before the notifications are sent. 

To use the notfications example to watch for recovery notifications perform the following steps.

1. Shutdown/stop all running instances of Coherence and Coherence clients.
   
2. In one of the existing Coherence terminal windows execute:
   

   $ bin\run-cache-server.cmd persistence

3. In a second Coherence terminal window start the notifications example:
   

   $ bin\run.cmd persistence notifications

   When the window displays ------Waiting for notifications. CTRL-C to interrupt.------ continue to the next step.
4. In a third Coherence terminal window execute:
   

   $ bin\run-cache-server.cmd persistence

5. Return to the window where the notifications example is running and you should see messages similar to:
   

   Fri Aug 12 11:22:58 EDT 2016 : PartitionedPofCache (recover.begin) Active recovery in progress
   Fri Aug 12 11:22:59 EDT 2016 : PartitionedPofCache2 (recover.begin) Active recovery in progress
   Fri Aug 12 11:23:38 EDT 2016 : PartitionedPofCache2 (recover.end) Recovery Completed (Duration=38799ms)
   Fri Aug 12 11:23:38 EDT 2016 : PartitionedPofCache2 (recover.begin) Active recovery in progress
   Fri Aug 12 11:23:41 EDT 2016 : PartitionedPofCache (recover.end)  Recovery Completed (Duration=42924ms)
   Fri Aug 12 11:23:41 EDT 2016 : PartitionedPofCache2 (recover.end)  Recovery Completed (Duration=3451ms)
   Fri Aug 12 11:23:42 EDT 2016 : PartitionedPofCache (recover.begin) Active recovery in progress
   Fri Aug 12 11:23:42 EDT 2016 : PartitionedPofCache (recover.end)  Recovery Completed (Duration=512ms)

   You may notice that there are multiple begin/end notifications for the recovery. This occurs because recovery may require one or more passes to recover all data.
   
6. Leave the two instances of Coherence running, they will be reused in the next section.
   
   

To use Coherence Query tool, you can use the query[.sh|.cmd] files located in the coherence/bin directory. The examples come with a script to run CohQL, which is the equivalent to running query.sh. To run commands from a file you specify the -f option followed by the file name to execute. You can also use the -c option which will exit CohQL once the commands have executed.

1. Ensure that you have two Coherence cache servers running.
   
2. Create a file example.cohql which contains the following commands, notice the semi-colon ";" at the end of each line. For this tutorial the script name is assumed to be example.cohql.
   

   list services;
   create snapshot 'test' 'PartitionedPofCache';
   list snapshots;
   recover snapshot 'test' 'PartitionedPofCache';
   remove snapshot 'test' 'PartitionedPofCache';
   

3. Run the Coherence Query tool, but using the newly created command file using a command similar to that shown below:
   

   $ bin\run persistence cohql -f example.cohql

   Note that if the file is not in the same directory as where the command was run, enter the fully qualified path name.
   Which should produce results similar to:

   . . . 
   redirecting Coherence logging to persistence.log...
   Results
   Service Name         Mode   Quorum Policy                                                       Current
   -------------------- ------ ------------------------------------------------------------------- -------
   PartitionedPofCache  active allowed-actions=distribution,restore,recover,cache-read,cache-write Idle
   PartitionedPofCache2 active allowed-actions=distribution,restore,recover,cache-read,cache-write Idle
   Creating snapshot 'test' for service 'PartitionedPofCache'
   Results
   "Success"
   Results
   "PartitionedPofCache": ["contacts", "test"]
   "PartitionedPofCache2": ["test"]
   Recovering snapshot 'test' for service 'PartitionedPofCache'
   Results
   "Success"
   Removing snapshot 'test' for service 'PartitionedPofCache'
   Results
   "Success"
   Coherence Command Line Tool
   CohQL>

4. Using the byecommand exit CohQL.
   

CohQL supports controlling error behavior using:
WHENEVER COHQLERROR THEN (CONTINUE | EXIT);
When specified CohQL will either ignore the error and CONTINUE or EXIT without processing additional commands.
EXIT is useful when you want to validate a snapshot or archived snapshot before you archive or retrieve. 

To add error support to your script:

1. Open the previously created script in an editor.
   
2. Add an exit command command on error and a suffix to the create snapshot command which should result in an error and the termination of the script.
   The updated script should resemble:
   
   

   whenever cohqlerror then exit;
   list services;
   create snapshot 'test' 'PartitionedPofCacheXYZ';
   list snapshots;
   recover snapshot 'test' 'PartitionedPofCache';
   remove snapshot 'test' 'PartitionedPofCache'
   

3. In the Coherence Query Tool window execute the following.  Note the addition of -c which will cause the query tool to execute its commands and then exit.

   $ bin\run persistence cohql -f example.cohql

   Which should produce results similar those below.
   

   redirecting Coherence logging to persistence.log...
   Results
   Service Name         Mode   Quorum Policy                                                       Current Operation
   -------------------- ------ ------------------------------------------------------------------- -----------------
   PartitionedPofCache  active allowed-actions=distribution,restore,recover,cache-read,cache-write Idle
   PartitionedPofCache2 active allowed-actions=distribution,restore,recover,cache-read,cache-write Idle
   Service 'PartitionedPofCacheXYZ' does not exist

The default CohQL command timeout is 60s. Many commands, such as archiving or validating a snapshot, can take considerably longer to complete. To specify a longer timeout use the alter session command.

For example:
ALTER SESSION SET TIMEOUT [timeout in MS]

To explore timeouts:

1. Start the Coherence Query tool using a command similar to:
   

    $ bin\run persistence cohql 

   
   
2. Specify a very short timeout and then executing a long query:
   

   CohQL> alter session set timeout 300000
   

   Which should result in:
   

   Results
   "CohQL statement timeout set to 5m"
   

3. Reset the timeout to 10ms:
   

   CohQL> alter session set timeout 10
   

   Which should result in:
   

   Results
   "CohQL statement timeout set to 10ms"
   

4. Rerun the select statement:
   

   CohQL> select count() from "contacts"

   Which should show results similar to:

   Error: (Wrapped) null

5. Optionally, reset the session timeout to 60 seconds and re-run the query.
   

   CohQL> alter session set timeout 60000
   Results
   "CohQL statement timeout set to 1m"
   CohQL> select count() from "contacts"
   Results
   10000
   

6. Exit CohQL and the two Coherence cache instances running, they will be used in the next section.