FAQ - Oracle NoSQL Database

1. General
   1. What are all these ports? When do I use which one?
   2. How do I set NoSQL DB parameters? What are policy parameters?
      1. Viewing current parameter values
      2. Changing parameters in the Admin Console
      3. Changing parameters in the CLI
      4. Policy
      5. Admin parameters
      6. Storage Node parameters
      7. Replication Node parameters
   3. How to apply a NoSQL DB update on an existing deployment
   4. How many Admin services should I deploy?
   5. How to create multiple Storage Nodes on the same machine
   6. How do I script NoSQL DB configuration?
   7. What is kvctl and why is it deprecated?
2. API
   1. How to efficiently delete a subtree of keys using multiple major keys
   2. Best practices when generating sequence values
   3. Are there any plans to support other languages beside Java
3. Troubleshooting
   1. What JVMs Can I Use to Run Oracle NoSQL Database?
   2. Why am I seeing an UnsupportedClassVersionError exception?
   3. Getting feedback while an administrative plan is running
   4. Where to find error information
   5. What is the format of the .log files?
   6. What is the format of the .perf files?
   7. What are the NoSQL DB service states?
   8. Why is my plan still running after an error has been reported?
   9. I configured my Storage Node registry port incorrectly
   10. I configured my Storage Node HA port range incorrectly

General

What are all these ports? When do I use which one?

A store deployment consists of many components, each running in its own process, which communicate with each other. The system administrator must define TCP/IP ports for each type of process on each Storage Node (SN).

1. The main port on each SN is the "registry port", which is owned by the Storage Node Agent (SNA). This is the port that is named by the "-port" argument to the java -jar kvstore.jar makebootconfig command. It is also used in the deploy-sn plan during configuration.

The documentation examples use 5000 as the registry port. Also, the run-kvlite.sh script uses port 5000 as a default value.

2. Each Admin process must have a port (the "Admin Console port") on which it listens for HTTP connections. This is specified in the "-admin" argument to the java -jar kvstore.jar makebootconfig command, and in the deploy-admin plan during configuration.

The documentation examples use 5001.

3. Finally, each SN must be given a range of ports (the "HA Range") to be used by any Replication Nodes hosted within the SN. The SNA manages this allotment of ports, and allocates one to each HA node on the SN.

You should use the registry port in the following situations:

• any use of the admin Command Line Interface (the java -jar kvstore.jar runadmin command), including initial configuration of the store. Note that although you're ultimately communicating with an Admin process, it is the Registry Port, not the Admin Console port, that you use in this situation.

• as part of the helperHostPort argument to the KVStoreConfig constructor in application code that will read/write to the store.

• as the value passed on the command line with "-port", to invoke utilities such as Load, Ping, and KVLite.

Similarly with example programs such as HelloBigDataWorld and SchemaExample.

You should use the Admin Console port in the target URL for the admin web UI:

    http://node1.example.com:5001/

Generally you can specify a port on any SN that hosts a component of the type you are trying to reach, and if necessary the system will redirect your request automatically and transparently. In the case of the Admin Console web UI, and the java -jar kvstore.jar runadmin command, this works as long as you specify an SN that is hosting an admin process.

Other than during configuration of an SN, there is normally no need to refer directly to ports in the HA Range, although it is certainly helpful to understand the HA Range when examining Ping results and perusing system log files.

How do I set NoSQL DB parameters? What are policy parameters?

NoSQL DB includes three types of services: Admin, Storage Node and Replication Node; each of which has a several configuration parameters. Some parameters must be supplied when the services are deployed, while others can be tweaked after deployment.

Viewing current parameter values

Parameters can be viewed and changed through the command line interface KVAdmin (also known as "java -jar kvstore.jar runadmin" or simply "the CLI"), or through the web-based Admin Console. In all cases, to change parameters, you first create a plan representing the change you want to make, then approve and execute the plan.

You can view current Parameter values using the Topology tab in the Admin Console. Click on the service instance of interest in the Topology display, and its current parameters will be shown in the right-hand panel.

To view parameter settings in the CLI, use one of the following commands:

show admin-params <adminId>
show repnode-params <groupId,nodeId>
show storagenode-params <storageNodeId>

Changing parameters in the Admin Console

You can use the Admin Console to change parameters in the "Plans & Configuration" tab. Choose the plan you wish to create from the menu labeled "Create a Plan". The plans for changing parameters are selected by menu items "Change StorageNode Parameters", "Change RepNode Parameters", and "Change Admin Parameters". Once you have selected a plan from the menu, the appropriate form will appear.

When changing Storage Node and Replication Node parameters, you must then choose from the presented menu the service's instance to which you want the changes to apply. When changing Replication Node parameters, you can choose to apply a change to a single Replication Node, or to ALL Replication Nodes.

When changing Admin parameters, the change will apply to ALL Admin instances.

The form thus displayed will show all the mutable parameters associated with the selected service, which can be edited in their text boxes. Click the button labeled "Create Plan" (or "Create and Execute Plan") to proceed.

During the execution of a parameter-changing plan, depending on which parameters are changed, services may be automatically restarted. If an Admin parameter change requires restarting of the Admin service, the Admin Console will lose its connection to the server. It may be necessary for you to click the browser's refresh button to re-connect to the Admin service. At this point the plan will be in the "interrupted" state, and must be completed manually by choosing "retry" in the plan's action menu.

Changing parameters in the CLI

All of the CLI commands used for creating parameter-changing plans share a similar syntax:

plan change-xxx-params [ParameterName=NewValue]...

All such commands can have multiple ParameterName=NewValue assignment arguments on the same command line. If NewValue contains spaces, then the entire assignment argument must be quoted within double quote marks. For example, to change the Admin parameter collectorPollPeriod, you would issue the command

kv-> plan change-all-admin-params "collectorPollPeriod=20 SECONDS"

The following commands are used to change service parameters:

plan change-repnode-params <groupId,nodeId> [assignments]

This command is used to change the parameters of a single Replication Node, which must be identified using the group and node numbers. The groupId,nodeId identifier must be given as a single argument with one embedded comma and no spaces.

plan change-all-repnode-params [assignments]

This command is used to change the parameters of ALL Replication Nodes in a store. No Replication Node identifier is needed in this case.

plan change-storagenode-params <storageNodeId> [assignments]

This command is used to change the parameters of a single Storage Node instance. The storageNodeId is a simple integer.

plan change-all-admin-params [assignments]

This command is used to change Admin parameters. Because each instance of Admin is part of the same replicated service, all instances of the Admin are changed at the same time, so no Admin identifier is needed in this command.

If an Admin parameter change requires the restarting of the Admin service, KVAdmin will lose its connection to the server. Under normal circumstances, KVAdmin will automatically re-connect after a brief pause, when the next command is given. At this point the plan will be in the "interrupted" state, and must be completed manually by issuing the "retry" command.

In all cases, you can choose either to create a plan, then approve and execute it in separate steps; or to create the plan and execute it in a single step by using the -execute option of the plan command.

Policy

Most Replication Node parameters are assigned to default values when a store is deployed. It can be inconvenient to adjust them after deployment, so NoSQL DB provides a way to set the defaults that are used during Replication Node deployment. These defaults are called Policy parameters.

You can set policy parameters in the CLI by using this command:

set policy [assignments]

You can set them in the Admin Console by choosing "Change Policy Parameters" from the main menu on the "Plans & Configuration" tab.

Admin parameters

The following parameters can be set on the Admin service:

adminLogFileCount=<Integer>

Sets the number of log files that are kept.

adminLogFileLimit=<Integer>

Limits the size of log files. After reaching this limit, the logging subsystem switches to a new log file.

latencyCeiling=<Integer>

If any Replication Node's latency exceeds this value, it is considered an "alertable" event. Such an event will produce a popup in the Admin Console, and it will be stored in the Admin's database as a critical event.

throughputFloor=<Integer>

Similar to latencyCeiling, throughputFloor sets a lower bound on Replication Node throughput. Lower throughput reports are considered alertable.

configProperties=<String>

This is an omnibus string of property settings for the underlying BDB JE subsystem. Its format is "property=value;property=value...".

collectorPollPeriod=<Long TimeUnit>

Sets the Monitor subsystem's delay for polling the various services for status updates.

javaMiscParams=<String>

This is an omnibus string that is added to the command line when the Admin process is started. It is intended for setting Java VM properties.

loggingConfigProps=<String>

Property settings for the Logging subsystem. The format of this string is like that of configProperties, above.

eventExpiryAge=<Long TimeUnit>

You can use this parameter to adjust how long the Admin stores critical event history. The default value is "30 DAYS".

Storage Node parameters

serviceLogFileCount=<Integer>

Sets the number of log files that are kept, for this Storage Node and for all Replication Nodes hosted on this Storage Node.

serviceLogFileLimit=<Integer>

Limits the size of log files. After reaching this limit, the logging subsystem switches to a new log file. This setting applies to this Storage Node and to all Replication Nodes hosted on this Storage Node.

haPortRange=<String>

Defines the range of port numbers available for assigning to Replication Nodes that are hosted on this Storage Node. A port is allocated automatically from this range when a Replication Node is deployed. The format of the value string is "lowport,highport".

haHostname=<String>

Sets the name of the network interface used by the HA subsystem.

Replication Node parameters

collectEnvStats=<Boolean>

If true, then the underlying BDB JE subsystem will dump statistics into the .stat file.

maxTrackedLatency=<Long TimeUnit>

The highest latency that will be included in the calculation of latency percentiles.

configProperties=<String>

Contains property settings for the underlying BDB JE subsystem. Its format is "property=value;property=value...".

javaMiscParams=<String>

A string that is added to the command line when the Replication Node process is started. It is intended for setting Java VM properties.

loggingConfigProps=<String>

Contains property settings for the Logging subsystem. The format of this string is like that of configProperties, above.

statsInterval=<Long TimeUnit>

Sets the collection period for latency statistics at this Replication Node.

cacheSize=<Long>

Sets the cache size in the underlying BDB JE subsystem.

How to apply a NoSQL DB update on an existing deployment

Oracle will inevitably release NoSQL DB software updates to add features and fix bugs. This entry describes how a user can install these (future) updates. NoSQL DB releases will be compatible with older releases from a protocol standpoint, thereby allowing a mix of old and new releases (at least adjacent releases) to coexist during the upgrade process. Note that it may be possible for a particular release to require a more extensive upgrade procedure than this note describes.

When NoSQL DB is first installed, it is placed in a KVHOME directory, which may be per-machine, or optionally be shared by multiple Storage Nodes (e.g. via NFS). In this note we refer to the existing KVHOME as OLD_KVHOME. It may be useful for installations to adopt a convention for KVHOME that includes the release number, e.g. always use a KVHOME like /var/kv/kv-x.y.z where "x.y.z" are the release.major.minor numbers. This can be easily achieved by simply unzip/untarring the distribution into a common directory (/var/kv in this example).

Installing new software requires that each node be restarted. NoSQL DB is a replicated system, so to avoid excessive failover events it is recommended that any node that is running as a MASTER be restarted after all those marked REPLICA. This command will tell you which nodes are MASTER and REPLICA:

java -jar kvstore.jar ping -host <hostname> -port <port>

Use the host and registry port for any active node in the store.

To upgrade to new software, for each node (machine) in the system do the following steps. They should be done serially to minimize possible performance impact:

1. Load new software in a new KVHOME, called NEW_KVHOME. If nodes share this directory via NFS this only needs to be done once for each shared directory
2. If you have configured the node to automatically start the Storage Node Agent on reboot via /etc/init.d, Upstart, or some other mechanism (e.g. using "java -jar KVHOME/lib/kvstore.jar start -root <kvroot> ..." ) first modify that script to point to NEW_KVHOME.
3. For each KVROOT (generally one per node):

Stop the storage node using the old code

 java -jar OLD_KVHOME/lib/kvstore.jar stop -root <kvroot> [-config <configfile>]

Restart the storage node using the new code

 java -jar NEW_KVHOME/lib/kvstore.jar start -root <kvroot> [-config <configfile>]

When done, OLD_KVHOME can be removed, first making sure that any administrative scripts or other files that reference OLD_KVHOME have been changed.

How many Admin services should I deploy?

The NoSQL DB Administrative processes are replicated in order to enhance reliability. It's important to continue to have administrative functionality available even in the event of node failure, so that you can continue to monitor, troubleshoot, and repair problems.

Additional Admin services are created and added to NoSQL DB though the deploy-admin plan. The information provided in the Admin Guide's "Identify your Replication Factor" section also applies to the number of Admin services that should be deployed. In general, we suggest that you deploy at least three Admin services.

How to create multiple Storage Nodes on the same machine

It is strongly recommended that Storage Nodes (SNs) be allocated one per machine in a production system for availability and performance reasons. If more than one SN is hosted on the same machine, multiple SNs are lost if that machine fails and data may become inaccessible. For early prototyping and testing, however, it can be useful to create multiple SNs on the same machine.

On a single machine a Storage Node is uniquely identified by its root director (KVROOT) plus a configuration file name, which defaults to "config.xml." This means you can create multiple SNs in two ways.

1. Create a unique KVROOT for each SN. For example, if you decide to put all SNs in the directory /var/kv/stores you might create and start more than one this way:

   mkdir /var/kv/stores/root1
   mkdir /var/kv/stores/root2
   mkdir /var/kv/stores/root3
   java -jar kvstore.jar makebootconfig -root /var/kv/stores/root1 -port 5000 
      -admin 5001 -harange 5005,5010
   java -jar kvstore.jar makebootconfig -root /var/kv/stores/root2 -port 5020 
      -harange 5025,5030
   java -jar kvstore.jar makebootconfig -root /var/kv/stores/root3 -port 5040 
      -harange 5045,5050
   java -jar kvstore.jar start -root /var/kv/stores/root1
   java -jar kvstore.jar start -root /var/kv/stores/root2
   java -jar kvstore.jar start -root /var/kv/stores/root3
   

2. Share a KVROOT but create a unique configuration file name for each SN. The default configuration file name for a Storage Node is "config.xml" but it can be changed with command arguments. In this example 3 SNs share the same KVROOT, /var/kv/root

   mkdir /var/kv/root
   java -jar kvstore.jar makebootconfig -root /var/kv/root -port 5000 
      -admin 5001 -harange 5005,5010 -config config1.xml
   java -jar kvstore.jar makebootconfig -root /var/kv/root -port 5020 
      -harange 5025,5030 -config config2.xml
   java -jar kvstore.jar makebootconfig -root /var/kv/root -port 5040 
      -harange 5045,5050 -config config3.xml
   java -jar kvstore.jar start -root /var/kv/root -config config1.xml
   java -jar kvstore.jar start -root /var/kv/root -config config2.xml
   java -jar kvstore.jar start -root /var/kv/root -config config3.xml
   

Both ways are fine for the system. A possible advantage of the former is to make it easier to put different SNs on different file systems. An advantage to the latter is that all log files for the store end up in the same directory if all SNs are used for the same store (KVROOT/<storename>/log).

How do I script NoSQL DB configuration?

You may find that you want to build the same NoSQL DB configuration repeatedly for testing purposes. The Admin CLI commands can be scripted in several ways.

Many uses of the Admin CLI are simple commands, such as java -jar kvstore.jar makebootconfig to initially configure a StorageNode, shown above. These are as amenable to scripting as any other UNIX commands and will not be discussed further here.

The interactive commands available in java jar kvstore.jar runadmin, among which are those used to create and execute plans, can be scripted in two ways. You can create a file containing the sequence of commands that you want to run, and run them in a batch using java -jar kvstore.jar runadmin -script.

For example, a script file named deploy.kvs could contain commands such as the following:

  configure mystore
  plan -execute deploy-datacenter boston
  plan -execute deploy-sn 1 localhost 5000
  plan -execute deploy-admin 1 5001

You could execute this script by issuing the command

  java -jar kvstore.jar runadmin -host localhost -port 5000 -script deploy.kvs

Another way to script these commands would be to use the -adminargs option to java -jar kvstore.jar runadmin. In this mode, each command becomes a separate invocation of the Admin CLI This allows you to use features of a more capable scripting language, such as a UNIX shell, and gives more flexibility for integrating these commands with other commands that are not available in the the interactive java -jar kvstore.jar runadmin environment.

For example, the same sequence of commands as in those above could be couched in a shell script:

  #!/bin/sh

  HOST=localhost
  PORT=5002
  HTTPPORT=5003

  java -jar kvstore.jar runadmin -host $HOST -port $PORT 
     -adminargs "configure mystore"
  java -jar kvstore.jar runadmin -host $HOST -port $PORT 
     -adminargs "plan -execute deploy-datacenter boston"
  java -jar kvstore.jar runadmin -host $HOST -port $PORT 
     -adminargs "plan -execute deploy-sn 1 localhost $PORT"
  java -jar kvstore.jar runadmin -host $HOST -port $PORT 
     -adminargs "plan -execute deploy-admin 1 $HTTPPORT"

What is kvctl and why is it deprecated?

In Oracle NoSQL DB 1.2.123, the two scripts for starting kvlite and executing control commands, bin/run-kvlite.sh and bin/kvctl, have been replaced by a java -jar lib/kvstore.jar command. This provides portability to all Java platforms, including Windows. The two scripts are deprecated, but will be supported for at least one release cycle.

The translation from the old script commands to the new -jar commands is as follows:

API

How to efficiently delete a subtree of keys using multiple major keys

The KVStore.multiDelete() methods can delete entire subtrees, but only on one (fully-specified) major key at a time.

In order to delete larger subtrees, rooted higher up within the major-key portion of the hierarchy, you can use a storeKeysIterator to find each full major key to be deleted, and then call multiDelete() on each one.

If the storeKeysIterator were allowed to finish, it would iterate over all keys in the desired subtree range, including minor keys within each major key. Therefore, the trick is to take just the first result key from the iterator, extract the major key portion for the call to multiDelete(), and then abandon the iterator and start over with a new one; rather than cycling all the way through a single iterator.

This is illustrated in the following sample code.

Key partialKey = ...;     // major key prefix

/*
 * Since we never take more than one key per iterator, a larger 
 * batch size would be a waste.
 */
final int batchSize = 1;
for (;;) {
    Iterator<Key> i = kv.storeKeysIterator
        (Direction.UNORDERED, batchSize, partialKey,
         null, Depth.DESCENDANTS_ONLY);
    if (!i.hasNext()) {
        break;
    }
    Key descendant = Key.createKey(i.next().getMajorPath());
    kv.multiDelete(descendant, null,
                   Depth.PARENT_AND_DESCENDANTS);
}

Best practices when generating sequence values

A NoSQL DB application can implement sequences in a fairly simplistic way by designating a record in the store to be the sequence. Simply pick a single key that identifies the sequence. The key can be anything the application chooses, for example, "app-sequence-1". The value for this record will be a long integer, which is the next available value in the sequence.

Use a read-modify-write operation to read the next available sequence, and increment it. Use KVStore.get() to read the value and putIfVersion() to write it. Like any read-modify-write operation, if putIfVersion() fails this means another client updated it, and the read-modify-write cycle should be retried.

Rather than incrementing the sequence by 1, increment it by a much larger value, say 500. This effectively assigns a block or range of sequence values to the client. The client should then "cache" this range of sequence numbers and assign record keys from this range. When the cached values have all been assigned, another read-modify-write operation should be performed.

Although the single sequence record will be a single point of contention among clients, this potential performance issue can be addressed by assigning sequences in large blocks as described. You could also allocate multiple independent sequences for different purposes, such as for different types of records or for key ranges.

Back to top
 

Are there any plans to support other languages beside Java

Troubleshooting

What JVMs Can I Use to Run Oracle NoSQL Database?

At this time, only Java SE 6 (jdk 1.6.0 u25) or later is supported.

Why am I seeing an UnsupportedClassVersionError exception?

Oracle NoSQL Database requires a Java SE 6 or later java runtime. If you see an error message like the following, you are probably trying to run it with an earlier version of Java.

Exception in thread "main" java.lang.UnsupportedClassVersionError:
    Bad version number in .class file

Getting feedback while an administrative plan is running

There are several ways to track the progress of an administrative command.

• The Topology tab of the Admin Console will refresh as NoSQL DB services are created and brought online.
• You can issue the verify command via the Topology tab or the Admin CLI concurrently as plans are executing. The verify will provide service status information as services come up.
• You can follow the storewide log via the Logs tab of the Admin Console or the CLI's logtail command.

Where to find error information

Error information for administrative plans is available as the plan executes and finishes. Errors are reported in the plan history each time an attempt to run the plan fails. The plan history can be seen via the command line interface KVAdmin (also known as "java -jar kvstore.jar runadmin" or simply "the CLI") show plan <planId> command, or in the Plans And Configuration tab of the Admin Console.

Other problems may occur asynchronously. You can learn about unexpected failures, service downtime, and performance issues through the critical events display in the Logs tab of the Admin Console, or through the CLI's show events command. Events come with a time stamp, and the description may contain enough information to diagnose the issue. In other cases, more context may be needed, and the administrator may want to see what else happened around that time.

The storewide log consolidates logging output from all services. Browsing this file may give the administrator a more complete view of activity during the problem period. It can be viewed via the Logs tab of the Admin Console or the CLI's logtail command, or by directly viewing the <storename>_N.log file in the <KVHOME>/<storename>/log directory. It's also possible to download the storewide log file using the Logs tab of the Admin Console.

What is the format of the .log files?

The .log files in the KVROOT/log directory contain trace messages from each NoSQL DB component. Each line is prefixed with the date of the message, its severity, and the name of the component which issued it. For example:

10-01-11 08:39:43:548 EDT INFO [admin1] Initializing Admin for store: kvstore

The line format is MM-dd-yy HH:mm:ss:SSS <java.util.logging.Level> [Component name] message. When looking for more context for events at a given time, use the timestamp and component name to narrow down the section of log to peruse. Serious exceptions are logged with the SEVERE log level, as well as being available in the Admin Console and CLI (java -jar kvstore.jar runadmin) show events command.

What is the format of the .perf files?

The administrative service collects server side throughput and latency statistics on a per-replication node basis. These statistics can be seen via the Admin Console's topology tab, the command line interface (CLI) show perf command, and in the <storename>.perf files available in the master Admin service's KVROOT log directory. The .perf files can also be located and downloaded via the Admin Console.

Throughput and latency are calculated for an interval of time determined by the statsInterval replication node parameter, which defaults to 10 seconds. Each line of the .perf file is for a single stats interval of a single replication node. Each line also applies to a single optype category.

The two optypes, Single and Multi, are used to understand the performance characteristics of different data operations. Some client API data operations pertain to a single data record, while others pertain to one or multiple records. For example, the KVStore.get() method fetches a single data record, while KVStore.multiGet() fetches one or more records. All single record operations are aggregated and recorded as optype Single, while all multi record operations are reported as optype Multi.

Latency information is calculated per data record. Since multi operations can amortize the per-operation overhead over many records, they can be significantly more efficient than single record operations. Statistics are kept per optype to help illumnate that difference.

For example, if the application issues KVStore.get(), KVStore.put(), KVStore.putIfVersion() and KVStore.multiGet() operations in one interval, all the multiGets are recorded as one Multi statistic line, and all the other operations are recorded together as one Single statistic line.

The following columns are present on each line:

Resource: the name of the replication node
Time: the start of the stats interval, formatted as MM/HH/YY hh:mm:ss
OpType: Single or Multi

Interval performance stats:

Total Ops: number of operations of that optype executed during the stats interval
PerSec: number of operations per second for that interval
Min: minimum latency, in milliseconds for that interval
Max: maximum latency, in milliseconds for that interval
Avg: average latency, in milliseconds for that interval
95th: latency of operations in the 95th percentile, for that interval
99th: latency of operations in the 99th percentile, for that interval

Cumulative performance stats:

Total Ops: number of operations of that optype executed during the lifetime of the replication node. Cumulative values are reset any time the replication node restarts
PerSec: number of operations per second the RN lifetime
Min: minimum latency, in milliseconds for the RN lifetime
Max: maximum latency, in milliseconds for the RN lifetime
Avg: average latency, in milliseconds for the RN lifetime
95th: latency of operations in the 95th percentile, for the RN lifetime
99th: latency of operations in the 99th percentile, for the RN lifetime

What are the NoSQL DB service states?

There are three types of NoSQL DB services: Admin, Storage Node, and Replication Node. Each service has a status that can be seen via the Topology tab in the Admin Console, the show topology command in the CLI (java -jar kvstore.jar runadmin), or java -jar kvstore.jar ping.

The status values are:

A healthy service begins in STARTING. It may transition to WAITING_FOR_DEPLOY for a short period before going on to RUNNING. ERROR_RESTARTING and ERROR_NO_RESTART indicate that there has been a problem that should be investigated. An UNREACHABLE service may only be in that state temporarily, although if that state persists, the service may be truly in an ERROR_RESTARTING or ERROR_NO_RESTART state.

Note that the Topology tab only shows abnormal service statuses. A service that is RUNNING will not display its status in that tab.

Why is my plan still running after an error has been reported?

Administrative plans may invoke remote services which execute asynchronously. Each plan step, or task, can run for a maximum amount of time which is configured by the task_timeout parameter, which defaults to 5 minutes. The task ends when it determines a success or error outcome, or if the timeout period is exceeded.

The asynchronous remote services may encounter errors which are reported directly back to the Admin Console and are displayed in a error dialog before the task has determined the outcome. Because of that, the user may learn of the error while the Admin service still considers the plan to be RUNNING and active. The plan will eventually see the error and will transition to an ERROR state.

I configured my Storage Node registry port incorrectly

If you have specified an invalid value for a Storage Node (SN) registry port, the Storage Node Agent will not start up properly. You'll be unable to see the SNA through the jps -m command, and it will not respond to the java -jar kvstore.jar ping command. The snaboot_0.log file in the SN's root directory will display error information. For example, if the registry port was already in use, the log might show:

10-03-11 22:47:59:525 EDT SEVERE [snaService] Failed to start SNA: Port already 
   in use: 1000; nested exception is: java.net.BindException: Permission denied

I configured my Storage Node HA port range incorrectly

If you have specified invalid values for the HA Port Range (described above in the FAQ), you will be unable to deploy a Replication Node (RN) or a secondary Administration process (Admin) on this SN. The problem will be discovered when you first attempt to deploy a store or a Admin Replica on that faulty SN. You will see these indications that the RN did not come up on this Storage Node:

• The Admin Console will display an error dialog that warns that this RN is in the ERROR_RESTARTING state. The Topology tab will also show this state in red, and after a number of retries, will indicate that the RN is in ERROR_NO_RESTART..
• The plan will go into ERROR state, and its detailed history, available by clicking on the plan in the Plans and Configuration tab of the Admin Console or through the java -jar kvstore.jar runadmin "show plan <planId>" command, will show an error message like this:

     Attempt 1
          state: ERROR
          start time: 10-03-11 22:06:12
          end time: 10-03-11 22:08:12
          DeployOneRepNode of rg1-rn3 on sn3/farley:5200 [RUNNING] failed. ....  
            Failed to attach to RepNodeService for rg1-rn3, see log, 
            /KVRT3/<storename>/log/rg1-rn3*.log, on host farley for more 
            information.
  

• The critical events mechanism, accessible via the Admin Console or CLI will show an alert that contains the same error information from the plan history.
• An examination of the specified .log file or the storewide log displayed in the Log tab of the Admin Console will show a specific error message, such as:

  [rg1-rn3] Process exiting
  java.lang.IllegalArgumentException: Port number 1 is invalid because the port
     must be outside the range of "well known" ports
  

The misconfiguration may be addressed with these steps. Some steps must be executed on the physical node which hosts the NoSQL DB Storage Node, while others can be done from any node which can access the Admin Console or Admin CLI.

1. Using the Admin Console or Admin CLI, cancel the deploy-store or deploy-admin plan which ran afoul of the misconfiguration.
2. On the SN node, kill the existing, misconfigured StorageNodeAgentImpl process and all its ManagedProcesses. You can distinguish them from other processes because they will have the parameter "-root <KVROOT>"
3. On the SN node, remove all files from the KVROOT directory.
4. On the SN node, recreate the storage node bootstrap configuration file in the KVROOT directory using the java -jar kvstore.jar makebootconfig command.
5. On the SN node, restart the storage node using java -jar kvstore.jar start.
6. Using the Admin Console or Admin CLI, re-deploy the storage node using the deploy-sn plan.

You have now returned to the same point where you previously experienced the error, and you can create and execute a deploy-store or deploy-admin plan which uses the same parameters as the initial attempt.