Changes since 3.2.76
Log File On-Disk Format Changes:
The change is forward compatible in that JE files created with release 3.2.* and earlier can be read when opened with JE 3.3.62. The change is not backward compatible in that files created with JE 3.3.62 cannot be read by earlier releases. Note that if an existing environment is opened read/write, a new log file is written by JE 3.3.62 and the environment can no longer be read by earlier releases.
A number of these on-disk format changes were made to reduce disk-space usage. [#15399]
Also note the binary and compile time incompatibilities between JE 3.3 and JE 3.2 described in the API Changes section.
- JE can now be configured to use a shared cache among multiple environments. By using the shared cache, multiple open environments in the same process will make better use of memory because the cache LRU algorithm is applied across all information in all environments sharing the cache. See EnvironmentConfig.setSharedCache() for more information. Two related stats were also added to EnvironmentStats: getNSharedCacheEnvironments and getSharedCacheTotalBytes. [#15267]
- JE now provides a "temporary" database mode, specified through DatabaseConfig.setTemporary() and StoreConfig.setTemporary(). Temporary databases operate internally in deferred-write mode to provide reduced disk I/O and increased concurrency. But unlike an ordinary deferred-write database, the information in a temporary database is not durable or persistent, and therefore temporary databases are lighter weight than deferred-write databases. More details can be found in the Getting Started Guide and in the javadoc for DatabaseConfig and StoreConfig.
In addition, two internal changes have been made to durable deferred-write database behavior. These changes are compatible with the existing use of deferred-write database, but may influence performance characteristics. These changes were necessary to ensure that the log cleaner operates properly for deferred-write databases.
- The JE checkpointer now effectively performs a Database.sync for all open deferred-write databases. If you don't want this behavior you can a) use a Temporary DB instead, or b) disable the checkpointer. Disabling the checkpointer is commonly used to reduce logging during a bulk load.
- JE now calls Database.sync for deferred-write databases when they are closed. If the data is not intended to be durable, we recommend using a Temporary DB instead.
- Added a new EnvironmentConfig parameter: je.cleaner.upgradeToLogVersion. This parameter is used to perform background upgrading of earlier format JE log files. All log files having a log version prior to the specified version will be cleaned -- migrating the log records forward to the end of the log and updating their format -- at a time when no other log cleaning is necessary. This parameter is intended for use in upgrading earlier format log files forward to the current log format version, for example, to take advantage of disk usage improvements. Note that earlier format log files are supported by later JE releases and log upgrading is optional. [#15365]
- JE now supports key prefixing. When enabled, the keys stored on a given internal b-tree nodes may be represented as a set of prefix/suffix combinations where the prefix is the common prefix of all keys for the internal b-tree node. This can improve storage efficiency for some key sets, both on-disk and in-memory, at the cost of computational efficiency. The degree of storage improvement is a function of the application's key values and access pattern. Key prefixing can be enabled with the
DatabaseConfig.setKeyPrefixing(boolean) method. [#15399]
- Added the ability to control caching policy on a per-operation basis. See Cursor.get/setCacheMode() and com.sleepycat.je.CacheMode to see how calls to Cursor "get" and "put" operations can specify LRU, MRU, or "pinned" cache hotness. [#15850]
- New stats have been added to meter random vs sequential disk io.
return operation and byte counts for the number of random and sequential disk IOs. All values are approximate and may differ from the actual number of operations/byte-counts depending on the type of disks and file system, disk geometry, and file system cache size. [#16086]
- A new stat has been added to gauge the cost of log file opens, which can be a factor for applications with large data sets. A large value for
EnvironmentStats.getNFileOpens() can indicate a need to tune the size of the file handle cache through increasing the
je.log.fileCacheSize property. Also, general improvements have been made to improve the file handle cache, and to remove a bottleneck in concurrency in that area. [#16166]
General Environment Changes
- JE btree nodes now use shared latches for a portion of the btree access path. This used to be available through the je.env.sharedLatches configuration property, and is now the default. This results in better concurrency for read operations. The je.env.sharedLatches parameter has been deprecated. [#12136]
- In the past, JE's cache management pinned the internal metadata representing application databases. That made the use of large numbers of databases problematic, because significant portions of the cache could be consumed by metadata. That has now been resolved, and there are no cache or performance limitations on the number of databases in a JE environment. [#13415]
- The per-lock memory overhead has been reduced. [#15512]
- Various parameters have been adjusted so that JE will run under the Google Android platform. Refer to the Android HOWTO document for details.
- Fixed a bug in Transaction.setLockTimeout and setTxnTimeout so that the timeout is applied for all isolation modes. Before, the timeout was not applied for ReadCommitted and Serializable isolation under certain circumstances. The problem was initially reported on this JE Forum thread. [#16017]
- Fixed a bug where an internal data structure was being accessed sequentially during logging, while blocking other threads that attempt to write. For applications where the data set does NOT fit in cache, the data structure was small and the sequential access had little impact. But for applications where the entire data set DOES fit in cache, the sequential access was causing noticeable performance and concurrency problems during writing. [#16037]
- Improved performance for applications having a very large JE cache and where the active data set fits in the cache. Before, in this scenario, checkpoints could take a long time to occur. This in turn prevented log cleaning from being performed until after the end of the checkpoint. Two changes were made:
- Cleaner utilization information is now updated during the checkpoint. The log cleaner will no longer wait until the end of the checkpoint to perform cleaning. Note that although cleaning takes place during the checkpoint, log files are still not deleted until the end of the checkpoint.
- A new environment configuration parameter -- je.checkpointer.highPriority -- can be used to reduce the time to perform a checkpoint. If set to true, the checkpointer uses more resources in order to complete the checkpoint in a shorter time interval. Btree latches are held and other threads are blocked for a longer period. Log cleaner record migration is performed by cleaner threads instead of during checkpoints. A shorter checkpoint is beneficial because a shorter recovery period will be required in the event of a crash. However, application response time may be longer during the checkpoint, and more cleaner threads may be required to maintain the configured je.cleaner.minUtilization.
- Improved concurrency when a large number of log files (more than fit in the open-file-cache) are being accessed. Before, the latch on the cache was held while opening the log file and reading the header. Now, only the latch on the individual file is held. Note that the size of the open-file-cache is configured with the je.log.fileCacheSize Environment configuration parameter. Normally the cache can be configured to a size large enough to hold all files being accessed. However, for very large data sets, this may not be practical. [#16166]
- The EnvironmentStats.getTotalLogSize statistic is now a slow stat (not returned when StatsConfig.setFast(true) is called) and a bug was fixed that caused the total to be incorrect and overflow in some cases. [#15985]
- Fixed a bug where lock timeouts (DeadlockException) would sometimes occur after twice the configured lock timeout interval. This fix may cause DeadlockException to occur more frequently in some applications because the configured lock timeout is now applied correctly. Note that the lock timeout is 0.5 seconds by default and may be configured via the setLockTimeout method in EnvironmentConfig and Transaction. [#16021]
Direct Persistence Layer (DPL), Collections and Bind packages
- Several validation checks have been added or corrected having to do with entity subclasses, which are @Persistent classes that extend an @Entity class.
- An IllegalArgumentException is now thrown when @PrimaryKey appears on an entity subclass. Before this was documented to be illegal but was not checked. [#15757]
- The DPL now throws IllegalArgumentException if an instance of an entity subclass is embedded in another entity. It has always been illegal to embed entities within other entities, but the check for entity subclasses was not correct and has been fixed. [#16077]
- In certain circumstances a NullPointerException was thrown before when attempting to embed an entity subclass, and this has also been corrected to throw IllegalArgumentException. [#16077]
- Optimize DPL marshaling for large numbers of embedded objects. Before, a sequential lookup was used to manage the object graph. Now, a HashMap is used. Many thanks to user624180 who suggested this in a forum post. [#16198]
- Fix a DPL bug that caused exceptions when using a class Converter for an instance containing non-simple fields. The problem was reported in this forum thread. Thanks to Stan Livitski for submitting a reproducible test case. [#16233]
- The StoredMap class now implements the Java 1.5 ConcurrentMap interface. Thanks to jahlborn for proposing this change in this OTN forum thread. [#16218]
- Add Java 1.5 generic type parameters where appropriate: EntryBinding, EntityBinding and all binding implementations; all references to the Map and Collection interfaces; etc.
- Deprecated the EnvironmentStats.getCacheDataBytes method. The new getDataBytes should be used instead. The getLockBytes method now includes the lock and transaction overhead. Before it returned only the lock overhead. [#15267]
- A number of fields in EnvironmentStats and TransactionsStats were changed from int to long to prevent overflows, which resulted in a change to the signature of a number of getXXX() methods. Regrettably, this constitutes an incompatible binary and compile time change from JE 3.2 to JE 3.3 for applications which used those getter methods. There is no incompatibility if your application merely instantiates an EnvironmentStat or TransactionStat or calls their toString() methods. [#15979]
- JE now requires Java 5 and uses generic types in method signatures. Most notably for the user, the methods which define custom comparators have been changed to these signatures:
DatabaseConfig.setBtreeComparator(Class<? extends Comparator<byte>>)
DatabaseConfig.setDuplicateComparator(Class<? extends Comparator<byte>>) If the application used a custom comparator defined as
that should now be declared as
public class MyCompare implements Comparator, Serializable
Please note that while these are fine:
public class MyCompare implements Comparator<byte>, Serializable
Comparator<byte> compareInstance = new MyCompare();
the following previously legal line will now provoke a compile error:
The compile error is puzzling and regrettably breaks compile time compatibility with JE 3.2 and earlier, but comes about because in Java, due to its type erasure based generics scheme, all the instances of a generic class have the same runtime class. Instead, the application has to apply the following cast:
dbConfig.setBtreeComparator((Class<? extends Comparator<byte>>) compareInstance.getClass());
- Database salvage mode is now available in the API through the com.sleepycat.je.util.DbScavenger class rather than solely from the DbDump -r command line. This helped to fix and clarify a mismatch between the Javadoc and the constructor parameters for DbDump in the JE 3.2.X code line. See the com.sleepycat.je.util.DbScavenger class. [#15385]
Configuration, Documentation, Test and Build Changes:
- JE 3.3 requires Java 220.127.116.11 or higher.
- Ant 1.7.0 is now required to build JE 3.3.
- In the past, the javadoc for JE's public classes was generated through an internal documentation process, and the actual content of the comments was not in the source itself. This was inconvenient for IDE users, because the javadoc contents would often not display in convenient places. This has been changed, and the javadoc content now resides in the source code in the normal fashion.
- Java developers often approach JE with past experience using SQL. To help illustrate the JE data model and API, we created a new example program which takes a list of common SQL queries and implements them using the Direct Persistent Layer API (DPL). Please refer to the Translating SQL Queries Example in JE Installation Notes for more information.
- The example.properties file is no longer used to document the configuration parameters that are used with EnvironmentConfig and EnvironmentMutableConfig. Instead, String constants have been added EnvironmentConfig which include that documentation. When calling setConfigParam these constants can be used to avoid hard-coding the parameter name. See EnvironmentConfig for details. [#16227]