Berkeley DB Java Edition 3.3.87 Change Log

Log File On-Disk Format Changes:

The change is forward compatible in that JE files created with release 3.2 and earlier can be read when opened with JE 3.3. The change is not backward compatible in that files created with JE 3.3 cannot be read by earlier releases. Note that if an existing environment is opened read/write, a new log file is written by JE 3.3 and the environment can no longer be read by earlier releases.

A number of these on-disk format changes were made to reduce disk-space usage. [#15399]

Also note the binary and compile time incompatibilities between JE 3.3 and JE 3.2 described in the API Changes sections.

The changes between 3.2 and 3.3.87 are described below. The changes are listed separately for each patch release, with the most recent patch release at the top.

Changes in 3.3.87

General Environment Changes

  1. Fix a bug that would allow the Total Memory Usage stat to become negative. This could prevent eviction from occurring, which could cause OutOfMemoryError for some applications. It is most likely to occur in an application with a large cache and frequent checkpoints, but it could occur in other scenarios as well. [#17462]

  2. Fix a bug that caused a NullPointerException during eviction when an exception occurs during logging, for example, an IOException when the disk is full. The NullPointerException masked the original exception (e.g., IOException). An example stack trace is below.
    Thanks to tkram01 for reporting this on OTN. [#17551]

  3. EnvironmentConfig.COMPRESSOR_PURGE_ROOT has been deprecated. This parameter no longer has any effect. Compression of the root node has no benefit and this feature has been removed. This feature was also bug-prone, as reported by penemue on OTN. [#17546]

  4. Fix a bug that could cause OutOfMemoryError when many records are read in a read-only environment, and the data set that is accessed does not fit in available memory. This applies to two types of applications:
    • User applications that open the Environment with the read-only ( EnvironmentConfig.setReadOnly(true)) property.
    • The JE utilities that open the Environment read-only: DbVerify, DbStat, and DbDump if neither the -r nor -R switch is specified.
    Thanks to user8694068 for reporting this on OTN and working with us to diagnose the problem. [#17590]

Changes in 3.3.75

General Environment Changes

  1. Fix a bug that somtimes causes two log cleaner threads (when multiple threads are configured or multiple app threads call Environment.cleanLog) to select the same file to clean. This can result in two attempts to delete the log file after cleaning. The second attempt causes an exception to be traced repeatedly, but no other side effect. The problem is transient and goes away if the Environment is closed and re-opened. An example of the exception is below. The symptom is that this exception occurs repeatedly, and the file is not present.
    2009-04-27 17:27:09:132:MEST SEVERE Cleaner deleteSafeToDeleteFilesLog file 0xab could not be deleted.
    This operation will be retried at the next checkpoint (JE 3.3.x) Couldn't delete /envHome/000000ab.jdb
    [#17079] (3.3.81)

  2. Fix a bug that could cause a LogFileNotFoundException when using an XAEnvironment, if a prepared transaction is not ended prior to a crash and then the prepared transaction is aborted after recovering from a crash. [#17022] (3.3.79)

  3. Fix a bug that prevented deferred-write record deletions from being made durable by Database.sync, if a crash occurs after Database.sync but prior to the next checkpoint. Under rare circumstances this could also result in a LogFileNotFoundException later when accessing the deleted entry. [#16864] (3.3.78)

  4. Fix a bug that prevents log cleaning from functioning properly when a temporary DB (DatabaseConfig.setTemporary) is large enough to overflow the JE cache. Also fix a bug that could in rare cirumstances cause an endless loop while performing log cleaning. [#16928] (3.3.78)

  5. Fix a bug that caused an infinite loop under certain timing dependent circumstances when using EnvironmentConfig.setSharedCache(true). This bug was reported in two different forum posts. Thanks to Kevin Jackson and Vyacheslav Lukianov for reporting it! [#16882] (3.3.78)

  6. Improve output of partial information in DatabaseEntry.toString. Thanks to archie172 for reporting this on OTN. [#16929] (3.3.78)

  7. Fix a bug in SecondaryCursor.dup that required the cursor to be initialized. This method, like Cursor.dup, may now be called whether the cursor is initialized or not. Thanks to archie172 for reporting this on OTN. [#16930] (3.3.78)

  8. Fix a bug that caused NullPointerException when opening an XAEnvironment under certain circumstances, when prepared but uncommitted transactions are present in the JE log and must be replayed during recovery. This bug was reported on the OTN forum by Matthieu Bentot -- thank you Matthieu! An example of the stack trace is below.
    Caused by: java.lang.NullPointerException: lsn1=21814149475543 lsn2=-1
    [#16774] (3.3.77)

  9. Fix a bug that caused LogFileNotFoundException in rare circumstances for an Environment having one or more Databases configured for duplicates (or one or more SecondaryIndexes with MANY_TO_XXX relationships). The bug only occurs when a single secondary/duplicate key value is associated with a large number of records/entities; specifically, the sub-Btree for a single duplicate key value must have at least 3 levels. [#16712] (3.3.76)

  10. Fix a bug that caused BufferOverflowException while writing transactional records. This could occur if multiple threads were writing to an environment while using the same Transaction. [#17204]

Direct Persistence Layer (DPL), Collections and Bind packages

  1. Fix a bug that causes the EvolveStats returned by EntityStore.evolve to contain zeros, even when entities are evolved and written, when no EvolveListener is configured. Thanks to jhalex on OTN for reporting this. [#17024] (3.3.78)

Changes in 3.3.75

General Environment Changes

  1. Fix a bug that caused the space taken by internal metadata in JE log files to increase over a long period of time. The rate of increase was slow in most cases, but in at least one observed case became rapid after a long time period and after the log cleaner became backlogged. To determine whether your JE log exhibits this problem, run
    java -jar je.x.y.z.jar DbPrintLog -h DIR -S
    and examine the line labeled MapLN on the left. If the amount of the log taken by MapLNs is 10% or greater, or if you see this number increasing steadily over time, then your application is probably experiencing this problem.

    By installing JE 3.3.75 or later, the excess disk space will automatically be reclaimed over time, as ordinary checkpoints and log cleaning occur. If you wish to recreate your database rather than wait for this to occur gradually, you can use DbDump and DbLoad to do so.

    We'd like to express our appreciation and sincere thanks to Jules and the other folks at Xoopit who reported this problem and helped us to diagnose it. We would not have found or fixed this problem as quickly as we did without their help.


  2. The Environment.scanLog method and related classes (LogScanner and LogScanConfig), have been removed from the public API. This API was added for s single customer with a special purpose requirement that is no longer applicable. This API is now private and not present in the documentation. [#16453]

Changes in 3.3.74

General Environment Changes

  1. Fix a problem that could cause LogFileNotFoundException in large databases. This could cause data loss and therefore all users are strongly recommended to upgrade to JE 3.3.74. This problem is found in all prior releases of JE.

    This bug can occur only for databases with a Btree that is 4 or more levels deep. For the default maximum entries per Btree node (128), this means the database must have grown to at least 10 million records to be a candidate for occurrance of this problem. This is only an approximation and may be larger or smaller if DatabaseConfig.setNodeMaxEntries has been called.


  2. Fix a problem where a static field instance in JE could continue to maintain a reference to internal data structures after an Environment is fully closed, preventing the Java GC from reclaiming the memory for the closed environment. This can occur when locking is disabled. This problem was originally mentioned in this forum post. [#16453]

  3. Fix a bug that caused NullPointerException during recovery (Environment construction) after deleting records in a database and then truncating or removing the database, as in the following stack trace:
    Caused by: java.lang.NullPointerException

  4. Fixed a bug in which RandomAccessFile.length() not being thread-safe can potentially cause unexpected resutls. [#16440]

Direct Persistence Layer (DPL), Collections and Bind packages

  1. Fix a bug that under certain circumstances causes "IllegalArgumentException: Not a key class" when calling EntityStore.getSubclassIndex, EntityStore.getPrimaryConfig, EntityStore.getSecondaryConfig, or PrimaryIndex.put, and a composite key class is used. [#16407]

  2. Fix a bug that causes class evolution to occur when an instance of an entity subclass is stored for the first time, after storing an instance of the entity (base) class. In some circumstances this could result in the following exception:
    Caused by: java.lang.NullPointerException
    at com.sleepycat.persist.impl.ComplexFormat.newInstance(
    at com.sleepycat.persist.impl.ComplexFormat$EvolveReader.newInstance(
    at com.sleepycat.persist.impl.PersistEntityBinding.readEntity(
    at com.sleepycat.persist.impl.PersistEntityBinding.entryToObject(
    at com.sleepycat.persist.PrimaryIndex.get(
    This exception could also be caused by two class changes that evolve the entity twice, such that the second change reverts the first change to the original version of the class. In other words, this could occur when there are three versions of a class A, A-1, A-2 and A-3, and A-3 is identical to A-1.

    Thanks to jhalex for reporting the problem in this OTN forum thread and helping to diagnose it. As mentioned in the forum thread, a workaround for this problem is to call EntityModel.registerClass for all entity subclasses.

Configuration, Documentation, Test and Build Changes:

  1. Clarify the javadoc for Database.put() and Cursor.put(). Change the phrase "If the key already appears in the database and duplicates are not configured, the existing key/data pair will be replaced." to "If the key already appears in the database and duplicates are not configured, the data associated with the key will be replaced."

Changes in 3.3.69

General Environment Changes

  1. Fix a bug that sometimes causes Environment.truncateDatabase and removeDatabase to hang, after opening an Environment with existing records in that Database. [#16329]

  2. Fix a bug that leaves the je.lck file open when an Environment cannot be opened for write access because another process has the same environment open for write access. When this occurs, the new EnvironmentLockedException is now thrown instead of an internal subclass of DatabaseException. [#16348]

  3. Fix a bug that causes an exception during Environment.close and prints the Environment stats to the console after the following message: "Local Cache Usage = ...". This bug could have caused a small memory leak for some applications that used the shared cache feature and that open and close many environments. [#16368]

  4. If an attempt to call Environment.cleanLog is made on a read-only Environment, IllegalStateException is now thrown. Before, the call to cleanLog was allowed, but log cleaning did not function properly. [#16368]

  5. Implement the -c flag of DbVerify. Before, this flag was in the usage command but was not implemented. Now, it causes DbVerify to check log cleaner metadata. [#16368]

  6. Fix a bug that causes OutOfMemoryError in DbVerify, Database.verify or Database.getStats, when the size of the data that is verified or examined exceeds the cache size. [#16368]

  7. Fix a bug that prevents opening an Environment as read-only under certain circumstances, or causes queries in a read-only Environment to return out of date, and possibly transactionally incorrect, data. The LogFileNotFoundException may be thrown when the problem occurs. [#16368]

  8. Fix a problem that reduces concurrency for multi-threaded applications using the Serializable isolation level. Before, when a reader performed an exact search for a record by key and the record was found, writers were blocked from inserting records past the last record in the database. Now, this blocking does not occur. Thanks to OTN user Alfranio who found this problem and identified the incorrect JE source code in this forum post. [#16393]

Direct Persistence Layer (DPL), Collections and Bind packages

  1. Fix a bug that prevents evolution of @SecondaryKey information in an entity subclass (a class that extends an @Entity class). The fix applies to adding, removing and renaming secondary key fields in an entity subclass. The problem was reported in this forum thread. [#16253]

  2. Add the ability to make fields persistent or not persistent independently of the use of the transient keywork for Java serialization. To override the normal rules for DPL field persistence, the @NotPersistent and @NotTransient annotations may be specified. Normally these keywords are not needed because the transient keyword is sufficient to mark a field as not persistent. However, some applications may wish for a field to be transient with respect to the DPL but not transient with respect to Java serialization, or vice-versa. The new annotations are provided for such applications. The feature was requested in this forum thread. [#16297]

  3. Add EntityCursor.setCacheMode and getCacheMode. See the class for more information. [#16239]

  4. The StoredMap class now implements the standard java.util.concurrent.ConcurrentMap interface. In the 3.3.62 release, the ConcurrentMap methods were implemented by the StoredMap class, but it was not declared to implement the ConcurrentMap interface. [#15382]

Changes in 3.3.62

New Features:

  1. JE can now be configured to use a shared cache among multiple environments. By using the shared cache, multiple open environments in the same process will make better use of memory because the cache LRU algorithm is applied across all information in all environments sharing the cache. See EnvironmentConfig.setSharedCache() for more information. Two related stats were also added to EnvironmentStats: getNSharedCacheEnvironments and getSharedCacheTotalBytes. [#15267]

  2. JE now provides a "temporary" database mode, specified through DatabaseConfig.setTemporary() and StoreConfig.setTemporary(). Temporary databases operate internally in deferred-write mode to provide reduced disk I/O and increased concurrency. But unlike an ordinary deferred-write database, the information in a temporary database is not durable or persistent, and therefore temporary databases are lighter weight than deferred-write databases. More details can be found in the Getting Started Guide and in the javadoc for DatabaseConfig and StoreConfig.

    In addition, two internal changes have been made to durable deferred-write database behavior. These changes are compatible with the existing use of deferred-write database, but may influence performance characteristics. These changes were necessary to ensure that the log cleaner operates properly for deferred-write databases.

    • The JE checkpointer now effectively performs a Database.sync for all open deferred-write databases. If you don't want this behavior you can a) use a Temporary DB instead, or b) disable the checkpointer. Disabling the checkpointer is commonly used to reduce logging during a bulk load.
    • JE now calls Database.sync for deferred-write databases when they are closed. If the data is not intended to be durable, we recommend using a Temporary DB instead.

  3. Added a new EnvironmentConfig parameter: je.cleaner.upgradeToLogVersion. This parameter is used to perform background upgrading of earlier format JE log files. All log files having a log version prior to the specified version will be cleaned -- migrating the log records forward to the end of the log and updating their format -- at a time when no other log cleaning is necessary. This parameter is intended for use in upgrading earlier format log files forward to the current log format version, for example, to take advantage of disk usage improvements. Note that earlier format log files are supported by later JE releases and log upgrading is optional. [#15365]

  4. JE now supports key prefixing. When enabled, the keys stored on a given internal b-tree nodes may be represented as a set of prefix/suffix combinations where the prefix is the common prefix of all keys for the internal b-tree node. This can improve storage efficiency for some key sets, both on-disk and in-memory, at the cost of computational efficiency. The degree of storage improvement is a function of the application's key values and access pattern. Key prefixing can be enabled with the DatabaseConfig.setKeyPrefixing(boolean) method. [#15399]

  5. Added the ability to control caching policy on a per-operation basis. See Cursor.get/setCacheMode() and to see how calls to Cursor "get" and "put" operations can specify LRU, MRU, or "pinned" cache hotness. [#15850]

  6. New stats have been added to meter random vs sequential disk io.
    return operation and byte counts for the number of random and sequential disk IOs. All values are approximate and may differ from the actual number of operations/byte-counts depending on the type of disks and file system, disk geometry, and file system cache size. [#16086]

  7. A new stat has been added to gauge the cost of log file opens, which can be a factor for applications with large data sets. A large value for EnvironmentStats.getNFileOpens() can indicate a need to tune the size of the file handle cache through increasing the je.log.fileCacheSize property. Also, general improvements have been made to improve the file handle cache, and to remove a bottleneck in concurrency in that area. [#16166]

General Environment Changes

  1. JE btree nodes now use shared latches for a portion of the btree access path. This used to be available through the je.env.sharedLatches configuration property, and is now the default. This results in better concurrency for read operations. The je.env.sharedLatches parameter has been deprecated. [#12136]

  2. In the past, JE's cache management pinned the internal metadata representing application databases. That made the use of large numbers of databases problematic, because significant portions of the cache could be consumed by metadata. That has now been resolved, and there are no cache or performance limitations on the number of databases in a JE environment. [#13415]

  3. The per-lock memory overhead has been reduced. [#15512]

  4. Various parameters have been adjusted so that JE will run under the Google Android platform. Refer to the Android HOWTO document for details.

  5. Fixed a bug in Transaction.setLockTimeout and setTxnTimeout so that the timeout is applied for all isolation modes. Before, the timeout was not applied for ReadCommitted and Serializable isolation under certain circumstances. The problem was initially reported on this JE Forum thread. [#16017]

  6. Fixed a bug where an internal data structure was being accessed sequentially during logging, while blocking other threads that attempt to write. For applications where the data set does NOT fit in cache, the data structure was small and the sequential access had little impact. But for applications where the entire data set DOES fit in cache, the sequential access was causing noticeable performance and concurrency problems during writing. [#16037]

  7. Improved performance for applications having a very large JE cache and where the active data set fits in the cache. Before, in this scenario, checkpoints could take a long time to occur. This in turn prevented log cleaning from being performed until after the end of the checkpoint. Two changes were made:
    1. Cleaner utilization information is now updated during the checkpoint. The log cleaner will no longer wait until the end of the checkpoint to perform cleaning. Note that although cleaning takes place during the checkpoint, log files are still not deleted until the end of the checkpoint.
    2. A new environment configuration parameter -- je.checkpointer.highPriority -- can be used to reduce the time to perform a checkpoint. If set to true, the checkpointer uses more resources in order to complete the checkpoint in a shorter time interval. Btree latches are held and other threads are blocked for a longer period. Log cleaner record migration is performed by cleaner threads instead of during checkpoints. A shorter checkpoint is beneficial because a shorter recovery period will be required in the event of a crash. However, application response time may be longer during the checkpoint, and more cleaner threads may be required to maintain the configured je.cleaner.minUtilization.

  8. Improved concurrency when a large number of log files (more than fit in the open-file-cache) are being accessed. Before, the latch on the cache was held while opening the log file and reading the header. Now, only the latch on the individual file is held. Note that the size of the open-file-cache is configured with the je.log.fileCacheSize Environment configuration parameter. Normally the cache can be configured to a size large enough to hold all files being accessed. However, for very large data sets, this may not be practical. [#16166]

  9. The EnvironmentStats.getTotalLogSize statistic is now a slow stat (not returned when StatsConfig.setFast(true) is called) and a bug was fixed that caused the total to be incorrect and overflow in some cases. [#15985]

  10. Fixed a bug where lock timeouts (DeadlockException) would sometimes occur after twice the configured lock timeout interval. This fix may cause DeadlockException to occur more frequently in some applications because the configured lock timeout is now applied correctly. Note that the lock timeout is 0.5 seconds by default and may be configured via the setLockTimeout method in EnvironmentConfig and Transaction. [#16021]

Direct Persistence Layer (DPL), Collections and Bind packages

  1. Several validation checks have been added or corrected having to do with entity subclasses, which are @Persistent classes that extend an @Entity class.
    • An IllegalArgumentException is now thrown when @PrimaryKey appears on an entity subclass. Before this was documented to be illegal but was not checked. [#15757]
    • The DPL now throws IllegalArgumentException if an instance of an entity subclass is embedded in another entity. It has always been illegal to embed entities within other entities, but the check for entity subclasses was not correct and has been fixed. [#16077]
    • In certain circumstances a NullPointerException was thrown before when attempting to embed an entity subclass, and this has also been corrected to throw IllegalArgumentException. [#16077]

  2. Optimize DPL marshaling for large numbers of embedded objects. Before, a sequential lookup was used to manage the object graph. Now, a HashMap is used. Thanks to user624180 who suggested this in a forum post. [#16198]

  3. Fix a DPL bug that caused exceptions when using a class Converter for an instance containing non-simple fields. The problem was reported in this forum thread. Thanks to Stan Livitski for submitting a reproducible test case. [#16233]

  4. The StoredMap class now implements the Java 1.5 ConcurrentMap interface. Thanks to jahlborn for proposing this change in this OTN forum thread. [#16218]

  5. Add Java 1.5 generic type parameters where appropriate: EntryBinding, EntityBinding and all binding implementations; all references to the Map and Collection interfaces; etc.

API Changes:

  1. Deprecated the EnvironmentStats.getCacheDataBytes method. The new getDataBytes should be used instead. The getLockBytes method now includes the lock and transaction overhead. Before it returned only the lock overhead. [#15267]

  2. A number of fields in EnvironmentStats and TransactionsStats were changed from int to long to prevent overflows, which resulted in a change to the signature of a number of getXXX() methods. Regrettably, this constitutes an incompatible binary and compile time change from JE 3.2 to JE 3.3 for applications which used those getter methods. There is no incompatibility if your application merely instantiates an EnvironmentStat or TransactionStat or calls their toString() methods. [#15979]

  3. JE now requires Java 5 and uses generic types in method signatures. Most notably for the user, the methods which define custom comparators have been changed to these signatures:
    DatabaseConfig.setBtreeComparator(Class<? extends Comparator<byte[]>>)
    DatabaseConfig.setDuplicateComparator(Class<? extends Comparator<byte[]>>)
    If the application used a custom comparator defined as
        public class MyCompare implements Comparator, Serializable
    that should now be declared as
        public class MyCompare implements Comparator<byte[]>, Serializable
    Please note that while these are fine:
        Comparator<byte[]> compareInstance = new MyCompare();
    the following previously legal line will now provoke a compile error:
    The compile error is puzzling and regrettably breaks compile time compatibility with JE 3.2 and earlier, but comes about because in Java, due to its type erasure based generics scheme, all the instances of a generic class have the same runtime class. Instead, the application has to apply the following cast:
    dbConfig.setBtreeComparator((Class<? extends Comparator<byte[]>>) compareInstance.getClass());

Utility Changes:

  1. Database salvage mode is now available in the API through the class rather than solely from the DbDump -r command line. This helped to fix and clarify a mismatch between the Javadoc and the constructor parameters for DbDump in the JE 3.2.X code line. See the class. [#15385]

Configuration, Documentation, Test and Build Changes:

  1. JE 3.3 requires Java or higher.

  2. Ant 1.7.0 is now required to build JE 3.3.

  3. In the past, the javadoc for JE's public classes was generated through an internal documentation process, and the actual content of the comments was not in the source itself. This was inconvenient for IDE users, because the javadoc contents would often not display in convenient places. This has been changed, and the javadoc content now resides in the source code in the normal fashion.

  4. Java developers often approach JE with past experience using SQL. To help illustrate the JE data model and API, we created a new example program which takes a list of common SQL queries and implements them using the Direct Persistent Layer API (DPL). Please refer to the Translating SQL Queries Example in JE Installation Notes for more information.

  5. The file is no longer used to document the configuration parameters that are used with EnvironmentConfig and EnvironmentMutableConfig. Instead, String constants have been added EnvironmentConfig which include that documentation. When calling setConfigParam these constants can be used to avoid hard-coding the parameter name. See EnvironmentConfig for details. [#16227]