Databases and MIDP, Part 1: Understanding the Record Management System

   


A key subsystem of the Mobile Information Device Profile (MIDP) is the Record Management System (RMS), an application programming interface (API) that gives MIDP applications local, on-device data persistence. On most MIDP-enabled devices today, RMS is the only facility for local data storage -- few devices support a conventional file system. As you might imagine, then, a thorough understanding of RMS is critical to writing any application that depends on persistent local data.

This article is the first in a series that will explore RMS and the larger issues surrounding its use in MIDP applications, such as interacting with external data sources like relational databases. We'll start by exploring what RMS has to offer and writing some simple RMS debugging aids.

Key Concepts

First, you need to understand the key concepts of the Record Management System.

Records

As you can gather from its name, RMS is a system for managing records. A record is an individual data item. RMS places no restrictions on what goes into a record: a record can contain a number, a string, an array, an image -- anything that a sequence of bytes can represent. If you can create a binary encoding of your data -- and a corresponding decoding -- then you can store it in a record, subject of course to any size restrictions the system imposes.

Many newcomers to RMS are confused by the term record. "Where are the fields?" they ask, wondering how the system subdivides individual records into discrete data sequences. The answer is simple: In RMS a record doesn't have any fields. Or, to put it more precisely, a record consists of a single binary field of variable size. The responsibility for interpreting the contents of a record falls entirely on the application. RMS provides the storage and a unique identifier, nothing else. While this division of labor complicates things for applications, it keeps RMS small and flexible -- important attributes for a MIDP subsystem.

At the API level, records are simply byte arrays.

Record Stores

A record store is an ordered collection of records. Records are not independent entities: each must belong to a record store, and all record access occurs through the record store. In fact, the record store guarantees that records are read and written atomically, with no possibility of data corruption.

When a record is created, the record store assigns it a unique identifier, an integer called the record ID. The first record added to a record store has a record ID of 1, the second a record ID of 2, and so on. A record ID is not an index: record deletions do not renumber existing records or affect the value of the next record ID.

Names are used to identify record stores within a MIDlet suite. A record store's name consists of 1 to 32 Unicode characters, and must be unique within the MIDlet suite that created the record store. In MIDP 1.0, record stores cannot be shared by different MIDlet suites. MIDP 2.0 optionally allows a MIDlet suite to share a record store with other suites, in which case the record store is identified by the names of the MIDlet suite and its vendor, along with the record store name itself.

Record stores also maintain time-stamp and version information so applications can discover when a record store was last modified. For close tracking, applications can register a listener to be notified whenever a record store is modified.

At the API level, a record store is represented by an instance of the javax.microedition.rms.RecordStore class. All RMS classes and interfaces are defined in the javax.microedition.rms package.

RMS Aspects

Before we look at some code, let's review some key information about RMS.

Storage Limits

The amount of memory available for record-based data storage varies from device to device. The MIDP specification requires devices to reserve at least 8K of non-volatile memory for persistent data storage. The specification does not place any limits on the size of an individual record, but space constraints will vary from device to device. RMS provides methods for determining the size of an individual record, the total size of a record store, and how much memory for data storage remains. Remember that persistent memory is a shared, scarce resource, so be frugal in its use.

MIDlet-Data-Size

Note that some MIDP implementations require you to define additional attributes related to storage requirements -- check device documentation for details.

Speed

Operations on persistent memory normally take longer than the equivalent operations on volatile (non-persistent) memory. Writing data, in particular, can take a long time on some platforms. For better performance, cache frequently accessed data in volatile memory To keep the user interface responsive, don't perform RMS operations from the MIDlet's event thread.

Thread Safety

RMS operations are thread-safe, but threads must still coordinate the reading and writing of data to the same record store, as with any shared resource. This coordination requirement applies to threads running in different MIDlets, because record stores are shared within the same MIDlet suite.

Exceptions

In general, methods in the RMS API throw one or more checked exceptions in addition to standard runtime exceptions like java.lang.IllegalArgumentException. The RMS exceptions are all part of the javax.microedition.rms package:

  • InvalidRecordIDException is thrown when an operation cannot be performed because the record ID is invalid.
  • RecordStoreFullException is thrown when no more space is available in the record store.
  • RecordStoreNotFoundException is thrown when the application tries to open a non-existent record store.
  • RecordStoreNotOpenException is thrown when an application tries to access a record store that has already been closed.
  • RecordStoreException is the superclass of the other four, and is thrown for general errors they don't cover.

Note that, for brevity, exception handling is simplified or omitted from some code samples in this series of articles.

Using RMS

The rest of this article describes basic record operations using the RMS API. Some of the operations are presented through the development of a utility class, RMSAnalyzer, for record store analysis. You can use RMSAnalyzer, as a debugging aid in your own projects.

Record Store Discovery

You obtain the list of record stores in a MIDlet suite by invoking RecordStore.listRecordStores(). This static method returns an array of strings, where each string represents the name of a record store owned by the MIDlet suite. If there are no record stores, the return is null.

The method RMSAnalyzer. analyzeAll() uses listRecordStores() as a way to call analyze() for each record store in the suite:

public void analyzeAll(){
    String[] names = RecordStore.listRecordStores();

    for( int i = 0;
         names != null && i < names.length;
         ++i ){
        analyze( names[i] );
    }
}

Note that the array identifies only the record stores of the owning MIDlet suite; that is, the one that created them. The MIDP specifications don't include any way to list the record stores of other MIDlet suites. In MIDP 1.0, no record store is visible outside the owning suite at all. In MIDP 2.0, the owning suite may designate a record store as shareable, but other MIDlet suites can use it only if they know its name.

Opening and Closing Record Stores

RecordStore.openRecordStore() is used to open, and optionally to create, a record store. This static method returns an instance of a RecordStore object, as shown in this version of RMSAnalyzer.analyze():

public void analyze( String rsName ){
    RecordStore rs = null;

    try {
        rs = RecordStore.openRecordStore( rsName, false );
        analyze( rs ); // call overloaded method
    } catch( RecordStoreException e ){
        logger.exception( rsName, e );
    } finally {
        try {
            rs.closeRecordStore();
        } catch( RecordStoreException e ){
            // Ignore this exception
        }
    }
}

The second parameter to openRecordStore() indicates whether the record store should be created if it does not already exist. If you're using MIDP 2.0, and want a MIDlet to open a record store created by a MIDlet in another suite, use this form of openRecordStore() instead:

...
String name = "mySharedRS";
String vendor = "EricGiguere.com";
String suite = "TestSuite";
RecordStore rs = 
      RecordStore.openRecordStore( name, vendor, suite );
...

The vendor and suite names must match what is defined in the MIDlet suite's manifest and application descriptor.

When you're finished with a record store, close it by calling RecordStore.closeRecordStore(), as in the analyze() method.

A RecordStore instance is unique within a MIDlet suite: once it's open, subsequent calls to openRecordStore() with the same name return the same object reference. This instance is shared by all MIDlets in the MIDlet suite.

Each RecordStore instance tracks the number of times it has been opened. The record store is not actually closed until closeRecordStore() has been called the same number of times. Using a record store after it's been closed will throw RecordStoreNotOpenException.

Creating Record Stores

To create a private record store, call openRecordStore() with the second parameter set to true:

...
// Create a record store
RecordStore rs = null;

try {
    rs = RecordStore.openRecordStore( "myrs", true );
} catch( RecordStoreException e ){
    // couldn't open it or create it
}
...

To perform a one-time initialization of a record store, check to see whether getNextRecordID() equals 1, immediately after opening the record store:

if( rs.getNextRecordID() == 1 ){
    // perform one-time initialization
}

Alternatively, to re-initialize a record store whenever it's empty, check the value returned by getNumRecords():

if( rs.getNumRecords() == 0 ){
    // record store is empty, re-initialize
}

To create a shareable record store (in MIDP 2.0 only), use the four-parameter variant of openRecordStore():

int     authMode = RecordStore.AUTHMODE_ANY;
boolean writable = true;

rs = RecordStore.openRecordStore( "myrs", true, 
       authMode, writable );

When the second parameter is true and the record store does not already exist, the last two parameters control its authorization mode and writability. The authorization mode determines whether other MIDlet suites will have access to the record store. The two possible modes are RecordStore.AUTHMODE_PRIVATE (only the owning MIDlet suite can access it) and RecordStore.AUTHMODE_ANY (any MIDlet suite can access it). The writability flag determines whether other MIDlet suites can change the record store -- if false, they can only read from the record store.

Note that the owning MIDlet suite can change the record store's authorization mode and writability at any time using RecordStore.setMode:

rs.setMode( RecordStore.AUTHMODE_ANY, false );

In fact, it's best to create a shared record store using AUTHMODE_PRIVATE, and to expose it only after it's been initialized.

Adding and Updating Records

You'll recall that records are byte arrays. Use RecordStore.addRecord() to add a new record to an open record store:

...
byte[] data = new byte[]{ 0, 1, 2, 3 };
int    recordID;

recordID = rs.addRecord( data, 0, data.length );
...

You can add an empty record by setting the first parameter to null. The second and third parameters specify the starting offset into the array and the total number of bytes to store from that offset. A successful add returns the new record's ID, otherwise an exception, like RecordStoreFullException, is thrown.

You can update a record at any time with RecordStore.setRecord():

...
int    recordID = ...; // some record ID
byte[] data = new byte[]{ 0, 10, 20, 30 };

rs.setRecord( recordID, data, 1, 2 ); 
    // replaces all data in record with 10, 20
...

You cannot add or update a record in chunks: you must build the entire record in memory as a byte array, and add or update it using a single call.

You can find out what record identifier the next call to addRecordStore() will return by calling RecordStore.getNextRecordID(). All current record identifiers will be less than this value.

In Part 2 of this series we'll look at strategies for converting objects and other data into byte arrays.

Reading Records

To read a record, use one of the two forms of RecordStore.getRecord(). The first allocates a byte array of the right size for you and copies the record data into it:

...
int    recordID = .... // some record ID
byte[] data = rs.getRecord( recordID );
...

The second form copies the data into a preallocated array, starting at a specified offset, and returns the number of bytes that were copied:

...
int    recordID = ...; // some record ID
byte[] data = ...; // an array
int    offset = ...; // the starting offset

int numCopied = rs.getRecord( recordID, data, offset );
...

The array must be large enough to hold the data, otherwise java.lang.ArrayIndexOutOfBoundsException will be thrown. Use the value returned by RecordStore.getRecordSize() to allocate an array that's big enough. In fact, the first form of getRecord() is equivalent to:

...
byte[] data = new byte[ rs.getRecordSize( recordID ) ];
rs.getRecord( recordID, data, 0 );
...

The second form is useful for minimizing memory allocations when iterating through a set of records. For example, you can use it along with getNextRecordID() and getRecordSize() to perform a brute-force search for all records in the record store:

...
int    nextID = rs.getNextRecordID();
byte[] data = null;

for( int id = 0; id < nextID; ++id ){
    try {
        int size = rs.getRecordSize( id );

        if( data == null || data.length < size ){
            data = new byte[ size ];
        }

        rs.getRecord( id, data, 0 );

        processRecord( rs, id, data, size ); // process it
    } catch( InvalidRecordIDException e ){
        // ignore, move to next record
    } catch( RecordStoreException e ){
        handleError( rs, id, e ); // call an error routine
    }
}
...

A better approach, however, is to use RecordStore.enumerateRecords() to iterate through the records. I'll cover the use of enumerateRecords() in Part 3 of this series.

Deleting Records and Record Stores

You delete records with RecordStore.deleteRecord():

...
int recordID = ...; // some record ID
rs.deleteRecord( recordID );
...

Once a record is deleted, any attempt to use it throws an InvalidRecordIDException.

You delete record stores themselves using RecordStore.deleteRecordStore():

...
try {
    RecordStore.deleteRecordStore( "myrs" );
} catch( RecordStoreNotFoundException e ){
    // no such record store
} catch( RecordStoreException e ){
    // somebody has it open
}
...

A record store can be deleted only if it is not currently open, and only by a MIDlet in the owning MIDlet suite.

Other Operations

A few other RMS operations remain, all of them methods of the RecordStore class:

  • getLastModified() returns the time of the last modification to the record store, in the same format returned by System.currentTimeMillis().
  • getName() returns the name of the record store.
  • getNumRecords() returns the number of records in the record store.
  • getSize() returns the total size of the record store, in bytes. The total size includes the sizes of all the records as well as the overhead required by the system to implement the record store.
  • getSizeAvailable() returns the number of bytes available for the record store to grow. Note that the actual size available may be smaller due to the overhead required to store individual records.
  • getVersion() returns the version number of the record store. The version number is a positive integer greater than zero that is incremented each time the record store is changed.

A MIDlet can also track the changes made to a record store by registering a listener using addRecordListener(), and later deregister it with removeRecordListener(). I'll discuss these listeners in Part 3 of this series.

The RMSAnalyzer Class

Part 1 of this series ends with the source code for the RMSAnalyzer class, our record store analyzer. To analyze a record store, do this:

...
RecordStore rs = ...; // open the record store
RMSAnalyzer analyzer = new RMSAnalyzer();
analyzer.analyze( rs );
...

By default, the analysis goes to the System.out stream and looks like this:

=========================================
Record store: recordstore2
    Number of records = 4
    Total size = 304
    Version = 4
    Last modified = 1070745507485
    Size available = 975950

Record #1 of length 56 bytes
5f 62 06 75 2e 6b 1c 42 58 3f  _b.u.k.BX?
1e 2e 6a 24 74 29 7c 56 30 32  ..j$t)|V02
5f 67 5a 13 47 7a 77 68 7d 49  _gZ.Gzwh}I
50 74 50 20 6b 14 78 60 58 4b  PtP k.x`XK
1a 61 67 20 53 65 0a 2f 23 2b  .ag Se./#+
16 42 10 4e 37 6f              .B.N7o    
Record #2 of length 35 bytes
22 4b 19 22 15 7d 74 1f 65 26  "K.".}t.e&
4e 1e 50 62 50 6e 4f 47 6a 26  N.PbPnOGj&
31 11 74 36 7a 0a 33 51 61 0e  1.t6z.3Qa.
04 75 6a 2a 2a                 .uj**     
Record #3 of length 5 bytes
47 04 43 22 1f                 G.C".     
Record #4 of length 57 bytes
6b 6f 42 1d 5b 65 2f 72 0f 7a  koB.[e/r.z
2a 6e 07 57 51 71 5f 68 4c 5c  *n.WQq_hL\
1a 2a 44 7b 02 7d 19 73 4f 0b  .*D{.}.sO.
75 03 34 58 17 19 5e 6a 5e 80  u.4X..^j^?
2a 39 28 5c 4a 4e 21 57 4d 75  *9(\JN!WMu
80 68 06 26 3b 77 33           ?h.&w3   

Actual size of records = 153
-----------------------------------------

This format is convenient for use when testing with the J2ME Wireless Toolkit. For testing on an actual device, you may wish to send the analysis output to a serial port or even across the network to a servlet. You can do so by defining your own class that implements the RMSAnalyzer.Logger interface and passing an instance of that class to the RMSAnalyzer constructor.

Accompanying this article is a J2ME Wireless Toolkit project called RMSAnalyzerTest that demonstrates the use of the analyzer:

package com.ericgiguere;

import java.io.*;
import javax.microedition.rms.*;

// Analyzes the contents of a record store.
// By default prints the analysis to System.out,
// but you can change this by implementing your
// own Logger.

public class RMSAnalyzer {

    // The logging interface.

    public interface Logger {
        void logEnd( RecordStore rs );
        void logException( String name, Throwable e );
        void logException( RecordStore rs, Throwable e );
        void logRecord( RecordStore rs, int id,
                        byte[] data, int size );
        void logStart( RecordStore rs );
    }

    private Logger logger;

    // Constructs an analyzer that logs to System.out.

    public RMSAnalyzer(){
        this( null );
    }

    // Constructs an analyzer that logs to the given logger.

    public RMSAnalyzer( Logger logger ){
        this.logger = ( logger != null ) ? logger :
                                           new SystemLogger();
    }

    // Open the record stores owned by this MIDlet suite
    // and analyze their contents.

    public void analyzeAll(){
        String[] names = RecordStore.listRecordStores();

        for( int i = 0;
             names != null && i < names.length;
             ++i ){
            analyze( names[i] );
        }
    }

    // Open a record store by name and analyze its contents.

    public void analyze( String rsName ){
        RecordStore rs = null;

        try {
            rs = RecordStore.openRecordStore( rsName, false );
            analyze( rs );
        } catch( RecordStoreException e ){
            logger.logException( rsName, e );
        } finally {
            try {
                rs.closeRecordStore();
            } catch( RecordStoreException e ){
                // Ignore this exception
            }
        }
    }

    // Analyze the contents of an open record store using
    // a simple brute force search through the record store.

    public synchronized void analyze( RecordStore rs ){
        try {
            logger.logStart( rs );

            int    lastID = rs.getNextRecordID();
            int    numRecords = rs.getNumRecords();
            int    count = 0;
            byte[] data = null;

            for( int id = 0;
                 id < lastID && count < numRecords;
                 ++id ){
                try {
                    int size = rs.getRecordSize( id );

                    // Make sure data array is big enough,
                    // plus add some for growth

                    if( data == null || data.length < size ){
                        data = new byte[ size + 20 ];
                    }

                    rs.getRecord( id, data, 0 );
                    logger.logRecord( rs, id, data, size );

                    ++count; // only increase if record exists
                }
                catch( InvalidRecordIDException e ){
                    // just ignore and move to the next one
                }
                catch( RecordStoreException e ){
                    logger.logException( rs, e );
                }
            }

        } catch( RecordStoreException e ){
            logger.logException( rs, e );
        } finally {
            logger.logEnd( rs );
        }
    }

    // A logger that outputs to a PrintStream.

    public static class PrintStreamLogger implements Logger {
        public static final int COLS_MIN = 10;
        public static final int COLS_DEFAULT = 20;

        private int          cols;
        private int          numBytes;
        private StringBuffer hBuf;
        private StringBuffer cBuf;
        private StringBuffer pBuf;
        private PrintStream  out;

        public PrintStreamLogger( PrintStream out ){
            this( out, COLS_DEFAULT );
        }

        public PrintStreamLogger( PrintStream out, int cols ){
            this.out = out;
            this.cols = ( cols > COLS_MIN ? cols : COLS_MIN );
        }

        private char convertChar( char ch ){
            if( ch < 0x20 ) return '.';
            return ch;
        }

        public void logEnd( RecordStore rs ){
            out.println( "\nActual size of records = "
                         + numBytes );
            printChar( '-', cols * 4 + 1 );

            hBuf = null;
            cBuf = null;
            pBuf = null;
        }

        public void logException( String name, Throwable e ){
            out.println( "Exception while analyzing " +
                         name + ": " + e );
        }

        public void logException( RecordStore rs, Throwable e ){
            String name;

            try {
                name = rs.getName();
            } catch( RecordStoreException rse ){
                name = "<unknown>";
            }

            logException( name, e );
        }

        public void logRecord( RecordStore rs, int id,
                               byte[] data, int len ){
            if( len < 0 && data != null ){
                len = data.length;
            }

            hBuf.setLength( 0 );
            cBuf.setLength( 0 );

            numBytes += len;

            out.println( "Record #" + id + " of length "
                         + len + " bytes" );

            for( int i = 0; i < len; ++i ){
                int    b = Math.abs( data[i] );
                String hStr = Integer.toHexString( b );

                if( b < 0x10 ){
                    hBuf.append( '0');
                }

                hBuf.append( hStr );
                hBuf.append( ' ' );

                cBuf.append( convertChar( (char) b ) );

                if( cBuf.length() == cols ){
                    out.println( hBuf + " " + cBuf );

                    hBuf.setLength( 0 );
                    cBuf.setLength( 0 );
                }
            }

            len = cBuf.length();

            if( len > 0 ){
                while( len++ < cols ){
                    hBuf.append( "   " );
                    cBuf.append( ' ' );
                }

                out.println( hBuf + " " + cBuf );
            }
        }

        public void logStart( RecordStore rs ){
            hBuf = new StringBuffer( cols * 3 );
            cBuf = new StringBuffer( cols );
            pBuf = new StringBuffer();

            printChar( '=', cols * 4 + 1 );

            numBytes = 0;

            try {
                out.println( "Record store: "
                             + rs.getName() );
                out.println( "    Number of records = "
                             + rs.getNumRecords() );
                out.println( "    Total size = "
                             + rs.getSize() );
                out.println( "    Version = "
                             + rs.getVersion() );
                out.println( "    Last modified = "
                             + rs.getLastModified() );
                out.println( "    Size available = "
                             + rs.getSizeAvailable() );
                out.println( "" );
            } catch( RecordStoreException e ){
                logException( rs, e );
            }
        }

        private void printChar( char ch, int num ){
            pBuf.setLength( 0 );
            while( num-- > 0 ){
                pBuf.append( ch );
            }
            out.println( pBuf.toString() );
        }
    }

    // A logger that outputs to System.out.

    public static class SystemLogger
                        extends PrintStreamLogger {
        public SystemLogger(){
            super( System.out );
        }

        public SystemLogger( int cols ){
            super( System.out, cols );
        }
    }

What's Next

In Part 2 we explore data mappings.

Reader Feedback
Excellent   Good   Fair   Poor  

If you have other comments or ideas for future technical tips, please type them here:

Comments:
If you would like a reply to your comment, please submit your email address:
Note: We may not respond to all submitted comments.