Databases and MIDP, Part 2: Data Mapping

   

by Eric Giguere
May 2004

As you discovered in Part 1 of this series, the Mobile Information Device Profile (MIDP) provides for data persistence through the Record Management System (RMS). MIDP support for persistence is limited to simple byte arrays, and records are read and written in their entirety, not field by field. Thus the RMS application programming interface (API) is very simple, but it requires applications to use a very unsophisticated binary format for data storage.

This article describes data mapping strategies you can use to encapsulate low-level storage operations so your applications can store and retrieve persistent data efficiently and effectively.

 

Core Classes for Manipulating Data



Writing data to a record store is really no different from sending a packet of data across a network to a server. The Connected Limited Device Configuration (CLDC), on which MIDP is based, includes standard data-manipulation classes drawn from the core library of the Java 2 Platform, Standard Edition (J2SE) that are particularly useful for RMS operations. A big benefit of using common interfaces is that MIDlets can interoperate more easily with applications running on standard and enterprise Java platforms.

 

Byte-Array Streams



A ByteArrayInputStream object transforms a byte array into an input stream, as this trivial example demonstrates:

...
byte[] data = new byte[]{ 1, 2, 3 };
ByteArrayInputStream bin = new ByteArrayInputStream( data );

int b;

while( ( b = bin.read() ) != -1 ){
    System.out.println( b );
}

try {
    bin.close();
} catch( IOException e ){
    // never thrown in this case
}
...

The input stream returns each byte in the array sequentially until it reaches the end of the array. Using mark() and reset() we can reposition the stream within the byte array at any time.

A ByteArrayOutputStream object captures data in a memory buffer for later transformation into a byte array:

...
ByteArrayOutputStream bout = new ByteArrayOutputStream();

bout.write( 1 );
bout.write( 2 );
bout.write( 3 );

byte[] data = bout.toByteArray();

for( int i = 0; i < data.length; ++i ){
    System.out.println( data[i] );
}

try {
    bout.close();
} catch( IOException e ){
    // never thrown in this case
}
...

The ByteArrayOutputStream's buffer grows automatically as data is written to the stream. The toByteArray() method copies captured data to a byte array. We can reuse the internal buffer for further capturing by calling reset().

 

Data Streams



DataInputStream transforms a raw input stream into primitive data types and strings:

...
InputStream in = ... // an input stream
DataInputStream din = new DataInputStream( in );    

try {
    int custID = din.readInt();
    String lastName = din.readUTF();
    String firstName = din.readUTF();
    long timestamp = din.readLong();

    din.close();
}
catch( IOException e ){
    // handle the error here
}
...

Data can be read in this way only if it has been written to the stream in the machine-independent format that DataInputStream expects. The class has methods to read most simple Java data types: readBoolean(), readByte(), readChar(), readShort(), readInt(), and readLong(). CLDC 1.1 implementations additionally support readFloat() and readDouble().There are also methods for reading byte arrays and unsigned values: readFully(), readUnsignedByte() and readUnsignedShort().

The readUTF() method reads strings up to 65,535 characters long that were encoded in UTF-8 format. To read a string written as a sequence of two-byte char values, an application must make multiple calls to readChar(), which assumes either that a delimiter identifies the end of the string, or that the length is already known. The length may be a fixed value or it may have been written to the stream immediately before the string.

The application reading the data must know the order in which the primitives were written in order to invoke the correct methods.

DataOutputStream writes strings and primitive data types to an output stream:

...
OutputStream out = ... // an output stream
DataOutputStream dout = new DataOutputStream( out );

try {
    dout.writeInt( 100 );
    dout.writeUTF( "Smith" );
    dout.writeUTF( "John" );
    dout.writeLong( System.currentTimeMillis() );

    dout.close();
}
catch( IOException e ){
    // handle the error here
}
...

The data is written in the same machine-independent format expected by DataInputStream. The class has methods to write most simple Java data types: writeBoolean(), writeByte(), writeChar(), writeShort(), writeInt(), and writeLong(). CLDC 1.1 implementations additionally support writeFloat() and writeDouble().

Strings are written using one of two methods. You can write strings of up to 65,535 characters encoded in UTF-8 format with writeUTF(), or call writeChars() to write a sequence of two-byte characters.

 

Basic Data Mappings



The standard data manipulation classes make basic data mapping easy.. Writing data to a record store is simply a matter of combining a DataOutputStream with a ByteArrayOutputStream and storing the resulting byte array:

...
RecordStore rs = ... // a record store
ByteArrayOutputStream bout = new ByteArrayOutputStream();
DataOutputStream dout = new DataOutputStream( bout );

try {
    dout.writeInt( 100 );
    dout.writeUTF( "Smith" );
    dout.writeUTF( "John" );
    dout.writeLong( System.currentTimeMillis() );
    dout.close();

    byte[] data = bout.toByteArray();
    rs.addRecord( data, 0, data.length );
}
catch( RecordStoreException e ){
    // handle RMS error here
}
catch( IOException e ){
    // handle IO error here
}
...

Reading data is just a matter of reversing these steps, using a DataInputStream and a ByteArrayInputStream:

...
RecordStore rs = ... // a record store
int recordID = ... // the record to read
ByteArrayInputStream bin;
DataInputStream din;

try {
    byte[] data = rs.getRecord( recordID );
     
    bin = new ByteArrayInputStream( data );
    din = new DataInputStream( bin );
    
    int id = din.readInt();
    String lastName = din.readUTF();
    String firstName = din.readUTF();
    long timestamp = din.readLong();

    din.close();

    ... // process data here
}
catch( RecordStoreException e ){
    // handle RMS error here
}
catch( IOException e ){
    // handle IO error here
}
...

Watch for null values when writing strings. In most cases, you'll write an empty string instead. If you don't, you'll need marker bytes to distinguish a null string from an empty string.

 

Simple Object Mappings



In many cases, the data you want to save is encapsulated in an object instance. Consider, for example, the class Contact, which holds contact information for a person:

package j2me.example;

// The contact information for a person

public class Contact {
    private String _firstName;
    private String _lastName;
    private String _phoneNumber;

    public Contact(){
    }

    public Contact( String firstName, String lastName,
                    String phoneNumber )
    {
        _firstName = firstName;
        _lastName = lastName;
        _phoneNumber = phoneNumber;
    }

    public String getFirstName(){
        return _firstName != null ? _firstName : "";
    }

    public String getLastName(){
        return _lastName != null ? _lastName : "";
    }

    public String getPhoneNumber(){
        return _phoneNumber != null ? _phoneNumber : "";
    }

    public void setFirstName( String name ){
        _firstName = name;
    }

    public void setLastName( String name ){
        _lastName = name;
    }

    public void setPhoneNumber( String number ){
        _phoneNumber = number;
    }
}

If the class is modifiable, the simplest way to introduce persistence is to add methods to map an object to and from a byte array:

public void fromByteArray( byte[] data ) throws IOException {
    ByteArrayInputStream bin = new ByteArrayInputStream(data);
    DataInputStream din = new DataInputStream( bin );

    _firstName = din.readUTF();
    _lastName = din.readUTF();
    _phoneNumber = din.readUTF();

    din.close();
}

public byte[] toByteArray() throws IOException {
    ByteArrayOutputStream bout = new ByteArrayOutputStream();
    DataOutputStream dout = new DataOutputStream( bout );

    dout.writeUTF( getFirstName() );
    dout.writeUTF( getLastName() );
    dout.writeUTF( getPhoneNumber() );

    dout.close();

    return bout.toByteArray();
}

Storing an object is then a simple operation:

...
Contact contact = ... // the contact to store
RecordStore rs = ... // the record store to use

try {
    byte[] data = contact.toByteArray();
    rs.addRecord( data, 0, data.length );
}
catch( RecordStoreException e ){
    // handle the RMS error here
}
catch( IOException e ){
    // handle the IO error here
}
...

Retrieving the object is just as easy:

...
RecordStore rs = ... // the record store to use
int recordID = ... // the record ID to read from
Contact contact = new Contact();

try {
    byte[] data = rs.getRecord( recordID );
    contact.fromByteArray( data );
}
catch( RecordStoreException e ){
    // handle the RMS error here
}
catch( IOException e ){
    // handle the IO error here
}
...

If the class is not modifiable, such as the standard Vector or Hashtable classes, you'll need to write a helper class. For example, here's a helper class that maps a list of non-null strings to a byte array:

package j2me.example;

import java.io.*;
import java.util.*;

// A helper class that converts transforms string
// vectors (vectors whose elements are instances of
// String or StringBuffer ) into byte arrays and vice-versa.

public class StringVectorHelper {

    // Reconstitutes a vector from a byte array.

    public Vector fromByteArray( byte[] data )
                                 throws IOException {
        ByteArrayInputStream bin =
                 new ByteArrayInputStream( data );
        DataInputStream din = new DataInputStream( bin );

        int count = din.readInt();
        Vector v = new Vector( count );

        while( count-- > 0 ){
            v.addElement( din.readUTF() );
        }

        din.close();

        return v;
    }

    // Transforms a vector into a byte array.

    public byte[] toByteArray( Vector v )
                               throws IOException {
        ByteArrayOutputStream bout =
                   new ByteArrayOutputStream();
        DataOutputStream dout = new DataOutputStream( bout );

        dout.writeInt( v.size() );

        Enumeration e = v.elements();
        while( e.hasMoreElements() ){
            Object o = e.nextElement();
            dout.writeUTF( o != null ? o.toString() : "" );
        }

        dout.close();

        return bout.toByteArray();
    }
}

Of course, what you're really doing with all this coding is developing an object serialization framework. A complete framework is beyond the scope of this discussion, but the following issues deserve careful consideration when you're making objects persistent:

  • Object creation. CLDC doesn't include reflection APIs, so you'll need some way to re-create persistent objects. It could be a constructor that accepts a stream or a byte array, or a more complicated factory class.
  • Versioning. If an object's data layout may change, you'll want to store a number at the start of the byte array that identifies the version of the object that was stored.
  • Object references. If one object refers to another, the relationship must be maintained. In some cases, you can replace the object reference with an index or a key that can be used to locate the object referred to, after deserialization. Otherwise, you must store a complete object graph, not just a single object.

You can avoid these issues or minimize their impact by saving and restoring only primitive Java data types and simple, self-contained objects. The fromByteArray() and toByteArray() methods shown so far are simple but effective means of making object persistence easy. You can also use these techniques to copy objects across a network: Once you have the object in byte-array form, it's a simple matter to send the array to another device or to an external server using a network connection, and to recreate the object at the other end. For example, the data classes you develop for your J2ME application can just as easily be used in a servlet developed for the Java 2 Platform, Enterprise Edition (J2EE).

 

Using Data Streams



The examples shown so far have dealt directly with byte arrays. Convenient as they are for saving and restoring a single set of data, dealing with raw byte arrays becomes cumbersome as soon as you want to store a sequence of data sets -- a list of objects, for example. A better approach is to separate the management of the byte array from the reading and writing of the data. For example, we can add these methods to our Contact class:

public void fromDataStream( DataInputStream din ) 
                            throws IOException {
    _firstName = din.readUTF();
    _lastName = din.readUTF();
    _phoneNumber = din.readUTF();
}

public void toDataStream( DataOutputStream dout ) 
                          throws IOException {
    dout.writeUTF( getFirstName() );
    dout.writeUTF( getLastName() );
    dout.writeUTF( getPhoneNumber() );
}

We can keep the existing fromByteArray and toByteArray methods for convenience, but we should rewrite them to use the new stream-oriented methods:

public void fromByteArray( byte[] data ) throws IOException {
    ByteArrayInputStream bin = new ByteArrayInputStream(data);
    DataInputStream din = new DataInputStream( bin );

    fromDataStream( din );
    din.close();
}

public byte[] toByteArray() throws IOException {
    ByteArrayOutputStream bout = new ByteArrayOutputStream();
    DataOutputStream dout = new DataOutputStream( bout );

    toDataStream( dout );
    dout.close();

    return bout.toByteArray();
}

Centralize the reading and writing of the data to ensure that it is always consistent.

 

What's Next



In this part of the series, you've learned some basic data-mapping techniques. In Part 3 I'll show you a more sophisticated approach to managing persistent data, one that enables your applications to store and retrieve objects composed of multiple fields of varying types.

Reader Feedback
Excellent   Good   Fair   Poor  

If you have other comments or ideas for future technical tips, please type them here:

Comments:
If you would like a reply to your comment, please submit your email address:
Note: We may not respond to all submitted comments.
false ,,,,,,,,,,,,,,,