Java Magazine Logo
Originally published in the May/June 2014 issue of Java Magazine. Subscribe today.

Leap Motion and JavaFX

by Johan Vos

Published May 2014

Use 3-D hand movements to interact with JavaFX applications.

Sooner or later, there might come a day when parents have to explain to their children what a keyboard is. Children will have a hard time understanding why their parents used such a strange device for many decades. Although today is not yet that day, we are already seeing devices that provide more-intuitive, natural input than a keyboard, a mouse, or a trackpad do. These devices do not necessarily replace existing input devices, but they can be very complementary.

It is important that the software we write and use today not be restricted to a limited set of input devices. The JavaFX platform can easily be integrated with new input devices. This article shows how the Leap Motion Controller device can be used as a user input device in JavaFX applications.

The Leap Motion Controller, a small device produced by the Leap Motion company, is equipped with infrared cameras and infrared LEDs. This device is capable of tracking hand and finger movements around the device in a clipped pyramid that has a radius of about 1 meter. The device is connected to a computer using a USB cable. Leap Motion provides native drivers for Windows, Linux, and Mac systems. On top of those drivers, a number of APIs for different programming languages are available. These APIs allow developers to create applications that leverage the Leap Motion Controller.

Did You Know?

JavaFX allows you to create client applications in which there is a large degree of independence between the layout and the input devices.
Fortunately, a Java API is available. We will now explore how you can use the Java API in a Java client application to interact with the Leap Motion Controller, but first we will briefly discuss the capabilities of the Leap Motion Controller.

For a thorough understanding of the Leap Motion Controller, see the Leap Motion website and the Leap Motion developer website.

Also visit the developer website to download the Leap Motion Software Development Kit (SDK), which allows you to create applications.

Using the Leap Motion SDK

Leap Motion provides an SDK for Windows, Mac OS, and Linux. The SDK contains a number of files, some of which are required for Java development, for example:

  • A Java library named LeapJava.jar, which provides the API we can use in Java applications Operating system–dependent native libraries that connect to the Leap Motion Service, which receives data from the Leap Motion Controller device over USB

To run Java applications that communicate with the Leap Motion Controller, the native libraries need to be in the native library path. This is achieved by starting the Java applications with the system property java .library.path pointing to the location of the native libraries for your particular OS, for example, java -Djava.library.path=/path/to/LeapSDK/lib/x64 on a 64-bit system. 

The Java library LeapJava.jar file should be in the classpath.

Johan Vos demonstrates how to use a Leap Motion Controller with a JavaFX application.

Leap Motion Controller Concepts

Because the Leap Motion Controller is capable of tracking hands and fingers in three directions, the retrieved data is obtained in a three-dimensional right-handed Cartesian coordinate system. The origin of this coordinate system is located at the center of the top of the device. The x, y, and z axes are pointing in the directions shown in Figure 1.

leapmotion-f1

Figure 1

All data obtained from the Leap Motion Controller is contained in instances of the com.leapmotion .leap.Frame class, which is part of the LeapJava.jar library. Depending on environment variables, the device sends frame data with a frequency between 30 Hz and 200 Hz.

There are two ways of obtaining the frame data, both of which require an instance of the com .leapmotion.leap.Controller class to operate:

  • By polling the Controller
  • By registering a Listener with the Controller, which will be notified when new data is available

When using JavaFX, the second approach is the most intuitive. The callback function on the Listener can be used to change properties in the data model of the application, and the JavaFX pulse thread will make sure the changes are reflected in the user interface. Because a pulse event is generated at most 60 times a second, the UI thread won’t be overloaded when the Leap Motion Controller provides more data than the graphical system can handle. 

The code in Listing 1 shows how to create a Controller and register a Listener in a JavaFX application. The LeapListener class we introduce here extends the com.leapmotion .leap.Listener class, as shown in Listing 2.

public class LeapConcepts extends Application {
    Controller c;
    Listener l;

    public void start (Stage stage) {
         …. // set up the scene and stage
        c = new Controller();
        l = new LeapListener(this);
        c.addListener (l);
    }
}

Listing 1

public class LeapListener extends Listener {

    @Override
    public void onConnect (Controller c) {
        … // device connected!
    }

    @Override
    public void onFrame (Controller c) {
        Frame frame = controller.frame();
    }
}

Listing 2

This Listener class will be called upon lifecycle events and whenever a frame containing tracked data is available. The onConnect method is called whenever the controller instance connects to a Leap Motion device. When, for some reason, the connection is broken, the onDisconnect method is called. Obviously, the onFrame method is called whenever a new frame with data is available.

Available Data

As mentioned before, all information we receive from the Leap Motion Controller is available on instances of the Frame class. This article gives a brief, nonexhaustive overview of the data that is available. The Java documentation on the Leap Motion website contains all the information for this class and the other classes.

The Leap Motion Controller is capable of tracking the position, direction, and movement of hands, fingers, and tools (for example, a pencil). Based on internal calculations, this allows the Leap Motion Service to also track a number of gestures, for example, swiping, tapping, and making a circular move.

If we want information on the tracked hands in a frame, we call frame.hands(). This method returns a HandList, which is an Iterable that can be used to obtain a number of tracked Hand instances. A Hand instance has a number of properties: for example, the location, the normal vector, and the velocity of the hand’s palm or the fingers that are detected on the hand.

Fingers can be tracked individually as well by calling frame.fingers(). This method returns a FingerList that can be used to obtain information about the detected fingers (for example, direction, location, and velocity).

Consequently, gestures are obtained by calling frame.gestures(), which returns a GestureList. Note that gestures will be added to the frame data only if the Controller is instructed to do so. Hence, it is good practice to enable gesture tracking in the onConnect method of the class extending the Listener class, as shown in Listing 3.

 @Override
    public void onConnect (Controller c) {
        c.enableGesture(Gesture.TYPE.TYPE_SWIPE);
        c.enableGesture(Gesture.TYPE.TYPE_KEY_TAP);
    }

Listing 3

The code in Listing 3 will make sure that when swipe or key-tap gestures are performed by the user, the corresponding information is added to the Frame instances that are provided.

Threading

Both the JavaFX platform and the Leap Motion Controller software have some specific requirements regarding threading. Fortunately, these requirements match very well.

When writing JavaFX applications, it is good practice to separate the view and the data. That is, user interface components are defined and glued together, but the data that describes their behavior (for example, the position of a circle or the width of a text field) is kept in JavaFX properties. These properties are altered whenever needed, but this does not mean that the user interface is redrawn every time a property changes.

It is a requirement, though, that changes to properties that can lead to changes in the user interface be executed on the JavaFX application thread.

The Leap Motion native library will create a new Java thread whenever a new Frame instance is available, and it will call Listener .onFrame()—if a Listener is registered with the Controller, that is.

The rate by which new threads are created is between 30 and 200 times a second. However, it turns out that a new thread is created only when the previous thread completed its work, that is, when the Listener.onFrame() method returns. This prevents a thread-flooding situation, where threads are created at a higher pace than they can be processed.

The combination of these two systems leads to an approach where the onFrame() method on the Listener is used to (directly or indirectly) change the properties of UI components, which are subsequently rendered. The schema shown in Figure 2 describes the flow.

leapmotion-f2

Figure 2

In this flow, the implementation of the Leap Motion Listener is using the Platform.runLater() approach to change properties of the JavaFX controls. This guarantees that those properties are modified only by the JavaFX application thread, as required by the JavaFX platform.

In practice, and especially in more-complex applications, it is often useful to add a step between the Listener implementation and the JavaFX controls. By doing so, the separation between the Leap Motion interface and your JavaFX application is even clearer.

Use Your Intuition

Only a little bit of code is required to integrate the Leap Motion Controller into existing applications. There are huge challenges and, hence, huge opportunities for determining the best way to react to hand movements. Intuition is very important in this area.
For brevity, in this article, we will use the approach where the implementation of the Listener modifies the properties of the JavaFX controls directly, using the Platform .runLater() approach.

We will demonstrate the flow with a very simple example: we will write a JavaFX application that displays a circle. The location and the radius of the circle are determined by hand movements. The full code for this example is available at this Bitbucket site.

As stated at the beginning of this article, JavaFX allows you to create client applications in which there is a large degree of independence between the layout and the input devices. We will first create a JavaFX application that is not dependent on the presence of the Leap Motion Controller. The code in Listing 4 generates a layout with a circle positioned in the middle.

public class LeapConcepts extends Application {
...
@Override
public void start (Stage primaryStage) {
        Circle circle = new Circle(20);
        circle.setFill(Color.GREEN);
        circle.translateXProperty().bind(centerX);
        circle.translateYProperty().bind(centerY);
        circle.radiusProperty().bind(radius);       
        StackPane root = new StackPane();
        root.getChildren().add(circle);
        Scene scene = new Scene(root, 300, 250);
        primaryStage.setScene(scene);
        primaryStage.show();
}
…
}

Listing 4

As you can see from this code, the location (via the translateX and translateY properties) and the radius (via the radius property) are bound to JavaFX properties. These properties are declared in our JavaFX application, and they are made available via public methods, as shown in Listing 5.

private final DoubleProperty centerY = 
new SimpleDoubleProperty(0);
    private final DoubleProperty centerX = 
new SimpleDoubleProperty(0);
    private final DoubleProperty radius = 
new SimpleDoubleProperty(10);
    
    public DoubleProperty centerX() {
        return centerX;
    }
    
    public DoubleProperty centerY() {
        return centerY;
    }
    
    public DoubleProperty radius () {
        return radius;
    }

Listing 5

So far, this application is very static. It shows a green circle with a fixed radius at a fixed position. We will now write a class that listens for Leap Motion Controller data. We do this by extending the Leap Motion Listener class, as shown in Listing 6.

public class MyLeapListener extends Listener {
   @Override
    public void onFrame(Controller controller) {
        Frame frame = controller.frame();
        HandList hands = frame.hands();
        if (!hands.isEmpty()) {
            Hand hand = hands.get(0);
            final float x = hand.palmPosition().getX();
            float y = hand.palmPosition().getY();
            float z = hand.palmPosition().getZ();
            Platform.runLater(() -> {
                app.centerX().set(x);
                app.centerY().set(z);
                app.radius().set(50.-y/5);
            });
        }
    }
}

Listing 6

In this class, we override the onFrame() method. Our imple-mentation of this method will be called when new frame data is available. The frame data is obtained by calling Frame frame = controller.frame();, where the controller instance is passed via the method invocation.

As shown in Listing 6, we can obtain the detected hands easily by calling HandList hands = frame .hands();. If no hands are detected, the hands.isEmpty() call will return true, and we do nothing. If at least one hand is detected, we obtain it by calling Hand hand = hands .get(0);. The position of the palm of the detected hand is obtained as a three-dimensional vector by calling hand.palmPosition().

We map the x and the z coordinates of the Leap Motion coordinate system with the translateX and translateY properties of the circle we created in the JavaFX application. The y coordinate is mapped to the radius property.

Note: Working with the Leap Motion APIs involves some mathematics. You have to transform the coordinates obtained from the Leap Motion Controller to pixels on the screen. The Leap Motion Controller coordinates are expressed in millimeters distance from the center of the top of the Leap Motion.

Because the onFrame method is called on a thread created by the native Leap Motion libraries, we cannot change JavaFX properties directly. Instead, we have to use the Platform.runLater() pattern in order to push the changes to the JavaFX properties onto the event queue.

The only remaining thing we have to do is to create an instance of our Listener and add it to a Controller. This is done in our application class, as shown in Listing 7.

private com.leapmotion.leap.Controller c;
private com.leapmotion.leap.Listener listener;

@Override
public void start (Stage primaryStage) {
    …
    c = new Controller();
    listener = new MyLeapListener(this);
    c.addListener(listener);
}

Listing 7

The Controller is created on the JavaFX application thread, as is the Listener. These methods return immediately, though, and they do not freeze the user interface. It is important to maintain a reference to the controller object, in order to prevent it from being garbage collected.

Simple Map Application

The example we have explored so far is very basic, but it outlines the core principles you have to use when integrating Leap Motion Controller data into your JavaFX application.

The possibilities are endless. We will close this article with a simple map application that is navigated using the Leap Motion Controller rather than with a mouse. The code for this map application is available here. Figure 3 shows a screenshot of this application.

leapmotion-f3

Figure 3

We apply the same core principle as in the previous example, which means that we first create a map application that does not have dependencies on the input device.

We create a MapArea instance that serves as the container where we put MapTile instances. A MapTile represents a 256-x-256-pixel piece of the world map at a given zoom level. We obtain these tiles from OpenStreetMap, which is an open data effort built by a community of mappers. The calculations required to map coordinates and zoom levels onto pixels is beyond the scope of this article. The MapArea class contains a loadTiles() method that will load new MapTile instances and their corresponding map images when needed. There are two potential causes for this:

  • The user is panning the map, and a new area is shown
  • The user is changing the zoom level, and a more detailed version of the map tiles should be rendered

All of this can be achieved via a traditional mouse or trackpad, but we can easily add a Leap Motion Listener that achieves the same end. We extend the Leap Motion Listener class and implement the onFrame method as shown in Listing 8

 @Override
    public void onFrame(Controller controller) {
        Frame frame = controller.frame();
        HandList handList = frame.hands();
        Iterator<Hand> handIterator = handList.iterator();
        while (handIterator.hasNext()) {
            Hand hand = handIterator.next();
            if (hand.isValid() && (hand.fingers().count() > 2)) {
                Vector palmPosition = hand.palmPosition();
                final float x = palmPosition.getX();
                final float z = palmPosition.getZ();
                final float y = palmPosition.getY();
                Platform.runLater(() -> {
                    if (Math.abs(x) > 10) {
                        area.moveX(x/10);
                    }
                    if (Math.abs(z) > 10) {
                        area.moveY(z/10);
                    }
                    if (Math.abs(y-150) > 20) {
                        area.moveZoom(y-150);
                    }
                });
            }
        }
    }

Listing 8

In this code, we once again detect the position of the hand. This time, however, we add an additional check: we will move or zoom only if the user opens his hand. We do this by asking how many fingers are detected. When a hand is closed, the Leap Motion Controller won’t count fingers on it. As long as we count at least two fingers, we decide the hand has been opened.

Next, we will move the map area proportional to the location of the hand, and the zoom level will be changed proportional to the height of the hand.

Learn More


 Leap Motion website

 Leap Motion developer website

 NightHacking video: Interview with Johan Vos on JavaFX and Leap Motion

Conclusion

As we showed in this article, only a little bit of code is required to integrate the Leap Motion Controller into existing applications. There are huge challenges and, hence, huge opportunities for determining the best way to react to hand movements. Intuition is very important in this area. Creative developers will probably have fun experimenting with this cool device.

The JavaFX platform provides a great environment for combining new input devices and existing or new Java code.

Special care is required when dealing with threading, but the JavaFX threading model allows for a perfect decoupling between the rendering of the user interface, and dealing with background events and computations.

Using JavaFX, developers can leverage the great graphical potential provided by the JavaFX controls. The whole Java platform is available for enriching the applications—for example, by providing communication to back-end systems.


vos-headshot


Johan Vos
started working with Java in 1995. He is a cofounder of LodgON, where he is working on Java-based solutions for social networking software. An enthusiast of both embedded and enterprise development, Vos focuses on end-to-end Java using JavaFX and Java EE.