Skeletal Tracking with JavaScript and the Leap Motion

When it comes to using JavaScript with next generation technology, the Leap Motion is one of my favourite devices to work with. With the Leap Motion, JavaScript is not an after thought. It isn’t a third party module hacked together by enthusiasts – it is developed with care and consideration by the Leap Motion team with each new API release. Their latest release of V2.1, currently in beta, is pretty darn exciting. In this article, I’ll give an overview of the new capabilities available to JavaScript developers.

Key Takeaways

• The Leap Motion API V2.1, currently in beta, provides JavaScript developers with new capabilities for skeletal tracking, including additional hand data and finger data.
• The API includes ‘hand.confidence’ which provides a value to indicate how certain the Leap Motion device is of its data. It also includes ‘hand.type’ to determine if it’s a left or right hand, ‘hand.pinchStrength’ to measure how close together the thumb and finger are, and ‘hand.grabStrength’ to represent whether the hand is open or closed.
• The Leap Motion API provides detailed finger data, including the type of each finger and whether it’s extended or not. It also provides data on the four finger bones in each finger, ordered from wrist to the tip of each finger.
• While the V2 Leap Motion API also supports VR headsets like the Oculus VR and provides access to the actual image data from the Leap Motion, these features are not yet available in the JavaScript API.

Keep Track of Those Hands with Greater Confidence

The API provides additional data for each overall hand. Hand data is available in frame.hands just as it was in the previous API, however in the updates we get a wonderful collection of new data that will make life easier for Leap Motion devs.

Hand Confidence

One of my favorite inclusions in the new API is the ability to know how certain the Leap Motion is of its data. hand.confidence provides a value from 0 to 1 which lets you know how certain what it is seeing matches the model of a hand it understands. Low values could mean it is not certain of overall hand data or it could mean it is uncertain of finger/bone positions. This allows you to check how sure the Leap Motion is before taking an action. I’d love to see this lead to much smoother interactions and hopefully encourage users to be very clear in their gestures.

Hand Type

hand.type works out whether the Leap Motion thinks it is seeing a left hand or a right hand. If your hand is in a fist, then it’s gonna struggle a bit with this distinction.

Pinch Strength

hand.pinchStrength is a wonderful new property which gives a value from 0 to 1 which represents how close together the thumb and finger are to each other. This could open up some interesting possibilities for gestures and controlling apps as a pinch is a nice and natural human gesture.

Grab Strength

hand.grabStrength returns a value from 0 to 1 just as pinch strength does, but it represents whether your hand is open flat or closed into a fist. The possibilities here are equally as exciting – Get your users to squeeze a virtual stress ball or grab and toss something across a virtual room.

New Finger Data

In the new skeletal API changes, every hand has always got five fingers and you can get a whole new set of data for each finger the Leap Motion can see.

Finger Type

You can read in what sort of finger each bit of data represents using finger.type. This object will be a value from zero to four:

• 0 is a thumb
• 1 is an index finger
• 2 is a middle finger
• 3 is a ring finger
• 4 is a pinky finger

Each time Leap Motion reads in hand data, it reads it in with all five fingers.

Finger Extended

If some fingers aren’t pointing out above the Leap Motion, the device records them as not extended using finger.extended = false.

As per the name “Skeletal”, we get data down to the bone

I’m not well versed in the bone structure of the hand, so I had the chance to learn something new here. My doctor would be proud of my expanded knowledge. Here’s hoping the metacarpus comes into casual conversation someday.

The Leap Motion provides data on the four finger bones in each finger, ordered from wrist to the tip of each finger: metacarpal, proximal phalanges, intermediate phalanges and distal phalanges. That makes no sense to most of society, so here’s a lovely diagram I found on Wikipedia Commons:

Finger bone details

It goes into more detail with each bone (I’ve arranged these from the wrist to the tip of the finger):

• finger.carpPosition – The x, y and z position of the base of the metacarpal bone.
• finger.mcpPosition – The position of the finger’s knuckle – the spot between the metacarpal and proximal phalanx bone.
• finger.pipPosition – The position of the spot between the proximal and the intermediate phalanges.
• finger.dipPosition – The position of the base of the distal bone where it connects to the intermediate phalanx.

It also has objects for each finger bone with more detail – finger.metacarpal, finger.proximal, finger.intermediate and finger.distal. Each one of these is represented in the Leap Motion API as a bone with the following details:

• bone.type – a value from 0 to 4 representing the type of bone:

• 0 – Metacarpal

• 1 – Proximal phalanx
• 2 – Intermediate phalanx
• 3 – Distal phalanx

• 4 – Forearm

• bone.basis – provides a two dimensional array that represents the vector of the bone. You can do some pretty crazy math with this to detect particular finger positions. I’ll need to fit in some mathematics practice before I attempt to use these values but it’s brilliant the potential is there!

• bone.prevJoint and bone.nextJoint – the coordinates of the end of the bone closest to the wrist and the end of the bone closest to the finger tip respectively.
• bone.length – the length of that bone in millimeters.
• bone.width – Leap Motion describes this best in their docs as “the average width of the fleshy part surrounding this bone”.
• bone.left – whether the bone is on the left hand (if not, then it’s on the right hand, there doesn’t seem to be a bone.right)
• bone.center – the midpoint of the bone
• bone.direction – the direction vector that the bone is in

Basically, that’s a good amount of data right there.

That Data in a Demo with Lots of Numbers

I threw together a quick demo to show just how much co-ordinate data is streaming through this API. The following demo shows the co-ordinates for each of the finger.carpPosition, finger.mcpPosition, finger.pipPosition and finger.dipPosition, along with basic data that is retrieved for the hand and fingers. It’s only a small fraction of the data available too!

The demo looks a bit like this:

You can try it out in action with your Leap Motion here.

There’s More to Come!

There is more to the V2 Leap Motion API than just these features, it also has support for VR headsets like the Oculus VR and also provides access to the actual image data from the Leap Motion. Sadly, these features aren’t available in the JavaScript API just yet but hopefully we’ll see them soon before the API leaves beta!

Other Resources

If you’re looking for more details on getting started with these APIs, the best place to go is the Leap Motion developer site:

• JavaScript SDK Documentation
• A gallery of Leap Motion JavaScript demos, check out the source code for some good insight!

Frequently Asked Questions about Skeletal Tracking with JavaScript and Leap Motion