Rajalakshmi Nandakumar, a doctoral student in computer science and engineering at the University of Washington, is the lead author in the study on FingerIO. ( photo)

WASHINGTON — As wearable devices grow and screens shrink, a team of Indian American researchers has developed a new sonar technology that allows you to interact with your smartphones and smartwatches by writing or gesturing on any nearby surface, such as a tabletop or sheet of paper, or even in mid-air.

Called FingerIO, it tracks fine-grained finger movements by turning a smartphone or smartwatch into an active sonar system using the device’s own microphones and speakers.

Because sound waves travel through fabric and do not require a line of sight, users can even interact with a phone inside a front pocket or smartwatch hidden under a sweater sleeve.

“You can’t type very easily onto a smartwatch display, so we wanted to transform a desk or any area around a device into an input surface,” said lead author Rajalakshmi Nandakumar, a doctoral student in computer science and engineering at the University of Washington.

“I don’t need to instrument my fingers with any other sensors. I just use my finger to write something on a desk or any other surface, and the device can track it with high resolution,” Nandakumar added.

FingerIO can accurately track two-dimensional finger movements to within eight millimeters, which is sufficiently accurate to interact with mobile devices.

Using FingerIO, one can use the flick of a finger to turn up the volume, press a button or scroll through menus on a smartphone without touching it, or even write a search command or text in the air rather than typing on a tiny screen.

“Using sound waves to track finger motion offers several advantages over cameras and other technologies like radar that require both custom sensor hardware and greater computing power,” explained Shyam Gollakota, senior author and assistant professor of computer science and engineering.

FingerIO turns a smartwatch or smartphone into a sonar system using the device’s own speaker to emit an inaudible sound wave.

That signal bounces off the finger, and those “echoes” are recorded by the device’s microphones and used to calculate the finger’s location in space.

“Acoustic signals are great, because sound waves travel much slower than the radio waves used in radar; you don’t need as much processing bandwidth, so everything is simpler,” Gollakota noted.

From a cost perspective, almost every device has a speaker and microphones, so you can achieve this without any special hardware.

The researchers employed a type of signal typically used in wireless communication called orthogonal frequency division multiplexing.

They demonstrated that it can be used to achieve high-resolution finger tracking using sound.

Their algorithms leverage the properties of OFDM signals to track phase changes in the echoes and correct for any errors in the finger location to achieve sub-centimeter finger tracking.

To test their approach, the researchers created a FingerIO prototype app for Android devices and downloaded it to an off-the-shelf Samsung Galaxy S4 smartphone and a smartwatch customized with two microphones, which are needed to track finger motion in two dimensions. Today’s smartwatches typically only have one, which can be used to track a finger in one dimension.

The average difference between the drawings and the FingerIO tracings was 0.8 centimeters for the smartphone and 1.2 centimeters for the smartwatch.

“Given that your finger is already a centimeter thick, that’s sufficient to accurately interact with the devices,” said co-author Vikram Iyer.

The team is now working on how FingerIO can be used to track multiple fingers moving at the same time and extending its tracking abilities into three dimensions by adding additional microphones to the devices.

The paper will be presented at the Association for Computing Machinery’s CHI 2016 conference in San Jose, Calif., in May.

Watch a demonstration of FingerIO:

(0) comments

Welcome to the discussion.

Keep it Clean. Please avoid obscene, vulgar, lewd, racist or sexually-oriented language.
Don't Threaten. Threats of harming another person will not be tolerated.
Be Truthful. Don't knowingly lie about anyone or anything.
Be Nice. No racism, sexism or any sort of -ism that is degrading to another person.
Be Proactive. Use the 'Report' link on each comment to let us know of abusive posts.
Share with Us. We'd love to hear eyewitness accounts, the history behind an article.