ITP Spring '22 → Connected Devices → Week 3

It's a short one this week because both Energy and Intangible Interaction have projects due next week and we spent most of the weekend working on those projects 🥲

For Intangible Interaction this week we were assigned to look at APDS9960 which has four IR sensors arranged in a way that can give us gesture information (in addition to proximity and color). I thought it'd be fun to use that in conjunction with this week's Connected Devices assignment to create an HID (Human Interface Device) with our Arduino. So the idea is to gesture left or right to navigate forwards or backwards in the Books app.

Wiring

The APDS9960 uses I2C so I connected it to POWER, GROUND, SCL, and SDA (I'm not 100% sure when the INTERRUPT pin should be hooked up though) and verified the wiring with this Arduino guide just in case.

Once I had it hooked up, I installed the corresponding Arduino library and tried the gesture example. I had some trouble getting it to work at first, and it turns out I wasn't gesturing close enough to the sensor and was also gesturing too quickly.

I tried looking through the sensor's datasheet for its optimal range but couldn't seem to find that information on a quick scan?

Code

Once I had the sensor going, I added in the keyboard functionality with the Arduino Keyboard library (an interesting note on serial port number changing when using Arduino as HID) and its keyboard press command. Thing to keep in mind: the keypress command assumes the key is continuously pressed unless we follow it up with a release() or releaseAll() command.

Initial gesture to arrow keys mapping (left) but ended up flipping the mapping because that felt more intuitive for flipping through a book (right).

I at first mapped my UP and RIGHT gestures to the right arrow and DOWN and LEFT to the left arrow on the keyboard (left video) but that actually didn't feel as intuitive, so I ended up flipping it (right video).

I initially thought this might be a fun idea for when I need to be hands free but still want to read a book on my device (when I'm blow drying my hair, for example), but I don't think this particular sensor is the right one for the job; I had to spend too much time making sure my gesture was just right. When I'm blow drying my hair, I want to flip to the next page with just a quick flick of my wrist.

Final code here

P.S. the book on the screen is Brandon Sanderson's latest Stormlight Archive, aka. the book I pre-ordered early 2020 in excitement but still haven't read because I've been scared I'd get sucked back in and lose a week of my life doing nothing but reading lol