[Web App] Second Hand Music – Hand-Tracked Musical Interaction

Interactive Experience
3.1k words

Overview

Second Hand Music is a web-based interactive music experience where users can play notes using hand gestures, without any physical instrument. The game uses MediaPipe for real-time hand tracking and p5.js for sound visualization, creating an intuitive and immersive gesture-controlled musical playground.

Inspired by music rhythm games like Tone Sphere and Cytus, it combines gesture-based note playing with a minimalist UI, making it accessible to all—except, unfortunately, for colorblind players, as the core mechanic is color-based.

We aimed to create a playful musical experience where users could compose experimental melodies with just their hands and a webcam. The idea was to make music creation accessible to people who don’t own instruments but enjoy exploring interactive sound experiences.

Game Modes

  • Free Mode: Fixed circles aligned with each finger’s assigned color—perfect for experimentation.
  • Game Mode: Randomized circle patterns appear on screen, challenging the player to match them and improve their score.

Key Features

Real-Time Hand Tracking with MediaPipe

  • Implemented gesture detection to allow users to interact with virtual notes using their fingers.
  • Optimized landmark tracking to ensure smooth and accurate motion detection.
Hand Tracking Setup
1
2
3
4
5
6
7
8
9
const hands = new Hands({
  locateFile: (file) => `https://cdn.jsdelivr.net/npm/@mediapipe/hands/${file}`,
});
hands.setOptions({
  maxNumHands: 2,
  modelComplexity: 1,
  minDetectionConfidence: 0.5,
  minTrackingConfidence: 0.5,
});

Interactive Sound System & Color Matching Mechanic

  • Created randomly generated interactive circles, each triggering a different musical note.
  • Implemented a color-matching mechanic: notes only play if the finger color matches the circle color.
Color Matching & Note Trigger
1
2
3
4
5
6
7
8
9
10
p.circleCrash = (x, y, finger_index) => {
  for (let i = 0; i < circles.length; i++) {
    if (p.dist(x, y, circles[i].x, circles[i].y) < circles[i].size) {
      if (p.matchingColorwithFinger(finger_index, circles[i].a)) {
        circles[i].size = 6;
        active_music[i].play();
      }
    }
  }
};

Gesture-Based Instrument Selection

  • Integrated hand gesture-based controls for switching between piano, bell, and electronic sounds.
  • Designed an intuitive UI where touching an instrument icon changes the active sound.
Gesture-Based Instrument Change
1
2
3
4
5
6
7
p.changeActiveInstrument = (x, y, finger_index) => {
  if (finger_index == 8) {
    if (p.dist(x, y, p.width / 2 - 182, 50) < 64) p.changeInstrument(1);
    if (p.dist(x, y, p.width / 2 - 32, 50) < 64) p.changeInstrument(0);
    if (p.dist(x, y, p.width / 2 + 118, 50) < 64) p.changeInstrument(-1);
  }
};

Vue.js Frontend & UI/UX Design

  • Built a structured, responsive interface with Vue.js for a clean and fluid user experience.
  • Created a simple yet effective rules section to onboard players quickly.

Animated Loader for Seamless UX

  • Developed a custom CSS loader to enhance user experience while assets load.
Loader Animation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
.loader-wrapper {
  width: 100%;
  height: 100%;
  position: absolute;
  top: 0;
  left: 0;
  background-color: #242f3f;
  display: flex;
  justify-content: center;
  align-items: center;
}

.loader {
  display: inline-block;
  width: 30px;
  height: 30px;
  position: relative;
  border: 4px solid #fff;
  animation: loader 2s infinite ease;
}

@keyframes loader {
  0%, 25% { transform: rotate(0deg); }
  50%, 75% { transform: rotate(180deg); }
  100% { transform: rotate(360deg); }
}