Process
Initial Methods
I began by exploring multiple hand-tracking methods, including color tracking and tracking the edges of the hands. The goal was to detect hands in various shapes and sizes while ensuring proper interaction with virtual or physical objects.
To improve accuracy, I attempted averaging out five points on the edge of the hands. This method was an attempt to better represent the hand’s shape and position in Unity. However, I encountered issues with the flexibility of hand shapes, where the tracking system failed to adjust properly to hands with varying forms (e.g., spread fingers vs. a fist).
Python:
Unity (Handtracker)
A more flexible approach
Bounding Box Tracking
The next approach was to use bounding boxes to track the hand. A bounding box is drawn around the largest contour, representing the hand’s position and size. The center of the box could then be used for hand control.
Python:
Unity (Handtracker)
While the bounding box helped in defining the hand’s position, the main issue occurred with the Z-axis: the size of the bounding box would change when the hand moved, but the Z-axis (depth) didn't track accurately, causing issues with spatial positioning in Unity.
Challenges
Problem with Z-Axis Movement
The main issue I faced with flexible hand shapes was that the object's tracked shape would continuously scale up and down without proper adjustment to the Z position of the hand. This led to inconsistent interactions where the hand’s depth was misrepresented in Unity, causing the virtual objects to either overlap or float incorrectly in the 3D space.
Solution
Center Tracking
To address this, I shifted focus from tracking the edges of the hand to tracking the center of the hand instead. By averaging the position of the center of the hand and using that as the reference point, I was able to achieve more stable and accurate tracking. This method ensured that even with varying hand shapes, the Z-axis movement was handled more reliably, leading to natural interactions between the hand and virtual objects.
Python:
Unity (Handtracker)
By averaging the hand's center and using that for movement, I avoided the problems of scaling and misalignment, providing a more intuitive control mechanism in Unity for hand interactions.
