Often presented as competing products on the market of low cost 3D sensors, the Kinect™ and the Leap Motion™ (LM) can actually be complementary in some scenario. We promote, in this paper, the fusion of data acquired by both LM and Kinect sensors to improve hand tracking performances. The sensor fusion is applied to an existing augmented reality system targeting the treatment of phantom limb pain (PLP) in upper limb amputees. With the Kinect we acquire 3D images of the patient in real-time. These images are post-processed to apply a mirror effect along the sagittal plane of the body, before being displayed back to the patient in 3D, giving him the illusion that he has two arms. The patient uses the virtual reconstructed arm to perform given tasks involving interactions with virtual objects. Thanks to the plasticity of the brain, the restored visual feedback of the missing arm allows, in some cases, to reduce the pain intensity. The Leap Motion brings to the system the ability to perform accurate motion tracking of the hand, including the fingers. By registering the position and orientation of the LM in the frame of reference of the Kinect, we make our system able to accurately detect interactions of the hand and the fingers with virtual objects, which will greatly improve the user experience. We also show that the sensor fusion nicely extends the tracking domain by supplying finger positions even when the Kinect sensor fails to acquire the depth values for the hand.
[1]
Frank Weichert,et al.
Analysis of the Accuracy and Robustness of the Leap Motion Controller
,
2013,
Sensors.
[2]
V. Ramachandran,et al.
The use of visual feedback, in particular mirror visual feedback, in restoring brain function.
,
2009,
Brain : a journal of neurology.
[3]
Nadine Warzée,et al.
3D augmented reality applied to the treatment of neuropathic pain
,
2012
.
[4]
Sander Oude Elberink,et al.
Accuracy and Resolution of Kinect Depth Data for Indoor Mapping Applications
,
2012,
Sensors.
[5]
G. Moseley,et al.
Using visual illusion to reduce at-level neuropathic pain in paraplegia
,
2007,
PAIN.
[6]
Herta Flor,et al.
Phantom-limb pain: characteristics, causes, and treatment
,
2002,
The Lancet Neurology.
[7]
Paul J. Besl,et al.
A Method for Registration of 3-D Shapes
,
1992,
IEEE Trans. Pattern Anal. Mach. Intell..
[8]
Adrian Kaehler,et al.
Learning opencv, 1st edition
,
2008
.
[9]
Antonis A. Argyros,et al.
Efficient model-based 3D tracking of hand articulations using Kinect
,
2011,
BMVC.