Hand Image Understanding via Deep Multi-Task Learning

Stem Module. The stem module consists of two 7× 7 convolutional layers with stride 2, and the channels are set to 64 and 128, respectively. Encoder. We employ the main-body of ResNet-50 [5] to implement the encoder. Specifically, the beginning conv1 together with the prediction head are removed, while the remaining conv2 x, conv3 x, conv4 x, and conv5 x are adopted to build the encoder module, and the number of repetitions are 3,4,5, and 6, respectively. Heat-Map Decoder. The heat-map decoder estimates the feature maps hms ft ∈ R256×64×64 to capture semantic features to encode the 2D hand pose. Similar with [10], the decoder also estimates the hms ∈ R21×128×128, based on the hms ft, to represent the locations of 21 hand key points, and the hms is used for intermediatesupervision. Skip connections between the encoder and the heat-map decoder are also adopted to favor the learning procedure. Mask Decoder. The target of the mask decoder is to estimate the feature maps mask ft ∈ R256×64×64 to capture semantic features that encode the hand segmentation mask. Similar to the heat-map decoder, the mask decoder also estimates the hand segmentation mask ∈ R1×256×256 based on the mask ft, and the mask is used for intermediate-supervision. Skip connections between the encoder and the mask decoder are also adopted to favor the learning. POF Decoder. The POF decoder aims to estimate the feature maps pof ft ∈ R256×64×64 to capture semantic features that encode 3D POF encoding. Similar to the