In this paper, we present an image retrieval architecture using specific feature vectors for the fashion domain. The triplet network is widely used in an image retrieval task with a deep-learning method. The vector from the deep neural network reflects general image characteristics such as color and pattern on the image. On the other hand, the characteristics of the fashion items, such as length, fit, detailed design part, etc., are not well combined in their vector because it is difficult to obtain the individual information of the domain by a deep convolutional neural network (CNN). To consider the specific characteristics of the fashion items, we proposed a fashion image retrieval network, which is called CoFirNet, applying domain-specific feature vectors extracted from the pretrained network based on a classification scheme to distinguish the domain property in an image. The domain-specific feature vector, conditional vector, are combined with the general characteristics feature vector to build the embedding vector for the image retrieval scheme to include the characteristics of fashion items. The proposed method is evaluated, and it shows that the objective and subjective performance were higher than the compared method on the fashion images. Therefore, we believe that the proposed scheme can be a useful method and system for a specialized image search system. etc.