Instanton effects in orientifold ABJM theory