Detailed analysis of flavor-changing decays of top quarks as a probe of new physics at the LHC