Transfer matrix analysis of the two-dimensional ANNNI model