On using norms for low-frequency words

When norms are used to select words of various frequency of occurrence in the language, great care must be exercised in the selection of low-frequency words. The stability of estimates of frequency for rare words hinges heavily on the size of the corpus on which the word count is based, and on whether the frequency index takes into account the distribution across samples making up the corpus. Although some frequency norms directly provide an index that takes dispersion into account (The American Heritage Word Frequency Book), others do not (e.g., Kucera & Francis, 1967). Researchers predominantly use the Kucera and Francis norms, and they routinely take the total frequency of occurrence in the corpus as their frequency index, with no correction for dispersion.