The Nesting-Depth of Disjunctive µ-Calculus for Tree Languages and the Limitedness Problem

In this paper we lift the result of Hashiguchi of decidability of the restricted star-height problem for words to the level of finite trees. Formally, we show that it is decidable, given a regular tree language Land a natural number kwhether Lcan be described by a disjunctive μ-calculus formula with at most knesting of fixpoints. We show the same result for disjunctive μ-formulas allowing substitution. The latter result is equivalent to deciding if the language is definable by a regular expression with nesting depth at most kof Kleene-stars. The proof, following the approach of Kirsten in the word case, goes by reduction to the decidability of the limitedness problem for non-deterministic nested distance desert automata over trees. We solve this problem in the more general framework of alternating tree automata.