Undecidability of the State Complexity of Composed Regular Operations

We consider the regularity-preserving operations of intersection and marked catenation and construct an infinite sequence Ci, i = 1, 2, . . ., of compositions formed from the two operations. We construct also an infinite sequence of polynomials Si, i = 1, 2, . . ., with positive integer coefficients. As a main result we prove that it is undecidable whether or not Si is a state complexity function of Ci. All languages needed are over a fixed alphabet with at most 50 letters. We also consider some implications and generalizations, as well as present some open problems.