Searching for intermediate mass black holes: understanding the data first