Autonomic Operation of a Large High Performance On-Line Compute Cluster