Experimental graybox quantum control