Hofstadter butterfly in optical multilayers