Nonperturbative superpotentials from membrane instantons in heterotic M-theory