Experimental measurement of a nonlinear entanglement witness by hyperentangling two-qubit states