Transformations of Theta-functions and analogues of Dedekind sums