Web services are becoming an essential technology for the development of current distributed applications. Therefore, the organizations must be aware of the possibilities and limitations of the web services and their enabling technologies related to interoperability, performance, security, etc. Benchmarking techniques can provide very useful insights about which technologies are viable and what are the current limitations of the available implementations of those technologies. Furthermore, well established benchmarks provide a way to carry out useful comparisons between two or more implementations. In this paper we present several issues of how web service implementations could be benchmarked. We describe the TPC-App benchmark and explain the most interesting issues of our implementations. Finally we present benchmarking results for the two predominant development platforms, .NET and J2EE.