Difference sets and the primes