Special points and Poincar\'e bi-extensions, with an Appendix by Bas Edixhoven