Discussion of “Some Controversial Multiple Testing Problems in Regulatory Applications”

This paper highlights some of the complexities involved with assessing regulatory submissions that involve multiple testing. It provides some useful observations on when certain gate keeping strategies are illogical and hence not appropriate for use in regulatory submissions. In particular, the section on “Testing Superiority and Non-Inferiority for Multiple Endpoints” discussed problems with using a sequential gate keeping strategy. In this section, sequential gate keeping is criticized in the situation where there are two endpoints of interest. The following observations on this section should be noted. In the example given, an applicant would like to demonstrate non-inferiority on one endpoint and then test for non-inferiority on a second endpoint. It is stated that if the pre-condition for testing the second endpoint is that the first endpoint demonstrates non-inferiority and that no superiority testing is performed, then this sequential gate keeping strategy will have a strong control of the ‘studywise’ type I error rate. Although this is true, it is of no practical use, as the testing of the first endpoint for non-inferiority and superiority utilizes the same 95% confidence interval. Hence, if superiority of the first endpoint was shown it will (and should) be claimed by an applicant. Thus, in reality, it has to be assumed that a test for superiority on the first endpoint has been performed. It is agreed that neither sequential chain defined in this section is logically sensible. The first chain is illogical as demonstrating superiority on the first endpoint and should not be necessary if the application could be successful if non-inferiority on the first endpoint is all that is required. Similarly, as already mentioned, the second chain is illogical, as if superiority on the primary endpoint is achieved; the gate keeping strategy should always ensure this claim is not prohibited by the failure of another endpoint to demonstrate non-inferiority. It is, therefore, good to see the limitations of sequential gate keeping in this situation being highlighted. Another practical consideration is the non-inferiority margin chosen by the applicant. If the margin is considered too large by the regulator then it would not be appropriate to conduct any further tests predefined by the sequential gate keeping strategy, unless a more acceptable narrower margin is also satisfied.