Further reflections on the Serial Gatekeeper
In some settings, we want to test the exploratory hypotheses after all preceding primary and/or key secondary endpoints have already been rejected. I am recently realized that such MTPs are called serial gatekeeping.
Per FDA guidance, two types of gatekeeping testing strategies are common in clinical trials, serial and parallel, determined by how the endpoints are tested within the primary family. The term serial strategy is applied when the endpoints of the primary family are tested as co-primary endpoints (notes: intersection-union testing). If all endpoints in the primary family are statistically significant at the same alpha level (e.g., \(\alpha\) = 0.05), the endpoints in the second family are examined. Here, the primary family can be treated as the family includes all primary/key secondary endpoints that deemed to be published in the labeling, and the second family can be treated as the family contains the endpoints we want to explore. The very key idea for the primary family, which is the serial gatekeeper, is that the endpoints in the second family are tested by any one of several possible methods (e.g., Holm procedure, the fixed-sequence method, or others). If, however, at least one of the null hypotheses of the primary family fails to be rejected, the primary family criterion has not been met and the secondary endpoint family is not tested.
A family is termed a serial gatekeeper if all hypotheses must be rejected within that family in order to proceed to the next family in the sequence. If family \(F_{i}\) (\(i = 1,2,\cdots,m - 1\)), is a serial gatekeeper, hypotheses in \(F_{i + 1}\) are tested iff \(\max_{j = 1,\ldots,n_{i}}{\widetilde{p_{ij}} \leq \alpha}\) (assume there are \(n_{i}\) individual hypotheses within family \(F_{i}\)). Mathematically, we can use the error rate function (introduced in Dmitrienko et al. (2008)) to explain:
\[e(I) = \underset{H_{I}}{SUP\;}{\Pr\left( \bigcup_{i \in I}^{}{\left\{ reject\; H_{i} \right\} \mid H_{I}} \right)},\]
for any \(I \subseteq \{ 1,2,\ldots,n\}\), where \(H_{I} = \bigcap_{i \in I}^{}H_{i}\) is the intersection of hypotheses \(H_{i}\) with \(i \in I\). Based on the paper by Zhiying Qiu 2018 (Link), we can use the upper bounds \(e^{*}( \cdot )\) to replace the error function, if it cannot be calculated easily. Let \(A_{i}\) be the set of accepted (not rejected) hypotheses in family \(F_{i}\). We can calculate \(e^{*}(A_{i})\) after testing \(F_{i}\) at level \(\alpha_{i}\) and then transfer the remaining amount of its local critical value \(\alpha_{i} - e^{*}(A_{i})\) to the family in the subsequent layer. Note that if all hypotheses in the \(F_{i}\) are rejected, then \(e^{*}\left( A_{i} \right) = e^{*}(\phi) = 0\). It means that all \(\alpha\) used in the serial gatekeeper will be transferred to the subsequent family in the next layer.
A common usage is serial gatekeeping with a single decision-making branch. Within each family \(F_{i}\ (i = 1,2,\ldots,m)\), all hypotheses are tested at the nominal \(\alpha\) level, for an example, the hypotheses in family \(F_{1}\) are testing using an intersection-union test, i.e. all hypotheses are rejected in \(F_{i}\) if \(p_{ij} < \alpha\), and all hypotheses are retained otherwise. A common MTP used in this case is fixed sequence procedure. For the bottom layer \(F_{m}\), any FWER-controlling test can be used.
The MTP process below is my attempt to use a graphical approach to visualize a serial gatekeeping strategy. In the example below (see Figure 1), the hypothesis tests for H5 to H8 will only begin after the hypothesis tests for H1 to H4 have all been rejected (\(\lim \epsilon \to 0\); in code we can assign some value like \(10^{-4}\) to \(\epsilon\)).
However, for the MTP used in the bottom layer \(F_{m}\), the Hochberg procedure should be used with cautions. Although it gains more power than Holm/Bonferroni (Hommel > Hochberg > Holm > Bonferroni, doi: Link), The Hommel and Hochberg procedures require the assumption of independence of the p-values since they are based on the Simes (1986) test, which makes this assumption, while the Holm and Bonferroni procedures do not have this restriction (these two procedures are assumption-free).
Per FDA, The Hochberg procedure is known to provide adequate overall alpha-control for independent endpoint tests and also for two positively-correlated dependent tests with standard test statistics, such as the normal Z, student’s t, and 1 degree of freedom chi-square. It is also a valid test procedure when certain conditions are met. Various simulation experiments for the general case (e.g., for more than two endpoints with unequal correlation structures) indicate that the Hochberg procedure usually will, but is not guaranteed to, control the overall Type I error rate for positively-correlated endpoints, but fails to do so for some negatively-correlated endpoints. Therefore, beyond the aforementioned cases where the Hochberg procedure is known to be valid, its use is generally not recommended for the primary comparisons of confirmatory clinical trials unless it can be shown that adequate control of Type I error rate is provided.
However, if the idea of “serial gatekeeping” is no more appropriate for the trial settings, i.e. we don’t want to test hypotheses in the family \(F_{i + 1}\) only if all hypotheses in \(F_{i}\) are rejected, then we should reach out to the idea as parallel gatekeeping. The conventional Holm/Hochberg (etc..) should be updated to truncated version by taking a convex combination of the critical values with the critical values of the Bonferroni test, in order to meet the separability condition (again, see Dmitrienko et al. (2008)).
More papers to read:
Applications of general multistage gatekeeping and graphical multiple testing strategies in a clinical trial setting”: https://www.sciencedirect.com/science/article/pii/S1551714417308133
Maurer W, Bretz F. A note on testing families of hypotheses using graphical procedures. Stat Med. 2014 Dec 30;33(30):5340-6. doi: 10.1002/sim.6267. Epub 2014 Jul 20. Erratum in: Stat Med. 2019 Jun 15;38(13):2504. doi: 10.1002/sim.8122. PMID: 25043831.
Appendix
