Spatial clusters can be detected using a wide range of statistical tests 1 2 , many of which are available in free software packages such as R 3 4 . Epidemiologists use local methods to detect clusters without a priori knowledge of their location, and to determine their significance. Because these cluster detection tests (CDTs) must reveal both the presence and location of clusters, performance studies have been constrained by the limitations of conventional estimation techniques. For example, a CDT may have maximum power for rejecting the null hypothesis (cluster absence), yet be incapable of accurately locating the simulated cluster. CDT performance is also a function of epidemiological and geographical context 1 5 6 7 8 9 10 11 . Furthermore, because epidemiological (e.g., incidence and relative risk) and geographical (e.g., spatial unit size and shape) factors tend to be intrinsically linked, their proper or common effects are difficult to evaluate. When evaluating the behavior of these CDTs in a particular region, limited knowledge can consequently be gleaned by simulating one or a few clusters in that region, and even less knowledge can be accrued from studies on other region.

Takahashi and Tango have proposed the concept of extended power (EP) 12 13 as a more accurate measure of CDT performance. This measure assesses both the probability that the null hypothesis is rejected and the accuracy of the cluster location. As such, it overcomes the inadequacy of conventional power measures. However, EP cannot eliminate the need to define what is meant by “an accurate” or “sufficiently accurate” location. The level of spatial accuracy depends upon context; for instance, an epidemiologist will require higher spatial accuracy for an ad hoc study than for a survey system. Takahashi and Tango therefore introduced a quantitative indicator of spatial accuracy, and summarized CDT performance using an EP curve in conjunction with this spatial accuracy indicator.

In this work, we propose a method that integrates the area under the EP curve (AUC_EP) in order to produce maps that provide a global overview of CDT performance over an entire study region.

The Auvergne region is characterized by low and medium mountains situated around a central plain. The at-risk population (see Methods) was heterogeneously distributed throughout sparsely populated areas (mainly borderland and mountainous) and highly populated urban areas. Figure 1 shows the size of the at-risk population in each cluster, which was assigned to its central SU.

Figure 2 demonstrates how CDT performance improved with increasing risk level. Clearly, the CDT could not detect clusters within regions with low number of births. For these clusters, performance only marginally improved, even at the highest risk combination (Figure 3).

CDT performance increased monotonically with the at-risk population size (Figure 3). We noted a stronger heterogeneity of CDT performance for the clusters with the largest populations, especially at intermediate risk levels (Figure 3); by this, we mean that clusters with nearly the same population size led to slightly different test performance behaviors. For example, Figure 4 shows test performance in detecting three clusters centered on SUs “43770” (red cluster in the figure), “03700” (blue cluster) and “03420” (green cluster), which had population sizes of 544, 558 and 545 births (mean number over 8 years), respectively. At the lowest risk level, the red cluster was the only one even marginally detected, whereas under other configurations, the blue cluster was best detected. The worst detection performance was exhibited with respect to the green cluster, particularly at intermediate risk levels. We note that the green cluster was the only borderland cluster.

Some summary statistics of the AUC_EP distributions are displayed in Table 1. Figure 5 shows two different extended power curves (and thus two different CDT behaviors) that have nearly equal AUC_EP. One of these clusters was centered on SU “03160”, the other on SU “63112”.

^amean number between 1999 and 2006.

Generation of one performance map from 221,000 datasets required about 5 days of computational time using the AUVERGRID grid.

Takahashi and Tango 13 have suggested using the AUC_EP to compare performance between CDTs. We used this synthetic indicator, suitable for compiling maps, to describe CDT performance. It thus fulfills our primary goal of realizing a systematic performance assessment of a CDT over an entire study area, rather than over only a few clusters. This mapping method, although using Takahashi and Tango’s extended power, is not dependent on this concept. Our method can use any other indicator that meets the requirements of being a scalar (i.e., a single measure of performance) indicating both the spatial accuracy of the detection and the capacity of cluster detection tests to reject the null hypothesis.

Interpretation of the AUC_EP requires further exploration, however. Although a higher AUC_EP clearly signifies stronger CDT performance, quite different behaviors can yield the same AUC_EP. As shown in Figure 5, different curves can possess very similar AUC_EP values. This figure shows the extended power curves “03160” and “63112”, whose AUC_EP values are nearly equal (0.931 and 0.932, respectively), but which reflect different CDT behaviors. The procedures used to construct these curves are described in detail within separate spreadsheets (see EP curve in the Additional file 3).

The curve “63112” is nearly horizontal, indicating that the EDCs (H₀ rejected, and cluster size l < maximum cluster size L) located the simulated cluster with high accuracy. As q increases, less tolerance is given to false positives until, eventually, only EDCs with at least one true positive and less than s false positives can contribute to the extended power. A near zero slope thus indicates that the same detected clusters, all of which contain less than s false positives, contribute to the extended power, regardless of q.

The intercept of curve “63112” is 0.939, meaning that eligible clusters (l < L), all of which contribute to the extended power (i.e., all clusters contain at least one true positive), were detected in 93.9% of the tests (H₀ rejected).

To summarize curve “63112”, the simulated cluster was not always detected (no H₀ rejection or EDC without true positive); however, provided that an EDC identified at least one true positive, the location was accurate (i.e., less than s false positives existed in the cluster).

In contrast, the curve “03160” yields the same AUC_EP, but is negatively sloped with an intercept of 0.951. Thus, the associated CDT produced more EDCs containing at least one true positive. The negative slope indicates that a higher proportion of these EDCs generated at least s false positives.

To summarize curve “03160”, the test rejected H₀ more often and/or produced more EDCs, but located the simulated cluster with less accuracy (i.e., this analysis produced more than s false positives).

One particular curve has intercept equal to 1 (q = 0) and a zero slope. An intercept equal to 1 implies that the CDT always rejects H₀ and that no false negatives exist in the EDCs. All detected clusters entirely overlap the simulated cluster, as in all other cases the weighting function W _{(l, s*, q=0)} is less than one. In addition, the zero slope indicates the perfect test that always exactly locates the simulated cluster. A perfect test always rejects H₀, and detected clusters always satisfy l = s* = s (i.e., generate no false positive or negative). The AUC_EP of a perfect test equals one, because in all other cases W _{(l, s*, q)} is less than one.

The intercept of an extended power curve can be regarded as a “quantitative” feature of CDT performance (all EDCs generating true positives contribute to the extended power), whereas the slope may be thought of as a “qualitative” feature of CDT performance, assessing location accuracy. The parameter q can, in fact, be regarded as a continuous indicator reflecting to what extent a detected cluster must accurately locate the simulated cluster to contribute to the performance measure.

As shown in Figure 5, however, if an entire curve is condensed into a single measure (such as the AUC), some information is lost, because CDTs with different behaviors (i.e., curves with different shapes) can yield the same performance value.

Consequently, the impact of CDT behavior on the extended power curve must be thoroughly explored, and behaviors relevant to a particular research or application need to be defined. Through such exploration, the extent to which the AUC_EP is a relevant performance measure, and the purposes for which it is most suited, can be determined.

The EP has the advantage of requiring only one arbitrarily set parameter. In this work, the parameter L, that determines the maximum allowed size for EDCs, has been set to 30 SUs. Takahashi and Tango 12 initially proposed to set the limit L to one fourth or one third of region size (in numbers of SUs). The authors stated that it was not unreasonable to assume that an actual cluster size will be less than such a limit. Such arguments are often open to dispute but in any case, it is an arbitrary decision. In our view, it would be more correct to set L according to the size s of the simulated cluster because, in the simulation, it is the “real” cluster. By construction, the consequences of this arbitrary setting are limited to the lowest values of q. Indeed, low values of q mean that EDCs with false positives are less penalized, and thus large clusters are allowed to contribute to EP. In our case (L = 30), only values of extended power for q ≤ 0.15 could be underestimated, and only if we consider that detected clusters more than 7.5 times larger than the simulated cluster (4 SUs) are still meaningful. At last, compared with L set to 30, computing AUC_EP with L equal to 221 (i.e. without an arbitrary limit) yields a difference in AUC_EP always less than 10^-5 in this work.

In producing our performance map, we chose to assign the AUC_EP value of a single cluster of four SUs to a single SU. Because two clusters centered on neighboring SUs likely contain common SUs, and the AUC_EP evaluates the detection of the entire cluster, visualizing performance on a single map can only be done in two ways. On the one hand, the AUC_EP of a cluster can be assigned to each of its SUs, or on the other hand, it can be assigned to a single, albeit arbitrarily chosen, SU. In the first solution, as each SU has a strong probability to be associated with more than one cluster, it is then necessary to compute a summary statistic, such as the mean, to produce a single map. In our view, it seems more comprehensible to arbitrary assign the performance measure for the whole cluster on a single SU. As we simulated more or less circular clusters, the central SU of the cluster was naturally chosen for this assignment. When simulating different cluster shapes, this choice will clearly be less obvious. We nevertheless recommend assigning the performance measure to the SU where the centroid of the cluster is located.

Authors who have studied CDT behavior mentioned its dependence on epidemiological and geographical factors 1 5 6 7 8 9 10 11 . Consistent with previously published results, the performance of Kulldorff’s spatial scan, and more generally, all local CDTs, improves in study regions of small SUs, large populations, high incidence of the studied phenomenon and for clusters with strong relative risk. Furthermore, as shown in Figure 4 and Table 1, the variation in AUC_EP among very similar simulated clusters (identical length, shape, population size and risk association) suggests that other factors influence CDT performance. To our knowledge, no other simulation study has been performed to both assess and visualize CDT performance over an entire region. Until now, authors have always considered a limited set of simulated clusters with particular epidemiological or geographical characteristics of interest. Consider the typical example of population size effect. To assess this effect, clusters are generally simulated in only a few arbitrarily chosen locations where a CDT behavior is assumed to be representative of its behavior in any other “similar” location. Usually, clusters in rural areas are compared with clusters in urban areas. Such studies are not sufficient to assess this factor that, as we have shown (Figure 3), has a strong relationship with CDT performance. Furthermore, population size cannot explain in itself all the variability in CDT performance.

However, some authors 21 have assessed performance on many randomly located clusters, which is a way to take into account the effect of spatial location without assessing it. It enabled them to assess the effect of factors such as relative risk or spatial resolution without the potential confounding effect of the spatial location. Still, this approach, while accounting for this effect, cannot quantify it.

Our systematic evaluation allows us to assess exactly when heterogeneity is most important, and thus within what population size range we can expect any other potential factor to have a maximum effect. In this work, we used predefined values for incidence and clustering characteristics (relative risk, shape, size and number) to generate performance maps. Epidemiologists should use reasonable values if a priori knowledge is available for some factors. However, the proper effect of any factor on CDT performance can be studied with this systematic evaluation, provided it uses suitable measure such as the AUC_EP.

Given that CDT performance depends on geographical and epidemiological context, the performance of these methods should be explored prior to monitoring a particular phenomenon in a given region. This work enables epidemiologists to study global CDT performance over an entire region. Furthermore, from a research viewpoint, our method seems beneficial for unraveling the proper effect of many factors, particularly geographical ones, on CDT performance.

AUCEP: Area under the curve of extended power; CDT: Cluster detection test; EDC: Eligible detected cluster; EP: Extended power; H0: Null hypothesis; H1: Alternative hypothesis; Iall: Incidence of all birth defects; Icv: Incidence of cardiovascular birth defects; RR: Relative risk.

The authors declare that they have no competing interests.

AG and LO conceived the design, performed the study and drafted the manuscript. AG was responsible for statistical programming and data analysis. JD, JG, IP, XL and JYB contributed to manuscript revision. All authors read and approved the final manuscript.