Inserm, CESP Centre for Research in Epidemiology and Population Health, U1018, Biostatistics team, F94807 Villejuif, France
Inserm, CESP Centre for Research in Epidemiology and Population Health, U1018, Hormones and Cardiovascular Disease team, F94807 Villejuif, France
Abstract
Background
The weighted estimators generally used for analyzing casecohort studies are not fully efficient and naive estimates of the predictive ability of a model from casecohort data depend on the subcohort size. However, casecohort studies represent a special type of incomplete data, and methods for analyzing incomplete data should be appropriate, in particular multiple imputation (MI).
Methods
We performed simulations to validate the MI approach for estimating hazard ratios and the predictive ability of a model or of an additional variable in casecohort surveys. As an illustration, we analyzed a casecohort survey from the ThreeCity study to estimate the predictive ability of Ddimer plasma concentration on coronary heart disease (CHD) and on vascular dementia (VaD) risks.
Results
When the imputation model of the phase2 variable was correctly specified, MI estimates of hazard ratios and predictive abilities were similar to those obtained with full data. When the imputation model was misspecified, MI could provide biased estimates of hazard ratios and predictive abilities. In the ThreeCity casecohort study, elevated Ddimer levels increased the risk of VaD (hazard ratio for two consecutive tertiles = 1.69, 95%CI: 1.631.74). However, Ddimer levels did not improve the predictive ability of the model.
Conclusions
MI is a simple approach for analyzing casecohort data and provides an easy evaluation of the predictive ability of a model or of an additional variable.
Background
Casecohort surveys produce incomplete data by design. A subcohort is selected by simple or stratified random sampling, all subjects are followed up and the events of interest are recorded. The phase1 variables are observed for the entire cohort, whilethe phase2 variables are only known for the casecohort sample, i.e., subjects belonging to the subcohort and all those presenting the event of interest
Various approaches have been described to estimate the proportional hazard model in casecohort surveys: Weighted estimators
No standard method exists for quantifying the usefulness or predictive ability of a model or an additional variable in the framework of casecohort surveys. The predictive ability can be measured in terms of calibration, which refers to the ability of a model to match predicted and observed values, when we are interested in individual predictions; or in terms of discrimination, which refers to the ability of a model to distinguish between subjects with or without a binary event, when we are interested in identifying a group of highrisk subjects. In the present work, we focus on discrimination.
As shown below, a naive measurement of predictive ability from casecohort data often leads to a biased estimate of the predictive ability because it varies with the censoring rate and thus depends on the subcohort size. Alternatively, because MI reconstitutes whole cohorts, any tool developed to estimate the predictive ability in the framework of cohort surveys can be applied to casecohort data, so we propose using the MI approach to estimate the predictive ability ofa model or of an additional variable and their standard errors.
The objectives of this study were 1) to evaluate MI for estimating hazard ratios when the distribution of the phase2 variable is misspecified; and 2) to present an adequate methodology for estimating the predictive ability of a model or of an additional variable in casecohort surveys. We performed a simulation study to validate the MI approach for estimating the predictive ability of a model or of an additional variable and to assess its potential limits. As an illustration, we analyzed casecohort data from the ThreeCity study
Methods
Incomplete observations and multiple imputation
Casecohort surveys are a particular type of incomplete observations, in which data are missing at random
where
where the factor 1 +
MI requires a model correctly reflecting the relationship between the incomplete variable and the outcome of interest. In casecohort surveys, we need to impute phase2 variable values for the noncases who do not belong to the subcohort. Under the rare disease assumption, we have shown that a simple generalized linear model, using all the complete data (cases and noncases) and including the case indicator among the explanatory variables, has to be considered
Predictive ability of a model and of a supplementary variable
Harrell
where π
In practice, we are often interested in estimating the predictive ability of an additional phase2 variable. Let
Simulation study
Two phase1 variables were simulated: a binary variable,
Five imputations were performed and 5 complete data sets were generated for each cohort. We estimated the log hazard ratios using MI and the "Borgan II" weighted estimator
Casecohort survey from ThreeCity study
Briefly, the 3CStudy was designed to examine the relationship between vascular diseases and dementia in a community housing 9,294 persons aged 65 years and over between 1999 and 2001 in three French cities. The detailed methodology has been previously described
We reassessed the relationship between plasma Ddimer levels and the risk of CHD and VaD, using MI and weighted estimators, and evaluated the predictive ability of Ddimer levels on both risks. We included the same explanatory variables as Carcaillon
For each outcome (CHD or VaD), it was necessary to reproduce the relationships among the incomplete variable, the outcomes and the confounder variables. For each outcome, we built an imputation model of tertiles of Ddimer levels, including the variables used in the proportional hazard model and the caseindicator. We estimated the predictive ability of proportional hazard models, without (
Results
Simulation study
The mean fraction of missing information about the effect of
Mean of the log hazard ratio estimates (Est), mean of the standard error estimates
Full cohort
Multiple imputation
Weighted estimator
Est
SE
MSE
Est
SE
MSE
Est
SE
MSE
Z_{2 }normally distributed
0.003
0.107
0.100
0.010
0.003
0.107
0.110
0.010
0.001
0.133
0.128
0.016
0.001
0.054
0.058
0.003
0.001
0.060
0.062
0.004
0.001
0.065
0.068
0.005
0.004
0.053
0.056
0.003
0.004
0.054
0.057
0.003
0.003
0.058
0.060
0.004
0.689
0.118
0.113
0.013
0.676
0.119
0.112
0.013
0.696
0.168
0.165
0.027
0.687
0.058
0.057
0.003
0.679
0.070
0.068
0.005
0.701
0.088
0.097
0.009
0.683
0.057
0.057
0.003
0.679
0.058
0.058
0.004
0.689
0.080
0.090
0.007
Z_{2 }log normally distributed
0.003
0.107
0.100
0.010
0.003
0.107
0.100
0.010
0.004
0.133
0.128
0.016
0.001
0.027
0.034
0.001
0.015
0.031
0.032
0.001
0.002
0.034
0.038
0.001
0.004
0.053
0.056
0.003
0.004
0.054
0.058
0.004
0.005
0.059
0.062
0.004
0.686
0.058
0.056
0.003
0.621
0.061
0.055
0.008
0.686
0.112
0.117
0.014
0.692
0.013
0.015
2e0^{4}
0.602
0.015
0.014
0.008
0.695
0.020
0.023
0.001
0.685
0.029
0.031
0.001
0.686
0.032
0.031
0.001
0.687
0.049
0.053
0.003
Z_{2 }uniformly distributed
0.007
0.181
0.175
0.031
0.007
0.181
0.175
0.031
0.007
0.197
0.188
0.035
0.001
0.092
0.087
0.008
0.004
0.094
0.088
0.008
0.002
0.098
0.095
0.009
0.003
0.090
0.090
0.008
0.002
0.090
0.090
0.008
0.004
0.093
0.093
0.009
0.690
0.120
0.116
0.013
0.680
0.121
0.115
0.013
0.694
0.166
0.169
0.028
0.695
0.069
0.063
0.004
0.656
0.075
0.066
0.006
0.698
0.087
0.082
0.007
0.690
0.058
0.054
0.003
0.689
0.059
0.055
0.003
0.698
0.081
0.081
0.007
^{a }MI estimates with imputation model:
The results concerning the consistency of the naive estimator of Harrell's
Mean of the predictive ability estimates (Est), mean of the standard error estimates
Est
SE
%
Est
SE
%
Est
SE
%
Full Cohort
0.518
0.033
0.012
0.727
0.032
0.015
0.733
0.029
0.014
0.524
0.033
0.013
0.747
0.031
0.015
0.733
0.029
0.014
0.006
0.010
0.009
3.7
0.020
0.007
0.007
91.6
0.049
0.010
0.010
100
NRI
0.007
0.017
0.019
4.8
0.071
0.030
0.033
52.5
0.167
0.034
0.035
99.9
IDI
2e^{4}
2e^{4}
3e^{4}
6.0
0.014
0.003
0.005
99.9
0.048
0.006
0.009
99.9
MI1000
0.518
0.033
0.012
0.724
0.032
0.016
0.733
0.029
0.014
0.526
0.033
0.013
0.745
0.031
0.016
0.783
0.027
0.014
Δ
0.008
0.012
0.010
3.4
0.021
0.008
0.008
90.6
0.049
0.010
0.011
100
NRI
0.009
0.019
0.017
1.5
0.076
0.033
0.033
64.8
0.172
0.037
0.036
100
IDI
3e^{4}
3e^{4}
4e^{4}
3.5
0.014
0.004
0.005
99.0
0.045
0.008
0.010
100
MI300
0.518
0.033
0.012
0.724
0.032
0.016
0.733
0.029
0.014
0.528
0.033
0.012
0.745
0.031
0.017
0.783
0.027
0.015
Δ
0.010
0.014
0.011
3.0
0.021
0.008
0.009
84.6
0.050
0.011
0.012
100
NRI
0.013
0.023
0.018
1.3
0.076
0.035
0.035
57.0
0.172
0.039
0.039
99.7
IDI
4e^{4}
4e^{4}
5e^{4}
1.8
0.014
0.005
0.006
87.5
0.046
0.010
0.012
100
CC1000
0.528
0.032
0.013
0.667
0.033
0.015
0.670
0.031
0.014
0.534
0.033
0.015
0.709
0.032
0.022
0.737
0.029
0.014
Δ
0.006
0.010
0.010
4.7
0.043
0.011
0.017
100
0.067
0.012
0.012
100
NRI
0.017
0.031
0.033
6.7
0.147
0.039
0.043
96.7
0.261
0.041
0.043
100
IDI
0.002
0.001
0.003
15.2
0.058
0.009
0.014
100
0.114
0.011
0.017
100
CC300
0.523
0.034
0.013
0.620
0.037
0.016
0.620
0.034
0.015
0.529
0.034
0.015
0.647
0.036
0.016
0.668
0.032
0.015
Δ
0.006
0.010
0.009
3.6
0.027
0.011
0.011
83.3
0.048
0.013
0.013
99.8
NRI
0.019
0.039
0.043
6.2
0.154
0.043
0.050
94.4
0.257
0.046
0.051
99.9
IDI
0.002
0.001
0.003
13.9
0.040
0.008
0.014
99.8
0.078
0.010
0.017
100
Results from 1000 simulations
Δ, Harrell's predictive value of the phase2 variable,
NRI, Net reclassification index by adding the phase2 variable,
Cohort, full cohort estimates; MI300, MI1000: multiple imputation estimates with subcohort sizes set, respectively, at 300 and 1,000; CC300, CC1000, casecohort estimates with subcohort sizes set, respectively, at 300 and 1,000
Table
Predictive ability of the two models and of the phase2 variable.
Full cohort
Multiple imputation
Est
SE
%
Est
SE
%
Z_{2 }normally distributed
0.518
0.033
0.012
0.518
0.033
0.012
0.524
0.033
0.013
0.526
0.033
0.013
Δ
0.006
0.010
0.010
3.7
0.008
0.012
0.010
3.4
0.733
0.029
0.014
0.733
0.029
0.014
0.783
0.027
0.013
0.783
0.027
0.014
Δ
0.049
0.010
0.010
100
0.049
0.010
0.011
100
Z_{2 }normally distributed
0.518
0.033
0.012
0.518
0.033
0.012
0.524
0.033
0.013
0.520
0.031
0.016
Δ
0.006
0.010
0.009
5.5
0.002
0.013
0.012
4.2
0.784
0.013
0.006
0.784
0.013
0.006
0.881
0.011
0.006
0.866
0.011
0.006
Δ
0.097
0.005
0.005
100
0.082
0.005
0.004
100
Z_{2 }uniformly distributed
0.532
0.055
0.019
0.532
0.055
0.019
0.540
0.055
0.019
0.541
0.055
0.020
Δ
0.008
0.015
0.013
2.2
0.009
0.017
0.013
4.0
0.733
0.029
0.014
0.733
0.029
0.014
0.781
0.027
0.012
0.785
0.027
0.012
Δ
0.048
0.009
0.009
100
0.052
0.010
0.010
100
Results of 1000 simulations
Δ, Harrell's predictive value of the phase2 variable,
Mean of the predictive ability estimates (Est), mean of the standard error estimates
Application to the ThreeCity study
The mean fraction of missing information about the effect of Ddimer was 4.9 and 3.7 per cent for CHD and VaD risks, respectively. Table
Estimates of hazard ratios (HR) and 95% confidence interval (CI) associated with Ddimer tertiles.
Multiple imputation estimates
Weighted estimates
HR (95% CI)
HR (95% CI)
Risk of CHD and DDimer^{a}
T1
1.00 (reference)
1.00 (reference)
T2
1.42 (0.992.04)
1.40 (0.972.04)
T3
1.32 (0.891.97)
1.30 (0.841.99)
Linear trend
1.14 (0.941.38)
1.13 (0.921.38)
Risk of VaD and DDimer^{b}
T1
1.00 (reference)
1.00 (reference)
T2
1.57 (0.633.93)
1.60 (0.634.09)
T3
2.77 (1.176.57)
2.93 (1.227.06)
Linear trend
1.69 (1.132.53)
1.74 (1.132.67)
CHD, cardiovascular heart disease; T1, tertile 1; T2, tertile 2; T3, tertile 3; VaD, vascular dementia
^{a }Adjusted for age, center, sex, body mass index, hypertension, hypercholesterolemia, diabetes, diabetes drugs, tobacco use
^{b }Adjusted for age, center, sex, educational level, body mass index, apolipoprotein
Harrell's
Predictive ability and 95% confidence interval (CI) of DDimer tertiles on cardiovascular heart disease (CHD) and vascular dementia (VaD) risks.
CHD
VaD
Estimate
95% CI
Estimate
95% CI
0.693
(0.6220.764)
0.865
(0.7870.943)
0.694
(0.6210.767)
0.874
(0.7980.950)
Δ
0.002
(0.0040.008)
0.009
(0.0110.029)
NRI
0.009
(0.0490.066)


IDI
0.001
(0.0010.003)
0.0004
(0.00020.0010)
Δ, Harrell's predictive ability of the phase2 variable
NRI, net reclassification improvement by adding the phase2 variable IDI, integrated discrimination index by adding the phase2 variable
Discussion
Use of a consistent estimator does not guarantee the absence of any bias for finite sample. We only showed that MI analysis of casecohort data provides unbiased estimates of the loghazard ratio when the imputation model and the proportional hazard model are correctly specified. The misspecification of the imputation model can originate from an erroneous choice of the distribution, or from wrongly assuming that the estimator of the imputation model is consistent and normal, or from the omission of some important explanatory variable. Imputations carried out using a misspecified distribution in the imputation model can provide biased estimates of hazard ratios, especially, if the specified distribution of the phase2 variable differs from the true one in terms of symmetry (lognormal versus normal distribution). The negative bias on a log hazard ratio of 0.69 was noticeable but not large when a lognormal variable was imputed according to a normal distribution (0.09 or 13%), but it is clearly a type of misspecification easily identified with diagnostic tools
In simulated data, for the phase1 variables, the precision of MI and full cohort estimates was similar and smaller than with the weighted estimator. For the phase2 variable, MI estimates were slightly more precise than weighted estimates. Globally, the mean squared errors were smaller with MI than with the weighted estimator, with one exception implying a normal imputation model for a lognormally distributed phase2 variable, an error which should easily be avoided.
There is no standard method for estimating the predictive ability of a model in the framework of casecohort surveys. We showed that the naive application of the
Analysis of the ThreeCity casecohort study was in agreement with our previous work
The imputation model must reflect the association between the incomplete variable, the outcome and the other explanatory variables. Therefore, variables included in the proportional hazard model as well as the stratification variables must be included in the imputation model. If a surrogate of the phase2 variable is available, it should also be included in the imputation model. On the other hand, multiple imputation approach can provide unbiased and more efficient estimates than weighted analysis even when no strong predictor of the phase2 variable is available
The number of requested imputations depends on the proportion of missing information which, in casecohort studies, is considerably smaller than the percentage of incompletely observed subjects. Rubin showed that with as much as 40 per cent information missing, M = 5 imputations provides an asymptotic relative efficiency was 0.97, and, with 50 per cent missing information, M = 10 provides an asymptotic relative efficiency of 0.98. Thus, a small number of imputations, 510, should suffice
The VaD risk increased with Ddimer tertiles. However, Ddimer inclusion did not significantly improve the predictive ability of the model for VaD risk. Computations of the
Conclusions
MI is a simple alternative approach to weighted analysis for analyzing casecohort surveys, obtaining correct estimates of the log hazard ratios and their standard errors, improving precision for the phase1 variable estimates, and providing at least the same precision as weighted estimators for phase2 variable estimates. It allows an easy evaluation of the predictive ability of the model and, more generally, any tool proposed in the framework of cohort studies can be applied to casecohort data using MI.
Abbreviations
MI: Multiple imputation; CHD: Coronary heart disease; VaD: Vascular dementia; NRI: Net reclassification index; IDI: Integrated discrimination index.
Competing interests
The authors declare that they have no competing interests.
Authors' contributions
HM conducted the literature review, simulations, data analyses and wrote the manuscript. LC conducted the analysis of the relationship between Ddimer levels and CHD and VaD risks and supervised the epidemiological aspects of the application to the ThreeCity study. MC conducted and supervised the writing of the manuscript. All authors have read the manuscript, are in agreement that the work is ready for submission to the journal, and accept responsibility for the manuscript's contents.
Acknowledgements
This study was supported by a grant from the Région ÎledeFrance. It used data from the ThreeCity study which is conducted under an agreement between the Institut National de la Santé et de la Recherche Médicale and the Université Victor SegalenBordeaux 2. This manuscript was not prepared in collaboration with the 3C study Steering Committee and does not necessarily reflect its opinions or views.
Prepublication history
The prepublication history for this paper can be accessed here: