Objective: We investigate which procedure selects the predictive model most trustworthy to reason on the effect of an intervention and support decision making. Methods: We study such a variety of model selection procedures in practical settings: finite sam- ples settings and without theoretical assumption of well-specified models. Beyond standard cross- validation or internal validation procedures, we also study elaborate causal risks. These build proxies of the causal error using “nuisance” re-weighting to compute it on the observed data. We evaluate whether empirically estimated nuisances, which are necessarily noisy, add noise to model selection. We compare different metrics for causal model selection in an extensive empirical study based on a simulation and three healthcare datasets based on real covariates. Results: Among all metrics, the mean squared error, classically used to evaluate predictive modes, is worse. Re-weighting it with propensity score does not bring much improvements. The R-risk, which uses as nuisances a model of mean outcome and propensity scores, leads to the best performances. Nuisance corrections are best estimated with flexible estimators such as a super learner. Conclusions: When predictive models are used to reason on the effect of an intervention, they must be evaluated with different procedures than standard predictive settings; using the R-risk from causal inference.