The extent to which the effect of risk factors on cognitive ageing is dependent on APOE epsilon4 remains unclear. The objective of this study is to examine whether APOE epsilon4 allele modifies the association between health behaviors and cognition in late midlife. Data are drawn from 5447 participants of the Whitehall II study, health behaviors were assessed in 1997-1999 (mean age = 55.6, Standard Deviation (SD) = 6.0) and APOE genotype and cognitive function in 2002-2004 (mean age = 60.9, SD = 5.9). Among APOE epsilon4 non-carriers, current smokers had lower scores on memory (difference in T-score = -2.49, 95%CI: -3.37, -1.60), reasoning (-2.88, 95%CI: -3.74, -2.01), phonemic (-2.66, 95%CI: -3.56, -1.76) and semantic (-2.38, 95%CI: -3.28, -1.47) fluency compared to never smokers. In APOE epsilon4 carriers, difference between current and never smokers was seen only for reasoning (-1.92, 95%CI: -3.31, -0.51). Interaction terms supported differential effects of smoking as a function of APOE epsilon4 status for memory (p = 0.01), and phonemic (p = 0.008) and semantic fluency (p = 0.02). Cognitive scores were lower among non-drinkers compared to moderate drinkers, among the sedentary participants and those who ate fruits and vegetable less than 2 times per day irrespective of APOE epsilon4 status. This study suggests that the APOE epsilon4 allele modifies the association of smoking but not that of other health behaviors - alcohol consumption, physical activity, fruit and vegetable consumption - with cognitive function in late midlife.