Studies in children have reported associations of screen time and background TV on language skills as measured by their parents. However, few large, longitudinal studies have examined language skills assessed by trained psychologists, which is less prone to social desirability. We assessed screen time and exposure to TV during family meals at ages 2, 3 and 5–6 years in 1562 children from the French EDEN cohort. Language skills were evaluated by parents at 2 years (Communicative Development Inventory, CDI) and by trained psychologists at 3 (NEPSY and ELOLA batteries) and 5–6 years (verbal IQ). Cross-sectional and longitudinal associations were assessed by linear regression adjusted for important confounders. Overall, daily screen time was not associated with language scores, except in cross-sectional at age 2 years, where higher CDI scores were observed for intermediate screen time. Exposure to TV during family meals was consistently associated with lower language scores: TV always on (vs never) at age 2 years was associated with lower verbal IQ (− 3.2 [95% IC: − 6.0, − 0.3] points), independent of daily screen time and baseline language score. In conclusion, public health policies should better account for the context of screen watching, not only its amount.