Little is known about cancer survivors' sexual health (SH)-particularly, from well after diagnosis and in cancers unrelated to sexuality. This study aimed to assess SH deterioration five years after diagnosis. We analyzed data from the French national VIe après le CANcer (VICAN) survey. Six items from the Relationship and Sexuality Scale were used to assess SH. Respondents were grouped according to an ascending hierarchical classification in four clusters: strong, moderate, and weak deterioration or stable (WD, SD, MD, or St). Out of 2195 eligible participants, 57.3% reported substantial SH deterioration as either SD (30.8%) or MD (26.5%), while WD and St accounted for 31.2% and 11.5% of respondents, respectively. Substantial deterioration was reported in all cancer sites (from 27.7% in melanoma to 83.1% in prostate). Treatment type, cancer sequelae, and pain, as well as psychological consequences (depression and anxiety, especially for younger patients) were associated with substantial SH deterioration. The same factors were identified after restricting the analysis to survivors of cancers unrelated to sexuality. Five years after diagnosis, the majority of cancer survivors reported SH deterioration. Interventions should be developed to improve SH regardless of cancer site. Particular attention should be paid to depression and anxiety, especially in younger survivors.