Cancer patients commonly experience psychological distress that may increase with the current COVID-19 pandemic. This prospective study aimed to measure post-traumatic stress disorder (PTSD) and anxiety in cancer patients following France’s first COVID-19-related lockdown, together with associated factors. Cancer patients receiving outpatient treatment or post-treatment follow-up completed a questionnaire which measured, among other things, PTSD (IES-R), anxiety (State-Trait Anxiety Inventory), and fear of cancer recurrence (FCR). Of the 1097 patients included in the study, 14.7% and 30.5% suffered from PTSD and anxiety, respectively. Patients afraid to come to hospital due to the risk of COVID-19 transmission (OR = 3.49, p < 0.001), those with a negative lockdown experience (OR = 0.98, p < 0.001), women (OR = 1.97; p = 0.009), and patients living alone (OR = 1.63, p = 0.045) were all more likely to have PTSD. Older patients (OR = 1.65, p = 0.020), women (OR = 1.62, p = 0.018), those with a higher FCR score (OR = 5.02, p < 0.001), patients unsatisfied with their cancer management (OR = 2.36, p < 0.001), and those afraid to come to hospital due to COVID-19 (OR = 2.43, p < 0.001) all had a higher risk of anxiety. These results provide a greater understanding of the psychological consequences of the COVID-19 pandemic in cancer patients and highlight the need to better integrate psychosocial support in pandemic response measures in order to guide health systems.