Introduction: Since the advent of HIV pre-exposure prophylaxis (PrEP), stigma has been shown to be a major barrier to its uptake and adherence. It is therefore essential to define the proportion of users who consider that PrEP can negatively impact their image and the factors associated with this perception. Method: We performed a multivariable logistic regression on data from the 2567 participants in the ANRS-PREVENIR study who answered the outcome question. Results: Almost one-third of the sample (comprising mostly cisgender men who have sex with men [94.3%]) considered that taking PrEP could give others a negative image of them. Younger participants (adjusted odds ratio [aOR] 0.98; 95% confidence interval [CI] 0.97-0.99) and more psychologically vulnerable participants (i.e., lower self-esteem score [aOR 0.98; 95% CI 0.96-0.99] and higher depression score [aOR 1.02; 95% CI 1.00-1.03]) were also more likely to have this perception. In contrast, participants encouraged to take PrEP by their main partner (aOR 0.67; 95% CI 0.51-0.88) and friends (aOR 0.79; 95% CI 0.66-0.95), and those who protected themselves more because they had knowledge of their most recent sexual partner's HIV status (aOR 0.83; 95% CI 0.69-0.99) and systematic use of PrEP and/or condoms during intercourse in the previous 3 months (aOR 0.80; 95% CI 0.67-0.96) were less likely to have this perception. Discussion: Given the strong interrelation between stigmatization (real or perceived), risky behaviours and adherence, our results emphasize the need for HIV prevention campaigns to promote a positive image of PrEP users. They also show that stigmatization and its effects need to be fully considered to improve HIV prevention offers to current and potential PrEP users who are most likely to be psychologically vulnerable. Trial registration: ClinicalTrials.gov NCT03113123.