Prostate cancer is the most frequently diagnosed cancer in men and a leading cause of cancer-related death. In recent decades, the development of immunotherapies has resulted in great promise to cure metastatic disease. However, prostate cancer has failed to show any significant response, presumably due to its immunosuppressive microenvironment. There is therefore growing interest in combining immunotherapy with other therapies able to relieve the immunosuppressive microenvironment. Radiation therapy remains the mainstay treatment for prostate cancer patients, is known to exhibit immunomodulatory effects, depending on the dose, and is a potent inducer of immunogenic tumor cell death. Optimal doses of radiotherapy are thus expected to unleash the full potential of immunotherapy, improving primary target destruction with further hope of inducing immune-cell-mediated elimination of metastases at distance from the irradiated site. In this review, we summarize the current knowledge on both the tumor immune microenvironment in prostate cancer and the effects of radiotherapy on it, as well as on the use of immunotherapy. In addition, we discuss the utility to combine immunotherapy and radiotherapy to treat oligometastatic metastatic prostate cancer.