In this review, we summarize available data regarding bone phenotypes in estrogen receptors alpha and beta, androgen receptor, and aromatase enzyme-deficient mice. We examine sex differences in the trabecular and cortical bone compartments and we discuss these findings in relation to known estrogen effects in humans. We also report how estrogen influences the responsiveness of the skeleton to exercise. Although uncertainties remain, it is clear that both estrogen and androgen are important for both male and female skeleton. Estrogen receptor alpha mainly through its classical signaling pathway is particularly important for the male mice skeleton while both estrogen receptors alpha and beta are required for female mice skeleton. These deletions also induce major hormonal alterations themselves impacting on bone metabolism. More investigations are needed to fully understand the respective role of all these receptors in periosteal expansion in both sexes and the way they affect the mechanical sensitivity of the periosteum.