Escherichia coli is a commensal of birds and mammals, including humans. It can act as an opportunistic pathogen and is also found in water and sediments. Since most population studies have focused on clinical isolates, we studied the phylogeny, genetic diversification, and habitat-association of 1,294 isolates representative of the phylogenetic diversity of more than 5,000, mostly non-clinical, isolates originating from humans, poultry, wild animals and water sampled from the Australian continent. These strains represent the species diversity and show large variations in gene repertoires within sequence types. Recent gene transfer is driven by mobile elements and determined by habitat sharing and by phylogroup membership, suggesting that gene flow reinforces the association of certain genetic backgrounds with specific habitats. The phylogroups with smallest genomes had the highest rates of gene repertoire diversification and fewer but more diverse mobile genetic elements, suggesting that smaller genomes are associated with higher, not lower, turnover of genetic information. Many of these small genomes were in freshwater isolates suggesting that some lineages are specifically adapted to this environment. Altogether, these data contribute to explain why epidemiological clones tend to emerge from specific phylogenetic groups in the presence of pervasive horizontal gene transfer across the species.