Frailty is influenced by numerous genetic and environmental factors. However, sex differences in how these factors affect frailty, and the gene-environment interplay among frailty and two of its well-established risk factors, unhealthy body mass index (BMI) and low education, are less clear. In a large sample of 42,994 Swedish twins, we used structural equation models to estimate the genetic (heritability) and environmental sources of variance in frailty, defined as the frailty index (FI), separately in men and women. Genetic and individual-specific environmental factors contributed approximately equally to the FI variance. The heritability of FI was slightly, but significantly, higher in women (52%) than in men (45%), yet we found only weak-to-no indication of different sources of genetic variance influencing frailty across sexes. We observed a small-to-moderate genetic overlap between FI and BMI, and that the correlation between FI and education was largely explained by environmental factors common to twins in a pair. Additionally, genetic factors accounted for more of FI variation at both low and high BMI levels, with similar patterns in both sexes. In conclusion, the twin-based heritability of frailty is higher in women than in men, and different mechanisms may underlie the associations of frailty with BMI and education.