The authors prospectively studied the effect of leisure-time physical activity level on hip fracture risk along with the influence of within-subject changes in activity levels, while taking possible confounding by other health behaviors and poor health into account. Analyses were based on pooled data from three population studies conducted in Copenhagen, Denmark. Among 13,183 women and 17,045 men, 1,121 first hip fractures were identified during follow-up. In comparison with being sedentary, the relative risk (RR) of hip fracture associated with being moderately physically active 2–4 hours per week was 0.72 (95% confidence interval (CI): 0.59, 0.89) in women and 0.75 (95% CI: 0.55, 1.03) in men after adjustment for confounders. Being in the most active leisure activity category did not decrease the risk of hip fracture further. Adjustment for poor health affected the risk estimates only modestly. Subjects who, during follow-up, reduced their physical activity level from the highest or the intermediate activity level to a sedentary level had a higher risk of hip fracture than did those who remained moderately physically active at the intermediate level (multivariate adjusted RR = 2.19, 95% CI: 1.00, 4.84 and RR = 1.89, 95% CI: 1.21, 2.95, for reduction from the highest and intermediate levels, respectively). There was no evidence of a fracture-protective effect from increasing physical activity. In conclusion, moderate levels of physical activity appear to provide protection against later hip fracture. Decline in the physical activity level over time is an important risk factor for hip fracture.