A randomised trial and economic evaluation of the effect of response mode on response rate, response bias, and item non-response in a survey of doctors
Top Cited Papers
Open Access
- 5 September 2011
- journal article
- research article
- Published by Springer Science and Business Media LLC in BMC Medical Research Methodology
- Vol. 11 (1), 126
- https://doi.org/10.1186/1471-2288-11-126
Abstract
Background Surveys of doctors are an important data collection method in health services research. Ways to improve response rates, minimise survey response bias and item non-response, within a given budget, have not previously been addressed in the same study. The aim of this paper is to compare the effects and costs of three different modes of survey administration in a national survey of doctors. Methods A stratified random sample of 4.9% (2,702/54,160) of doctors undertaking clinical practice was drawn from a national directory of all doctors in Australia. Stratification was by four doctor types: general practitioners, specialists, specialists-in-training, and hospital non-specialists, and by six rural/remote categories. A three-arm parallel trial design with equal randomisation across arms was used. Doctors were randomly allocated to: online questionnaire (902); simultaneous mixed mode (a paper questionnaire and login details sent together) (900); or, sequential mixed mode (online followed by a paper questionnaire with the reminder) (900). Analysis was by intention to treat, as within each primary mode, doctors could choose either paper or online. Primary outcome measures were response rate, survey response bias, item non-response, and cost. Results The online mode had a response rate 12.95%, followed by the simultaneous mixed mode with 19.7%, and the sequential mixed mode with 20.7%. After adjusting for observed differences between the groups, the online mode had a 7 percentage point lower response rate compared to the simultaneous mixed mode, and a 7.7 percentage point lower response rate compared to sequential mixed mode. The difference in response rate between the sequential and simultaneous modes was not statistically significant. Both mixed modes showed evidence of response bias, whilst the characteristics of online respondents were similar to the population. However, the online mode had a higher rate of item non-response compared to both mixed modes. The total cost of the online survey was 38% lower than simultaneous mixed mode and 22% lower than sequential mixed mode. The cost of the sequential mixed mode was 14% lower than simultaneous mixed mode. Compared to the online mode, the sequential mixed mode was the most cost-effective, although exhibiting some evidence of response bias. Conclusions Decisions on which survey mode to use depend on response rates, response bias, item non-response and costs. The sequential mixed mode appears to be the most cost-effective mode of survey administration for surveys of the population of doctors, if one is prepared to accept a degree of response bias. Online surveys are not yet suitable to be used exclusively for surveys of the doctor population.Keywords
This publication has 31 references indexed in Scilit:
- Quality Indicators for the Development and Didactics of Ultrasound Courses in Continuing Medical EducationUltraschall in der Medizin - European Journal of Ultrasound, 2011
- Getting Physicians to Respond: The Impact of Incentive Type and Timing on Physician Survey Response RatesHealth Services Research, 2010
- The "Medicine in Australia: Balancing Employment and Life (MABEL)" longitudinal survey - Protocol and baseline data for a prospective cohort study of Australian doctors' workforce participationBMC Health Services Research, 2010
- Methods to increase response to postal and electronic questionnairesEmergencias, 2009
- Combining Web-Based and Mail Surveys Improves Response Rates: A PBRN Study From PRIME NetAnnals of Family Medicine, 2009
- A very low response rate in an on‐line survey of medical practitionersAustralian and New Zealand Journal of Public Health, 2008
- Mail Versus Internet SurveysEvaluation & the Health Professions, 2007
- Mixing Web and Mail Methods in a Survey of PhysiciansHealth Services Research, 2006
- Total cost/response rate trade-offs in mail survey research: impact of follow-up mailings and monetary incentivesIndustrial Marketing Management, 2003
- Econometric methods for fractional response variables with an application to 401(k) plan participation ratesJournal of Applied Econometrics, 1996