Validity of a questionnaire survey: the role of non-response and incorrect answers

Abstract

Errors in questionnaire surveys are usually of one of two sources: non-responses or incorrect answers. The aim was to investigate the validity of a questionnaire survey and to estimate the respective bias of these answers. Of 9,283 subjects selected to receive a questionnaire by post, 3,949 (43%) responded, and, of these, 3,400 correctly reported their Swedish social security number. Answers in the questionnaire survey were given as proportions of the claims registered at local insurance offices. In the group of respondents who had correctly reported their social security number, the answers were compared individually with the registrations in dental insurance claims. In Sweden, these claims are labeled with the patient's social security number and it is thereby possible to make such comparisons. It was shown that errors were caused by non-response and also by respondents giving incorrect answers. Incorrect answers accounted for approximately one-third of the total bias. The remaining bias was caused by a non-response error. It is concluded that questionnaire studies have a bias caused by both non-response and incorrect answers and that together these can be substantial. Scientific reports that include questionnaire surveys must describe the procedure carefully. If possible, other sources of information should be considered.