A ‘3 step’ IRT strategy for evaluation of the use of sum scores in small studies with questionnaires using items with ordered response levels : ‘To make a lot out of nothing’, from the Swedish locution: ‘Att koka soppa på en spik’
Author: Brodin, Ulf B
Date: 2014-11-28
Location: Sal John, Widerströmska huset, Tomtebodavägen 18A, Karolinska Institutet, Solna
Time: 09.00
Department: Inst för lärande, informatik, management och etik / Dept of Learning, Informatics, Management and Ethics
Abstract
This study is focused on a strategy for a basic evaluation of a questionnaire at an early state (i.e. only a limited sample from the target population is available).
Several questionnaires are constructed within the medical research to investigate phenomenon which cannot be explicitly measured (latent variables). In many cases, these questionnaires are applied on a limited sample (less than 100 subjects), without any proper evaluation of its basic characteristic.
This thesis presents a ‘3- step’ strategy for evaluation of questionnaires, where only a limited sample is available for the investigation. Only questionnaires, containing items with a common set of answer alternatives, are considered. The answer alternatives are in terms of an ordered scale to measure the underlying, latent, variable. In all cases, the intention is that a sum score will be a relevant measure of the status of a subject. The higher the score, the more of the latent characteristic is attached to the subject. This implies that all items should represent one common dimension.
The investigation is carried out in three steps and is focused on whether the following criteria, at least approximately, correspond to the intentions which the researcher was supposed to have in mind, at the construction of the questionnaire: 1. All items cooperate; 2. All items work together towards a common dimension; 3. Non-coherent/irrelevant items, as well as non-coherent answer profiles can be identified; 4. The subjects can be reasonably ranked, based on the sum score, on the latent scale; 5. The sum score can be transformed, via a statistical model, to a relevant interval scaled measure; 6. The set of items covers reasonably the intended population; 7. The item set is sufficient for an estimate of a person’s position on the latent scale; 8. Defined subgroups perceive the questionnaire in the same way; 9. A straight forward sum score is a sufficient measure (sufficient statistic). Otherwise, an elaborated model, with item specific discrimination, is considered.
Step 1: Non parametric statistical analysis according to ‘The Mokken scale analysis’ (1 - 4); Step 2: A parametric model according to the Rasch approach (3, 5 - 8); and Step 3: Can further information be gained from an extended model? (9).
This strategy was applied on 5 studies, all carried out with the intention to use the sum score as a relevant measure of persons’ status on the underlying latent scale. Each study involves more than one questionnaire. The ’3 – step’ strategy was applied on 13 questionnaires within these 5 studies, where the intention was to use a sum score.
Results: Already Step 1 reveals most of the basic weaknesses of the questionnaire, such as weak or contradictory items, deficient correlations between items and a violation of an invariant ordering of the item across low to high scoring persons. These findings are also confirmed in later steps, where further characteristics can be revealed. It also turned out that a well behaved questionnaire according to the ‘Mokken requirements’ is a prerequisite for a reasonable parametric approach. Furthermore, in most cases the number of items appears to be too small and that the item set has an insufficient coverage to reasonably estimate a person measure for all subjects belonging to the intended population. But, the ‘3 steps’ can constitute a comprehensive message for a basic improvement of the questionnaire.
Several questionnaires are constructed within the medical research to investigate phenomenon which cannot be explicitly measured (latent variables). In many cases, these questionnaires are applied on a limited sample (less than 100 subjects), without any proper evaluation of its basic characteristic.
This thesis presents a ‘3- step’ strategy for evaluation of questionnaires, where only a limited sample is available for the investigation. Only questionnaires, containing items with a common set of answer alternatives, are considered. The answer alternatives are in terms of an ordered scale to measure the underlying, latent, variable. In all cases, the intention is that a sum score will be a relevant measure of the status of a subject. The higher the score, the more of the latent characteristic is attached to the subject. This implies that all items should represent one common dimension.
The investigation is carried out in three steps and is focused on whether the following criteria, at least approximately, correspond to the intentions which the researcher was supposed to have in mind, at the construction of the questionnaire: 1. All items cooperate; 2. All items work together towards a common dimension; 3. Non-coherent/irrelevant items, as well as non-coherent answer profiles can be identified; 4. The subjects can be reasonably ranked, based on the sum score, on the latent scale; 5. The sum score can be transformed, via a statistical model, to a relevant interval scaled measure; 6. The set of items covers reasonably the intended population; 7. The item set is sufficient for an estimate of a person’s position on the latent scale; 8. Defined subgroups perceive the questionnaire in the same way; 9. A straight forward sum score is a sufficient measure (sufficient statistic). Otherwise, an elaborated model, with item specific discrimination, is considered.
Step 1: Non parametric statistical analysis according to ‘The Mokken scale analysis’ (1 - 4); Step 2: A parametric model according to the Rasch approach (3, 5 - 8); and Step 3: Can further information be gained from an extended model? (9).
This strategy was applied on 5 studies, all carried out with the intention to use the sum score as a relevant measure of persons’ status on the underlying latent scale. Each study involves more than one questionnaire. The ’3 – step’ strategy was applied on 13 questionnaires within these 5 studies, where the intention was to use a sum score.
Results: Already Step 1 reveals most of the basic weaknesses of the questionnaire, such as weak or contradictory items, deficient correlations between items and a violation of an invariant ordering of the item across low to high scoring persons. These findings are also confirmed in later steps, where further characteristics can be revealed. It also turned out that a well behaved questionnaire according to the ‘Mokken requirements’ is a prerequisite for a reasonable parametric approach. Furthermore, in most cases the number of items appears to be too small and that the item set has an insufficient coverage to reasonably estimate a person measure for all subjects belonging to the intended population. But, the ‘3 steps’ can constitute a comprehensive message for a basic improvement of the questionnaire.
List of papers:
I. Brodin Ulf, Fors Uno, Bolander Laksov Klara. The application of Item Response Theory on a teaching strategy profile questionnaire. BMC Biomedical Education. 2010;10:14.
Fulltext (DOI)
Pubmed
View record in Web of Science®
II. Nissell M., Brodin, U., Christensen K., Rydelius P-A. The Imperforate Anus Psychosocial Questionnaire (IAPSQ): Its construction and psychometric properties. Child and Adolescent Psychiatry and Mental Health. 2009;3:15.
Fulltext (DOI)
Pubmed
III. Brodin U., Fors UGH, Olsson Gunilla M. Adolescent Adjustment Profile - revised and investigated by means of an Item Response Theory approach. [Manuscript]
IV. Adler M., Hetta J., Isacsson G., Brodin U. An Item Response Theory evaluation of three depression assessment instruments in a clinical sample. BMC Med Res Methodol. 2012 Jun 21;12(1):84.
Fulltext (DOI)
Pubmed
View record in Web of Science®
V. Adler Mats, Brodin Ulf. An IRT validation of the Affective Self Rating Scale. Nordic Journal of Psychiatry. 2011;65(6-9:396-402.
Fulltext (DOI)
Pubmed
View record in Web of Science®
I. Brodin Ulf, Fors Uno, Bolander Laksov Klara. The application of Item Response Theory on a teaching strategy profile questionnaire. BMC Biomedical Education. 2010;10:14.
Fulltext (DOI)
Pubmed
View record in Web of Science®
II. Nissell M., Brodin, U., Christensen K., Rydelius P-A. The Imperforate Anus Psychosocial Questionnaire (IAPSQ): Its construction and psychometric properties. Child and Adolescent Psychiatry and Mental Health. 2009;3:15.
Fulltext (DOI)
Pubmed
III. Brodin U., Fors UGH, Olsson Gunilla M. Adolescent Adjustment Profile - revised and investigated by means of an Item Response Theory approach. [Manuscript]
IV. Adler M., Hetta J., Isacsson G., Brodin U. An Item Response Theory evaluation of three depression assessment instruments in a clinical sample. BMC Med Res Methodol. 2012 Jun 21;12(1):84.
Fulltext (DOI)
Pubmed
View record in Web of Science®
V. Adler Mats, Brodin Ulf. An IRT validation of the Affective Self Rating Scale. Nordic Journal of Psychiatry. 2011;65(6-9:396-402.
Fulltext (DOI)
Pubmed
View record in Web of Science®
Institution: Karolinska Institutet
Supervisor: Fors, Uno
Issue date: 2014-10-29
Rights:
Publication year: 2014
ISBN: 978-91-7549-709-9
Statistics
Total Visits
Views | |
---|---|
A ...(legacy) | 1097 |
A ... | 217 |
Total Visits Per Month
September 2023 | October 2023 | November 2023 | December 2023 | January 2024 | February 2024 | March 2024 | |
---|---|---|---|---|---|---|---|
A ... | 0 | 0 | 0 | 0 | 1 | 0 | 1 |
File Visits
Views | |
---|---|
Thesis_Ulf_Brodin.pdf(legacy) | 453 |
Thesis_Ulf_Brodin.pdf | 412 |
Spikblad_Ulf_Brodin.pdf(legacy) | 179 |
Errata_Ulf_Brodin.pdf(legacy) | 123 |
Spikblad_Ulf_Brodin.pdf | 49 |
Thesis_Ulf_Brodin.pdf(legacy) | 39 |
Errata_Ulf_Brodin.pdf | 21 |
Top country views
Views | |
---|---|
China | 361 |
United States | 290 |
Sweden | 127 |
Germany | 108 |
South Korea | 47 |
France | 17 |
Ireland | 12 |
Netherlands | 10 |
Poland | 9 |
United Kingdom | 8 |
Top cities views
Views | |
---|---|
Shenzhen | 250 |
Sunnyvale | 63 |
Stockholm | 57 |
Ashburn | 55 |
Seoul | 42 |
Nürnberg | 34 |
Kiez | 27 |
Woodbridge | 13 |
Dublin | 12 |
Kraków | 8 |