Table 1 Study internal and external validity coding criteria
Criteria description | Scoring for criteria | |
|---|---|---|
Internal validity criteria | ||
1 | Was the method of randomization appropriate? | Y = 1; N = 0 |
2 | Were baseline groups equivalent on important demographic measures? | Y = 1; N = 0; UK = 0 |
If No, was analysis conducted to estimate/adjust for effect of demographic measure on outcomes? | Y = 1; N = 0; UK = 0 | |
3 | Did the design of the study include comparison to a no treatment control group or a group with either no technology or no tailoring? | Y = 1; N = 0 |
4 | Was retention rate ≥ 80% at post-test/post-intervention follow-up? | Y = 1; N = 0 |
5 | Were Outcome Measurement instruments valid? Was there a description of instrument reliability/validity (reference or coefficients) OR did they use a well-established known valid measure? | Y = 1; N = 0 |
6 | Was an objective measure of behavior change used or did they rely solely on self-report measures? | Objective = 1; Self-report only = 0 |
7 | Was power analysis reported to determine sample size? | Y = 1; N = 0 |
8 | Were analyses conducted with consideration for missing data that maintain fidelity of the randomization (e.g. intention to treat, imputation)? Note: if 100% retention then N/A | Y = 1; N = 0 |
9 | Was the intervention based on theory? | Y = 1; N = 0 |
External validity criteria | ||
1 | Were recruitment methods and/or inclusion & exclusion criteria sufficiently described? | Both = 1, Either = 0.5; None = 0 |
2 | Were participation/recruitment rates provided OR Are analyses reported on the similarity and differences between participants versus either those who decline or the intended target audience (individuals or settings)? | Y = 1; N = 0 |
3 | Was a large heterogeneous sample used? Was the representativeness of participants described? Was a homogenous/heterogeneous sample sought for target population? Do the exclusion criteria used reduce the generalizability of findings? | Generalizable population = 1 |
4 | Was the representativeness of the setting described? Was the study conducted in an uncontrolled/controlled setting? Can their findings only be generalized to the limited conditions within which the research was carried out? | Generalizable setting (real-life) = 1; Controlled = 0 |
5 | Were all participants who entered trial accounted for at its conclusion i.e. Are data on attrition by condition reported OR was dropout rate described? | Y = 0.5; N = 0 |
Are drop-outs' compared to completers OR are the dropout's characteristics and reasons for drop-out described? | Y = 0.5; N = 0 | |
6 | Was the use of comparison conditions relevant to real-world decisions? (the computer-tailored treatment group was compared with either non-technology or non-tailored or alternative programs rather than no treatment) | Y = 1; N = 0 |
7 | Are data on the costs presented? | Y = 1; N = 0 |
8 | Was there sufficient description of the intervention, including: method of tailoring, duration and intensity (amount of contact time required)? | Y = 1; N = 0 |
9 | Are data reported on maintenance or longer-term effects? | Short-term = 0; Medium term = 0.5; Long-term = 1 |