How to Report Data Collection Tools, Validity, and Reliability?

SAER Journal · Academic Writing Guide

Clear reporting of data collection instruments allows readers to understand what was measured, how consistently it was measured, and how the resulting scores should be interpreted. Scales, tests, questionnaires, interview guides, observation forms, devices, and other instruments should be described with sufficient detail in the Methods section.

What Is a Data Collection Instrument?

It is any tool used to measure or record a variable, characteristic, experience, behaviour, or outcome, including scales, questionnaires, tests, observation forms, interview guides, clinical measures, devices, and institutional records.

How Should a Scale or Test Be Described?

  • Full instrument name
  • Developer and source
  • Year
  • Adaptation study, where applicable
  • Number of items
  • Subscales
  • Response format
  • Scoring procedure
  • Meaning of high/low scores
  • Prior validity and reliability evidence
  • Measurement evidence in the present study

Validity and Reliability

Validity concerns the evidence supporting the intended interpretation of scores. It depends on purpose, population, context, and use.

Reliability concerns measurement consistency and error. A reliable measure is not automatically valid.

Internal Consistency

Cronbach's alpha and McDonald's omega are commonly used. There is no universal alpha cut-off; interpretation should consider purpose, item count, construct, and use.

For multidimensional scales, subscale coefficients should often be reported rather than only a total-score coefficient.

Other Reliability Evidence

Test-retest: temporal stability.

Inter-rater: agreement among raters, assessed with measures such as Cohen's kappa, Fleiss' kappa, ICC, or percentage agreement depending on the data and design.

Content Validity

Content validity examines whether the items adequately represent the intended construct. Expert review may address relevance, coverage, clarity, and cultural appropriateness. Content-validity indices may be reported when appropriate.

Construct Validity

Evidence may include factor analysis, convergent validity, discriminant validity, hypothesis testing, and expected group differences.

Exploratory Factor Analysis

  • KMO
  • Bartlett's test
  • Extraction method
  • Rotation method
  • How the number of factors was determined
  • Factor loadings
  • Cross-loadings
  • Explained variance

Parallel analysis, scree plots, theory, and interpretability can be considered together rather than relying only on eigenvalues greater than one.

Confirmatory Factor Analysis

CFA tests a prespecified measurement model. Reporting may include:

  • Factor structure
  • Standardised loadings
  • Estimator
  • Fit indices
  • Error covariances and justification
  • Alternative-model comparisons

Model Fit

Common indices include χ²/df, CFI, TLI, RMSEA with confidence intervals, and SRMR. Fit should be interpreted collectively rather than from a single cut-off.

Convergent and Discriminant Validity

Convergent validity: factor loadings, AVE, and CR may be used.

Discriminant validity: HTMT, Fornell-Larcker, and interfactor correlations may be considered. Decision rules should be justified with appropriate references.

Criterion and Known-Groups Validity

Criterion validity evaluates expected relationships with an external criterion; known-groups validity examines whether groups expected to differ actually show the anticipated differences.

Scale Adaptation

  • Translation
  • Back-translation
  • Expert review
  • Cultural adaptation
  • Pilot testing
  • Factor structure
  • Reliability
  • Convergent/discriminant validity
  • Criterion validity

Scale Development

  • Conceptual definition
  • Item generation
  • Expert review
  • Pilot testing
  • Item analysis
  • EFA
  • CFA
  • Reliability
  • Validity evidence

When feasible, EFA and CFA should be conducted in independent samples.

Questionnaires, Researcher-Developed Forms, and Tests

Not every questionnaire is a scale. Researcher-developed demographic forms should simply state what information they collect.

For achievement or knowledge tests, KR-20 and item difficulty/discrimination indices may be appropriate depending on the scoring format.

Single Items, Reverse Scoring, and Item Removal

Cronbach's alpha is not meaningful for a single-item measure.

Reverse-scored items must be recoded correctly before analysis.

Items should not be deleted solely to increase alpha; theory, item-total relations, factor loadings, cross-loadings, and content coverage should all be considered.

Measurement Invariance

When comparing groups, researchers may examine configural, metric, scalar, and strict invariance to determine whether the construct is measured comparably across groups, languages, cultures, countries, or time points.

Permissions and Copyright

Instrument use depends on copyright and licensing conditions. Full item sets should not automatically be reproduced in a manuscript, especially for copyrighted measures.

Quality in Qualitative Research

Qualitative studies should not mechanically import quantitative validity-reliability language. Depending on methodology, quality may be discussed through:

  • Credibility
  • Transferability
  • Dependability
  • Confirmability
  • Reflexivity

Member checking and inter-coder agreement are not mandatory in every qualitative tradition; strategies should fit the methodological approach.

Interview Guides and Device Measurements

Interview guides: report development, main questions, expert review, pilot interviews, and use of probes where relevant.

Devices/laboratory measures: report model, manufacturer, calibration, protocol, and technical details required for reproducibility.

Reporting Example

Example: The X Scale, developed by Yılmaz and Demir, contains 18 items across three subscales. Items are rated from 1 (strongly disagree) to 5 (strongly agree), with higher scores indicating higher levels of X. The original study supported a three-factor structure and reported subscale internal-consistency coefficients between .78 and .89. In the present study, Cronbach's alpha for the total scale was .87.

Common Mistakes

  • Naming a scale without describing it
  • Failing to cite the original development study
  • Ignoring the adaptation source
  • Not reporting reliability in the current sample
  • Treating alpha as evidence of validity
  • Calculating alpha for a single item
  • Reporting only total alpha for separately analysed subscales
  • Using eigenvalue > 1 alone for factor retention
  • Relying on one fit index in CFA
  • Using modification indices without scientific justification
  • Deleting items only to increase alpha
  • Using quantitative quality language mechanically in qualitative work

Before Submission, Check

  • Are all instruments clearly described?
  • Is the original source cited?
  • Is the adaptation source cited where relevant?
  • Are item count, subscales, and scoring explained?
  • Is score interpretation clear?
  • Are reliability coefficients reported for the present sample?
  • Are validity analyses appropriate to the study aim?
  • Are EFA/CFA procedures sufficiently described?
  • Are fit indices interpreted together?
  • Are convergent and discriminant validity assessed when needed?
  • Are item removals and model modifications scientifically justified?
  • Are qualitative quality criteria consistent with the methodology?