Research Methods and Program Evaluation
Expert-defined terms from the Advanced Certificate in Oppositional Defiant Disorder course at LearnUNI. Free to read, free to share, paired with a professional course.
ABAB design – a single‑case experimental format that alternates baseline… #
related: single‑case, reversal design Example: observing ODD behaviors during two treatment periods separated by return to usual care. Practical use: isolates treatment effects in small samples. Challenge: ethical concerns when withdrawing effective interventions.
ABCD matrix – a planning tool that aligns Activities, Behaviors, Conditio… #
related: logic model, program mapping Example: mapping a parent‑training program (Activities) to reduce child aggression (Behaviors) under home‑environment conditions (Conditions) to achieve lower ODD symptom scores (Desired outcomes). Practical use: clarifies program components. Challenge: oversimplification of complex interactions.
Adaptive design – a study structure that allows modifications based on in… #
related: interim analysis, flexible trial Example: adjusting sample size of a CBT efficacy trial for ODD after early effect size estimates. Practical use: increases efficiency and ethical acceptability. Challenge: requires rigorous statistical planning.
Baseline assessment – initial measurement of variables before any interve… #
related: pre‑test, initial screening Example: administering the Disruptive Behavior Rating Scale before starting a parent‑training module. Practical use: establishes a reference point for change. Challenge: baseline reactivity may alter participant behavior.
Behavioural coding – systematic observation and categorization of observa… #
related: observational methods, inter‑rater reliability Example: coding instances of defiance during a classroom observation using a predefined scheme. Practical use: provides objective data on ODD manifestations. Challenge: requires extensive training and time.
Bias (research) – systematic error that skews study findings away from th… #
related: confounding, selection bias Example: recruiting only families who can afford intensive therapy may overestimate effectiveness. Practical use: identify and mitigate biases during design. Challenge: some biases are difficult to eliminate fully.
Case study – an in‑depth investigation of a single individual, family, or… #
related: qualitative method, narrative analysis Example: detailed description of a child with severe ODD undergoing multimodal treatment. Practical use: generates rich contextual insights. Challenge: limited generalizability.
Cluster randomization – random assignment of groups (e #
g., schools, clinics) rather than individuals. related: group‑level randomization, intraclass correlation Example: randomizing whole classrooms to receive a school‑based ODD program. Practical use: reduces contamination across participants. Challenge: requires larger sample sizes and complex analysis.
Coding scheme – a set of rules for assigning symbols or categories to obs… #
related: behavioural coding, reliability Example: a 5‑point scale for rating intensity of oppositional statements. Practical use: standardizes data collection. Challenge: must be validated for the target population.
Concurrent triangulation – a mixed‑methods approach where qualitative and… #
related: mixed methods, data integration Example: pairing teacher questionnaires with focus‑group interviews about an ODD program. Practical use: strengthens validity through multiple perspectives. Challenge: requires careful coordination and analytic expertise.
Content validity – the extent to which a measure represents all facets of… #
related: construct validity, face validity Example: ensuring an ODD symptom checklist covers aggression, defiance, and non‑compliance. Practical use: supports the relevance of assessment tools. Challenge: often relies on expert judgment.
Control group – participants who do not receive the experimental interven… #
related: comparison group, standard care Example: children receiving usual counseling while another group receives a new parent‑training protocol. Practical use: isolates the effect of the intervention. Challenge: ethical concerns if control receives inferior care.
Cost‑effectiveness analysis – an economic evaluation comparing the relati… #
related: cost‑benefit, economic evaluation Example: calculating cost per point reduction on the ODD rating scale for a school‑based program versus individual therapy. Practical use: informs resource allocation. Challenge: assigning monetary value to psychosocial outcomes.
Cross‑sectional study – observational research that captures data at a si… #
related: snapshot, prevalence study Example: surveying prevalence of ODD symptoms across different grade levels. Practical use: quick assessment of current status. Challenge: cannot infer causality.
Data saturation – the point at which additional data no longer generate n… #
related: thematic saturation, qualitative adequacy Example: after 12 parent interviews, no new themes about barriers to program participation emerge. Practical use: guides sample size for qualitative components. Challenge: subjective determination.
Data triangulation – the use of multiple data sources or methods to corro… #
related: methodological triangulation, source triangulation Example: combining teacher reports, parent questionnaires, and direct observations of child behavior. Practical use: enhances credibility of results. Challenge: integrating disparate data types.
Delphi technique – a structured communication process that gathers expert… #
related: expert panel, consensus method Example: developing a set of core outcome measures for ODD interventions. Practical use: leverages collective expertise without face‑to‑face meetings. Challenge: potential for attrition of experts over rounds.
Design effect – the factor by which sample size must be inflated in clust… #
related: intraclass correlation, cluster sampling Example: a design effect of 1.5 indicates 50% more participants needed than in an individually randomized trial. Practical use: accurate power calculations. Challenge: estimating intraclass correlation accurately.
Ecological validity – the degree to which study findings generalize to re… #
related: external validity, generalizability Example: an ODD program tested in a controlled clinic may have limited ecological validity when applied in community schools. Practical use: informs decisions about scaling up. Challenge: balancing experimental control with naturalistic environments.
Effect size – a quantitative measure of the magnitude of a treatment effe… #
related: Cohen’s d, standardized mean difference Example: a Cohen’s d of 0.8 indicates a large reduction in ODD symptoms after intervention. Practical use: facilitates comparison across studies. Challenge: requires consistent measurement scales.
Ethical clearance – formal approval from an institutional review board (I… #
related: informed consent, human subjects protection Example: obtaining IRB approval before recruiting families for a randomized ODD trial. Practical use: ensures participant safety and rights. Challenge: navigating complex regulatory requirements.
External validity – the extent to which study results apply to other popu… #
related: generalizability, transportability Example: findings from an urban clinic may not transfer to rural schools. Practical use: guides decisions about broader implementation. Challenge: often limited in tightly controlled efficacy trials.
Factor analysis – a statistical technique that reduces many variables int… #
related: dimensionality, construct validation Example: confirming that an ODD questionnaire loads onto three factors: aggression, defiance, and non‑compliance. Practical use: refines measurement instruments. Challenge: requires large sample sizes.
Fidelity monitoring – systematic assessment of whether an intervention is… #
related: implementation quality, adherence Example: using a checklist to verify that therapists follow the manualized CBT protocol for ODD. Practical use: links implementation to outcomes. Challenge: adds workload for staff.
Focus group – a moderated discussion with a small group of participants t… #
related: qualitative interview, group dynamics Example: gathering parent feedback on barriers to attending weekly ODD workshops. Practical use: captures rich, collective insights. Challenge: potential dominance by outspoken participants.
Grounded theory – a qualitative methodology that builds theory inductivel… #
related: constant comparative method, emergent coding Example: developing a model of family coping mechanisms during ODD treatment. Practical use: yields theory directly grounded in participant experiences. Challenge: intensive analytic process.
Hybrid design – a study that combines elements of efficacy and effectiven… #
related: type‑2 hybrid, implementation science Example: testing a validated ODD program while simultaneously assessing real‑world adoption in schools. Practical use: accelerates translation from research to practice. Challenge: balancing internal and external validity.
Implementation science – the study of methods to promote the systematic u… #
related: dissemination, fidelity Example: investigating factors that influence adoption of an evidence‑based ODD curriculum across districts. Practical use: bridges the gap between evidence and practice. Challenge: multidisciplinary nature demands diverse expertise.
Incidence rate – the number of new cases of a condition occurring in a de… #
related: prevalence, onset Example: measuring how many children develop ODD symptoms during a school year. Practical use: informs preventive program planning. Challenge: requires longitudinal tracking.
Informed consent – the process by which participants voluntarily agree to… #
related: assent, ethical standards Example: obtaining parental consent and child assent before enrolling in a behavioral intervention trial. Practical use: protects autonomy and legal compliance. Challenge: ensuring comprehension in diverse populations.
Instrument reliability – the consistency of a measurement tool across tim… #
related: test‑retest, inter‑rater reliability Example: a Cronbach’s alpha of .92 for an ODD symptom checklist indicates high internal consistency. Practical use: assures stable measurement. Challenge: maintaining reliability across different settings.
Inter‑rater reliability – the degree of agreement among different observe… #
related: kappa statistic, coder agreement Example: two clinicians independently rating the severity of defiant behavior and achieving a κ of .85. Practical use: validates observational data. Challenge: requires thorough coder training.
Iterative refinement – a cyclical process of testing, evaluating, and imp… #
related: pilot testing, feedback loops Example: revising a parent‑training manual after each cohort provides feedback on clarity. Practical use: enhances relevance and usability. Challenge: time‑intensive and may delay full rollout.
Judge‑adjudicated outcome – an outcome determined by an independent exper… #
related: blinded assessment, expert rating Example: a clinical psychologist rating overall functional improvement after an ODD program. Practical use: reduces bias from participants. Challenge: availability of qualified judges.
Latent variable – an unobserved construct inferred from multiple observed… #
related: factor analysis, structural equation modeling Example: “behavioral dysregulation” inferred from aggression, impulsivity, and non‑compliance items. Practical use: captures complex constructs. Challenge: model specification can be complex.
Logistic regression – a statistical model used to predict the probability… #
related: odds ratio, binary classification Example: modeling the odds of treatment completion based on family income and baseline severity. Practical use: identifies predictors of dichotomous outcomes. Challenge: assumes linearity of log‑odds.
Mixed‑methods research – an approach that integrates quantitative and qua… #
related: convergent design, complementarity Example: combining standardized ODD rating scales with parent interviews to evaluate program impact. Practical use: offers comprehensive insight. Challenge: requires expertise in both paradigms and careful integration.
Monitoring and evaluation (M&E) – systematic processes for tracking progr… #
related: performance indicators, outcome evaluation Example: monthly fidelity checks and end‑line outcome assessments for a school‑based ODD intervention. Practical use: informs continuous improvement. Challenge: resource‑intensive data collection.
Multiple imputation – a statistical technique that replaces missing data… #
related: missing data handling, MAR assumption Example: generating five imputed datasets for incomplete parent questionnaire responses. Practical use: reduces bias from missingness. Challenge: assumptions must be justified.
Naturalistic observation – recording behavior in the participant’s usual… #
related: field observation, ecological validity Example: observing child‑parent interactions in the home during routine activities. Practical use: captures authentic behavior. Challenge: limited control over extraneous variables.
Nominal group technique – a structured brainstorming method where partici… #
related: consensus building, idea prioritization Example: identifying barriers to ODD program adoption among school staff. Practical use: ensures equal participation. Challenge: may still be influenced by dominant personalities during discussion.
Observational study – research that assesses relationships among variable… #
related: correlational design, cohort study Example: tracking the association between parenting style and ODD symptom trajectories. Practical use: useful when experimental manipulation is unethical. Challenge: cannot establish causation.
Outcome measure – a variable used to assess the effect of an intervention #
related: dependent variable, endpoint Example: change in the Child Behavior Checklist ODD subscale score after six months of treatment. Practical use: provides evidence of program impact. Challenge: selecting measures that are sensitive, reliable, and relevant.
Participatory evaluation – an approach that actively involves stakeholder… #
related: stakeholder engagement, co‑production Example: teachers and parents collaborate to define success criteria for an ODD classroom management program. Practical use: enhances relevance and buy‑in. Challenge: balancing diverse interests and maintaining methodological rigor.
Pilot study – a small‑scale preliminary investigation conducted to test f… #
related: feasibility testing, trial run Example: implementing a brief version of a parent‑training module with ten families before full rollout. Practical use: identifies logistical issues early. Challenge: limited statistical power for efficacy conclusions.
Power analysis – a calculation that determines the sample size needed to… #
related: Type II error, sample size determination Example: using G*Power to estimate 120 participants required to detect a medium effect size (d = 0.5) with 80% power. Practical use: prevents under‑powered studies. Challenge: accurate effect size estimates are often unavailable.
Pre‑test/post‑test design – a quasi‑experimental format that measures par… #
related: before‑after study, repeated measures Example: administering the ODD rating scale at program entry and after 12 weeks of treatment. Practical use: simple way to assess change. Challenge: lacks a control group, vulnerable to history effects.
Process evaluation – assessment of how a program is implemented, includin… #
related: implementation evaluation, formative assessment Example: tracking attendance rates and therapist adherence to protocol during an ODD skills group. Practical use: informs real‑time adjustments. Challenge: requires detailed data collection systems.
Qualitative content analysis – systematic coding and categorizing of text… #
related: thematic analysis, coding framework Example: analyzing interview transcripts to extract common parental concerns about managing oppositional behavior. Practical use: makes large text datasets manageable. Challenge: subjectivity in theme identification.
Random assignment – the process of allocating participants to study condi… #
related: randomization, allocation concealment Example: using a computer‑generated sequence to assign families to either the experimental CBT group or treatment‑as‑usual. Practical use: balances known and unknown confounders. Challenge: maintaining allocation concealment in field settings.
Randomized controlled trial (RCT) – the gold‑standard experimental design… #
related: efficacy trial, superiority design Example: a multi‑site RCT evaluating a new school‑based ODD curriculum against standard practice. Practical use: provides high internal validity. Challenge: often costly and may have limited ecological validity.
Reliability coefficient – a statistic that quantifies the consistency of… #
related: Cronbach’s alpha, intraclass correlation Example: a test‑retest reliability of .88 for an ODD symptom inventory over two weeks. Practical use: assures stable measurement across time. Challenge: high reliability does not guarantee validity.
Replication study – a study that repeats the methods of an earlier invest… #
related: reproducibility, confirmatory research Example: conducting a second trial of a parent‑training program in a different geographic region. Practical use: strengthens evidence base. Challenge: subtle methodological differences can affect outcomes.
Response bias – systematic tendency of participants to answer in a social… #
related: social desirability, acquiescence Example: parents underreporting child aggression to appear effective caregivers. Practical use: incorporate validity scales or indirect measures. Challenge: difficult to eliminate entirely.
Risk assessment – systematic identification and evaluation of potential h… #
related: safety monitoring, adverse event reporting Example: evaluating the likelihood of increased irritability during a stimulant medication trial for ODD. Practical use: informs mitigation strategies. Challenge: unpredictable adverse events.
Sample bias – distortion that occurs when the selected sample does not re… #
related: selection bias, representativeness Example: recruiting only families with internet access for an online ODD program. Practical use: assess sampling frame carefully. Challenge: may limit external validity.
Sampling frame – a list or mechanism from which study participants are dr… #
related: population definition, recruitment source Example: using school enrollment records as the sampling frame for a community ODD prevalence survey. Practical use: defines the universe of potential participants. Challenge: incomplete frames can lead to coverage error.
Scale development – the process of creating, testing, and refining a meas… #
related: item generation, validation Example: constructing a new 20‑item questionnaire to assess parental self‑efficacy in managing ODD behaviors. Practical use: fills gaps where existing tools are inadequate. Challenge: requires extensive psychometric testing.
Secondary data analysis – re‑examining existing data sets to answer new r… #
related: archival research, data mining Example: analyzing a national health survey to explore demographic correlates of ODD diagnosis. Practical use: cost‑effective and time‑saving. Challenge: limited control over data quality and variable selection.
Self‑report measure – an instrument in which participants directly provid… #
related: questionnaire, survey Example: a parent‑completed rating scale of child oppositional behaviors. Practical use: efficient data collection. Challenge: prone to recall bias and social desirability effects.
Single‑case experimental design – a methodological approach focusing on i… #
related: ABAB, multiple baseline Example: tracking a child’s defiant episodes before, during, and after a behavior‑modification plan. Practical use: allows precise causal inference for rare or heterogeneous cases. Challenge: generalizability is limited.
Statistical significance – the probability that an observed effect is not… #
05. related: p‑value, null hypothesis testing Example: a p‑value of .03 indicating a significant reduction in ODD scores after treatment. Practical use: informs hypothesis testing. Challenge: does not convey effect magnitude or practical importance.
Structural equation modeling (SEM) – a multivariate statistical technique… #
related: path analysis, latent growth modeling Example: modeling how parental stress mediates the effect of a training program on child ODD outcomes. Practical use: integrates measurement and structural aspects. Challenge: requires large sample sizes and advanced expertise.
Subgroup analysis – examination of treatment effects within specific part… #
related: moderator analysis, stratified analysis Example: comparing outcomes for boys versus girls in an ODD intervention trial. Practical use: identifies differential efficacy. Challenge: increases risk of Type I error if not pre‑specified.
Survey methodology – the set of procedures for designing, administering,… #
related: sampling, questionnaire design Example: using a mixed‑mode (online and paper) survey to collect parent reports on child behavior. Practical use: standardizes data collection. Challenge: non‑response and measurement error.
Therapeutic alliance – the collaborative relationship between therapist a… #
related: rapport, working relationship Example: measuring alliance scores to predict adherence to an ODD treatment protocol. Practical use: predicts treatment outcomes. Challenge: difficult to quantify consistently.
Treatment fidelity – the degree to which an intervention is delivered as… #
related: implementation fidelity, protocol compliance Example: using a fidelity checklist to ensure each session of a CBT program for ODD includes all core components. Practical use: links delivery quality to outcomes. Challenge: monitoring can be resource‑intensive.
Triangulation of data sources – combining information from different orig… #
g., teachers, parents, self‑reports) to validate findings. related: multi‑informant assessment, cross‑validation Example: confirming reductions in ODD symptoms by aligning teacher rating scale scores with parent questionnaires and classroom observations. Practical use: strengthens confidence in results. Challenge: discrepancies among informants require careful interpretation.
Validity – the extent to which a measurement instrument accurately captur… #
related: construct validity, criterion validity Example: demonstrating that an ODD scale predicts future disciplinary actions (predictive validity). Practical use: ensures meaningful interpretation of scores. Challenge: multiple forms of validity must be established.
Variable operationalization – defining abstract concepts in measurable te… #
related: construct definition, measurement Example: operationalizing “parental involvement” as the number of training sessions attended and homework completion rate. Practical use: facilitates data collection. Challenge: oversimplification may miss nuanced aspects.
Variable control – strategies used to hold potential confounding factors… #
related: covariate adjustment, blocking Example: controlling for baseline severity when comparing two ODD interventions. Practical use: isolates the effect of the independent variable. Challenge: impossible to control all extraneous influences.
Validity evidence – empirical data supporting the accuracy of a measureme… #
related: convergent validity, discriminant validity Example: correlating a new ODD questionnaire with the established Child Behavior Checklist to demonstrate convergent validity. Practical use: builds confidence in instrument use. Challenge: requires multiple studies across diverse samples.
Weighting – applying statistical adjustments to compensate for unequal pr… #
related: survey weighting, design weights Example: assigning higher weights to under‑represented rural schools in a national ODD prevalence survey. Practical use: improves representativeness. Challenge: complex calculations and potential increase in variance.
Yield (research) – the amount of useful information or knowledge generate… #
related: productivity, knowledge translation Example: a high‑yield study produces actionable recommendations for scaling an ODD program. Practical use: assesses efficiency of research investment. Challenge: yield is often subjective and context‑dependent.