Choosing the right statistical test

Most people arrive at this question too late. They have finished collecting data, opened SPSS, and now need to know which menu to click. By then the useful decisions have already been made, and some of them are wrong.

The test is not chosen at the analysis stage. It is chosen when you write your objectives, and everything after that is execution.

Start with the objective

Look at the verb in each of your specific objectives. It tells you what kind of analysis you need before you know anything about your data.

This is why supervisors write "your instruments do not align with your objectives." An objective that says compare needs a design with at least two groups and an instrument that assigns respondents to them. An objective that says predict needs an outcome variable and predictors measured on the same respondents. If the objective and the instrument disagree, no choice of test will rescue the analysis.

Write your objectives, then write down the test each one implies, then design the instrument to produce exactly the data that test needs. In that order.

The three questions that settle it

Once your objectives are stable, three questions determine the test.

1. What is the question asking?

Difference, relationship, or prediction. These are not interchangeable. "Is there a difference in attendance between programme and non-programme schools" is a comparison. "Does attendance rise as meal frequency rises" is a relationship. "How much of the variation in attendance is explained by meal frequency, class size and distance from home" is a prediction.

2. What kind of variable is your outcome?

This matters more than anything else and is where most errors begin.

Nominal. Categories with no order. Sex, county, marital status, whether a respondent voted. You can count these and nothing else.

Ordinal. Ordered categories with unequal or unknown gaps. Education level, income bracket, a single Likert item. You know that "agree" is more than "neutral" but not by how much.

Continuous. Real numbers where the gaps are equal and meaningful. Age in years, test score, attendance rate, income in shillings.

The common failure is treating ordinal data as continuous without saying so. See the section on Likert scales below, because this is the single most argued point in postgraduate vivas.

3. How many groups, and are they independent?

Two groups or more than two changes the test. So does whether the groups contain different people (independent) or the same people measured twice (paired).

Comparing men and women is independent. Comparing the same pupils before and after an intervention is paired. Using an independent t-test on paired data throws away the pairing and understates your ability to detect an effect.

The decision table

Two notes on this table.

A chi-square test tells you that two categorical variables are associated. It does not tell you how strongly. Report Cramér's V alongside it or the finding is incomplete.

Correlation is not regression. Correlation says two things move together. Regression says how much of one is explained by the other, and in which direction you are reading the relationship. Objectives that say "determine the influence of" need regression, not correlation, however tempting the simpler option is.

Assumptions, and what to do when they fail

Every parametric test assumes things about your data. Examiners increasingly ask whether you checked. The honest answer for most submitted theses is no.

Normality. t-tests, ANOVA and linear regression assume the outcome is roughly normally distributed within groups. Check with a histogram and a Q-Q plot first, because they show you the shape. Shapiro-Wilk gives a formal test but is oversensitive in large samples, where it will reject normality for departures too small to matter. With samples above about 30 per group, the central limit theorem makes these tests fairly robust to moderate non-normality anyway.

Homogeneity of variance. ANOVA and the independent t-test assume the groups have similar spread. Levene's test checks this. If it fails, use Welch's t-test or Welch's ANOVA, which do not make the assumption.

Independence of observations. Each respondent contributes one observation, unaffected by the others. This is the assumption most often violated and least often discussed. If you sampled whole classrooms, pupils within a classroom are more alike than pupils across classrooms, and ordinary regression will report standard errors that are too small. Clustered standard errors or a multilevel model is the correct response.

Linearity, for regression. Plot the residuals. If the relationship curves, a straight line will misrepresent it.

When an assumption fails, you have three options in order of preference: transform the variable (log transformation handles right-skewed data like income well), use the rank-based equivalent from the table above, or use a robust method. What you should not do is proceed silently and hope nobody checks.

The Likert problem

A single Likert item is ordinal. The distance between "agree" and "strongly agree" is not known to be the same as the distance between "neutral" and "agree", so the mean of a single item is not strictly meaningful.

A Likert scale is different from a Likert item. If you sum or average several items that measure the same construct, and the scale shows acceptable internal consistency, treating the total as continuous is widely accepted and defensible.

So:

• Single item, reported on its own: use medians, frequencies and rank-based tests.
• Multi-item scale with a reported Cronbach's alpha: you may treat the composite as continuous, and you should say in the methodology that you have done so and why.

The failure is not choosing one or the other. It is choosing one and not saying which.

Sample size

A very large number of Kenyan postgraduate proposals arrive at 384 respondents. That figure comes from Cochran's formula for estimating a proportion in a large population at 95% confidence with a 5% margin of error, assuming maximum variability.

It is a perfectly good number if your objective is to estimate a proportion. If your objective is to detect a difference between groups or to fit a regression with six predictors, it answers a question you did not ask.

For comparative and predictive designs, justify the sample with a power analysis: state the effect size you want to be able to detect, the power you want (0.80 is conventional), and the alpha (0.05 is conventional). G*Power does this free. For regression, a common rule of thumb is a minimum of 10 to 20 observations per predictor, which is a floor rather than a justification.

Whatever you do, say what you did and why. A supervisor will accept a modest sample that is honestly justified far more readily than a large one that arrived by copying.

Eight mistakes that get proposals returned

1. Choosing the test after seeing the data. Running several tests and reporting the one that reached p < 0.05 inflates your false positive rate. Decide in the proposal.

2. Three t-tests instead of one ANOVA. Comparing three groups pairwise means three chances to find a difference that is not there. Run the ANOVA, and only then do post-hoc comparisons.

3. Correlation described in causal language. "Attendance was influenced by meal provision" is a causal claim. A correlation coefficient does not support it. Either change the design or change the sentence.

4. Chi-square with sparse cells. If more than 20% of expected counts fall below 5, the test is unreliable. Collapse categories or use Fisher's exact test.

5. Linear regression on a yes-or-no outcome. If the outcome is binary, use logistic regression. Linear regression will predict probabilities above 1 and below 0.

6. Reporting p-values without effect sizes. "Significant at p < 0.05" tells a reader that an effect probably exists, not whether it matters. Report Cohen's d, eta squared, R², or an odds ratio, and interpret it in the units of the study.

7. No assumption checks anywhere in the chapter. Increasingly asked about in vivas, and easy to do.

8. Instruments that cannot produce the data the objectives require. An objective that compares two groups needs an instrument that identifies which group each respondent belongs to. This sounds obvious and is the most common structural failure we see.

A worked example

Study: Influence of school feeding programmes on pupil attendance in public primary schools.

Objective 1. To establish the extent of implementation of school feeding programmes in the study area.

Descriptive. Frequencies and percentages for whether a school runs a programme, how many days a week, and how many pupils it reaches. No inferential test is needed and none should be forced.

Objective 2. To compare pupil attendance rates between schools with and without a feeding programme.

A comparison of two independent groups. The outcome, attendance rate as a percentage, is continuous. So: independent samples t-test, after checking normality within each group and homogeneity of variance. If attendance is strongly skewed, which it often is because most schools cluster near the top, use Mann-Whitney U instead. Report the mean difference and Cohen's d, not just the p-value.

Objective 3. To determine the influence of meal frequency on pupil attendance.

Prediction, with a continuous outcome. Linear regression of attendance rate on meals per week. Because pupils are nested within schools and the programme operates at school level, the analysis should be at school level, or use clustered standard errors if it is at pupil level. Report R² and the coefficient with its confidence interval, and say what a one-meal increase corresponds to in attendance points.

Notice that the three objectives require three different analyses, and that the instrument has to capture attendance, programme status and meal frequency for the same schools. That is a design decision, made at proposal stage.

A note on software

SPSS, Stata, R and Jamovi will all run every test in the table above. None of them will tell you that you chose the wrong one. The menu does not know what your objective says.

Jamovi and R are free. SPSS is what most Kenyan universities teach and what most supervisors can read. Use what your supervisor can follow, unless you have a reason not to.

When this is worth getting help with

This page covers the decisions you can make yourself, and most researchers can make them with a little care.

Get advice when the structure of your data does not fit the table: repeated measures on the same people over several time points, respondents nested inside schools or clinics or counties, missing data that is not missing at random, or a scale you built yourself that needs validating before you can analyse anything with it. Those situations need a judgement about design, not a lookup.

Probit Research advises on research design, methodology and analysis. We work with you on your own data and your own writing. We do not write theses, dissertations or assignments, and the point of every engagement is that you leave able to defend the choices yourself.