Statistical Good Practice Guidelines

1.1 Introduction

This guide is concerned with some fundamental ideas of analysis of data from surveys. The discussion is at a statistically simple level; other more sophisticated statistical approaches are outlined in our guide Modern Methods of Analysis. Our aim here is to clarify the ideas that successful data analysts usually need to consider to complete a survey analysis task purposefully.

An ill-thought-out analysis process can produce incompatible outputs and many results that never get discussed or used. It can overlook key findings and fail to pull out the subsets of the sample where clear findings are evident. Our brief discussion is intended to assist the research team in working systematically; it is no substitute for clear-sighted and thorough work by researchers. We do not aim to show a totally naïve analyst exactly how to tackle a particular set of survey data. However, we believe that where readers can undertake basic survey analysis, our recommendations will help and encourage them to do so better.

Part 1 outlines a series of themes, after an introductory example. Different data types are distinguished in Section 1.2. Section 1.3 looks at data structures; simple if there is one type of sampling unit involved, and hierarchical with e.g. communities, households and individuals. In Section 1.4 we separate out three stages of survey data handling - exploration, analysis and archiving - which help to define expectations and procedures for different parts of the overall process. We contrast the research objectives of description or estimation (Section 1.5), and of comparison (Section 1.6) and what these imply for analysis. Section 1.7 considers when results should be weighted to represent the population - depending on the extent to which a numerical value is or is not central to the interpretation of survey results. In Section 1.8 we outline the coding of non-numerical responses. The use of ranked data is discussed in brief in Section 1.9.

In Part 2 we look at the ways in which researchers usually analyse survey data. We focus primarily on tabular methods, for reasons explained in Section 2.1. Simple one-way tables are often useful as explained in Section 2.2. Cross-tabulations (Section 2.3) can take many forms and we need to think which are appropriate. Section 2.4 discusses issues about 'accuracy' in relation to two- and multi-way tables. In Section 2.5 we briefly discuss what to do when several responses can be selected in response to one question.

Cross-tabulations can look at many respondents, but only at a small number of questions, and we discuss profiling in Section 2.6, cluster analysis in Section 2.7, and indicators in Sections 2.8 and 2.9.

[TOP]

1.2 Data Types

Introductory Example: On a nominal scale the categories recorded, usually counted, are described verbally. The 'scale' has no numerical characteristics. If a single one-way table resulting from simple summarisation of nominal (also called categorical) scale data contains frequencies:

there is little that can be done to present exactly the same information in other forms. We could report highest frequency first as opposed to alphabetic order, or reduce the information in some way e.g. if one distinction is of key importance compared to the others:

On the other hand, where there are ordered categories, the sequence makes sense only in one, or in exactly the opposite, order:

We could reduce the information by combining categories as above, but also we can summarise, somewhat numerically, in various ways. For example, accepting a degree of arbitrariness, we might give scores to the categories:

and then produce an 'average score' - a numerical indicator - for the sample of:

This is an analogue of the arithmetical calculation we would do if the categories really were numbers e.g. family sizes.

The same average score of 3.33 could arise from differently patterned data e.g. from rather more extreme results:

Hence, as with any other indicator, this 'average' only represents one feature of the data and several summaries will sometimes be needed.

A major distinction in statistical methods is between quantitative data and the other categories exemplified above. With quantitative data, the difference between the values from two respondents has a clearly defined and incontrovertible meaning e.g. "It is 5C° hotter now than it was at dawn" or "You have two more children than your sister". Commonplace statistical methods provide many well-known approaches to such data, and are taught in most courses, so we give them only passing attention here.

In this guide we focus primarily on the other types of data, coded in number form but with less clear-cut numerical meaning, as follows. Binary - e.g. yes/no data - can be coded in 1/0 form; while purely categorical or nominal data - e.g. caste or ethnicity - may be coded 1, 2, 3... using numbers that are just arbitrary labels and cannot be added or subtracted. It is also common to have ordered categorical data, where items may be rated Excellent, Good, Poor, Useless, or responses to attitude statements may be Strongly agree, Agree, Neither agree nor disagree, Disagree, Strongly disagree. With ordered categorical data the number labels should form a rational sequence, because they have some numerical meaning e.g. scores of 4, 3, 2, 1 for Excellent through to Useless. Such data supports limited quantitative analysis, and is often referred to by statisticians as 'qualitative' - this usage does not imply that the elicitation procedure must satisfy a purist's restrictive perception of what constitutes qualitative research methodology.

[TOP]

1.3 Data Structure

SIMPLE SURVEY DATA STRUCTURE: the data from a single-round survey, analysed with limited reference to other information, can often be thought of as a 'flat' rectangular file of numbers, whether the numbers are counts/measurements, or codes, or a mixture. In a structured survey with numbered questions, the flat file has a column for each question, and a row for each respondent, a convention common to almost all standard statistical packages. If the data form a perfect rectangular grid with a number in every cell, analysis is made relatively easy, but there are many reasons why this will not always be the case and flat file data will be incomplete or irregular. Most importantly:-

* Surveys often involve 'skip' questions where sections are missed out if irrelevant e.g. details of spouse's employment do not exist for the unmarried. These arise legitimately, but imply different subsets of people respond to different questions. 'Contingent questions', where not everyone 'qualifies' to answer, often lead to inconsistent-seeming results for this reason. If the overall sample size is just adequate, the subset who 'qualify' for a particular set of contingent questions may be too small to analyse in the detail required.

* If some respondents fail to respond to some questions (item non-response) there will be holes in the rectangle. Non-informative non-response occurs if the data is missing for a reason unrelated to the true answers e.g. the interviewer turned over two pages instead of one! Informative non-response means that the absence of an answer itself tells you something, e.g. you are almost sure that the missing income value will be one of the highest in the community. A little potentially informative non-response may be ignorable, if there is plenty of data. If data are sparse or if informative non-response is frequent, the analysis should take account of what can be inferred from knowing that there are informative missing values.

HIERARCHICAL DATA STRUCTURE: another complexity of survey data structure arises if the data are hierarchical. A common type of hierarchy is where a series of questions is repeated say for each child in the household, and combined with a household questionnaire, and maybe data collected at community level.

For analysis, we can create a rectangular flat file, at the 'child level', by repeating relevant household information in separate rows for each child. Similarly, we can summarise information for the children in a household, to create a 'household level' analysis file. The number of children in the household is usually a desirable part of the summary; this "post-stratification" variable can be used to produce sub-group analyses at household level separating out households with different numbers of child members. The way the sampling was done can have an effect on interpretation or analysis of a hierarchical study. For example if children were chosen at random, households with more children would have a greater chance of inclusion and a simple average of the household sizes would be biased upwards: it should be corrected for selection probabilities.

Hierarchical structure becomes important, and harder to handle, if there are many levels where data are collected e.g. government guidance and allocations of resource, District Development Committee interpretations of the guidance, Village Task Force selections of safety net beneficiaries, then households and individuals whose vulnerabilities and opportunities are affected by targeting decisions taken at higher levels in the hierarchy. In such cases, a relational database reflecting the hierarchical structure is a much more desirable way than a spreadsheet to define and retain the inter-relationships between levels, and to create many analysis files at different levels. Such issues are described in the guide The Role of a Database Package for Research Projects. Any one of the analysis files may be used as we discuss below, but any such study will be looking at one facet of the structure, and several analyses will have to be brought together for an overall interpretation. A more sophisticated approach using multi-level modelling, described in our guide on Modern Methods of Analysis, provides a way to look at several levels together.

[TOP]

1.4 Stages of Analysis

It is often worth distinguishing the three stages of exploratory analysis, deriving the main findings, and archiving.

EXPLORATORY DATA ANALYSIS (EDA) means looking at the data files, maybe even before all the data has been collected and entered, to get an idea of what is there. It can lead to additional data collection if this is seen to be needed, or savings by stopping collecting data when a conclusion is already clear, or existing results prove worthless. It is not assumed that results from EDA are ready for release as study findings.

* EDA usually overlaps with data cleaning; it is the stage where anomalies become evident e.g. individually plausible values may lead to a way-out point when combined with other variables on a scatterplot. In an ideal situation, EDA would end with confidence that one has a clean dataset, so that a single version of the main datafiles can be finalised and 'locked' and all published analyses derived from a single consistent form of 'the data'. In practice later stages of analysis often produce additional queries about data values.

* Such exploratory analysis will also show up limitations in contingent questions e.g. we might find we don't have enough currently married women to analyse their income sources separately by district. EDA should include the final reconciliation of analysis ambitions with data limitations.

* This phase can allow the form of analysis to be tried out and agreed, developing analysis plans and program code in parallel with the final data collection, data entry and checking. Purposeful EDA allows the subsequent stage of deriving the main findings to be relatively quick, uncontroversial, and well organised.

DERIVING THE MAIN FINDINGS: the second stage will ideally begin with a clear-cut clean version of the data, so that analysis files are consistent with one another, and any inconsistencies, e.g. in numbers included, can be clearly explained. This is the stage we amplify upon, later in this guide. It should generate the summary findings, relationships, models, interpretations and narratives, and first recommendations that research users will need to begin utilising the results.

Of course one needs to allow time for 'extra' but usually inevitable tasks such as:-

* follow-up work to produce further more detailed findings, e.g. elucidating unexpected results from the pre-planned work.

* a change made to the data, each time a previously unsuspected recording or data entry error comes to light. Then it is important to correct the database and all analysis files already created that involve the value to be corrected. This will mean repeating analyses that have already been done using, but not revealing, the erroneous value. If that analysis was done "by mouse clicking" and with no record of the steps, this can be very tedious. This stage of work is best undertaken using software that can keep a log: it records the analyses in the form of program instructions that can readily and accurately be re-run.

ARCHIVING means that data collectors keep, perhaps on CD, all the non-ephemeral material relating to their efforts to acquire information. Obvious components of such a record include:-

(i) data collection instruments,

(ii) raw data,

(iii) metadata recording the what, where, when, and other identifiers of all variables,

(iv) variable names and their interpretations, and labels corresponding to values of categorical variables,

(v) query programs used to extract analysis files from the database,

(vi) log files defining the analyses, and

(vii) reports.

Often georeferencing information, digital photographs of sites and scans of documentary material are also useful. Participatory village maps, for example, can be kept for reference as digital photographs.

Surveys are often complicated endeavours where analysis covers only a fraction of what could be done. Reasons for developing a good management system, of which the archive is part, include:-

* keeping the research process organised as it progresses;

* satisfying the sponsor's (e.g. DFID's) contractual requirement that data should be available if required by the funder or by legitimate successor researchers;

* permitting a detailed re-analysis to authenticate the findings if they are questioned;

* allowing a different breakdown of results e.g. when administrative boundaries are redefined;

* linking several studies together, for instance in longer-term analyses carrying baseline data through to impact assessment.

[TOP]

1.5 Population Description as the Major Objective

In the next section we look at the objective of comparing results from sub-groups, but a more basic aim is to estimate a characteristic like the absolute number in a category of proposed beneficiaries, or a relative number such as the prevalence of HIV seropositives. The estimate may be needed to describe a whole population or sections of it.

In the basic analyses discussed below, we need to bear in mind both the planned and the achieved sampling structure.

Example: Suppose 'before' and 'after' surveys were each planned to have a 50:50 split of urban and rural respondents. Even if we achieved 50:50 splits, these would need some manipulation if we wanted to generalise the results to represent an actual population split of 70:30 urban:rural. Say we wanted to assess the change from 'before' to 'after' and the achieved samples were in fact split 55:45 and 45:55. We would have to correct the results carefully to get a meaningful estimate of change.

Samples are often stratified i.e. structured to capture and represent particular segments of the target population. This may be much more sophisticated than the urban/rural split in the previous paragraph. Within-stratum summaries serve to describe and characterise each of these parts individually. If required by the objectives, overall summaries, which put together the strata, need to describe and characterise the whole population.

It may be fine to treat the sample as a whole and produce simple, unweighted summaries if (i) we have set out to sample the strata proportionately, (ii) we have achieved this, and (iii) there are no problems due to hierarchical structure. Non-proportionality arises from various quite distinct sources, in particular:-

* Case A: often sampling is disproportionate across strata by design, e.g. the urban situation is more novel, complex, interesting or accessible, and gets greater coverage than the fraction of the population classed as rural.

* Case B : sometimes particular strata are bedevilled with high levels of non-response, so that the data are not proportionate to stratum sizes, even when the original plan was that they should be.

If we ignore non-proportionality, a simple-minded summary over all cases is not a proper representation of the population in these instances.

The 'mechanistic' response to 'correct' both the above cases is (1) to produce within-stratum results (tables or whatever), (2) to scale the numbers in them to represent the true population fraction that each stratum comprises, and then (3) to combine the results.

There is often a problem with doing this in case B, where non-response is an important part of the disproportionality: the reasons why data are missing from particular strata often correspond to real differences in the behaviour of respondents, especially those omitted or under-sampled, e.g. "We had very good response rates everywhere except in the north. There a high proportion of the population are nomadic, and we largely failed to find them." Just scaling up data from settled northerners does not take account of the different lifestyle and livelihood of the missing nomads. If you have largely missed a complete category, it is honest to report partial results making it clear which categories are not covered and why.

One common 'sampling' problem arises when a substantial part of the target population is unwilling or unable to cooperate, so that the results in effect only represent a limited subset - those who volunteer or agree to take part. Of course the results are biased towards e.g. those who command sufficient resources to afford the time, or e.g. those who habitually take it upon themselves to represent others. We would be suspicious of any study which appeared to have relied on volunteers, but did not look carefully at the limits this imposed on the generalisability of the conclusions.

If you have a low response rate from one stratum, but are still prepared to argue that the data are somewhat representative, the situation is at the very least uncomfortable. Where you have disproportionately few responses, the multipliers used in scaling up to 'represent' the stratum will be very high, so your limited data will be heavily weighted in the final overall summary. If there is any possible argument that these results are untypical, it is worthwhile to think carefully before giving them extra prominence in this way.

[TOP]

1.6 Comparison as the Major Objective

One sound reason for disproportionate sampling is that the main objective is a comparison of subgroups in the population. Even if one of two groups to be compared is very small, say 10% of the total number in the population, we now want roughly equally many observations from each subgroup, to describe both groups roughly equally accurately. There is no point in comparing a very accurate set of results from one group with a very vague, ill-defined description of the other; the comparison is at least as vague as the worse description. The same broad principle applies whether the comparison is a wholly quantitative one looking at the difference in means of a numerical measure between groups, or a much looser verbal comparison e.g. an assessment of differences in pattern across a range of cross-tabulations.

If for a subsidiary objective we produce an overall summary giving 'the general picture' of which both groups are part, 50:50 sampling may need to be re-weighted 90:10 to produce a quantitative overall picture of the sampled population.

The great difference between true experimental approaches and surveys is that experiments usually involve a relatively specific comparison as the major objective, while surveys much more often do not. Many surveys have multiple objectives, frequently ill defined, often contradictory, and usually not formally prioritised. Along with the likelihood of some non-response, this tends to mean there is no sampling scheme which is best for all parts of the analysis, so various different weighting schemes may be needed in the analysis of a single survey.

[TOP]

1.7 When Weighting Matters

Several times in the above we have discussed issues about how survey results may need to be scaled or weighted to allow for, or 'correct for', inequalities in how the sample represents the population. Sometimes this is of great importance, sometimes not. A fair evaluation of survey work ought to consider whether an appropriate trade-off has been achieved between the need for accuracy and the benefits of simplicity.

If the objective is formal estimation, e.g. of total population size from a census of a sample of communities, we are concerned to produce a strictly numerical answer, which we would like to be as accurate as circumstances allow. We should then correct as best we can for a distorted representation of the population in the sample. If groups being formally compared run across several population strata, we should try to ensure the comparison is fair by similar corrections, so that the groups are compared on the basis of consistent samples. In these cases we have to face up to problems such as unusually large weights attached to poorly-responding strata, and we may need to investigate the extent to which the final answer is dubious because of sensitivity to results from such subsamples.

Survey findings are often used in 'less numerical' ways, where it may not be so important to achieve accurate weighting e.g. "whatever varieties they grow for sale, a large majority of farm households in Sri Lanka prefer traditional red rice varieties for home consumption because they prefer their flavour". If this is a clear-cut finding which accords with other information, if it is to be used for a simple decision process, or if it is an interim finding which will prompt further investigation, there is a lot to be said for keeping the analysis simple. Of course it saves time and money. It makes the process of interpretation of the findings more accessible to those not very involved in the study. Also, weighting schemes depend on good information to create the weighting factors and this may be hard to pin down.

Where we have worryingly large weights, attaching to small amounts of doubtful information, it is natural to want to put limits on, or 'cap', the high weights, even at the expense of introducing some bias, i.e. to prevent any part of the data having too much impact on the result.

The ultimate form of capping is to express doubts about all the data, and to give equal weight to every observation. The rationale, not usually clearly stated, even if analysts are aware they have done this, is to minimise the maximum weight given to any data item. This lends some support to the common practice of analysing survey data as if they were a simple random sample from an unstructured population. For 'less numerical' usages, this may not be particularly problematic as far as simple description is concerned. Of course it is wrong - and may be very misleading - to follow this up by calculating standard deviations and making claims of accuracy about the results which their derivation will not sustain!

[TOP]

1.8 Coding

We recognise that purely qualitative researchers may prefer to use qualitative analysis methods and software, but where open-form and other verbal responses occur alongside numerical data it is often sensible to use a quantitative tool. From the statistical viewpoint, basic coding implies that we have material, which can be put into nominal-level categories. Usually this is recorded in verbal or pictorial form, maybe on audio- or videotape, or written down by interviewers or self-reported. We would advocate computerising the raw data, so it is archived. The following refers to extracting codes, usually describing the routine comments, rather than unique individual ones which can be used for subsequent qualitative analysis.

By scanning the set of responses, themes are developed which reflect the items noted in the material. These should reflect the objectives of the activity. It is not necessary to code rare, irrelevant or uninteresting material.

In the code development phase, a large enough range of the responses is scanned to be reasonably sure that commonly occurring themes have been noted. If previous literature, or theory, suggests other themes, these are noted too. Ideally, each theme is broken down into unambiguous, mutually exclusive and exhaustive, categories so that any response segment can be assigned to just one, and assigned the corresponding code value. A 'codebook' is then prepared where the categories are listed and codes assigned to them. Codes do not have to be consecutive numbers. It is common to think of codes as presence/absence markers, but there is no intrinsic reason why they should not be graded as ordered categorical variables if appropriate, e.g. on a scale such as fervent, positive, uninterested/no opinion, negative.

The entire body of material is then reviewed and codes are recorded. This may be in relevant places on questionnaires or transcripts. Especially when looking at 'new' material not used in code development, extra items may arise and need to be added to the codebook. This may mean another pass through material already reviewed, to add new codes e.g. because a particular response is turning up more than expected.

From the point of view of analysis, no particular significance attaches to particular numbers used as codes, but it is worth bearing in mind that statistical packages are usually excellent at sorting, selecting or flagging, for example, 'numbers between 10 and 19' and other arithmetically defined sets. If these all referred to a theme such as 'forest exploitation activities of male farmers' they could easily be bundled together. It is of course impossible to separate out items given the same code, so deciding the right level of coding detail is essential at an early stage in the process.

When codes are analysed, they can be treated like other nominal or ordered categorical data. The frequencies of different types of response can be counted or cross-tabulated. Since they often derive from text passages and the like, they are often particularly well-adapted for use in sorting listings of verbal comments - into relevant bundles for detailed non-quantitative analysis.

[TOP]

1.9 Ranking & Scoring

A common means of eliciting data is to ask individuals or groups to rank a set of options. The researchers' decision to use ranks in the first place means that results are less informative than scoring, especially if respondents are forced to choose between some nearly-equal alternatives and some very different ones. A British 8-year-old offered baked beans on toast, or fish and chips, or chicken burger, or sushi with hot radish might rank these 1, 2, 3, 4 but score them 9, 8.5, 8, and 0.5 on a zero to ten scale!

Ranking is an easy task where the set of ranks is not required to contain more than about four or five choices. It is common to ask respondents to rank, say, their best four from a list of ten, with 1 = best, etc. Accepting a degree of arbitrariness, we would usually replace ranks 1, 2, 3, 4, and a string of blanks by pseudo-scores 4, 3, 2, 1, and a string of zeros, which gives a complete array of numbers we can summarise - rather than a sparse array where we don't know how to handle the blanks. A project output paper available via the SSC website explores this in more detail.

Where the instructions were to rank as many as you wish from a fixed, long list, we would tend to replace the variable length lists of ranks with scores. One might develop these as if respondents each had a fixed amount, e.g. 100 beans, to allocate as they saw fit. If four were chosen these might be scored 40, 30, 20, 10, or with five chosen 30, 25, 20, 15, 10, with zeros again for unranked items. These scores are arbitrary e.g. 40, 30, 20, 10 could instead be any number of choices e.g. 34, 28, 22, 16 or 40, 25, 20, 15; this reflects the rather uninformative nature of rankings, and the difficulty of post hoc construction of information that was not elicited effectively in the first place.

Having reflected and having replaced ranks by scores we would usually treat these like any other numerical data, with one change of emphasis. Where results might be sensitive to the actual values attributed to ranks, we would stress sensitivity analysis more than with other types of numerical data, e.g. re-running analyses with (4, 3, 2, 1, 0, 0, ...) pseudo-scores replaced by (6, 4, 2, 1, 0, 0 , ...). If the interpretations of results are insensitive to such changes, the choice of scores is not critical.

Converting Ranks to Scores for an ad hoc Assessment of Methods of Communication Available to Farmers by Savitri Abeyasekera, Julie Lawson-Macdowell & Ian Wilson. This is an output from DFID-funded work under the Farming Systems Integrated Pest Management Project, Malawi and DFID NRSP project R7033, Methodological Framework for Combining Qualitative and Quantitative Survey Methods.

[TOP]