Find information about

Section 3 Issues and concerns in research and consultation

3.02 Representativeness: how important is it?

Great value is often attached to ideas about representativeness in research and consultation. This is because researchers often want to be able to generalise their findings to a larger population. In fact, this can only be done if the sample under study is statistically representative of the larger population from which it was drawn.

True statistical representativeness is actually very difficult and costly to achieve. In fact, it is doubtful whether sampling bias can be sufficiently removed to achieve truly representative statistical samples and it is not a matter of certainty that sample findings can be generalised to the larger population; calculations are needed to state the degree of confidence that can be placed on statistics based on samples.

Often results are claimed to be representative, although these claims do not stand up to scrutiny when sampling strategies are examined. Attempts to achieve statistical representativeness in research and consultation exercises are rarely feasible or successful and may hinder the use of approaches that might be more realistic, meaningful and useful.

Decisions about whether a survey sample design should be based on the goal of statistical representativeness will depend on the purpose of the survey. To be strictly statistically representative, a sample should be selected on a random or probability basis. See the section on sampling for more information. This is only really necessary if you wish to be able to make claims about the wider population on the basis of your sample by calculating inferential statistics. This is rarely a major concern of general satisfaction surveys and would be costly to achieve.

It is more likely to be important that samples for general satisfaction surveys are distributed across the target population in proportion, so that key variables such as area office patches or tenants and owners are included in the same proportion that they occur in the overall population. Some kind of stratified or quota sampling is likely to be adequate for most purposes. Sample size is also a key issue. Procedures to select cases to fulfil quotas should also minimise systematic bias. All these issues are dealt with in more detail in the section on sampling.

Research for this guidance suggests that some landlords (and service users themselves) are concerned about low response rates to surveys. Poor response rates do, of course, undermine representativeness and it is right to be concerned about this. However, the problem may be more fundamental than finding a technical fix by offering incentives and so on.

A more fundamental rethink about appropriate methods may be required; although research for this guidance suggests that some landlords are now reconciled to the difficulties of strict statistical representativeness in surveys, other landlords’ concerns about achieving an elusive strict statistical representativeness may undermine their confidence in the validity of other research approaches and methods that might be more meaningful and useful.

It is important to understand that using statistics is not the only way of being representative and of being able to generalise from research findings. The pursuit of representativeness and generalisability is present in the use of other methods and approaches but the concepts may be treated differently. For example, most qualitative research researchers want to know ‘what’s going on’, so it is important to study the processes that will clarify and validate ‘what’s going on’. Therefore, sampling is not based on probability theory and randomness, nor are samples drawn on the basis of how statistically representative they are of a larger population. Rather, samples are selected through ‘theoretical sampling’ which enables sampling of whatever phenomena will provide valuable information on issues of importance and people are sampled on the basis that they are ‘cultural experts’ who are able to shed light on ‘what’s going on’.

Generalisability in qualitative research is gained through constantly comparing between different groups undergoing the same processes. For example, if the researcher thinks they have a reasonably good descriptive understanding of one event, then they should also examine other similar events and compare them. This enables common themes and differences across different groups to be identified.

We think the ideas of ‘meaningful representativeness’ or ‘participatory representativeness’ are often more useful in research and consultation and apply across the whole range of approaches as part of a strategic approach to consultation. Meaningful representation recognises that some people will not be interested in giving their views, whatever efforts are made by their landlord to seek them, but does not absolve the landlord of the responsibility to ensure that they have a number of opportunities to do so if they wish. Meaningful or participatory representativeness would aim to ensure that all tenants and residents have opportunities to make their views known, through the use of a variety of diverse methods, conducted in ways that are as inclusive as possible and designed to provide actionable information for the landlord.

Practice Point

• Be clear about the purpose and the use of your research. Pursuing statistical representativeness may not be the most appropriate way to achieve the depth and clarity of understanding about topics that you need. Your research purpose and use of findings may require you to use approaches and methods that do not always treat representativeness as statistical.

Previous page Next page