You are here: home how we work good practice how to gather views on service quality techniques and tools quantitative methods

Find information about

 How we work

Section 4 Techniques and Tools

4.03 Planning for high quality surveys

This section looks at a number of issues that affect quality in survey design, data collection, analysis and reporting. It considers response rates, the use of contractors and in-house surveys, resources for surveys, different types of surveys, representativeness in sampling for surveys, high quality analysis and common errors and reporting.

Promoting better response rates in surveys

Surveys often claim to be representative in some way, but whether this is the case will depend on the nature and quality of the sample selected and the response rate.

Response rates vary for different types of survey; face-to-face surveys should achieve response rates of 60-70 per cent. Postal or telephone surveys are likely to achieve between 30-50 per cent. Response rates are said to be declining, although many satisfaction survey reports do not include this information. However, for some surveys it is not possible to calculate a true response rate because of the way the sample is chosen. See the section on sampling.

It may be that response rates differ for sub-groups of service users, across different geographical areas and tenure groups. It is important to consider what measures might be necessary to encourage responses from the full range of service users. This should be considered in advance and where possible steps taken to ensure sufficient responses are received to analyse the data in the way intended. It may be necessary to oversample certain sub-groups. The use of incentives such as prize draws and inclusion of a freepost reply or stamped addressed envelope for postal surveys may encourage responses. For face-to-face surveys, fieldwork procedures must be adequate to reflect the likely times that people will be at home and must ensure that sufficient calls are made to each address. This will usually be three to four calls at different times of the day and week.

Survey procedures will need to be adapted to accommodate respondents with literacy issues, with care and support needs and people whose first language is not English. This might mean briefing interviewers to bring in a translator when they come across a non-English speaker and avoiding the use of a postal or self-completion survey. See the sections on making research and consultation as inclusive as possible and ethical research practice.

All surveys should briefly outline their purpose to allow people to give informed consent to their participation. It is usual to assure participants of the confidentiality and anonymity of their responses. Strict anonymity may not be necessary; if the survey asks if participants would like information to be sent to them, it will be necessary to ensure that it is possible to do this and participants can be asked to provide their details.

Promoting better response rates; checklist

√ Adopt a utilisation focus to planning and designing the survey.

√ Consider how best to use publicity to promote the survey and encourage responses.

√ Involve tenants and residents in the design of the scope of the survey.

√ Get tenants’ organisations on board to help publicise the survey and encourage participation.

√ Ensure that fieldwork procedures are adequate to provide sufficient opportunities for participation and consider the use of incentives to promote responses.

√ Adopt procedures to accommodate service users with literacy or support needs and those whose first language is not English.

Who should conduct general surveys?

Typically large-scale general satisfaction surveys have been conducted by external consultants who design and manage the survey process and analyse and report the findings. Options for managing research and consultation are discussed more fully elsewhere.

The use of external contractors has often been to provide an assessment that is seen as more independent than that likely to be provided if the work is conducted in-house. This is sometimes seen as more acceptable by service users themselves or by councillors and board members.

Each survey should have a brief or specification outlining the objectives, key stages of the project and outputs, including information required for quality assurance, even if the survey is to be conducted in-house.

Appropriate skills and capacity to manage large surveys are not always available in-house. Large surveys are likely to require substantial project management even if they are contracted out and will benefit from the use of dedicated research staff time.

However, surveys are expensive and managing at least some of the elements of the work in-house could provide cost savings.

A further option is to consider whether the contract for such surveys could include an element of skills transfer or capacity building for staff.

The active involvement of tenants or residents and other key stakeholders in the design of the scope of the survey and in the interpretation of the findings may allay fears about a lack of independence.

Contractors should have a track record in survey work and should work to a professional code of conduct or ethics. Market research and social survey companies should work to the Market Research Society Code of Conduct. Other contractors, however small, should work within a quality control framework such as the Social Research Association Ethical Guidelines. For a fuller discussion see the section on ethical research practice.

Contractors should be asked to provide information for quality assurance purposes including:

a) Survey design procedures – how the specific questions and question wording was decided upon.

b) Piloting procedures – how the survey was tested out.

c) Sampling strategy – sample size, sample strategy and statistical information to assess the representativeness of the achieved sample.

d) Response rates, including by key factors such as tenure or area.

e) Commentary on response rates achieved.

f) Fieldwork procedures; measures to promote response rates; the procedures adopted to accommodate service users with support needs and those whose first language is not English.

g) Ethical issues; for example provisions to ensure confidentiality, anonymity, informed consent where appropriate.

For more guidance on research commissioning and management, see the section on options for organising and managing research and consultation.

Practice point

• It may matter less exactly who conducts the survey in terms of carrying out the technical tasks, than a sense that there have been sufficient opportunities for all key stakeholders to influence the scope of the survey, the interpretation of the findings and decisions about what actions should be taken as a result. Consider establishing a critical reference or advisory group to be involved in this way.

Resources for surveys

There should be a specific budget for service user research and consultation.

How long it will take to conduct a survey will depend on sample size, sample dispersion, length of questionnaire and the form of administration. It may also depend on the available field force if interviewers are to be used. Postal surveys need to build in at least two weeks for initial replies, then a further two weeks after a reminder and so on for any subsequent reminders. Surveys conducted face-to-face will be more expensive and time consuming than postal or telephone surveys.

There may be scope for some organisations to develop a more strategic approach to research commissioning, perhaps working in partnership with others in the locality to share expertise and spread costs. See the section on budgeting for research and consultation.

Planning for high quality surveys: checklist

√ Ensure there is a survey brief outlining the objectives, key stages of the project and outputs, even if the survey is to be conducted in-house.

√ Consider whether you need to contract out large-scale surveys to specialists. If so, this should be by competitive tender.

√ Ensure any contractor has a track record in surveys of this kind and that they work to a professional code of conduct or ethics.

√ Ask contractors to provide detailed technical information for quality assurance purposes.

Different types of surveys

Surveys can be designed and administered in a number of different ways. The best format to be used depends both on the specific group of service users of interest and the survey topic and will affect the type, amount, quality and reliability of the data collected. The pros and cons of different survey types are discussed in Figure 4.1.

Figure 4.1 The pros and cons of different types of surveys Face-to-face surveys • Surveys conducted face-to-face are able to collect fuller, more complex data. • The use of an interviewer gives more control over who actually answers the questions. This will be important with strict statistically representative sampling designs. • Designed with care and administered well they will generally have better response rates than other types of survey. • They are likely to be more expensive than other options. Postal or self-completion surveys • These are less reliable, need to be shorter than face-to-face surveys and use simple, “tick boxes” types of questions. • They can be cost effective and provide anonymity which may prompt a better response rate for more sensitive topics. • Whilst many organisations may prefer postal surveys on cost grounds, it may not always be the most appropriate approach. • There is a higher risk that some groups will be over or under-represented, such as those with language, literacy difficulties or with support needs. Telephone surveys • These need to be relatively short and straightforward. • Some categories of people will be systematically under-represented. • Telephone surveys may be useful for some service specific surveys where there is a contact number for each person from which to draw a sample. Web surveys • At present, web based or e-mail surveys are of limited value in customer research in public service contexts because the distribution of access to the web is not evenly spread across all sections of the population.

Representativeness in surveys: how important is it?

Decisions about whether a survey sample design should be based on the goal of statistical representativeness will depend on the purpose of the survey. Attempts to achieve statistical representativeness in research and consultation exercises are rarely feasible or successful and may hinder the use of approaches that might be more realistic, meaningful and useful. Great value is often attached to ideas about representativeness in research and consultation and this is discussed more fully in Section 3.

Ensuring high quality analysis and reporting of survey data

Survey data is not always analysed as fully as it could be. This is probably because too much data is collected without a clear sense of exactly what it is for. If the use of the data has been designed in from the start, it should be clear how all the questions are to be analysed.

Simple descriptive statistics such as frequency counts and percentages will usually be necessary. It may also be useful to analyse the experience or views of different sub-groups within the research population, such as tenants in different areas, types of properties or those with different demographic characteristics.

The interpretation and presentation of survey findings must be believable, credible and valid to a critical reference or advisory group. It is difficult to undertake quantitative analysis in a genuinely participatory way, but checking emerging findings back with an advisory group in a more iterative way may assist the process of interpretation and validation.

The survey analysis should be written up in a way that clearly presents the key findings and addresses the original information requirements. This should be comprehensive in relation to the aims of the survey, but need not be complicated or extensive.

Tables, figures and charts should be used to convey key points, not necessarily every point.

Avoiding errors and misrepresentation in analysis

There are a number of common errors and misrepresentations in analysis. To avoid these, the following checks should be made.

• Ensure that all the data collected has been fully analysed including sub-group analysis and that the analysis addresses the original information requirements.

• Ensure that categories have not been combined to distort the findings, such as “very” and “fairly satisfied”.

• Make sure comparisons of answers across different groups are on a ‘like-for-like’ basis.

• Ensure clarity about how missing data or ‘don’t knows’ have been treated and ensure that where these are a large proportion of the total responses this is noted as a finding, rather than being disregarded.

• Ensure that appropriate statistical procedures have been used and reported.

• Make sure that claims about the wider research population are not made on the basis of very small sample or sub-sample sizes.

• Ensure the number base of the figures presented is quoted.

• Where there are low numbers of cases ensure that the absolute figures are quoted rather than percentages.

• Make sure the results from key sub-groups are analysed.

• Check that the scale used on figures and charts does not distort the findings.

• Ensure that claims are not made about the wider applicability of the findings unless the sampling strategy supports such statistical inference.

• Make sure that the interpretation of survey findings is grounded in the evidence presented.

• Use the indicators reported to identify issues for further investigation.

• Ensure that any apparent inconsistencies or contradictory findings are highlighted and alternative interpretations are given.

Reporting survey findings

Presentation formats should be appropriate for the audience and made available, as appropriate and practicable, in suitable formats such as Braille, tapes and other languages. Outputs should be available in other media such as presentations, newsletters and visual displays.

Ensure that all service users, staff, committee members and councillors are informed of the results and their feedback invited. The findings should make it clear what the implications of the research are for service delivery and any further research that’s needed to more fully understand the results.

Reporting survey findings: checklist

√ Ensure the findings are written up in an appropriate and accessible way that identifies the implications for service delivery and the actions to be taken as a result.

√ Consider whether outputs should be available in other formats and media.

√ Ensure that all service users, staff, committee members and councillors are informed of the results and their feedback invited.

√ Consider what further research and consultation needs still exist.

Previous page Next page

Terms & Conditions | Freedom of Information | bookmark this website | © Crown Copyright