Find information about

Section 4 Techniques and Tools

4.02 Planning and designing useful surveys

A key principle of quality in research and consultation is that the research design must be led by purpose and use. It is essential that the purposes of the research and how the findings will be used are clear. This section addresses the elements of a utilisation-focus to the planning and design of surveys.

Think about the purpose of the survey

It is important to be clear about the purpose of any general survey. Undertaking a survey may raise expectations for change that can’t be met either because the information asked for is not precise or clear enough to be actionable or because resources or other factors are not in place to support change. It is important to be clear about what can be changed and to be frank with service users about what they can expect to see happen as a result of any survey. Satisfaction surveys should not be undertaken simply to elicit general approval ratings that do not provide actionable information or for PR purposes.

Practice point

• Surveys should have a clear purpose and focus on the usefulness of the data rather than being done to elicit general approval ratings. The uses to which the data will be put and by whom should be able to be specified in advance.

Ask the right questions of the right people

Different terminology is used by social landlords to refer to people in relation to housing. These terms of reference include tenants, residents, owners, shared owners, customers, service users. It is important to be clear about who should be answering the questions. This may be very strictly defined for example, the legal tenant, or may be more flexible, such as any adult living in the property. Exactly what is meant by whatever term is used should be clearly defined. The working definition used may miss out an important section of the community of research interest, so, for example, a focus on tenants will not include the views of young people and others in the community who are not legal tenants.

It also important to define the research population to make sure you ask for information based on recent experience. If the focus of the survey is assessing general satisfaction and views, then a sample based on the whole target population will be appropriate, such as all tenants or residents. If the survey is asking about specific services, then it will be important to ensure that you target actual service users. If it is attempting to do both, those who do not have recent experience of using a particular service should not be asked questions about it. More guidance is available in the section on sampling.

Consider whether change over time is important

Surveys may produce baseline data to allow monitoring over time and comparison with other organisations or to allow benchmarking. If this is the intention, standardised question wording should be used. Landlords may feel tied to using previous questions even though many things may have changed so it’s worth considering very carefully how necessary this is.

Keep it short

Questionnaire length, the way the questionnaire is administered and the way questions are worded all affect the quality of the data. As a general rule, ask as few questions as possible and only ask questions that are necessary.

Simulate how the data will be used

Before data is collected, it’s useful to simulate how the data will be used. This can be done with fabricated findings, in other words, with the kind of findings you expect to get. This can be done in a real-enough way to help you make concrete decisions about what changes are needed to make the data as useful as possible. As a result of this kind of exercise, you might decide not to ask particular questions or decide that you need to ask alternative or additional questions if you are to be able to analyse the data in the way you wish.

It would be worthwhile to involve a mix of people in this exercise who are likely to have different interests and perspectives, including tenants or residents themselves. This will help to ensure that the results from each question will be believable, credible and valid to those people.

Think about what contributes to service quality

Make a distinction between questions which ask about processes that enable a service to be provided (for example, the speed of response to a repair request) and questions about the outcomes of those services (for example, the quality of the work achieved).

To avoid questionnaires becoming an unstructured collection of loosely or unrelated questions, identify the different characteristics or dimensions of service quality and use each dimension as a section heading if several questions are to be asked on it. For example;

• access - such as convenient opening hours;

• reliability – such as keeping appointment times; and

• responsiveness – such as providing appointment times.

This approach will provide more useful information to improve services.

Go beyond the headline figures

It is also important to go beyond ‘headline’ approval ratings. For example, a useful satisfaction survey will ask questions that provide richer data in addition to questions that merely enable simple number counts such as ‘90% of tenants are satisfied with the services provided by the landlord’. A useful survey will ask follow-up questions about respondents’ reasons for expressing satisfaction or dissatisfaction. These questions will elicit information about what characteristics or aspects of the service are important to service users. In this way, valuable information about expectations can be gained and the landlord can then either continue to provide the service in the same way or make changes. For more guidance see the sections on assessing satisfaction and expectations, including the use of different types of rating scales and Servqual.

Consider scope for sharing resources

Whilst there is no endorsed standardised survey in Scotland, there may be scope for greater sharing of approaches; this might include omnibus surveys across different local authority departments or groups of housing associations. See the section on sharing ideas and resources amongst landlords.

Check that all questions asked are relevant to your organisation

Sometimes standardised questions are used by consultants without regard for the specific information requirements and local context of the commissioning organisation. A utilisation-focus will help to ensure that questions are suitably adapted.

Only ask for socio-economic and demographic data that you can use

Surveys often ask for socio-economic and demographic data such as age, gender, disability, household type, employment status, minority ethnic group and income. It’s worth asking how you intend to use this data. In some cases, sample sizes will be too small to allow any meaningful sub-group analysis. If you are interested in the particular views of a smaller sub-group or community such as lone parents or a specific minority ethnic community, you will need to conduct some complementary, more focused research. See, in particular, the sections on making research as inclusive as possible, focus groups and participatory methods.

If your purpose is to build up a profile of the client base across the whole organisation, this approach would only be valid if the sample is a statistically representative one. See the section on sampling for more guidance. If your purpose is to allow analysis of the views and experiences of particular groups and to identify scope for follow-up work, as long as you are clear about how you will be able to act on the data provided, the research will be worth doing. Asking for information about income is more difficult; some landlords have included this in general surveys to allow them to undertake affordability analysis. However, the amount of missing data and the unreliability of the data collected in this way undermines its accuracy and usefulness. A specific exercise with a more focused and sensitive approach is likely to yield better data for such purposes.

Test out the questions and the questionnaire beforehand

The wording of questions and the whole questionnaire should be piloted or tested before the survey starts with a group of people who can comment on design and administration aspects. This provides an opportunity to make last-minute changes that could enhance the quality of the data.

Planning and designing useful surveys; checklist

√ Be clear about the purpose of a general survey and whose views it is seeking. It may be more effective to target particular service users rather than conducting a general survey.

√ Clearly specify the target population for your survey.

√ Ask questions that are important to tenants or residents as well as questions that concern landlords.

√ Be clear about what changes in services may be realistically expected as a result of the survey.

√ Consider carefully how important it is to track changes over time; do not automatically ask the same questions as earlier surveys unless there is a good reason to do so.

√ Design use into the survey; ask for precise and actionable information and be clear about the intended use of every question in the survey.

√ Consider how the data is to be analysed in advance; discard any questions that are not essential, and discard or appropriately reword questions that are likely to elicit a low response.

√ Be clear in advance about who will use the survey results and exactly how the results are to be used to change services.

√ Always tell service users what changes have resulted from the last time they were consulted and acknowledge the value of their input. This will help to encourage future responses by demonstrating the actions that have resulted.

Previous page Next page