U.S. Department of Housing and Urban Development, Office of Policy Development and Research. A Guide to Evaluating Crime Control of Programs in Public Housing. Washington, DC: Prepared for the U.S. Department of Housing and Urban Development by KRA Corporation; 1997.  pp. 6.1-6.15.

View entire document

 

Chapter 6
   How Do You Get the Information You Need for Your Evaluation?

After you have completed your evaluation design, you are ready to begin collecting information, the process commonly referred to by evaluators as the data collection phase. This chapter will provide you with steps and suggestions to help you in this process. The information you collect will provide some immediate feedback on whether the program has been effective in reaching its objective(s).

Information collection generally consists of six steps. These steps were introduced earlier in this manual as part of the process of developing your evaluation plan.

Step 1: Determine the kinds of information you need for your evaluation

Your outcome objectives, if stated in measurable terms, will guide the decision about the kinds of information needed. As discussed in chapter 2, both quantitative and qualitative performance measures will be needed to describe program outcomes. You will need to collect information that can be used to demonstrate that your program or program activities have been effective.

For example if one of your general participant outcome objectives is to integrate children from the housing complex into ongoing leagues or other competitive activities in the surrounding community, that objective, stated in measurable terms, will determine the types of information needed. That objective stated in measurable terms may be:

To increase the number of extracurricular activities that youths participate in and to reduce their incidence of behavioral problems as reported by crime statistics, their parents, and self-reports by the youngsters themselves.

Note that any given objective may have multiple measures. In this example, the types of information you will need to assess attainment of this objective are:

Specifying the information needed will ensure that you do not collect more than you need. It also keeps the cost and time required for the evaluation to a minimum. Given that most Public Housing Agencies (PHAs) have limited resources, you will want to collect only information that is actually needed for the evaluation.

Step 2: Identify the best sources for the information you need

Every data element usually has a range of potential information sources, including:

To decide the best sources for information, ask yourself three questions:

1. What sources are likely to provide the most accurate information?

2. What sources are the least costly or time consuming?

3. Does the information collection pose an undue burden on the sources?

Having accurate data sources for the evaluation is the most important factor. For example, it may be less costly or time consuming to obtain information about services from interviews with program staff, but staff may not provide as accurate information about services as case records could. When you interview staff, you are relying on their memories, but when you review case records, you should be able to obtain information about what actually did happen.

If you choose to use case records to obtain evaluation information, however, you will need to make sure that staff are consistent in recording evaluation information in the records. Sometimes case record reviews can be difficult to use for evaluation purposes because they are incomplete or do not report either participant or service-related information in a consistent manner.

Step 3: Select or develop data collection instruments

There is a variety of types of data collection instruments, including:

Choosing a particular survey methodology will depend on your PHA's budget, the number and type of questions you need to ask, number of residents being surveyed, availability of staff and/or residents to conduct the surveys, and other factors. Types of surveys include mail, telephone, and inperson. Most likely, you will need to develop your own questionnaire to collect information that specifically addresses your particular program objectives. This is not a complicated process. The discussion on the following pages outlines the steps in developing surveys. Each type of data collection instrument has both advantages and disadvantages, which are listed in the chart on the following page.

[[  insert the chart on the Advantages and Disadvantages of Different
Dab Collection Instruments on page 6-5]]

Developing a mail survey. The success of a mail survey depends on obtaining the cooperation of the resident, who must be motivated to complete and return the survey within the time specified. Therefore, the design of any mail survey must convey to the resident the following:

The cover letter. of the simplest ways to convey the importance of the survey is through a cover letter. If possible, the letter should be personalized with the resident's name rather than using a generic "Dear Resident." The cover letter should spell out the purpose and objectives of the survey, who is being asked to respond to the survey and why, and how the resident was selected to receive the questionnaire. The letter should be signed by an official such as the PHA project manager or some other person to show that the study has been given a high profile.

The letter must include a contact person and telephone number in case the resident has questions. In addition, the letter should include a pre-addressed, stamped envelope for returning the survey. An example of a cover letter appears on the following page.

Sample Cover Letter
Violence Prevention Program Resident Survey

[Date]

[Name]
[Address]
[Address]

Dear [Name of Resident]:

I am writing to ask you to be a part of an important study being conducted by the [name of PHA]. The purpose of this study is to learn more about how the [name of program] has affected residents and their families. We would like to get feedback from residents about the violence prevention services that have been provided through the [name of program]. By completing the enclosed questionnaire, you can help us to determine if this program is working and, if necessary, to make changes that will benefit you and other residents.

You were randomly selected to complete this questionnaire because you participated in the [name of program]. Your answers will be kept strictly confidential and will not in any way affect your current or future eligibility for housing at [name of PHA] or any other services you may be receiving. Please take a few moments to complete the enclosed survey and return it in the enclosed pre-addressed, stamped envelope by [date].

Your prompt return of the enclosed questionnaire is extremely important to the success of this evaluation. If you have any questions about the survey, please call [name of contact person] at [telephone number].

Sincerely,

[Name of PHA]

Clear instructions for completing the survey. The instructions for completing a self-administered mail survey are extremely important to ensure that the resident reports accurate data. In self-administered questionnaires the question wording and sequencing can have an effect on data quality. The resident who is completing the survey must first comprehend the questions. To ensure that the resident understands each question and each of the possible answers, you will need to be certain that:

Using existing data collection instruments. Many existing instruments can be used to assess participant outcomes. The Center for the Study & Prevention of Violence (CSPV), University of Colorado at Boulder maintains VIOEVAL, a database that stores reference information about survey instruments. Examples of survey references include the self-reporting delinquency scale, the self-efficacy scale, the violence scale, and the self-esteem scale. Contact information for CSPV is included in the resources section at the end of this manual.

Before you select a standardized assessment instrument for your evaluation, be sure to ask an outside professional for advice, and ask administrators of similar programs about their experiences using the instrument. In addition, you should review each item on the instrument to ensure that the information it asks for is consistent with your expectations about how program participants will change.

If you are unable to find an appropriate existing instrument to assess participant outcome objectives, you will need to develop your own. Developing your own outcome assessment instrument is a complex process and may require the assistance of an expert to ensure its usefulness for your evaluation.

Criteria for selecting or developing an appropriate instrument. Whether you decide to use an existing instrument or to develop your own, the instrument you use should meet the following criteria.

Cultural Relevance

Your forms should be sensitive to issues and concerns of your participant group. You will want to know:

  • Do participants understand the terms used on the forms? Is the language similar to their everyday language? Is the language at a level everyone can understand?
  • Are concepts that are dealt with in the questionnaires and forms familiar to participants?
  • Are questions asked in a thoughtful and nonintrusive manner?
  • Do the questions support the values of the participant group?

Step 4: Establish procedures for collecting information

Once you decide what type of instrument you will use to collect evaluation information, you must establish a set of procedures to ensure that this information will be collected in a consistent and systematic manner. Everyone involved in collecting evaluation information must be trained in these procedures:

Informed Consent

An important part of implementing an evaluation is ensuring that your participants are aware of what you are doing and that they are cooperating with the evaluation voluntarily. People should be allowed their privacy, and therefore they have the right to refuse to give any personal or family information, the right to refuse to answer any questions, and even the right to refuse to be a part of the evaluation at all.

The best way to handle this is to explain the evaluation activities and what will be required of them as part of the evaluation effort. People should be told that their names will not be used and that the information they provide will not be linked to them. Then have them sign an informed consent form that documents that they understand the scope of the evaluation, that they agree (or disagree) to participate, that they understand what is expected of them, and that they understand that they have the right to refuse to give any information and may drop out of the evaluation at any time. If children are involved, then you must get the permission of their parents or guardians before the children participate in the evaluation.

A sample informed consent form appears at the end of this chapter.

It is very useful to develop a manual that describes exactly what is expected in the information collection process. This helps maintain the quality of the evaluation effort, especially when new staff are hired.

Case record extraction form. If you are using program records as a source of information, you will need to develop a case record extraction form to use. This form provides a place for recording all of the information from participant records needed for the evaluation. Similarly, if you plan to interview program staff, you must develop interviews that focus specifically on the evaluation's information needs. Sometimes, in developing these instruments, you or your evaluator may decide that certain types of information would be "interesting" to collect. However, if the information does not relate directly to your program or outcome objectives, you should resist this urge.

Observation forms or checklists. Another popular evaluation technique is the observation form or checklist. These tools are useful to record information about the environment where the program is located, the number of residents participating when the program is visited, and the behaviors of participating residents and service providers or staff.

Step 5: Pretest the information collection instruments and procedures

Before you begin collecting evaluation information, you will need to pretest your instruments and procedures. The pretest will determine whether the instruments and procedures obtain the information that you want, are not excessively burdensome, and are appropriate for your participant population. Use the pretest information to make any necessary revisions before you begin your evaluation. The kinds of information that can be obtained from a pretest include:

You may pretest your instruments with a small number of individuals or program records. You should instruct individuals involved in the pretest to take notes and make comments on the process of using the instruments. These notes and comments may be reviewed to determine whether changes are needed in the instruments or procedures.

You must also review the completed instruments to assess the number of incomplete answers, unlikely answers, comments included in the margins, or other indicators that revisions are necessary. Generally, you will probably need to improve the wording of some questions and instructions for the respondent, as well as delete or add items.

Step 6: Conduct and monitor data collection

After you have completed steps 1 through 5, you are ready to begin collecting evaluation information. This process should be carefully monitored. Monitoring will ensure consistency in the data collection process and that everyone adheres to the time intervals established for collecting information from individual participants.

As part of the monitoring process, you may want to establish a schedule for submitting completed data collection instruments to the evaluation team. This will ensure that instruments are not lost and that confidentiality is maintained. Completed data collection instruments should be treated confidentially; it is a good idea to have completed forms submitted immediately to a member of the evaluation team.

To ensure quality control, information collection staff, particularly if they also are program staff, must be fully educated about the importance of carefully administering and consistently completing evaluation instruments. The first priority of program staff is usually providing services or training to participants, and little effort is given to collecting evaluation information. Encourage your staff to focus on evaluation collection as an important aspect of providing services or training to program participants.

Quality Control Procedures

To make sure that information is being collected appropriately, you must also implement quality control procedures. These will have been stated in your evaluation plan and are noted in chapter 5. Quality control is an essential feature of the data collection phase and must be implemented on an ongoing basis throughout the process. Nothing is more damaging to an evaluation effort than information collection instruments that have been incorrectly or inconsistently administered or that are incomplete.

Once evaluation information is collected, you can begin to analyze it. This process may take place on an ongoing basis or after all data have been collected. The procedures for analyzing and interpreting the evaluation information are discussed in the following chapter.

Sample Informed Consent Form

[Name of PHA] would like you to participate in the evaluation of [program name]. Your participation is important to us and will help us assess the effectiveness of the program. As a resident of [complex name] we will ask you to [complete a questionnaire, answer questions in an interview, or other tasks].

We will keep all of your answers confidential. Your name will never be included in any reports and none of your answers will be linked to you in any way. The information that you provide will be combined with information from everyone else participating in the study.

[If information/data collection includes questions about drug abuse or other illegal activity, the program should make clear its legal obligation to report this information and should let the participant know that confidentiality will be broken in these cases.]

You do not have to participate in the evaluation. Even if you agree to participate now, you may stop participating at any time or refuse to answer any question. Refusing to be part of the evaluation will not affect the services you receive in [program name].

If you have any questions about the study, you may call [name and telephone number of evaluator, housing director, or community advocate].

By signing below, you confirm that this form has been explained to you and that you understand it.

Please check one:

  • AGREE TO PARTICIPATE
  • DO NOT AGREE TO PARTICIPATE

Signed:  _______________________
Date:     _______________________