Bureau of Justice Assistance. Urban Street Gang Enforcement. Washington, DC: Prepared
U.S. Department of Justice, Bureau of Justice Assistance by the Institute for Law and Justice, Inc.; 1997. pp. 97-98 and 102-103.
View entire document
A process evaluation documents and analyzes the early development and actual implementation of the strategy or program, assessing whether strategies were implemented as planned and whether expected output was actually produced. Examples of output include:
Other output measures are listed on a quarterly reporting form completed by departments funded under BJA's Urban Street Gang Program. A copy of this form is included in Exhibit 4 which is not attached.
Detailed information about the program as it was actually implemented is invaluable for determining what worked and what did not. A thorough process evaluation should include the following elements:
Describing the Program Environment
Before a program's effectiveness can be judged, it is important to understand its operating environment. The program analysis stage (see Chapter 3) produces a detailed description of the environment at the time of planning. Significant changes in the environment should be documented during program implementation to help determine whether similar results may be expected in other communities or whether the results are site specific. That is, if the environment is unique, then the results achieved in that setting may not be replicable elsewhere.
Describing the Process
Good ideas do not always yield good results. Therefore, to understand the tasks to be performed and the scope of effort, a clear description of the implementation process is required. This step also will aid in replicating the effort in other environments. Describing the implementation process would involve elements such as:
Describing and Measuring Program Operations
The process evaluation must describe the way the gang suppression strategy worked, or failed to work, using quantitative and qualitative data. Questions to consider in assessing the process evaluation include the following:
Identifying Intervening Events
Local programs operate in continually changing environments. To identify the effects of various intervening external factors on program outcomes, an intervening variable analysis is employed. The intervening variables that may affect a program are numerous, complex, and varied and may or may not be able to be controlled or eliminated. These variables are discussed later in this chapter in the Impact Evaluation section. To assess outcomes correctly, these variables will need to be identified, interpreted, and described.
Collecting Process Data
The process evaluation should begin during the program planning phase and continue
through program implementation. Two main categories of data should be collected. The first
data source includes official police, city/county, census, and other data gathered during
the initial program planning and analysis phase. These data will help determine whether
program outcomes may be expected in similar jurisdictions.
The second data source includes interviews with and observations of participants. Observations should begin with early program development and continue throughout program implementation. Major planning activities as well as enforcement activities are of interest. Observers will answer questions such as the following:
Interviews with key participants also should be conducted to complement information attained from observation. These interviews should reveal the reactions of patrol officers, investigators, and others to the program's development, noting their observations about difficulties encountered and associated explanations, as well as suggested solutions. An open-ended format for observations is suggested so that observers are not limited in their focus. However, the protocol for interviews should be more structured to ensure consistency and validity.