Research Methods in Crime and Justice
Chapter 13Evaluation Research Methods
Evaluation Research Methods
Evaluation research has become a critical part of criminal justice administration.The analysis of a program’s efficiency and effectiveness is an important part of a criminal justice practitioner’s job.Evaluation researchconsiders the effectiveness, efficiency, and the unintended consequences of a program, process or policy.
Evaluation Research Basics
Like the qualitative and non-reactive research traditions, evaluation research is not a single methodology.Instead,evaluation researchis any applied research method that considers the effectiveness, efficiency, and the unintended consequences of a program, process or policy.
Types of Evaluation Research
There are two commonly used methods in evaluation research.Acost/benefit analysisrefers to a technique that compares the costs and benefits of an existing or future program, process or policy.Themultiple methods approachinvolves the use of multiple data gathering techniques to evaluate a program, process or policy.
The Benefits of Evaluation Research
Evaluation research is effective at;Demonstrating the effectiveness or efficiency of an existing program, process orpolicy.Monitoring the effectiveness or efficiency of new programs, processes orpolicies,Evaluating the unintended consequences of a program, process orpolicy, and/orInforming the debate on controversial programs, processes and policies.
The Limitations of Evaluation Research
Evaluation research;Can evaluate but cannotdetermine the underlying outcome or objective of a program, process or policy.Is pointlesswhen a program, process or policy is mandatory or legally required.
The Evaluation Research Process
ACase Study in EvaluationResearch (The Police Protective Custody Process)In most states, there is a process by which the police can remove abused and/or neglected children from their homes and place them in temporary housing.This process is expensive and in most cases results in considerable physical and psychological disruption.Withrow (2001) and Withrow and Bolin (2003) were asked to conduct an evaluation of the Police Protective Custody process in Kansas.
The Evaluation Research Process
Askinga Research Question in EvaluationResearchThe research question in evaluation research is whether the program or policy is performing as it was intended to.Is it effective?Is it efficient?Are their unintended consequences or opportunities?
The Evaluation Research Process
Askinga Research Question in EvaluationResearch – cont’dThe PCC evaluation had the following research objectives (questions).To document the PPCprocess.To determine what factors affect a child’s length ofstay.To make recommendations for reducing the number of PPCreferrals.To make recommendations for reducing a child’s lengthof stay(Withrow, 2001 p.i).
The Evaluation Research Process
Conductinga Literature Review in EvaluationResearchThe primary literature in evaluation research is other evaluation studies on the same topic.Many of these studies are in the possession of criminal justice and allied agencies.
The Evaluation Research Process
Conductinga Literature Review in EvaluationResearch – cont’dThere were actually two versions of the PPC report.The scholarly journal included literature from scholarly sources, most notably research on the effect of separation during the PPC process.The version written for the Wichita Children’s Home included only a nominal amount of scholarly literature because it was intended to be consumed by practitioners.
The Evaluation Research Process
Refiningthe Research Question in EvaluationResearchMost evaluation research projects do not include formal hypotheses.Instead they ask;Did the policy, process or program have its intended effect?Was the policy, process or program implemented efficiently?Did the policy, process or program result in unintended consequences?
The Evaluation Research Process
Refiningthe Research Question in EvaluationResearch – cont’dExplicit hypotheses were not developed for the PPC study.Instead Withrow and Bolin made several predictions about the relationships between the variables used in the study.Thesepredictions enabledthento at least begin the analysis.
The Evaluation Research Process
DefiningConcepts and Creating Measures in EvaluationResearchIn evaluation research,conceptualization and operationalization involve developing measures for policyor program success, efficiency and/or effectiveness.Evaluation researchers must be mindful that stakeholders may have a different view on how to measure a program’ssuccess.
The Evaluation Research Process
DefiningConcepts and Creating Measures in EvaluationResearch – cont’dMany of the concepts in the PPC study (e.g. foster care, respite, etc.) were defined by law.Most of the other variables (e.g. parents drug use) were dichotomous (i.e. yes or no).This is common in criminal justice research.
The Evaluation Research Process
DefiningConcepts and Creating Measures in EvaluationResearch – cont’dThere were competing perspectives on what constitutes the success of the PPC process.Policeofficers associated success with the ability to refer a child to the WCH without restriction.Socialworkers defined success in terms of whether they were able to find a suitable long term living situation for the child.Prosecutorsdefined success in terms of their ability to convince a judge to approve a child in need of care petition.Representativesfrom social services agencies who were concerned about expenditures were defined success as the ability to limit services to children who did not need them.
The Evaluation Research Process
DefiningConcepts and Creating Measures in EvaluationResearch – cont’dTo overcome the inevitable conflict associated with alternative views of ‘success’ the researchers chose to focus on producing accurate descriptions of how the system worked.This, they believed, would enable policy makers to make more informed decisions
The Evaluation Research Process
Designing a Method in EvaluationResearchThe actual method or methods used by evaluation researchers depends on the type of research question asked.The use of multiple methods has become particularly popular in evaluation research.
The Evaluation Research Process
Designing a Method in EvaluationResearch – cont’dBecause the PPC had four objectives, no single research method would have sufficed.The researchers used a multiple methods in our evaluation.focus groupSecondary analysisArchival analysisInterviews
The Evaluation Research Process
Designing a Method in EvaluationResearch – cont’dFocus groups with representatives from the affected agencies allowed us to;Document the PPC process, andIdentify redundancies in the process.Secondary analysis of data on admitted children allowed us to identify the factors that affected;A child’s length of stay, andWhether a child would be admitted to state custody.
The Evaluation Research Process
Collecting Data in Evaluation ResearchThe process by which evaluation researchers collect data depends on the specific methods they use.There are a two important considerations specific to evaluation research.Gaining full access to data during an evaluation is sometimes difficult.Criminal justice data is notoriously incomplete, inaccurate, and inconsistent.
The Evaluation Research Process
Collecting Data in Evaluation Research – cont’dIn the PPC study the focus group required considerable coordination.The practitioners involved in the PPC process are busy.So finding a single day when they all could meet was a rather daunting task.The data set for the secondary analysis included information on 6,600 children.This data set had to be ‘cleaned’ to facilitate the analysis.In order to avoid the possibility of violating a child’s privacy, the data set did not contain (at the researcher’s request) names and other personal identifiers.
The Evaluation Research Process
Analyzing and Interpreting Data in Evaluation ResearchThe analyzes and interprets of the evaluation’s results depends on;the nature of the research methods used,the data produced by the research methods, andthe overall objectives of the study.
The Evaluation Research Process
Analyzing and Interpreting Data in Evaluation Research – cont’dThe PPC study produced (for the first time) a flow chart of the PPC process that enabled policy makers to ‘see’ system redundancies.The secondary analysis found that children were more likely to be taken from their parents if;they had previously received state social or welfare services,they had parents who used illegal drugs,they had previously been abused, and/orthey had limited parental supervision.
The Evaluation Research Process
Analyzing and Interpreting Data in Evaluation Research – cont’dThe PPC study made two primary recommendations.The PPC process should be managed by a team of professionals.Police officers should only use the PPC process for children who;are being or have been abused or neglected,have received any type of social or welfare services,Have limited parental supervision, andAre younger than 16 years of age.
The Evaluation Research Process
Communicating the Findings from Evaluation ResearchMost evaluation studies are published ‘in-house’.Prior to the release of the report it is prudent to allow affected individuals to review the report for accuracy.Often evaluation researchers are asked to present their findings in multiple formats to different audiences.On occasion the results of an evaluation report will cause a public controversy.
The Evaluation Research Process
Communicating the Findings from Evaluation Research - cont’dThe initial report (for agency consumption) was brief to accommodate busy practitioners.Another version of the report was published in a scholarly journal.In both situations the reports clearly identified the limitations of the study.
Getting to the Point
Evaluation researchis designed to determine whether a program, process or policy is achieving its intended outcome and/or resulting in any unintended consequences.Evaluation research is becoming increasingly critical to criminal justice practice.
Getting to the Point
Evaluation research typically involves either;acost/benefit analysis, in which the costs associated with a policy or program are weighed against the benefits, oramultiple methods approach, in which various data collection methods are used to evaluate a policy or program.
Getting to the Point
Evaluation research is effective at demonstrating and monitoring the effectiveness or efficiency of new and existing programs, processes or policies.It can also identify the unintended consequences of and provide objective information about controversial programs, processes and policies.
Getting to the Point
Evaluation research is not effective;without agreed upon objectives, orwhen a program is mandated by law, policy or higher authority.
Getting to the Point
Evaluation research can be exploratory, descriptive or explanatory depending on the researcher’s objective.It is almost always applied because the researcher usually intends for the results to influence policies and programs.
Getting to the Point
Although evaluation research reports can be found in scholarly journals, the overwhelming majority of them are in the possession of;various government agencies,privately sponsored research organizations, andindividual criminal justice agencies.Such reports are typically available through the internet or can be made available through a direct request to the individual agency in possession of the report.
Getting to the Point
Evaluation research projects may begin with general research questions or explicit hypotheses depending on the nature of the project.In either case, the primary purpose is to investigate whether a program, process or policy is having its intended results.
Getting to the Point
In evaluation research, researchers typically start by defining and operationalizing policy or program effectiveness.Because criminal justice agencies are publically funded, the researchers who evaluate criminal justice programs should be mindful of the various political objectives when developing measures of program effectiveness.
Getting to the Point
The actual method or methods used by evaluation researchers to collect data depends on the type of research question(s) asked.The use of multiple methods has become particularly popular in evaluation research.
Getting to the Point
Gaining access to data during evaluation research is sometimes difficult.In many cases, individuals who make the policy or work in the program are less than willing to give information, lest the policy or program be found to be unsuccessful or inefficient.
Getting to the Point
Criminal justice data is also notoriously incomplete, inaccurate and inconsistent.As a result, evaluation researchers should expect that they will need to spend considerable time preparing the data for analysis.
Getting to the Point
The interpretation process in evaluation research is not much different than that required in any research project.But evaluation researchers should bear in mind that criminal justice data are notoriously incomplete and inconsistent.This limitation should always been acknowledged in the interpretation of the results.
Getting to the Point
Evaluation researchers are often asked to present their findings in multiple formats and to multiple audiences.When the media get involved with disseminating the results of the research, evaluation researchers should be mindful of the complicated dynamics of public opinion and messaging.
Research Methods in Crime and Justice
Chapter 13Evaluation Research Methods
0
Embed
Upload