The schedule moving forward…

The schedule moving forward…
TodayEvaluation ResearchRemaining time = start on quantitative data analysisThursdayMethodology section dueQuantitative analysis continued (start Exercise 10)Next TuesdayExercise 10 DueMaahsreturns Exercise #9Research proposal (everything is now online) is due on Dec 21st(Monday)
Evaluation Research
PURPOSErather than specific methodMuch more popular in last 20 yearsA form of “applied” researchResults, though designed to impact decision-making, often have no impact
Types of Evaluation Research
Needs assessmentCost-benefit studyMonitoring studyProgram evaluation
Doing evaluation (program) research
Research questions / measurement as Job #1Political context how do measureOperationalize “outcome” of interest“Response” variableNEED AGREEMENT: what is the program trying to do?Operationalize “processes”Intermediate objectivesContext of”
Research designs for evaluation
ExperimentalProblems, ethics (placebo)The “black box” issueQuasi-experimental designsTime Series (multiple time series)Qualitative evaluationsLow birth weight study
Evaluation Research ISSUES
Evaluation research as “MESSY”Administrative control, context of “real life”Importance of looking at “black box”EthicsThe intervention itself can raise issuesControl group membersViolating Ethics 101 (Tuskegee)Evaluation impacts people’s lives
The (non) use of findings
Purely rational/scientific world vs. the one we inhabitNixon panel on pornScared Straight3 Strikes and You’re Out legislationStudies gone wrong…Fire the evaluator (or dismiss as pointy headed idiot) and keep the programWhy not trust/use the results?Scientist/practitioner gap“True believers”Vested interests
Bivariate Analysis
Backed up with alittle inferentialstatistics……yeah baby!
Review
Descriptive statisticsPurpose?Types?These are “univariate” statisticsExplanatory researchAttempt to demonstrate cause-effectNecessary Criteria?
Demonstrating Associations
Bivariate (2 variables) analysisThere are a number of ways to do thisThe method you choose depends largely on characteristics of the two variablesLevels of Measurement?
Contingency Tables (cross tabs)
Some find these very intuitive…others struggleIt is very easy to misinterpret these crittersConvention: the independent variable is on the top of the table (dictates columns) and the dependent variable is on the side (dictates rows).What is in the individual “cells?”Frequencies (number of cases that fit criteria)Convert to Percentages: a way to standardize cells and make relationships more apparent
Example
A survey of 10,000 U.S. residentsResearch question: is one’s political view related to attitudes towards police?What is the DV and IV?In constructing a table, what goes where?
An Example
The Percentages of Interest
Inferential Statistics
Researchers are typically not interested in whether there is a relationship in thesampleWant to know about the populationWhy might there be a difference between what you find in the sample and what actually exists in the pop?Even with a probability sample, there is always sampling errorWithout a probability sample = error + bias
Inferential Statistics II
Cannot assume that the relationship in your sample is true in populationBUT:probability theoryallows us to estimate:“The likelihood of obtaining a particular finding if in the population, there was no relationship” OR“The likelihood of obtaining a particular finding assuming the null hypothesis”
Test Statistics and Significance Tests
What does “statistically significant” mean?Not due to sampling errorWhat do you need to do to test for statistical significance?A “test statistic”An indicator of how different the sample is from “no relationship”The level of error that is acceptable (.05, .01)Degrees of Freedom
The Test Statistic for Contingency Tables
Chi Square, orχ2CalculationObserved frequenciesExpected frequenciesCan get confusing calculating these critters by handIntuitive: how different are the observed cell frequencies from the expected (under null hypothesis) cell frequenciesDegrees of Freedom:(# of Rows -1) (# of Columns -1)
The Chi-Square Sampling Distribution(Assuming Null is True)
Conventional Significance Testing
Calculate the test statisticSet your “Type II” error levelThe risk of being wrong that you are willing to live withFind the “critical region” within the distribution for your sample statisticHow far out on the curve does your test statistic have to get before you can reject the null hypothesisDecide whether to reject, or fail to reject the null hypothesis
Significance Testing in SPSS
Within your crosstabs, select “Chi-Square”SPSS gives you the χ2 valueWhat you would calculate based on the observed and expected cell frequenciesSPSS also gives you “p”The exact probability of obtaining this χ2, if indeed the null hypothesis was correctIn newer versions of SPSS, the p values are listed under “sig” or “significance” or something similar
Review of Contingency Tables
Level of MeasurementBoth IV and DV are Nominal or Ordinal (Categorical data)Constructing a tableIV on top (columns), DV on bottom (rows)With this format, select “column percentages”
Review of Inferential Statistics
Purpose of inferential statisticsFigure out the odds that a finding from sample is due to “chance” or “sampling error”ProcessAssume null hypothesis is correctCalculate the odds of obtaining your particular finding under this assumptionIf the odds are very low, you may be suspicious that the null hypothesis is incorrectAt some point (research sets level), you say that the odds are so low that you are going to go ahead and reject the null hypothesis
Review of Chi-Square
A measure of how different observed (from your sample data) cell frequencies are from what would be expected under the null hypothesis (e.g., no relationship)The Chi-square distribution changes shape with different degrees of freedomThis is because, by definition, the more cells you have, the higher χ2 getsdf = (R-1)(C-1)
Interpreting Chi-Square
Chi-square has no intuitive meaning, it can range from zero to very largeThe real interest is the “p value” associated with the calculated chi-square valueThis is the exact probability of obtaining the chi-square if in the population there was no relationshipIn other words, the exact probability of finding that chi-square, under the null hypothesis