Collecting Data for Program Evaluation at the Department ...

Collecting Data for Program Evaluation at the Department of State
By Dean OlsenSocial Science AnalystEvaluation and Measurement UnitUS Department of State
A data collectionoverview
Evaluation Design, Question(s), and Logic ModelsData IntegrityDifferences in DataData Collection Methods and AnalysisConsiderations for selecting data collection methodsData Collection for Public Diplomacy Evaluations at the Department of State
Evaluation designs, questions, and logic models
The purpose for the evaluationWho is the audience?How will the evaluation be used?Evaluation question(s)Types of evaluation designsRandomized experimentsQuasi-experimentalNon-experimentalPre/Post-TestNon-experimentalTime SeriesAnalytical framework – Logic models
Program Logic Model
Your Planned Work
Your Intended Results
Resources/ Inputs
Activities
Output
Outcome
Impact
What does success look like?
Evaluation Logic Model
Evaluation Question
Type of Answer/ Evidence Needed
Method for Data Collection
Sampling/ Selection Approach
Data Analysis Method
Method
Source
SurveyInterviewFocus GroupObservationTest/ExamDocument Review
ParticipantsParentsTeachersTrainersStakeholdersProgram documents
Data Integrity
ReliabilityIsthe ability to produce consistent results and indicates whetheran evaluationis likely toproduce thesame results if conducted again.ValidityIsabout coming as close to accurately measuring something as possible. For example, if you want to assess a child’s math skills, the math test you administer to them must be validated such that it is measuring the child’s actual math ability and not somethingelse, such as their test taking skills.Source:Corporation for National and Community Service,http://www.nationalservice.gov/sites/default/files/resource/Description_of_Audio_Data_Collection_for_Program_Evaluation.pdf
Differences in Data
Secondary and Primary DataData some else collectedData you collectDisadvantagesAdvantagesQuantitative DataIsinformation about quantities; that is, information that can be measured and written down withor expressed by numbers.Qualitative DataIs information that cannot be expressed by numbers and is descriptive in nature.
Data Collection Methods and Analysis
QuantitativeMethods – Surveys, Polls, QuestionnairesData someone else has already collected – Agency databases, public opinion polls, program performance statistics/metrics, publications, journals, books, magazines, etc.Analysis – Statistics, graphs, chartsQualitativeMethods – Structured and unstructured interviews, focus groups, open ended questionnaires, observation, document reviewsAnalysis – Content analysis, word clouds
Considerations for selecting data collection methods
Evaluation Question(s)Nature, scope, and purpose of inquiryType of evaluation designAvailability of fundsTime factorPrecision required
Data collection for Public Diplomacy Evaluations at the State Department
Stakeholders and ContactsUnderstanding the culture, audience and environmentScheduling, Relationship Building, and SnowballingThird Party VendorsWho else do you think I should be talking to?
Questions?