Proposed program
Lecture Overview • General considerations in qualitative design • Types of qualitative evaluation • A case example • What to include in your proposal
Understanding Qualitative Evaluation • In quantitative research, the data are in the form of numbers; in qualitative research, the data are in the form of words • The approaches entail different kinds of questions • You can combine approaches • Quantitative methods are assumed to be more objective: Is this necessarily true? • Text states that many audiences prefer quantitative: Is this always the case?
Understanding Qualitative Evaluation • In quantitative research, the data are in the form of numbers; in qualitative research, the data are in the form of words • The approaches entail different kinds of questions • You can combine approaches • Quantitative methods are assumed to be more objective: Is this necessarily true? • Text states that many audiences prefer quantitative: Is this always the case?
Types of Qualitative Approaches • Interviews, focus groups, observation • Focus on meaning • The importance of context • Can complement quantitative approaches • For your project, you can design a mixed-methods evaluation study: – One of the components of the design can complement the other (e.g., a quantitative study in which you interview a subset of participants) – A mixed-methods approach should be logical. Ask: Is this an appropriate design for my research question?
Case Example
Project Enterprise: A Community Based Program • Serves clients living below the poverty line (mostly clients of color) • Peer-group, micro-lending model: – Based on “international development” model – Peer-group training, small micro-credit loans, shared accountability for loan payback • Highly successful – Exceptionally high loan payback rate – Successful businesses have impacted the local community Research Question: What Makes This Program so Effective? • Meetings with the staff • Interview questions developed collaboratively • Recruitment at regular meetings • Individual on-site appointments • Audiotaped interviews • Qualitative analysis of findings • Report findings to staff
Key Technique in Interviewing: Probing • The goal of probing is to use clear questions to understand meaning from the perspective of the interviewee • The interviewer does not suggest answers to the interviewee • Interviewees answer in their own words and at their own pace • The goal is to use open-ended questions that aid interviewees in describing the meaning they attach to the event or experience Content Analysis of Interview Responses • The goal of content analysis is to derive a list of themes to capture interview responses • Each response must be categorized under one—and only one—theme • In our example, the role of PE in life goals was coded using three thematic categories: – Leadership – Support – Direction • Those categories were used to write the final report to the organization
What to Include in Your Proposal
In Your Proposal… If you choose a qualitative approach: • Be sure to outline the exact method(s) you’ll use • If you’re conducting interviews, remember to include only the number of questions you need to capture the information you want
Lecture Overview • Analysis of findings: general considerations • Types of analyses • Dissemination of findings • Challenges in disseminating findings (poor use of evaluation results) • What to include in your proposal
The Purpose of Analysis in Evaluation In program evaluation, the goal in analyzing data is primarily to translate data into a meaningful message: • Meaningful to the evaluator: Do the findings contribute to the research literature? • Meaningful to the setting: Can the findings guide higher-level planning about programming? • Meaningful to others who design programs: Can the findings inform the development of new programs? • Meaningful to clients/consumers: Do the findings demonstrate what worked about the program (and what didn’t work)?
Common Types of Analyses • Did a positive outcome occur for participants? • Did control variable(s) influence change (e.g., socioeconomic status, education)? • Did certain groups benefit more than others (e.g., gender)? • Did participants who attended more sessions show more positive change than those who attended fewer sessions? • Did participants believe that they benefited from the program? • Was the program implemented as planned (process)?
Common Types of Analyses (cont.) • How did participants’ outcomes compare to nonparticipants (control/comparison group)? – Check whether groups that were not randomly assigned differed in some way (e.g., motivation, pretest scores on outcome indicator) and control for those variables – Compare amount of change in intervention group and control/comparison group • More detailed analysis of outcomes (to rule out rival explanations of positive findings) – Groups were different to begin with – Attrition (maybe riskier cases dropped out of intervention; more stable cases dropped out of control/comparison group)
Dissemination: Why Disseminate Findings? • To inform future implementations of program • To justify efforts (e.g., Was it worth the time/money?) • To impact policy and programming decisions • To contribute to the scholarly knowledge base • To help other institutions grappling with similar issues • To establish your reputation as an expert
Who Do You Disseminate Your Findings To? • Clinical staff and administrators • Clients • Parents/families • Community leaders • Similar settings • Funding source • Policy makers
Types of Dissemination: Written Reports • Cater report to audience (may even write different versions) • Executive summary—outlines research questions; provides brief description of program; highlights main findings, implications, and recommendations • Journal article: reporting to an academic audience (select journal relevant to audience)
Other Types of Dissemination • Speeches to organizations of practitioners • Conference presentations • Interactions with peers in institutional/organizational settings • Professional development workshops • Mass media coverage
Poor Use of Evaluation Findings • Organizational reluctance to change • New practices may conflict with old beliefs • Strong commitment to program in its current form • Unfavorable evaluation can lower morale and reduce administrative commitment and funding • Policy makers may tolerate ineffective program if outside funders want to continue funding it
In All Proposals… • Describe how you will analyze your data – How will you make sense of your findings? –Assume optimally effective results • Describe your dissemination plan –What format will it take? –Who will you disseminate to?
