Qualitative research & evaluation methods

(3rd ed.). Patton, Michael Quinn. (2002). Thousand Oaks, CA: Sage.

1. The nature of qualitative inquiry (3-29)
*Three kinds of qualitative data: interview, observation, and document

The purposes of research and evaluation are different, but research can be evaluative: "When one examines and judges accomplishments and effectiveness, one is engaged in evaluation. When this examination of effectiveness is conducted systematically and empirically through careful data collection and thoughtful analysis, one is engaged in evaluation research" (p. 10)

*Criteria to judge the quality of the findings:
evaluation standards: utility, feasibility, propriety, accuracy
nontraditional criteria: trustworthiness, diversity of perspectives, clarity of voice, credibility of the inquirer to primary users of the findings.

Patton noted, "In qualitative inquiry, the researcher is the instrument. The credibility of qualitative methods, therefore, hinges to a great extent on the skills, competence, and rigor of the person doing fieldwork--as well as things going on in a person's life that might prove a distraction" (p. 14)

Patton discussed the purposes of open-ended responses and participant observation for evaluative research: "the open-ended responses permit one to understand the world as seen by the respondents. The purpose of gathering responses to open-ended questions is to enable the researcher to understand and capture the points of view of other people without predetermining those points of view through prior selection of questionnaire categories. . . . The task for the qualitative researcher is to provide a framework within which people can respond in a way that represents accurately and thoroughly their points of view about the world..." (p. 21)

In order to overcome the limitations pertaining to what people say, Patton called for participant observation for evaluation research. "The purpose of observational analysis is to take the reader into the setting that was observed" (p. 23).

[In my evaluation research, I utilized participant observation, written out in depth in chapter 3, and made available direct quotes from the observed to provide readers with their points of view.]

2. Strategic themes in qualitative inquiry (37-73)
* Patton presented some analysis strategies: unique case orientation (capturing the details of the individual case being studied), inductive analysis and creative synthesis, holistic perspective (focusing on complex interdependence and providing it as a complex system), context sensitivity (attentive to a social, historical, and temporal context), voice/reflexivity (reflective on my own voice)

With regard to reflexivity, Patton wrote: "Reflexivity has entered the qualitative lexicon as a way of emphasizing the importance of self-awareness, political/cultural consciousness, and ownership of one's perspective" (p. 64) He suggested using the first-person, active voice in this light.

Patton seems to be skeptical about paradigmatic division. He argued that a paradigmatic strategy indeed can be a hindrance for evaluation research. He argued:

"Such prejudices and paradigmatic blinders limit methodological choices, flexibility, and creativity. Adherence to a methodological paradigm can lock researchers into unconscious patterns of perception and behavior that disguise the biased, predetermined nature of their methods "decision." " (p. 71)

Further, he argued that "the qualitative-quantitative debate oversimplified and often confused methodological and philosophical issues" (p. 71), as in reality, there are the cases that notably consistent methods and philosophy in a certain paradigm have proven problematic.

3. Variety in qualitative inquiry: Theoretical orientations (75-142)
Patton introduces 16 qualitative inquiries, including phenomenology and grounded theory, which I think are the most relevant to my dissertation research. So I will take a close look at them.

Phenomenology
Foundational question: what is the meaning, structure, and essence of the lived experience of this phenomenon for this person or group of people?

Phenomenon can be emotion, a relationship, a program, or a culture. The focus is on describing the essence of people's experience in a certain phenomenon and developing a worldview. It usually involves participant observation and in-depth interviews. For example, the essence of being a participant in a particular program can be a kind of phenomenological study. [My chapter 4 is a definitely phenomenological study in that sense.]

Grounded theory
Foundational question: What theory emerges from systematic comparative analysis and is grounded in fieldwork so as to explain what has been and is observed?

Popular in social sciences and methodology with emphasis on generating, instead of testing, theory. Patton asserts that all the approaches are grounded in the empirical world, but the conceptualizations of what to ask vary from one inquiry to another. "While the phrase "ground theory" is often used as a general reference to inductive, qualitative analysis, as an identifiable approach to qualitative inquiry, it consists of quite specific methods and systematic procedure" (p. 126). Thus, he argued:

"As a matter of philosophical distinctness, then, grounded theory is best understood as fundamentally realist and objectivist in orientation, emphasizing disciplined and procedural ways of getting the researcher's biases out of the way but adding healthy doses of creativity to the analytic process." (p. 128-129).

[To the extent that I compared the experience of the research participants with that of others in other programs in Chapter 4, it has a systematic comparison emphasized in grounded theory, and yet, this analysis is not so much grounded theory as phenomenology. On the other hand, Chapter 5 is more linked to grounded theory as I intend to provide some methodological tools to researchers. But I also doubt that it can be called grounded theory as I do not develop theory out of the fieldwork. ]

4. Particularly appropriate qualitative applications (143-205)
Patton holds pragmatism in his approach to methodology: "Pragmatic and utilitarian frameworks can guide qualitative inquiry on their practical and applied underpinnings without having to be attached to or derived from a theoretical tradition." (p. 145)

Patton suggests various evaluation models:

Goal-free evaluation
"doing fieldwork and gathering data on a broad array of actual effects or outcomes, then comparing the observed outcomes with the actual needs of program participants" (p. 169). In general, it purposefully suspends the researcher's judgment.

[Then, I can take up goal-free evaluation for the embedded study. The participants want to find out ways to help increase retention rate, in the end, they suggested a way. There is no way to check the actual effects, but from the participants' perspective, the research brought about some solutions they looked for. ]

Transaction models: responsive and illuminative evaluation

The transaction model of evaluation that Ernie House (1978) called concentrates on case studies and "places prime emphasis on perception and knowing as a transactional process between researcher and research participant" (p. 172). Its variations include illuminative evaluation, developed in educational evaluation, and Stake's (1975) responsive approach to evaluation, which requires to have "face-to-face contact with people in the program and learn about diverse stakeholder's perspectives, experiences and concerns. This responsive evaluation has been integrated into Guba and Lincoln's (1989) "responsive constructivist evaluation" known as the fourth generation evaluation, which is based on naturalistic inquiry.
Connoisseurship studies
It places the evaluator's perceptions and expertise at the center of the evaluation process, "drawing heavily on his or her own judgments about what constitutes excellence" (172).

Interactive and participatory applications:

Placing a greater emphasis on human interaction in designing evaluation strategies, Patton wrote:

Qualitative methods may also be perceived as more personal because of their inductive nature. This means that, again, rather than imposing on people or a program some predetermined model or hypotheses, the results unfold in a way that takes into account idiosyncrasies, uniqueness, and complex dynamics. Finally, qualitative methods may be perceived as more humanistic and personal simply by avoiding numbers. (p. 176)

Action research, action learning, reflective practice
i.e. adult educators interviewing students as case studies, then through cross-case analysis they deepened their sensitivity to new immigrant students and developed their professional skills.
Participatory research and evaluation
Film: Awakening (1990)--a real story of participatory research in a mental hospital

Patton provided an overview of a growing popularity of participatory research:
"Interest in participatory research has exploded in recent years, especially as an element of larger community change efforts (Stoecker 1999). The principal researcher trains the coresearchers to observe, interview, reflect, and/or keep careful records or diaries. Those involved come together periodically to share in the data analysis process. The purpose of such shared inquiry is typically to elucidate and improve the nature of practice in some arena of action" (183).

In particular, participatory evaluation emphasizes "working with people" (183), thus, Patton wrote: "The process is facilitated by the researcher, but is controlled by the people in the program or community. They undertake a formal, reflective process for their own development and empowerment" (183).

Patton noted that participatory evaluation has been used with great success by a number of NGOs and private voluntary organization in the Third World.

Patton also drew attention to various connotations of participatory inquiry, saying that there is no definitive definition for participatory evaluation. Patton, in particular, emphasized teaching inquiry logic and skills as the purpose of participatory evaluation, as "the process of participatory self-evaluation, in and of itself, provided useful learning experience for participants" (184).

[Here Patton suggests participatory evaluation/inquiry as a pedagogical form.]

Finally, he noted, "these participatory approaches often employ qualitative methods because those methods are understandable, teachable, and usable by people without extensive research training." (185)

[Patton regards participatory paradigm from the perspective of pragmatism and links qualitative methods as strong candidates for that paradigmatic inquiry.]

8. Qualitative analysis and interpretation (431-534)

Patton discussed about summative vs. formative evaluation: Summative evaluations are usually accompanied by a full report with data, analysis, and recommendations, whereas formative evaluation is conducted for program improvement and may not generate a written report. Findings are often presented orally to the staff or funders. Patton wrote:
"The methods are qualitative, the purpose is practical, and the analysis is done throughout fieldwork; no written report is expected beyond a final outline of observations and implications. Academic theory takes second place to understanding the program's theory of action as actually practices and implemented" (435)

[The VT research can be viewed as formative evaluation to give insights into the OP program, which is not academic theory oriented and rather concentrates on providing critical insights from the insiders' perspective.]

Theory-based analysis approaches: Patton discusses phenomenological analysis and ground theory analysis in great detail, along with qualitative comparative analysis and analytic induction.

Analytic induction seems to be the most relevant to my approach. It originally emphasized discovering universal causal generalizations, but gradually moved away to qualitative inquiry and comparative case analysis including examination of preconceived hypotheses. In this approach, "researchers actively seek to disconfirm emerging hypotheses through negative case analysis, that is, analysis of cases that hold promise for disconfirming emerging hypotheses and that add variability to the sample. In this way, the originators of the method sought to examine enough cases to assure the development of universal hypothesis." (Jane Gilgun 1995:268-269, We shared something special: the moral discourse of incest perpetrators, Journal of marriage and the family, 57, 265-81). Analytic induction consists of re-examinations.

9. Enhancing the quality and credibility of qualitative analysis (541-597)

In traditional scientific research, among the criteria for judging the credibility are objectivity, validity, systematic rigor of fieldwork procedures, triangulation, etc. The common aim is "to use qualitative methods to describe and explain phenomena as accurately and completely as possible so that their descriptions and explanations correspond as closely as possible to the way the world is and actually operates" (p. 546)

Social construction and constructivist criteria: Lincoln and Guba (1986) proposed credibility (for internal validity), transferability, dependability and confirmability. In combination, they viewed these criteria as addressing trustworthiness (in lieu of rigor). Fundamentally, they are "suspicious of causal explanations and empirical generalizations applied to complex human interactions and cultural system" (546)

Critical change criteria: another stream of qualitative inquiry aims at social and political change. Thus consequential validity matters in addressing the credibility of the inquiry.

[For the analysis of my dissertation study, what criteria should be applied? The set of criteria should stem from the hypotheses of participatory (video) research. What does participatory research claim to be able to do? Does my study prove or disprove it? How do I make judgment? --My judgment should be grounded in constructivist perspective, the fourth generation evaluation. Then, what I need to verify is the claims of participatory research. ]