Qualitative research & evaluation methods

(3rd ed.). Patton, Michael Quinn. (2002). Thousand Oaks, CA: Sage.

1. The nature of qualitative inquiry (3-29)
*Three kinds of qualitative data: interview, observation, and document

The purposes of research and evaluation are different, but research can be evaluative: "When one examines and judges accomplishments and effectiveness, one is engaged in evaluation. When this examination of effectiveness is conducted systematically and empirically through careful data collection and thoughtful analysis, one is engaged in evaluation research" (p. 10)

*Criteria to judge the quality of the findings:
evaluation standards: utility, feasibility, propriety, accuracy
nontraditional criteria: trustworthiness, diversity of perspectives, clarity of voice, credibility of the inquirer to primary users of the findings.

Patton noted, "In qualitative inquiry, the researcher is the instrument. The credibility of qualitative methods, therefore, hinges to a great extent on the skills, competence, and rigor of the person doing fieldwork--as well as things going on in a person's life that might prove a distraction" (p. 14)

Patton discussed the purposes of open-ended responses and participant observation for evaluative research: "the open-ended responses permit one to understand the world as seen by the respondents. The purpose of gathering responses to open-ended questions is to enable the researcher to understand and capture the points of view of other people without predetermining those points of view through prior selection of questionnaire categories. . . . The task for the qualitative researcher is to provide a framework within which people can respond in a way that represents accurately and thoroughly their points of view about the world..." (p. 21)

In order to overcome the limitations pertaining to what people say, Patton called for participant observation for evaluation research. "The purpose of observational analysis is to take the reader into the setting that was observed" (p. 23).

[In my evaluation research, I utilized participant observation, written out in depth in chapter 3, and made available direct quotes from the observed to provide readers with their points of view.]

2. Strategic themes in qualitative inquiry (37-73)
* Patton presented some analysis strategies: unique case orientation (capturing the details of the individual case being studied), inductive analysis and creative synthesis, holistic perspective (focusing on complex interdependence and providing it as a complex system), context sensitivity (attentive to a social, historical, and temporal context), voice/reflexivity (reflective on my own voice)

With regard to reflexivity, Patton wrote: "Reflexivity has entered the qualitative lexicon as a way of emphasizing the importance of self-awareness, political/cultural consciousness, and ownership of one's perspective" (p. 64) He suggested using the first-person, active voice in this light.

Patton seems to be skeptical about paradigmatic division. He argued that a paradigmatic strategy indeed can be a hindrance for evaluation research. He argued:

"Such prejudices and paradigmatic blinders limit methodological choices, flexibility, and creativity. Adherence to a methodological paradigm can lock researchers into unconscious patterns of perception and behavior that disguise the biased, predetermined nature of their methods "decision." " (p. 71)

Further, he argued that "the qualitative-quantitative debate oversimplified and often confused methodological and philosophical issues" (p. 71), as in reality, there are the cases that notably consistent methods and philosophy in a certain paradigm have proven problematic.

3. Variety in qualitative inquiry: Theoretical orientations (75-142)
Patton introduces 16 qualitative inquiries, including phenomenology and grounded theory, which I think are the most relevant to my dissertation research. So I will take a close look at them.

Phenomenology
Foundational question: what is the meaning, structure, and essence of the lived experience of this phenomenon for this person or group of people?

Phenomenon can be emotion, a relationship, a program, or a culture. The focus is on describing the essence of people's experience in a certain phenomenon and developing a worldview. It usually involves participant observation and in-depth interviews. For example, the essence of being a participant in a particular program can be a kind of phenomenological study. [My chapter 4 is a definitely phenomenological study in that sense.]

Grounded theory
Foundational question: What theory emerges from systematic comparative analysis and is grounded in fieldwork so as to explain what has been and is observed?

Popular in social sciences and methodology with emphasis on generating, instead of testing, theory. Patton asserts that all the approaches are grounded in the empirical world, but the conceptualizations of what to ask vary from one inquiry to another. "While the phrase "ground theory" is often used as a general reference to inductive, qualitative analysis, as an identifiable approach to qualitative inquiry, it consists of quite specific methods and systematic procedure" (p. 126). Thus, he argued:

"As a matter of philosophical distinctness, then, grounded theory is best understood as fundamentally realist and objectivist in orientation, emphasizing disciplined and procedural ways of getting the researcher's biases out of the way but adding healthy doses of creativity to the analytic process." (p. 128-129).

[To the extent that I compared the experience of the research participants with that of others in other programs in Chapter 4, it has a systematic comparison emphasized in grounded theory, and yet, this analysis is not so much grounded theory as phenomenology. On the other hand, Chapter 5 is more linked to grounded theory as I intend to provide some methodological tools to researchers. But I also doubt that it can be called grounded theory as I do not develop theory out of the fieldwork. ]

4. Particularly appropriate qualitative applications (143-205)
Patton holds pragmatism in his approach to methodology: "Pragmatic and utilitarian frameworks can guide qualitative inquiry on their practical and applied underpinnings without having to be attached to or derived from a theoretical tradition." (p. 145)

Patton suggests various evaluation models:

Goal-free evaluation
"doing fieldwork and gathering data on a broad array of actual effects or outcomes, then comparing the observed outcomes with the actual needs of program participants" (p. 169). In general, it purposefully suspends the researcher's judgment.

[Then, I can take up goal-free evaluation for the embedded study. The participants want to find out ways to help increase retention rate, in the end, they suggested a way. There is no way to check the actual effects, but from the participants' perspective, the research brought about some solutions they looked for. ]

Transaction models: responsive and illuminative evaluation

The transaction model of evaluation that Ernie House (1978) called concentrates on case studies and "places prime emphasis on perception and knowing as a transactional process between researcher and research participant" (p. 172). Its variations include illuminative evaluation, developed in educational evaluation, and Stake's (1975) responsive approach to evaluation, which requires to have "face-to-face contact with people in the program and learn about diverse stakeholder's perspectives, experiences and concerns. This responsive evaluation has been integrated into Guba and Lincoln's (1989) "responsive constructivist evaluation" known as the fourth generation evaluation, which is based on naturalistic inquiry.
Connoisseurship studies
It places the evaluator's perceptions and expertise at the center of the evaluation process, "drawing heavily on his or her own judgments about what constitutes excellence" (172).

Interactive and participatory applications:

Placing a greater emphasis on human interaction in designing evaluation strategies, Patton wrote:

Qualitative methods may also be perceived as more personal because of their inductive nature. This means that, again, rather than imposing on people or a program some predetermined model or hypotheses, the results unfold in a way that takes into account idiosyncrasies, uniqueness, and complex dynamics. Finally, qualitative methods may be perceived as more humanistic and personal simply by avoiding numbers. (p. 176)

Action research, action learning, reflective practice
i.e. adult educators interviewing students as case studies, then through cross-case analysis they deepened their sensitivity to new immigrant students and developed their professional skills.
Participatory research and evaluation
Film: Awakening (1990)--a real story of participatory research in a mental hospital

Patton provided an overview of a growing popularity of participatory research:
"Interest in participatory research has exploded in recent years, especially as an element of larger community change efforts (Stoecker 1999). The principal researcher trains the coresearchers to observe, interview, reflect, and/or keep careful records or diaries. Those involved come together periodically to share in the data analysis process. The purpose of such shared inquiry is typically to elucidate and improve the nature of practice in some arena of action" (183).

In particular, participatory evaluation emphasizes "working with people" (183), thus, Patton wrote: "The process is facilitated by the researcher, but is controlled by the people in the program or community. They undertake a formal, reflective process for their own development and empowerment" (183).

Patton noted that participatory evaluation has been used with great success by a number of NGOs and private voluntary organization in the Third World.

Patton also drew attention to various connotations of participatory inquiry, saying that there is no definitive definition for participatory evaluation. Patton, in particular, emphasized teaching inquiry logic and skills as the purpose of participatory evaluation, as "the process of participatory self-evaluation, in and of itself, provided useful learning experience for participants" (184).

[Here Patton suggests participatory evaluation/inquiry as a pedagogical form.]

Finally, he noted, "these participatory approaches often employ qualitative methods because those methods are understandable, teachable, and usable by people without extensive research training." (185)

[Patton regards participatory paradigm from the perspective of pragmatism and links qualitative methods as strong candidates for that paradigmatic inquiry.]

8. Qualitative analysis and interpretation (431-534)

Patton discussed about summative vs. formative evaluation: Summative evaluations are usually accompanied by a full report with data, analysis, and recommendations, whereas formative evaluation is conducted for program improvement and may not generate a written report. Findings are often presented orally to the staff or funders. Patton wrote:
"The methods are qualitative, the purpose is practical, and the analysis is done throughout fieldwork; no written report is expected beyond a final outline of observations and implications. Academic theory takes second place to understanding the program's theory of action as actually practices and implemented" (435)

[The VT research can be viewed as formative evaluation to give insights into the OP program, which is not academic theory oriented and rather concentrates on providing critical insights from the insiders' perspective.]

Theory-based analysis approaches: Patton discusses phenomenological analysis and ground theory analysis in great detail, along with qualitative comparative analysis and analytic induction.

Analytic induction seems to be the most relevant to my approach. It originally emphasized discovering universal causal generalizations, but gradually moved away to qualitative inquiry and comparative case analysis including examination of preconceived hypotheses. In this approach, "researchers actively seek to disconfirm emerging hypotheses through negative case analysis, that is, analysis of cases that hold promise for disconfirming emerging hypotheses and that add variability to the sample. In this way, the originators of the method sought to examine enough cases to assure the development of universal hypothesis." (Jane Gilgun 1995:268-269, We shared something special: the moral discourse of incest perpetrators, Journal of marriage and the family, 57, 265-81). Analytic induction consists of re-examinations.

9. Enhancing the quality and credibility of qualitative analysis (541-597)

In traditional scientific research, among the criteria for judging the credibility are objectivity, validity, systematic rigor of fieldwork procedures, triangulation, etc. The common aim is "to use qualitative methods to describe and explain phenomena as accurately and completely as possible so that their descriptions and explanations correspond as closely as possible to the way the world is and actually operates" (p. 546)

Social construction and constructivist criteria: Lincoln and Guba (1986) proposed credibility (for internal validity), transferability, dependability and confirmability. In combination, they viewed these criteria as addressing trustworthiness (in lieu of rigor). Fundamentally, they are "suspicious of causal explanations and empirical generalizations applied to complex human interactions and cultural system" (546)

Critical change criteria: another stream of qualitative inquiry aims at social and political change. Thus consequential validity matters in addressing the credibility of the inquiry.

[For the analysis of my dissertation study, what criteria should be applied? The set of criteria should stem from the hypotheses of participatory (video) research. What does participatory research claim to be able to do? Does my study prove or disprove it? How do I make judgment? --My judgment should be grounded in constructivist perspective, the fourth generation evaluation. Then, what I need to verify is the claims of participatory research. ]

Using visual data in qualitative research

Banks, Marcus. (2007). London, UK: Sage

Validity in Qualitative Research

Robin Whittemore, Susan K. Chase and Carol Lynn Mandle (2001). Qualitative Health Research. 11 (4), 522-537.

Abstract:
Much contemporary dialogue has centered on the difficulty of establishing validity criteria in qualitative research. Developing validity standards in qualitative research is challenging because of the necessity to incorporate rigor and subjectivity as well as creativity into the scientific process. This article explores the extant issues related to the science and art of qualitative research and proposes a synthesis of contemporary viewpoints. A distinction between primary and secondary validity criteria in qualitative research is made with credibility, authenticity, criticality, and integrity identified as primary validity criteria and explicitness, vividness, creativity, thoroughness, congruence, and sensitivity identified as secondary validity criteria.

Reading the historical development of validity issues, the authors discussed three types of tensions.

1. Tension Between Qualitative and Quantitative Research

"Despite the incongruency between quantitative epistemology and qualitative methodology, translated standards of validity have proven to be useful criteria for demonstrating rigor and legitimacy of qualitative research. Guba and Lincoln (1989) advanced this work by proposing standards that were more reflective of specific threats to qualitative research." (pp. 523-524)

2. Tension Between Epistemological Purism and Pluralism

3. Tension Between Rigor and Creativity

"Creativity must be preserved within qualitative research, but not at the expense of the quality of the science. Creative work supports the discovery of the not yet known, going beyond previously established knowledge and challenging accepted thinking (Marshall, 1990). However, it is essential that qualitative work should be highly creative at the same time that it is analytically rigorous and explicit (Patton //--page break--// 1990). Creativity in and of itself will not contribute to sound science. Elegant and innovative thinking can be balanced with reasonable claims, presentation of evidence, and the critical application of methods." (pp. 526-527)


The authors pointed out there is no championing terms to articulate validity criteria in qual. Yet, they call Lincoln and Guba (1985, Naturalistic Inquiry: "credibility" for validity ) "the staying power." Yet, they argued, "validity is an accurate term and does provide the opportunity for criteria to be developed that are reflective of the tenets of the interpretive perspective" (pp. 527).


The authors wrote:

"Validity cannot be assumed, and presentation of research findings must invite the opportunity for critical reflection by consumers. The importance of explicating “how we claim to know what we know” (Altheide & Johnson, 1994, p. 496) is as essential as the claim to what we know." (p 535)

[It implies validity is self-reflection on the research process: authentic, rigor, moral]

Moments, Mixed Methods, and Paradigm Dialogs

Denzin, Norman K. (2010). Qualitative Inquiry. 16 (6), 419-427. doi: 10.1177/1077800410364608

Denzin examined mixed methods community, who calls for triangulation in qualitative inquiry. The paradigm wards of the 1970s and 80s, he noted, were played out against a post-Vietnam wars and civil right movements, resulting in flourishing qualitative program evaluation. Now, he sees the current paradigm war plays out against federal interventions, regulating sets of auditing criteria. He calls this current moment the “politics of evidence” (p. 425) Against this positivistic backlash, Denzin called for moral research:

Inquiry has always been and will always be a moral, political, value-laden enterprise. We seek only a politics of hopes, models of social justice to lead us forward.

As members of a larger moral community, we all need to draw together so we can share our problems and experiences with these new discourses. We need a moral and methodological community that honors and celebrates paradigm and methodological diversity. As fellow travelers we need research agendas that advance human rights and social justice through multiple forms of qualitative research. (425-426)

A Critique of Experimentalism

Howe, Kenneth R. (2004). Qualitative Inquiry, 10 (1), 42-61. doi: 10.1177/1077800403259491

Howe criticized neoclassical experimentalism—appropriating medical research as a model for all science research--saying “it gets the shoe on the wrong foot” (p. 49); On the other hand, mixed-methods experimentalism—predominantly in education field--is an advance over neoclassical experimentalism. Yet, it also places quantitative methods and centering technocratic perspective, what works for the research, and randomized experiments, thus Howe wrote this, too, is retrograde. He argued that it focuses on the question “whether interventions are effective in achieving given outcomes” (p.54).

Howe argued that there is no preset of criteria to determine what constitutes “scientific” research. Instead, researchers should decide what is counted as science or scientific, he said, adding such decisions are value laden.

Pointing out the pervasive influence of interpretivism on social science, Howe drew attention on Cronbach’ (1975) generalizations decay and Giddens’s (1976) double hermeneutic. What Cronbach meant by generalization decay is generalizations are fallible and subject to change over time. Double hermeneutic means that there are two types of interpretations in social science—one by researchers and the other by their audience. Howe pointed out, “For ‘critical’ researchers, making generalizations decay—generalizations documenting oppressive relationships, in particular—is an explicit goal of social research” (51-52).

Criticizing the mainstream mixed methods experimentalism, Howe looked into mixed methods interpretivism. He stressed that the outcome of this type of research are open to end and determined by stakeholders, thus, for him, inclusion and dialogue are central to this research approach. He put qualitative methods at the front versus QUAN. More importantly, the role of participants divides this approach from mixed methods experimentalism.

In conclusion, Howe noted that the contrast between the technocratic experimentalism and democratic interpretivism gets at the political dimensions of educational research. [Here Howe’s democratic interpretivism can be framed as “participatory” research, although he does not mention.]

Qualitative Inquiry Under Fire

: Toward a new paradigm dialogue. Denzin, N. K. (2009). Walnut Creek, CA: Left coast press.

1. Introduction (13-24)
Denzin defined the current regime of qualitative research as conservative evidence based or scientifically based research (SBR). He finds this trend in a nostalgia toward positivism and sees it as a continuation of the paradigm wars of the 1980s, being implemented as part of a conservative political backlash. "American postpositivists in the 1990s and the in the first decade of the new century were aligned with SBR movements that were gaining force under conservative political regimes..."(p. 21)
Denzin stated that "Mixed methods are direct descendants of classical experimentalism and the triangulation movement of the 1970s" (p. 17, Denzin 1989 "The research act: a theoretical introduction to sociological methods") The advent of the mixed-methods movement called for the paradigm wars of the 1980s, asking whether methods and perspectives could be combined and compatible. Morse(2003) warns that ad-hoc mixing of methods would threat validity; others deconstruct validity; Hesse-Biber and Leavy (2008) emphasized emerging methods that blur the boundaries between QUAL and QUAN methods and seeks to reveal subjugated knowledge underpinning everyday life. Amid the blurring boundariesDenzin noted an emerging confluence that focuses on the value of multivocal texts, as Guba and Lincoln (2005) suggested, which emphasizes a shared commitment to social justice and human rights. Within this context, Denzin write: "It is time to take stack, time, time to be reflexively self-critical" (p. 21)

2. Interpretive research after 9/11/01 (25-38)
Denzin argued that the conservative backlash after 9/11 attempted to regulate interpretive inquiry by enforcing bio-medical and evidence-based methods. Thus, he called for a radical change in critical interpretive inquiry.

4. The Elephant in the living room or extending the conversation about the politics of evidence (57-82)
Denzin describes the evidence-based model of research in qualitative inquiry as an elephant sitting in a living room. The evidence-based research movement and emerging standards and guidelines for conducting qualitative inquiry is at the brink of an global tendency.
Denzin pointed out three basic positions on the issue of evaluative criteria: foundational, quasifoundational, and nonfoundational. Foundational argues for a shared set of criteria, QUAL or QUAN, such as validity, credibility, transferability, confirmability, warrantibility, tranparency); Quasifoundational concerns about criteria for qualitative research, such as reflexivity, theoretical grounding, iconic; nonfoundationalists "stress the importance of understanding versus prediction. They conceptualize inquiry within a moral frame, implementing an ethic rooted in the concepts of care, love, and kindness" (p. 61).

For Denzin, "Inquiry is always political and moral" (p. 82). Likewise, "Objectivity and evidence are political and ethical terms" (p. 82). He stresses that ways of knowing are always partial, moral, and political.

5. The art of interpretation: The stories we tell one another (85-114)

The Practice of Qualitative Research

Sharlene Nagy Hesse-Biber & Patricia Leavy. (2006). Sage Publications.

Chapter 4. In-Depth Interview (pp. 119-145)

[I chose this chapter to discuss among the VTW research group members. It is written in a plain yet engaging manner with some examples to illustrate the authors' points.]

The authors introduce the significance of in-depth or intensive interview a common method for qualitative research. In comparison with quantitative interview, they argue, qualitative interviews, which are often involved with in-depth interview, should be "knowledge-producing conversation" (p. 128) based upon reciprocal relationships between interviewees and interviewers as well as a sense of shared authority, which regards interviewees as experts.

Similarly, defining the qualitative interview as "co-creation of meaning" (p. 134), the authors call for reflexivity among researchers. In their term:

"The heart of the qualitative interview requires much reflexivity, that is, sensitivity to the important 'situational' dynamics between the researcher and researched that can impact the creation of knowledge" (p. 135)

In the discussion of dealing with difference between researchers and researched, the authors talk about the notions of insiders and outsiders. They argue that one's status as insider/outsider is fluid and prone to change in the course of research. In general, they agree, insiders can bring much in-depth data, yet state the advantage of being an outsider:

"By not belong to a specific group, you may be viewed as more unbiased by your respondent. Likewise, you may be more likely to ask things that you would otherwise take for granted as "shared knowledge" and in fact learn that your participation have their own way of viewing a given issue." (p. 140)

The authors suggest to be mindful of the importance of difference between researchers and researched.

In the discussion of analysis and interpretation of interview data, the authors emphasize the importance of negative cases that do not fit the overall problems, as they are often the most informative. As a way of triangulating data, they suggest to try hard to disprove a researcher's presumed ideas.

Qualitative Researching

Mason, Jennifer. (1996). Qualitative Researching. London, Thousand Oaks, and New Delhi: Sage.



Generating Qualitative Data: Interviewing (35-59) [research method training for VT Summer Research Group].

Generating Qualitative Data: Observation, Documents and Visual Data (60-82)

Doing Qualitative Research Differently

: Free Association, Narrative and the Interview Method. Hollway, Wendy, & Jefferson, Tony. (2000). London, Thousand Oak, and New Delhi: Sage.

Introduction: the Need to do Research Differently (pp.1-6)

The authors question the veracity of accounts that a research subject tell during interview. For them, one of the reasons for believing is democratic and related to power differentials between researcher and researched. Feminists, according to the author, have been conscious of this issue of power differentials and advocated the principles of giving voice to voiceless women. However, the authors claim that researchers cannot give voice to others, quoting a statement by Reissman (1993, Narrative Analysis, p. 3):
"we cannot give voice" since we "do not have direct access to another's experience. We deal with ambiguous representations of it--talk, text, interaction and interpretation"

[This argument is not logical. Giving voice does not mean directly accessing someone's experience.]

From the standpoint of critical realism (Bunge, 1993; Watkins, 1994-5), the authors argue that researchers interpret their subject's experience only through ambiguous representations of it, yet there is a relationship between people's representations and their experiences. Here the emphasis is given to that the relationship, tracked only by the researcher, depends on the researcher's particular view.

Producing Data with defended subjects (26-54)
From their pilot studies, the authors delineate four distinct approaches to interview: traditional, feminist, narrative, and clinical case-study. While traditional approaches focus on interview technique, feminists are conscious of power balance between interviewer and interviewee. In narrative approaches, an interviewee becomes a story-teller, thus the interview agenda is open to development, depending on the interviewee's experiences. Clinical case-study emphasizes a researcher's reflexivity on their own involvement in the interview process, such as anxiety, personal experience, etc.

The authors propose biographical-interpretative method. It emulates a biographer's role to listen to someone's story and make a coherent account out of the story-telling, Gestalt, explaining, "a whole which is more than the sum of its parts, an order or hidden agenda" (p. 34). The authors propose four principles of this method: (1) the more open the better, (2) elicit stories (i.e. Tell me about a time when you feel unsafe.), (3) avoid why questions to obtain concrete meanings from the interviewee. Why questions usually induce empty, cliché-ridden discussion, the authors assert. (4) Follow up using respondents' ordering and phrasing by taking a note.

The authors emphasize free association that comes unconsciously during interview.