Jackson, Edward T., & Kassam, Yusuf. (1998). West Hartford, CT: Kumarian.
The authors define participatory evaluation as a strategic too for development projects. They stated that the essence of participatory evaluation lies in shared knowledge between local community and specialists (or professionals). Through shared knowledge, they argued that (1) participatory evaluation better serves the interests of both the local beneficiaries and development agencies; (2) it democratizes and enriches the assessment of development; (3) it enhances the benefits for the stakeholders.
Historically examining various participatory evaluation projects, the authors proposed the definition of participatory evaluation:
"Participatory evaluation is a process of self-assessment, collective knowledge production, and cooperative action in which the stakeholders in a development intervention participate substantively in the identification of the evaluation issues, the design of the evaluation, the collection and analysis of data, and the action taken as a result of the evaluation findings. . . . This approach evaluation employs a wide range of data collection and analysis techniques, both qualitative and quantitative, involving fieldwork, workshops, and movement building. " (p. 3)
With respect to fourth generation evaluation that Guba and Lincoln proposed, the authors argued that this approach replaced the concept of participation with that of stakeholder interaction; it separates the participation from the rhetoric concern of "empowerment" in the 1970s and 1980s, thus making the participatory evaluation methodologically more achievable. Yet, the authors argued that in participatory evaluation, all the stakeholders are not equal and thus, there should be a bias in favor of the least powerful in the process of participatory evaluation. This perspective makes the authors argument distinct from other collaborative forms of assessment. [Here, participatory evaluation is written in the context of macro development projects, namely to criticize co-opted participatory evaluation as a tool kit, thus the direct comparison with fourth generation evaluation does not sound feasible.]
Among the issues related to participatory evaluation, the authors empathized the importance of facilitation skills to make a safe space in which every stakeholder can freely speak.
Saturday, October 2, 2010
Friday, October 1, 2010
Fourth generation evaluation
Egon Guba & Yvona Lincoln. (1989). Newbury Park, London, and New Delhi: Sage.
Forward (7-19)
The authors criticized that pervasively positivistic approaches to evaluation treat evaluation as a scientific process that misses its social, political, value-oriented nature. Instead, they called for a constructivist approach, arguing that evaluation outcomes "represent meaningful constructions that individual actors or groups of actors form to 'make sense' of the situations in which they find themselves" (p. 8). Evaluation, therefore, is a process of negotiation. They called this evaluation paradigm fourth generation evaluation, empathizing the nature of construction and negotiation among stakeholders. In this light, they noted that such evaluation should be action-oriented. Here, the role of evaluator is defined as the orchestrator of the negotiation process:
"If there is to be a course of action with which most stakeholders can agree, it can only be arrived at through negotiation [sic] . . . If that goal is to be achieved, the evaluator must play a larger role by far than simply that of technician-gathering-information; instead he or she must be the orchestrator of the negotiation process, which in the final analysis is the guts of the evaluation." (p. 10)
The authors also pointed out that the ethics involved in research should go beyond informed consent, harm, and confidentiality. To make research more ethical, they argued that it should allow full participative involvement among its stakeholders and others in the evaluation process such that they partake in the research as equal partners in the aspects of design, implementation, interpretation, and evaluation. [Here, the authors are arguing for participatory research and empathized that they should participate in the evaluation process.]
Besides constructivist methodology, they discuss "responsive focusing" as one of the two elements of fourth generation evaluation. Developing the concept of responsive evaluation by Robert Stake, they argued that the participants of the evaluation, stakeholders, "have the right to place their claims, concerns, and issues on the table for consideration (response), irrespective of the value system to which they adhere" (p. 12). With these claims, concerns, and issues, responsive focusing is progressed to resolve them.
The Coming of Age of Evaluation (21-49)
Looking back the history of evaluation for more than a century, the authors defined the first generation evaluation as measurement; the second as description; the third as judgment. Here the evaluator assumes the role of judge while retaining the technical and descriptive functions as well. One of the authors criticisms on the first three generations is "overcommitment to the scientific paradigm of inquiry"(p. 35). Against this background, the authors develop their evaluation generation based on the concept of responsive evaluation, first proposed by Stake (1975, Evaluating the arts of education, Columbus, OH: Merrill): It "determines parameters and boundaries through an interactive, negotiated process that involves stakeholders and that consumes a considerable portion of the time and resources available." Thus, constructivism is a natural methodology for them to pursue this new emerging paradigm of evaluation.
Similar to the research methodology that the authors argued for, they proposed the following criteria for judging the adequacy of fourth generation evaluation:
A. Trustworthiness
(in lieu of validity, reliability and objectivity that stem from the positivist paradigm). Instead, they put forward credibility, dependability, and confirmability.
1) Credibility focuses on "establishing the match between the constructed realities of respondents (of stakeholders) and those realities as represented by the evaluator and attributed to various stakeholders" (p. 237). Here, the objective neutral reality is refuted. Some of the techniques to enhance credibility are prolonged engagement, persistent observation, peer debriefing, negative case analysis, progressive subjectivity ("monitoring the evaluator's own privilege, for instance, by recording his or her developing construction. In other words, watching out what the evaluator discovers more than what they were looking for. Here the authors write, "If the inquirer 'finds' only what he or she expected to find, initially, or seems to become 'stuck' or 'frozen' on some intermediate construction, credibility suffers.), and member checking.
2) Dependability (in lieu of reliability)
The authors assert that reliability is concerned with the stability of the data over time, but argued that methodological changes and shifts in construction inevitably result in different research products, thus important is not to bring about stable data but to make such changes and shifts trackable ("publicly inspectable" -p. 242). This enables an inquiry audit. The authors assert that this audit should be accompanied by the concern of confirmability.
3) Confirmability (in lieu of objectivity)
Whereas objectivity in the conventional paradigm is rooted in method, confirmability is rooted in the data themselves, argued the authors: "This means that data (constructions, assertaions, facts, and so on) can be tracked to their sources, and that the logic used to assemble the interpretations into structurally coherent and corroborating wholes is both explicit and implicit in the narrative of a case study" (p. 243)
B. The hermeneutic process of its own quality control
The authors empathized the hermeneutic and dialectic process for quality control. A simple way of doing is to analyze any type of feedback immediately on receipt and incorporate into the emerging joint as the process continues. In so doing, the authors argued that the possibility of the evaluator's prejudice can shape the evaluation results is virtually zero as long as their own construction is "laid on the table along with all the others and made to withstand the same barrage of challenge, criticism, and counterexample as any others..."(p. 244).
C. The authenticity criteria
1) Ontological authenticity: Through the provision of various experience, the authors argued, the evaluation process should enable the respondents to improve their consciousness of the world.
2) Educative authenticity: It represents " the extent to which individual respondents' understanding of and appreciation for the constructions of others [sic] outside their stakeholding group are enhanced" (p. 248)
3) Catalytic authenticity: It is related to the purpose of evaluation. It asks whether the evaluation triggers some forms of action or decision making. The authors argued that evaluation should take into consideration the call for "getting theory into action" (p. 249). This process has to do with the ownership of the participants. They put it:
"When action is jointed negotiated, it should follow that action is "owned" by participants and, therefore, as the research has shown, more willingly carried out." (p. 250)
4) Tactical authenticity: This concept compensates certain conditions that prohibit such actions. It refers to "the degree to which stakeholders and participants are empowered to act" in the process of evaluation (p. 250) How can it be demonstrated? The authors suggested soliciting testimony, following up and rendering judgment. They suggested asking the following questions: "Was it participatory? [Here fourth genetaion evaluation is the very same concept with participatory evaluation] Have all stakeholders felt that they or their representatives have had a siginificant role in the process? Are all participants more skilled than previously in understanding and utilizing power and negotiation techniques? If the answers to those questions are uniformed yes, then tactical authenticity has probably been achieved" (p. 250)
Forward (7-19)
The authors criticized that pervasively positivistic approaches to evaluation treat evaluation as a scientific process that misses its social, political, value-oriented nature. Instead, they called for a constructivist approach, arguing that evaluation outcomes "represent meaningful constructions that individual actors or groups of actors form to 'make sense' of the situations in which they find themselves" (p. 8). Evaluation, therefore, is a process of negotiation. They called this evaluation paradigm fourth generation evaluation, empathizing the nature of construction and negotiation among stakeholders. In this light, they noted that such evaluation should be action-oriented. Here, the role of evaluator is defined as the orchestrator of the negotiation process:
"If there is to be a course of action with which most stakeholders can agree, it can only be arrived at through negotiation [sic] . . . If that goal is to be achieved, the evaluator must play a larger role by far than simply that of technician-gathering-information; instead he or she must be the orchestrator of the negotiation process, which in the final analysis is the guts of the evaluation." (p. 10)
The authors also pointed out that the ethics involved in research should go beyond informed consent, harm, and confidentiality. To make research more ethical, they argued that it should allow full participative involvement among its stakeholders and others in the evaluation process such that they partake in the research as equal partners in the aspects of design, implementation, interpretation, and evaluation. [Here, the authors are arguing for participatory research and empathized that they should participate in the evaluation process.]
Besides constructivist methodology, they discuss "responsive focusing" as one of the two elements of fourth generation evaluation. Developing the concept of responsive evaluation by Robert Stake, they argued that the participants of the evaluation, stakeholders, "have the right to place their claims, concerns, and issues on the table for consideration (response), irrespective of the value system to which they adhere" (p. 12). With these claims, concerns, and issues, responsive focusing is progressed to resolve them.
The Coming of Age of Evaluation (21-49)
Looking back the history of evaluation for more than a century, the authors defined the first generation evaluation as measurement; the second as description; the third as judgment. Here the evaluator assumes the role of judge while retaining the technical and descriptive functions as well. One of the authors criticisms on the first three generations is "overcommitment to the scientific paradigm of inquiry"(p. 35). Against this background, the authors develop their evaluation generation based on the concept of responsive evaluation, first proposed by Stake (1975, Evaluating the arts of education, Columbus, OH: Merrill): It "determines parameters and boundaries through an interactive, negotiated process that involves stakeholders and that consumes a considerable portion of the time and resources available." Thus, constructivism is a natural methodology for them to pursue this new emerging paradigm of evaluation.
Similar to the research methodology that the authors argued for, they proposed the following criteria for judging the adequacy of fourth generation evaluation:
A. Trustworthiness
(in lieu of validity, reliability and objectivity that stem from the positivist paradigm). Instead, they put forward credibility, dependability, and confirmability.
1) Credibility focuses on "establishing the match between the constructed realities of respondents (of stakeholders) and those realities as represented by the evaluator and attributed to various stakeholders" (p. 237). Here, the objective neutral reality is refuted. Some of the techniques to enhance credibility are prolonged engagement, persistent observation, peer debriefing, negative case analysis, progressive subjectivity ("monitoring the evaluator's own privilege, for instance, by recording his or her developing construction. In other words, watching out what the evaluator discovers more than what they were looking for. Here the authors write, "If the inquirer 'finds' only what he or she expected to find, initially, or seems to become 'stuck' or 'frozen' on some intermediate construction, credibility suffers.), and member checking.
2) Dependability (in lieu of reliability)
The authors assert that reliability is concerned with the stability of the data over time, but argued that methodological changes and shifts in construction inevitably result in different research products, thus important is not to bring about stable data but to make such changes and shifts trackable ("publicly inspectable" -p. 242). This enables an inquiry audit. The authors assert that this audit should be accompanied by the concern of confirmability.
3) Confirmability (in lieu of objectivity)
Whereas objectivity in the conventional paradigm is rooted in method, confirmability is rooted in the data themselves, argued the authors: "This means that data (constructions, assertaions, facts, and so on) can be tracked to their sources, and that the logic used to assemble the interpretations into structurally coherent and corroborating wholes is both explicit and implicit in the narrative of a case study" (p. 243)
B. The hermeneutic process of its own quality control
The authors empathized the hermeneutic and dialectic process for quality control. A simple way of doing is to analyze any type of feedback immediately on receipt and incorporate into the emerging joint as the process continues. In so doing, the authors argued that the possibility of the evaluator's prejudice can shape the evaluation results is virtually zero as long as their own construction is "laid on the table along with all the others and made to withstand the same barrage of challenge, criticism, and counterexample as any others..."(p. 244).
C. The authenticity criteria
1) Ontological authenticity: Through the provision of various experience, the authors argued, the evaluation process should enable the respondents to improve their consciousness of the world.
2) Educative authenticity: It represents " the extent to which individual respondents' understanding of and appreciation for the constructions of others [sic] outside their stakeholding group are enhanced" (p. 248)
3) Catalytic authenticity: It is related to the purpose of evaluation. It asks whether the evaluation triggers some forms of action or decision making. The authors argued that evaluation should take into consideration the call for "getting theory into action" (p. 249). This process has to do with the ownership of the participants. They put it:
"When action is jointed negotiated, it should follow that action is "owned" by participants and, therefore, as the research has shown, more willingly carried out." (p. 250)
4) Tactical authenticity: This concept compensates certain conditions that prohibit such actions. It refers to "the degree to which stakeholders and participants are empowered to act" in the process of evaluation (p. 250) How can it be demonstrated? The authors suggested soliciting testimony, following up and rendering judgment. They suggested asking the following questions: "Was it participatory? [Here fourth genetaion evaluation is the very same concept with participatory evaluation] Have all stakeholders felt that they or their representatives have had a siginificant role in the process? Are all participants more skilled than previously in understanding and utilizing power and negotiation techniques? If the answers to those questions are uniformed yes, then tactical authenticity has probably been achieved" (p. 250)
Subscribe to:
Posts (Atom)
