International: The pitfalls of monitoring and evaluating gender equality programmes

A summary of Part I “Capturing change in women’s realities: The challenges of monitoring and evaluating our work” a paper by Srilatha Batliwala* and Alexandra Pittman.** Monitoring and evaluation now form an integral part of women’s rights and gender equality programmes as we attempt to measure how effectively we work. But are the frameworks we use able to perform this ambitious task? In their paper Capturing change in women’s realities: The challenges of monitoring and evaluating our work Srilatha Batliwala and Alexandra Pittman assess the “ifs,” the “whys” and the “hows” of monitoring and evaluation (M&E) in women’s rights, women’s empowerment and gender equality. They observe that “over the past few decades, important strides have been made in developing ways of capturing a whole range of abstract but vital social realities, and particularly in trying to quantify them.

These efforts have been the result of the realization that when policies, resources, and strategies are applied towards building more equitable, sustainable, rights-affirming, inclusive and peaceful societies, we have to devise ways of checking whether they are working effectively or not - whether they are producing the changes we wish to see.”

However, the authors argue that the push to incorporate monitoring and evaluation into our work has led to problematic assumptions regarding our ability to measure and assess the changes we are trying to achieve and the outcomes of our efforts to do so. Too often it is taken for granted that the tools we use to measure change are effective, and assumed that the measurement of change enhances our ability to make or accelerate positive change. With the increasing demand – particularly from donors – for increasingly elaborate M&E systems, the authors say that it is urgent to interrogate assumptions about M&E.

It is challenging to measure social change, particularly in the context of women’s rights, gender equality and women’s empowerment work. In summarizing the greatest challenge of this measurement, the authors quote a seasoned activist who says:

“When you work for women’s interests, it’s two steps forward – if you’re really smart and very lucky! – and at least one step back. In fact, it’s often two or three steps back! And those steps back are, ironically, often evidence of your effectiveness: because they represent the threat you have posed to the power structure, and its attempt to push you back. Sometimes, even your ‘success stories’ are nothing more than ways the power structure is trying to accommodate and contain the threat of more fundamental change by making small concessions.”

The authors acknowledge the difficulty of transforming certain power relations such as North-South, race, sexuality, and caste, but argue that transforming gender power relations is the last frontier of social change. This is because patriarchal norms are embedded and normalized in all power structures, so changing them is twice as challenging. Furthermore, gender power is situated within public and private institutions and relationships, so a shift in one arena may not lead to changes in another.

Why do we measure change?

Batliwala and Pittman begin their paper with a discussion of this critical question. Theoretically, there are five basic objectives to M&E:

* To learn how change happens - what works and what doesn’t - in order to refine strategies for greater impact;

* To analyze our role in the change process and identify cause-effect relationships;

* To empower our constituencies so that they can sustain and expand change;

* To be accountable and credible to donors and constituencies;

* To advance advocacy for social justice in order to mobilize support for our agenda.

The reality, however, is that M&E is often done because:

* Donors require it;

* It helps to make the case for obtaining funding;

* It supports advocacy work.

They argue that the pressures contained in the latter set of reasons “convert measurement from an activity designed to aid learning into one that evaluates performance, and so distorts the purpose and potential value of our M&E work.” A feminist approach to M&E would be motivated by the first set objectives rather than the second.

What isn’t working

Batliwala and Pittman have analyzed over fifty M&E frameworks and tools used by women’s organizations and assessed their strengths and limitations. In summary, the main challenges are as follows:

1. Few M&E frameworks actually enable us to understand how change happens or how gender relations have been altered. Linear frameworks typically describe goals and related activities so that at the end of a project cycle, it is only possible to say whether or not the goals were achieved, and not whether change was achieved. Even frameworks that assess changes in communities or constituencies do not help us to understand whether there is sustainable change. As the activist quoted earlier observed, sometimes the power structures make small concessions, but resist sustained transformations and so a strategy that worked once might not work a second time.

2. Current M&E frameworks may not help us to know what to measure. They do not usually provide an avenue for the testing of their own assumptions or underlying theories of change. Even long-standing measures like GDP are now questioned because they don’t provide an adequate picture of people’s lived realities. Although M&E frameworks should be a learning aid for future interventions, in practice M&E indicators become an end in themselves or a tool of enforcement and accountability.

3. Most tools assume that change occurs in a linear way, but social change is a complex process that cannot be reduced to neat sets of causal relationships. What often ends up happening when organizations use this kind of linear framework is that their focus is on the tool itself – how to meet the expected achievements – rather than on the larger picture.

4. Problematic macro-political assumptions underlie many M&E frameworks. Batliwala and Pittman say it is assumed that “democratic rights, law and order, an impartial judiciary and police, an independent media etc., etc. are inevitably present, surrounding change processes in a larger safety net.” This is far from the reality in most contexts, particularly in the developing world, where humanitarian, economic, health and political crises abound. Any M&E framework that overlooks these macro-political factors is fundamentally flawed.

5. Most M&E tools do not provide for tracking negative change, reversals and backlash, yet whenever women’s rights work seriously challenges patriarchal structures there are often setbacks. It is important to have tools that capture the ‘two steps forward one step back’ reality.

6. Current M&E tools cannot adequately measure gender equality work that is less tangible but very important. Organizations find it difficult to measure the impact of work such as research, capacity building or attitude change and resort to measuring their processes, outreach and outputs rather than the effect of their work.

7. There are false binaries and dichotomies contained in many M&E approaches: macro versus micro, quantitative versus qualitative, success versus failure and so on. Instead of integrating seemingly opposite values, most approaches create a hierarchy.

8. Change isn’t always visible within a six-month or three-year grant period. M&E frameworks are usually bound to a particular time frame and there is often very little that can be realistically measured in a short period of time.

9. Few of the current assessment methods available are gendered or feminist. Gender analysis frameworks are not the same as feminist evaluation.

A feminist M&E approach would:

* Have tools to unpack the nature of gender and social inequalities;

* Treat gender and social inequalities as systemic and therefore examine how interventions address them as such;

* Not seek to attribute change to particular actors, but to understand contributing actors and factors;

* Break the hierarchy between the evaluator and the “evaluated” and respect the knowledge of both; and

* View M&E as a political activity, not a value-free assessment, and use it as part of the change process.

Batliwala and Pittman conclude their paper with the following call to action: “Given these myriad and serious challenges, it is crucial to construct new feminist M&E approaches and tools that combine the strengths of some of our existing tools, overcome their more serious shortcomings, and adequately capture the complexity of gender equality work and the social dynamic within which it occurs.”

The paper Capturing change in women’s realities: The challenges of monitoring and evaluating our work is the initial product of AWID’s action-research initiative to develop principles and tools for feminist approaches to monitoring and evaluation. In 2010, working in collaboration with women’s rights organizations and networks, this initiative will publish further tools and resources on this topic.

By Kathambi Kinoti and Sanushka Mudaliar

We welcome comments and suggestions on the analysis in this article. Please use the comments section below or email your feedback to contact@awid.org.

__________________________

* Srilatha Batliwala is Scholar Associate with AWID’s Building Feminist Movements and Organizations Strategic Initiative.

** Alexandra Pittman is a consultant with AWID’s Where is the Money for Women’s Rights Strategic Initiative.