For some of us, it is a nightmare that the Internet is “forever”. However, as I said in my previous post about “historians” and “curators”, it is a wonderful tool to ensure that the historical record remains true and “historical”. I just need to find the time to share this public information in a coherent way. But I will.
So, I cannot help it.
I love anything to do with Rule of Law, especially anything to do with Afghanistan, because I learnt a lot from my Afghan colleagues AND I always wonder how the US Government’s justice sector program is progressing. There was a time when I would read and dissect what SIGAR was stating, both for my employer and my Afghan colleagues. Nowadays, I hope there are others who are on top of this, doing the same thing. Here is the latest SIGAR report: https://www.sigar.mil/pdf/quarterlyreports/2018-10-30qr.pdf.
I cannot find anything that gives me information about Rule of Law issues in Afghanistan. Zilcho, zero, nada. Maybe I am doing the wrong Google search… because the only thing I find are articles written by implementers. Which I find is a disservice to those who take working on Rule of Law issues seriously. Even USG-funded websites that described what the USG was doing in Afghanistan in the justice sector have been obliterated.
The USG changes contractors, periodically. So, whatever the previous contractor did -at the USG demand/direction- is totally forgotten when the new contractor wins the contract. Who cares, right? Yet, whoever has been trying to be a “historian”, or “curator” of the project, is all of a sudden left without any historical background. Why? Because, understandably so, the new contractor does not want to recognize the enormous work the previous group performed.
Luckily, the Internet is forever. IMHO, it is my Afghan counterparts who need to weave the story of what the USG and the Afghan Government accomplished for many, many years. I will just share what I think ought to be checked by anyone doing some research. There is so much available, if we dig deep enough. I wish someone would have done this for me…
Late on Sunday afternoon, doing some research on whether recipient countries might pay lip service on Rule of Law/justice sector reform donor programs, I came across this article in the Mexican Law Review on Rule of Law, titled, “International Support for Justice Reform: Is It Worthwhile? written by Luis Pásara, a Peruvian lawyer, sociologist and professor.
I have just skimmed through it, although I already know I want to study it in full because certain premises of his coincide with some of my thoughts. While it addresses justice reform in Latin America, the author touches on universal problems. Mr. Pásara’s conclusion is worth quoting:
On the one hand, it is important to keep in mind that establishing the Rule of Law is a broader and more difficult task than reforming the justice system. Therefore, building a better justice system is not enough to establish the Rule of Law; the former is just a component of the latter. The quality of the laws, the legal culture, the actual social and economic inequalities, and the role played by the government —among other elements— are important and complex components of the process of building the Rule of Law.
On the other hand, internationally-funded programs of justice system reform are not able to produce deep changes, which are badly needed for both a better justice system and the establishment of the Rule of Law, in the receiving countries. Clearly, such programs are not able to “fundamentally reshape the balances of power, interests, historical legacies, and political traditions of the major political forces in recipient countries. They do not neutralize dug-in antidemocratic forces. They do not alter the political habits, mind-sets, and desires of entire populations” and “[o]ften aid cannot substantially modify an unfavorable configuration of interests or counteract a powerful contrary actor.”175 That is why international aid in the area of justice has not delivered a new justice system in receiving countries. It simply could not do it.
But there is some room for improvement. Taking into account the analysis made in this article, some concrete suggestions can be proposed for the many people, acting in good faith in the international agencies and who are willing to find ways to do a better job of improving justice systems in the region:
Knowledge is a must. No decision about the area, content, size, timing or amount of a project should be made without detailed knowledge of the subject in the country where the work is to be done.
Learn what others produced. To gain knowledge of the prevailing conditions mainly requires: collecting the information that already exists, paying attention to national actors’ perceptions and analysis, taking advantage of the knowledge of international experts who have gained experience in that particular country, and evaluating other agencies’ experience in the field.
National actors and a clear strategy are needed. The conditions required to develop a project include: a core of national actors who are truly committed to the reform goals, and a strategy —to be designed jointly with national actors— with well-defined short, medium, and long-term goals within the project.
National actors have a crucial say. The implementation phase of any project needs to have a partnership of national and international actors, but the last word should be said by national actors who know better and ultimately are responsible for the reform process in their country.
Monitoring and evaluation are indispensable. Project implementation needs continuous monitoring and project evaluation presents opportunities to learn about both achievements and failures. External reviews of the projects —including work done by academic researchers— are powerful tools for a critical analysis on what works and what does not. Reticence to share information with capable peers is, in the long term, a way of wasting resources.
If these remedies —and other possible changes— are introduced to alter the performance of international actors and agencies, they may dramatically increase the level of quality of the outcomes of justice reform projects.
A friend of mine recently commented on Monitoring & Evaluation (M&E) processes, which made me ponder as to why they are found baffling by the average person, no matter how many years of experience and education that person may have.
I discovered that many proposal evaluators get confused when reading the proposed M&E section and will acknowledge without compunction that they just could not quite follow what the organization writing the M&E plan was actually proposing. I have also witnessed intelligent individuals turn glassy eyed at hearing about the M&E work plan’s development, that includes outputs vs. outcomes, inputs vs. indicators, activities vs. results, and the concept of an “iterative adaptation”.
Below I share some of the M&E resources that I found helpful in trying to understand what different donors had in mind when referring to the elusive “monitoring for results” in capacity building projects. However, I have yet to find answers to my concerns about conflicts of interest and other problems in M&E and program implementation:
- Who are the evaluators?
- Evaluating the competition: There is an inherent conflict of interest when the evaluators are hired to do M&E work on an implementing entity and they themselves are competitors in the contracting/grant implementation world. This situation places the implementer in a very vulnerable position, as the competitor/evaluator is in the enviable position of learning proprietary information.
- Evaluating a former employer:
- When a disgruntled or aggrieved former employee is hired to evaluate the former employer’s work by the donor, who is aware of the complaints and grievances of this former employee, the integrity and the objectivity of the evaluation are in peril.
- When a former employee is knowingly hired by the donor to evaluate that former employee’s own work, there is an inherent conflict of interest that taints the evaluation from its very beginning. How unbiased can that former employee be?
- How does one ensure true transparency in the M&E process?
- Learning from failure:
- Will the program implementer that the M&E shows is failing in certain aspects of the project not worry about the potential risk of losing the project to a competitor?
- Donors face budgetary pressures to work on successful programs. But M&E points out to what does not work, what needs improvement. If the M&E plan is done internally, by the implementer itself, there are conflicts between those program experts who want to apply the learned lessons of the M&E -even if it means revising the program, readjusting it, or removing parts of the program that don’t work, and those administrators who mostly pay attention to the bottom line and do not want to see the program shrink at all. One could argue the same conflicts exist between donor and contractor. See the tension?
- Learning from failure:
How can you guarantee complete accuracy of the data being entered into a database?
- Self-assessment via an implementer’s internal M&E process relies on the honesty, good faith, and accuracy of the employees providing the data and those entering the data. However, when the donor is under immense pressure to produce results, the temptation to churn information that may not be verifiable is real.
- The same issues above apply to third parties hired by the donor to gather the implementers’ data and produce charts and graphs that make beautiful infographics for future publications. However, who monitors these third parties, who may be using flawed algorithms or erroneous excel sheet mathematical equations?
So, is M&E really that difficult to understand? I have my own theory on why Rule of Law/Justice Sector projects are so hard to assess, but this is for another day. Here is a list of methodologies and other resources for you to decide:
- USAID’s Human and Institutional Capacity Development (HICD)
- the Maturity Model (MM)
- the Staged Capacity Building Model (SCBM)
- a Problem-Driven Iterative Adaptation (PDIA)
- Managing for Results Framework of the State Department
- the Governance Practitioner’s Notebook published by the OECD
- UNDP’s Measuring Capacity
- UNDP’s User’s Guide on How to Measure Rule of Law, Justice and Security Programmes
- The UN’s Rule of Law Indicators