Musings on Monitoring & Evaluation (M&E) of Justice Sector Reform Programs

Some U.S. Government agencies were late in understanding the importance of M&E to determine the impact that foreign assistance programs were having.  In the last few years, I always kept hearing that we needed to answer then Secretary of State Hillary Clinton’s “so what?” question regarding how effective our international aid projects were.

Many multi-million dollar programs had no internal nor external M&E experts to provide guidance.  In Afghanistan, for example, the U.S. Embassy’s 2013 rule of law strategy failed to incorporate any performance measures.  (For an interesting report that reveals what the problems relating to M&E were at the time, I suggest you read the Special Inspector General for Afghanistan Reconstruction (SIGAR) audit).

Through evaluation tools, M&E programs aim to demonstrate program impact.  This, in turn, provides feedback to guide program implementation staff to enhance future programming by identifying planned and unplanned results to allow donors, implementers and host country beneficiaries to understand what works and does not work, how to maximize efficiencies, and address any issues that might arise before they become a problem or a cataclysmic risk.

In government contracts, the Statement of Work (SOW) may provide the indicators to be used.  Sometimes, the implementer may develop a series of iterative evaluations as well, which might include a training evaluation and an audit, a trainee-satisfaction survey, a mentoring plan, and -depending on the program- a public outreach component.

Performance indicators may combine the Foreign Assistance Framework Indicators (F-Indicators), as well as customized indicators, with the goal to develop and utilize indicators that measure outputs and impact in the short, medium and long-term of the project.

Of course, the most perfect and all-encompassing M&E plan will not work unless both donors, implementers and beneficiaries take into account the critical risks inherent in, or coming from, the place of performance, and agree on some critical assumptions that, at the very least, encompass three contexts: political, security, and operational.

What I have learnt is that decision-makers and bureaucrats from both the government side and the corporate side make choices and issue “diktats” without having had the benefit of operating in the environment where the program is being carried out.  I never gave it much thought until I witnessed it first-hand.  Therefore, it is imperative that the “experts” who are hired to handle M&E issues understand that they may be dealing with people who have little or no knowledge of the hurdles the technical staff face day in and day out.

Sometimes, the mere fact that electricity is not available or the internet connection does not work, may mean that M&E data cannot be incorporated into a database.

While I applaud the importance of M&E in program management, I see some problem areas:

  1. Who monitors and evaluates the authenticity and the accuracy of the M&E plan and its implementation in-house?  In other words, if I am the donor, would I fully trust the contractor or grantee to monitor and evaluate itself?
  2. If the donor hires a third-party to do an independent M&E of a program, how comfortable can the donor and implementer be that the third-party will do an unbiased and truly objective M&E assessment?  What are the chances that the M&E firm will have a former implementer employee evaluating the very same program that person put in place?

Rule of law programs are not immune from a myriad of conflicts of interest.  Who pays attention to these things?

Human Trafficking or Trafficking in Persons (TIP)

In my experience, I have found that there tends to be a disconnect in program implementation between those who are technical experts and those who actually administer the government contract.

Government (USG) officials, companies, and NGOs have expressed concern about the risk of Human Trafficking in global supply chains, including in federal contracts. Victims originate from almost every region of the world; the top three countries of origin of federally identified victims in fiscal year (FY) 2017 were the United States, Mexico, and the Philippines.

Human Trafficking or Trafficking in Persons (TIP) is defined in the Trafficking Victims Protection Act of 2000 (TVPA). It is about people being bought and sold as chattel, involving the acquisition of a human being through the use of force, fraud, or coercion for the purpose of exploiting the individual for profit through forced labor or prostitution.  The USG efforts to combat such trafficking, also incorporated in the United Nation’s Sustainable Development Goal 8.7, are to prevent it, protect the victims, and prosecute the traffickers. A fourth “p”, involves “partnership” between the public and private sectors, with state and local organizations, and with survivors.

The annual State Department TIP Report classifies countries into tiers based on a government’s efforts to comply with the “minimum standards for the elimination of trafficking” found in TVPA.  Although the U.S. is a Tier 1 country according to the 2017 TIP Report, it is also “a source, transit, and destination country for men, women, transgender individuals, and children— both U.S. citizens and foreign nationals—subjected to sex trafficking and forced labor. Trafficking occurs in both legal and illicit industries, including in commercial sex, hospitality, sales crews, agriculture, fishing, manufacturing, janitorial services, construction, restaurants, health and elder care, salon services, fairs and carnivals, peddling and begging, and domestic service.

The National Defense Authorization Act (NDAA) allows the USG to terminate a contract if the prime contractor or subcontractor commits acts that directly support or advance trafficking in persons, such as:  confiscating an employee’s identity or immigration documents; offering employment using fraudulent or misleading pretenses; charging placement or recruitment fees; and providing housing that fails to meet the host country housing and safety standards.

Judicial reform programs might include capacity training on TIP, and it is important to understand how TIP issues can affect the federal contractor in its day-to-day business affairs.

Additional resources:

Thoughts on the Evolution of a Rule of Law Justice Sector Program

Ideally, the genesis of an international Rule of Law “development” project should entail the meetings of the minds between the donor and the beneficiary.  Some of us like to say that there has to be “buy-in” from the host government as well as the individual host organizations that might be involved in the project.  This would show the cooperative efforts of all concerned towards what could be the key goals of the mission:  strengthening of the rule of law and the fostering of  accountability and transparency.  For example, a specific justice sector program’s mission might be to build the capacity of the host country’s criminal justice sector institutions through improving the ability of their professional staff to deliver fair and effective justice services to citizens.  Sometimes, the initial focus is to help a country build its police and prosecutorial capacity through formal training programs of academic instruction.

What I have discovered is that what the donor organization (which may include the program implementer as well) wants to see happen may not necessarily be best suited to the way the project should be carried out.  One does not become an international development practitioner overnight.  While an expatriate technical advisor may have stellar credentials from his or her prosecutorial days in their particular state or country, they may have never lived in a different environment other than their own.  This can lead to disastrous results because there is a lack of understanding of, and maybe a lack of empathy for, the recipient of the technical advice.

A long time ago, I witnessed a foreign “expert” deliver a lecture on American jurisprudence and individual rights to an academic group in a socialist country.  The audience was barely curious and did not seem to engage.  What the “expert” did not realize was the group’s lack of understanding of what he believed were common concepts, until someone asked “what is the right to privacy?”  Once it became obvious that there had been such a gulf between the lecturer and the trainees, the “expert” was able to correct the situation and begin to provide examples that the local nationals could finally relate to!

I was at the very beginning of my professional “Rule of Law” work, and it was a fine lesson for me too:  borrowing from the Spaniards, there are many “Ruperto el Experto” types, but few that meet the “experto crede Ruperto”  standard.

Rule of Law Resources

Below is an incomplete list of Rule of Law resources that I have found very helpful throughout my work.  Because of my concentration on Afghanistan in recent years, some of these resources include specific Afghan projects.  However, some of the references and issues are applicable to other rule of law and justice sector contexts.  I will be supplementing this list as best I can as time goes by.

Afghan Women Network (AWN)

Afghanistan Legal Documents Exchange Center (ALDEC)

Afghanistan Legal Education Project (ALEP)

Afghanistan Research and Evaluation Unit

Center for International Governance Innovation (CIGI)

Harvard Islamic Research Studies

Hauser Global Law School Program: (GIRoA Legal System and Research)

Humanitarian Crisis in Afghanistan

Independent National Legal Training Center

International Committee Red Cross (ICRC) Afghanistan

International Legal Foundation

International Network to Promote Rule of Law (INPROL)

Law Library of Congress

LexisNexis – The Rule of Law

Max-Planck-Institute Organization and Jurisdiction of the Newly Established Afghan Courts

Relief Web International (Afghanistan)

The Afghanistan NGO Safety Office (ANSO)

The Siracusa International Institute for criminal justice and human rights

United States Institute of Peace

University of Michigan Law Library

Women for Women

Yale Law School – Lillian Goldman Law Library

Justice Sector Support Programs Worldwide

When it comes to U.S. Government foreign assistance to rule of law development programs, or international justice sector support programs, we need to look at the Democracy, Human Rights, and Governance  (DHRG) section of the U.S. Government’s website on all foreign assistance programs, since rule of law projects fall under this section.  If you visit the site, you will discover that for Fiscal Year 2018 $1.8 billion have been planned for the DHRG sector, out of $25.8 billion.  Peace & Security as well as the Health sectors and Humanitarian Assistance take the biggest pieces of the pie in foreign assistance.

Although DHRG is relatively small in the scope of U.S. international development support, I believe it is a critical component in what we do as a country and as a people.