Category: Evolution of a program

Of Interest: SIGAR’s Audit Report on PROMOTE Program in Afghanistan.

jssp2017Leave a comment

The name of the audit says it all, don’t you think?

“Promoting Gender Equity in National Priority Programs (Promote):  USAID Needs to Assess This $216 Million Program’s Achievements and the Afghan Government’s Ability to Sustain Them”

There was so much excitement at the time of the program’s announcement by the then USAID’s Administrator at the U.S. Institute of Peace (USIP), which I remember well.   It was a grandiose $216 million project with the expectation that other international donors would contribute an additional $200 million in funds.  SIGAR’s recommendations are three:

1. Conduct an overall assessment of Promote and use the results to adjust the program and measure future program performance….

2. Provide written guidance and training to contracting officer’s representatives on maintaining records in a consistent, accurate manner. …

3. Conduct a new sustainability analysis for the program.

Of the SIGAR recommendations to USAID above, I find #2 quite sad, because in my experience, record keeping has deteriorated to the point of oblivion.  Institutional knowledge has waned in many organizations, whether they belong to the private sector or the public one.

Anyone involved in government contracting work ought to read this audit, because it highlights some major flaws in how we run (or not!) multi-million dollar taxpayer-funded programs.

You can read more about the audit at the Stars and Stripes.

Of Interest: SIGAR’s New Lessons Learnt Report on Afghanistan

jssp2017Leave a comment

The Special Inspector General for Afghanistan Reconstruction (SIGAR) released at the end of May 2018 its latest “lessons learnt” report covering the period 2002-2017.  It is chock full of information for anyone who is interested in “development” work.

For those who follow “rule of law” issues, and especially for those who actually are working in Afghan-related projects, the following excerpts strike, at least to me, a very familiar chord.

Even within dispute resolution, the U.S. government chose to focus on formal rule of law, rather than informal rule of law, also called traditional dispute resolution (TDR). TDR in Afghanistan employs a varying mixture of traditional, community-specific norms, and Islamic legal principles, or sharia. 1015 As early as 2007, international legal experts highlighted the coalition’s inattention to informal justice, even though an estimated 80–90 percent of Afghan disputes are handled through TDR, and many Afghans have more faith in it than in formal dispute resolution. 1016

The U.S. government spent more than $1 billion on rule of law programming in Afghanistan between 2003 and 2015, of which less than $100 million (approximately 10 percent) was spent on enhancing informal rule of law. 1022 State’s 2009 rule of law strategy—the only one it drafted—recognized the importance of TDR to Afghans, even calling it a “pillar” of the coalition’s effort; however, the balance between funding for formal and informal rule of law programming did not seem to reflect this recognition. 1023  Worse, the kind of dispute resolution promoted by formal rule of law programming was not only considered corrupt, but also foreign to most rural Afghans.

In my many years of working in international settings doing “development” work, I have found that one of the biggest problems is overcoming individual egos and the posturing that comes with those egos.  “Development” work is not just altruistic:  there is a lot of money to be made and prestige to be gained.  There is a door that is always “revolving” between the implementer, the donor, the supervising entity, the inspecting authority, the academicians, and other intellectuals.  It is human nature.

However, once in a while, it is good to read that some of the lonely and knowledgeable individuals who pointed out some of the flaws in design were vindicated.   Note that I don’t use the term “expert”, because, in my work, most experts are really “Rupertos the “expertos””.  (My own label for the last few years has been “subject matter expert”!).

Program Implementation – The Alphabet Soup of M&E: PDIA, HICD, MM, SCBM, etc.

jssp2017Leave a comment

A friend of mine recently commented on Monitoring & Evaluation (M&E) processes, which made me ponder as to why they are found baffling by the average person, no matter how many years of experience and education that person may have.

I discovered that many proposal evaluators get confused when reading the proposed M&E section and will acknowledge without compunction that they just could not quite follow what the organization writing the M&E plan was actually proposing.   I have also witnessed intelligent individuals turn glassy eyed at hearing about the M&E work plan’s development, that includes outputs vs. outcomes, inputs vs. indicators, activities vs. results, and the concept of an “iterative adaptation”.

Below I share some of the M&E resources that I found helpful in trying to understand what different donors had in mind when referring to the elusive “monitoring for results” in capacity building projects.  However, I have yet to find answers to my concerns about conflicts of interest and other problems in M&E and program implementation:

  • Who are the evaluators?
    • Evaluating the competition:  There is an inherent conflict of interest when the evaluators are hired to do M&E work on an implementing entity and they themselves are competitors in the contracting/grant implementation world.  This situation places the implementer in a very vulnerable position, as the competitor/evaluator is in the enviable position of learning proprietary information.
    • Evaluating a former employer:
      • When a disgruntled or aggrieved former employee is hired to evaluate the former employer’s work by the donor, who is aware of the complaints and grievances of this former employee, the integrity and the objectivity of the evaluation are in peril.
      • When a former employee is knowingly hired by the donor to evaluate that former employee’s own work, there is an inherent conflict of interest that taints the evaluation from its very beginning.  How unbiased can that former employee be?
  • How does one ensure true transparency in the M&E process?
    • Learning from failure:
      • Will the program implementer that the M&E shows is failing in certain aspects of the project not worry about the potential risk of losing the project to a competitor?
      • Donors face budgetary pressures to work on successful programs.  But M&E points out to what does not work, what needs improvement.  If the M&E plan is done internally, by the implementer itself, there are conflicts between those program experts who want to apply the learned lessons of the M&E -even if it means revising the program, readjusting it, or removing parts of the program that don’t work, and those administrators who mostly pay attention to the bottom line and do not want to see the program shrink at all.  One could argue the same conflicts exist between donor and contractor.  See the tension?

  •  How can you guarantee complete accuracy of the data being entered into a database?

    • Self-assessment via an implementer’s internal M&E process relies on the honesty, good faith, and accuracy of the employees providing the data and those entering the data.  However, when the donor is under immense pressure to produce results, the temptation to churn information that may not be verifiable is real.
    • The same issues above apply to third parties hired by the donor to gather the implementers’ data and produce charts and graphs that make beautiful infographics for future publications.  However, who monitors these third parties, who may be using flawed algorithms or erroneous excel sheet mathematical equations?

So, is M&E really that difficult to understand?  I have my own theory on why Rule of Law/Justice Sector projects are so hard to assess, but this is for another day.  Here is a list of methodologies and other resources for you to decide:

 

SIGAR’s Advice on Program Implementation.

jssp2017Leave a comment

The Special Inspector General for Afghanistan Reconstruction (SIGAR) produced an audit of a Department of Defense (DOD) $635 million program in Afghanistan -the Task Force for Business and Stability Operations (TFBSO), which yields some self-evident and interesting points:

Taking the following actions might improve such an entity’s ability to implement programming and achieve results:

• Define the entity’s mission, scope, and objectives in clear and measureable terms.
• Authorize the entity for longer than 1-year intervals to reduce uncertainty about its future and allow it time to plan ahead for its projects.
• Direct the entity to:

o Develop contract planning policies that emphasize the importance of understanding host-country or local dynamics and obtaining buy-in from all stakeholders before executing a project;
o Develop and implement action plans to minimize the award of  oncompetitive and sole-source contracts;
o Develop and implement action plans to ensure that its staff has adequate training and experience in developing contract requirements and providing contract oversight;
o Work with a single primary contract administration office when developing performance work statements to ensure consistency in drafting requirements;
o Develop management systems to track project metrics, civilian travel, and government-furnished equipment;
o Develop and implement a document retention policy; and
o Develop monitoring, evaluation, and sustainment plans for all projects so that their economic impacts can be accurately measured and sustained, and if necessary, assets can be transferred to an enduring partner.

SIGAR mentions that DOD was given the opportunity to comment on the audit.  Something that struck me was SIGAR’s comment to DOD’s comment, which -in my experience- is the crux of development aid or foreign assistance (emphasis in bold below is mine):

It is important to understand the difference between projects that met or partially met their contractual deliverables and projects that actually met or partially met their program objectives.  DOD is correct in observing that this report finds the contracts directly supporting 16 TFBSO projects generally met their contract deliverables and that contracts directly supporting 12 projects partially met their contract deliverables (or in one case, met them after significant delay). However, just because some TFBSO contractors met their contract deliverables in whole or in part does not necessarily mean that the projects they supported had successful or sustainable outcomes. For example, there are several documented cases where TFBSO contractors completed construction and equipment of a facility, but TFBSO was unable to locate a private company able to operate and maintain it, leading that facility to fall into a state of disuse or disrepair.  Furthermore, as we note in the report, because TFBSO did not consistently track outcomes data, such as the jobs created and government revenues generated by their projects, TFBSO was generally unable to demonstrate whether its projects met its overall objectives to “reduce violence, enhance stability, and support economic normalcy in Afghanistan.”

At the end of the day, what worries me is that many in government and the private sector voice these concerns; however,  not often are the solutions offered taken into account, or, worse still, they are quickly forgotten.  In my own experience, these concerns and suggestions have been made for decades.  I have a theory -which I will try to articulate later- as to why we seem to reinvent the wheel…

 

Eliminating Violence Against Women

jssp2017Leave a comment

Below is an illustration from a booklet published by IDLO a few years ago that was used by the Afghan Attorney General’s office to explain the Law on the Elimination of Violence Against Women.  It provides a glimpse of what was then the educational campaign needed to reach the many rural and remote provinces, communities and government officials who did not know about the laws affecting the rights of women.
The OECD has interesting statistics that show countries’ ranks in the world of violence against women (VAW).  In 2014 Jamaica appears as the least violent country, while Guinea appears to have been last.  The World Health Organization (WHO) produced an Ethical and and safety recommendations for intervention research on domestic violence against women, which focuses on “health-based interventions to address VAW…”.  The WHO Infographics shows why VAW has health-based repercussions.

Keeping things in perspective, from an international development lens, it was not until 1994 that the U.S. Government enacted the Violence Against Women’s Act, and the U.S. Department of Justice’s Office on Violence Against Women (OVW) did not open until a year later.

Of note:  Even though the U.S. Government has not ratified the UN’s 1979 Convention on the Elimination of All Forms of Discrimination against Women (CEDAW), many U.S. Government funded Rule of Law programs incorporate CEDAW in their gender-based and VAW projects.  By the way, Afghanistan ratified CEDAW on March 5th, 2003.

 

Musings on Monitoring & Evaluation (M&E) of Justice Sector Reform Programs

jssp2017Leave a comment

Some U.S. Government agencies were late in understanding the importance of M&E to determine the impact that foreign assistance programs were having.  In the last few years, I always kept hearing that we needed to answer then Secretary of State Hillary Clinton’s “so what?” question regarding how effective our international aid projects were.

Many multi-million dollar programs had no internal nor external M&E experts to provide guidance.  In Afghanistan, for example, the U.S. Embassy’s 2013 rule of law strategy failed to incorporate any performance measures.  (For an interesting report that reveals what the problems relating to M&E were at the time, I suggest you read the Special Inspector General for Afghanistan Reconstruction (SIGAR) audit).

Through evaluation tools, M&E programs aim to demonstrate program impact.  This, in turn, provides feedback to guide program implementation staff to enhance future programming by identifying planned and unplanned results to allow donors, implementers and host country beneficiaries to understand what works and does not work, how to maximize efficiencies, and address any issues that might arise before they become a problem or a cataclysmic risk.

In government contracts, the Statement of Work (SOW) may provide the indicators to be used.  Sometimes, the implementer may develop a series of iterative evaluations as well, which might include a training evaluation and an audit, a trainee-satisfaction survey, a mentoring plan, and -depending on the program- a public outreach component.

Performance indicators may combine the Foreign Assistance Framework Indicators (F-Indicators), as well as customized indicators, with the goal to develop and utilize indicators that measure outputs and impact in the short, medium and long-term of the project.

Of course, the most perfect and all-encompassing M&E plan will not work unless both donors, implementers and beneficiaries take into account the critical risks inherent in, or coming from, the place of performance, and agree on some critical assumptions that, at the very least, encompass three contexts: political, security, and operational.

What I have learnt is that decision-makers and bureaucrats from both the government side and the corporate side make choices and issue “diktats” without having had the benefit of operating in the environment where the program is being carried out.  I never gave it much thought until I witnessed it first-hand.  Therefore, it is imperative that the “experts” who are hired to handle M&E issues understand that they may be dealing with people who have little or no knowledge of the hurdles the technical staff face day in and day out.

Sometimes, the mere fact that electricity is not available or the internet connection does not work, may mean that M&E data cannot be incorporated into a database.

While I applaud the importance of M&E in program management, I see some problem areas:

  1. Who monitors and evaluates the authenticity and the accuracy of the M&E plan and its implementation in-house?  In other words, if I am the donor, would I fully trust the contractor or grantee to monitor and evaluate itself?
  2. If the donor hires a third-party to do an independent M&E of a program, how comfortable can the donor and implementer be that the third-party will do an unbiased and truly objective M&E assessment?  What are the chances that the M&E firm will have a former implementer employee evaluating the very same program that person put in place?

Rule of law programs are not immune from a myriad of conflicts of interest.  Who pays attention to these things?

Human Trafficking or Trafficking in Persons (TIP)

jssp2017Leave a comment

In my experience, I have found that there tends to be a disconnect in program implementation between those who are technical experts and those who actually administer the government contract.

Government (USG) officials, companies, and NGOs have expressed concern about the risk of Human Trafficking in global supply chains, including in federal contracts. Victims originate from almost every region of the world; the top three countries of origin of federally identified victims in fiscal year (FY) 2017 were the United States, Mexico, and the Philippines.

Human Trafficking or Trafficking in Persons (TIP) is defined in the Trafficking Victims Protection Act of 2000 (TVPA). It is about people being bought and sold as chattel, involving the acquisition of a human being through the use of force, fraud, or coercion for the purpose of exploiting the individual for profit through forced labor or prostitution.  The USG efforts to combat such trafficking, also incorporated in the United Nation’s Sustainable Development Goal 8.7, are to prevent it, protect the victims, and prosecute the traffickers. A fourth “p”, involves “partnership” between the public and private sectors, with state and local organizations, and with survivors.

The annual State Department TIP Report classifies countries into tiers based on a government’s efforts to comply with the “minimum standards for the elimination of trafficking” found in TVPA.  Although the U.S. is a Tier 1 country according to the 2017 TIP Report, it is also “a source, transit, and destination country for men, women, transgender individuals, and children— both U.S. citizens and foreign nationals—subjected to sex trafficking and forced labor. Trafficking occurs in both legal and illicit industries, including in commercial sex, hospitality, sales crews, agriculture, fishing, manufacturing, janitorial services, construction, restaurants, health and elder care, salon services, fairs and carnivals, peddling and begging, and domestic service.

The National Defense Authorization Act (NDAA) allows the USG to terminate a contract if the prime contractor or subcontractor commits acts that directly support or advance trafficking in persons, such as:  confiscating an employee’s identity or immigration documents; offering employment using fraudulent or misleading pretenses; charging placement or recruitment fees; and providing housing that fails to meet the host country housing and safety standards.

Judicial reform programs might include capacity training on TIP, and it is important to understand how TIP issues can affect the federal contractor in its day-to-day business affairs.

Additional resources:

Thoughts on the Evolution of a Rule of Law Justice Sector Program

jssp2017Leave a comment
Ideally, the genesis of an international Rule of Law “development” project should entail the meetings of the minds between the donor and the beneficiary.  Some of us like to say that there has to be “buy-in” from the host government as well as the individual host organizations that might be involved in the project.  This would show the cooperative efforts of all concerned towards what could be the key goals of the mission:  strengthening of the rule of law and the fostering of  accountability and transparency.  For example, a specific justice sector program’s mission might be to build the capacity of the host country’s criminal justice sector institutions through improving the ability of their professional staff to deliver fair and effective justice services to citizens.  Sometimes, the initial focus is to help a country build its police and prosecutorial capacity through formal training programs of academic instruction.

What I have discovered is that what the donor organization (which may include the program implementer as well) wants to see happen may not necessarily be best suited to the way the project should be carried out.  One does not become an international development practitioner overnight.  While an expatriate technical advisor may have stellar credentials from his or her prosecutorial days in their particular state or country, they may have never lived in a different environment other than their own.  This can lead to disastrous results because there is a lack of understanding of, and maybe a lack of empathy for, the recipient of the technical advice.

A long time ago, I witnessed a foreign “expert” deliver a lecture on American jurisprudence and individual rights to an academic group in a socialist country.  The audience was barely curious and did not seem to engage.  What the “expert” did not realize was the group’s lack of understanding of what he believed were common concepts, until someone asked “what is the right to privacy?”  Once it became obvious that there had been such a gulf between the lecturer and the trainees, the “expert” was able to correct the situation and begin to provide examples that the local nationals could finally relate to!

I was at the very beginning of my professional “Rule of Law” work, and it was a fine lesson for me too:  borrowing from the Spaniards, there are many “Ruperto el Experto” types, but few that meet the “experto crede Ruperto”  standard.