Program Implementation – The Alphabet Soup of M&E: PDIA, HICD, MM, SCBM, etc.

A friend of mine recently commented on Monitoring & Evaluation (M&E) processes, which made me ponder as to why they are found baffling by the average person, no matter how many years of experience and education that person may have.

I discovered that many proposal evaluators get confused when reading the proposed M&E section and will acknowledge without compunction that they just could not quite follow what the organization writing the M&E plan was actually proposing.   I have also witnessed intelligent individuals turn glassy eyed at hearing about the M&E work plan’s development, that includes outputs vs. outcomes, inputs vs. indicators, activities vs. results, and the concept of an “iterative adaptation”.

Below I share some of the M&E resources that I found helpful in trying to understand what different donors had in mind when referring to the elusive “monitoring for results” in capacity building projects.  However, I have yet to find answers to my concerns about conflicts of interest and other problems in M&E and program implementation:

  • Who are the evaluators?
    • Evaluating the competition:  There is an inherent conflict of interest when the evaluators are hired to do M&E work on an implementing entity and they themselves are competitors in the contracting/grant implementation world.  This situation places the implementer in a very vulnerable position, as the competitor/evaluator is in the enviable position of learning proprietary information.
    • Evaluating a former employer:
      • When a disgruntled or aggrieved former employee is hired to evaluate the former employer’s work by the donor, who is aware of the complaints and grievances of this former employee, the integrity and the objectivity of the evaluation are in peril.
      • When a former employee is knowingly hired by the donor to evaluate that former employee’s own work, there is an inherent conflict of interest that taints the evaluation from its very beginning.  How unbiased can that former employee be?
  • How does one ensure true transparency in the M&E process?
    • Learning from failure:
      • Will the program implementer that the M&E shows is failing in certain aspects of the project not worry about the potential risk of losing the project to a competitor?
      • Donors face budgetary pressures to work on successful programs.  But M&E points out to what does not work, what needs improvement.  If the M&E plan is done internally, by the implementer itself, there are conflicts between those program experts who want to apply the learned lessons of the M&E -even if it means revising the program, readjusting it, or removing parts of the program that don’t work, and those administrators who mostly pay attention to the bottom line and do not want to see the program shrink at all.  One could argue the same conflicts exist between donor and contractor.  See the tension?
  •  How can you guarantee complete accuracy of the data being entered into a database?

    • Self-assessment via an implementer’s internal M&E process relies on the honesty, good faith, and accuracy of the employees providing the data and those entering the data.  However, when the donor is under immense pressure to produce results, the temptation to churn information that may not be verifiable is real.
    • The same issues above apply to third parties hired by the donor to gather the implementers’ data and produce charts and graphs that make beautiful infographics for future publications.  However, who monitors these third parties, who may be using flawed algorithms or erroneous excel sheet mathematical equations?

So, is M&E really that difficult to understand?  I have my own theory on why Rule of Law/Justice Sector projects are so hard to assess, but this is for another day.  Here is a list of methodologies and other resources for you to decide:

 

Fund for Peace: The Bottom 100

Channel surfing while in the city of Buenos Aires, this commercial on CNN International caught my eye.  It turned out to be what I had suspected:  a commercial about the Bottom 100, a project involving the Fund for Peace.

The Bottom 100 allows one to “[s]ee life at the other end of the world’s rich list… [and] gives a face and voice to the millions in poverty who struggle with nothing.”

(Disclosure:  I am on the fund’s board).

Of Interest: Afghanistan and The Book of Kings or Shahnameh

As I have mentioned many a time, I have a soft spot in my heart for Afghanistan, and the work I was involved with for years opened a window into a fascinating world of beauty and history.  I used to pester colleagues with my “Of Interest” emails, in which I would relate things that made me ponder.  Now, I shall not just ponder, but actually read The Book of Kings:

Samangan is one of Afghanistan’s 34 provinces.  What I did not know, is that it is also the setting of an epic love story, that comes from the Persian equivalent of the Odyssey and the Ilyad:  The Book of Kings, or Shahnameh.

The story, written in verse around 1,000 years ago by Persian poet Ferdowsi, tells how a mighty warrior, Rustam, makes it all the way to Samangan, seeking his lost horse.  While the guest of the king, Rustam retires to his chambers, after enjoying a sumptuous meal with the king, only to be woken up by the king’s daughter, Tahmineh, who declares her love for the warrior.  That one night of passion, that results in a marriage, yields a son, whom Rustam will only meet in battle many years hence.

The woeful story of Rustam and Tahmineh and their son Sohrab starts like this:

STORY OF SOHRÁB
O ye, who dwell in Youth’s inviting bowers,
Waste not, in useless joy, your fleeting hours,
But rather let the tears of sorrow roll,
And sad reflection fill the conscious soul.
For many a jocund spring has passed away,
And many a flower has blossomed, to decay;
And human life, still hastening to a close,
Finds in the worthless dust its last repose.
Still the vain world abounds in strife and hate,
And sire and son provoke each other’s fate;
And kindred blood by kindred hands is shed,
And vengeance sleeps not—dies not, with the dead.
All nature fades—the garden’s treasures fall,
Young bud, and citron ripe—all perish, all.

And now a tale of sorrow must be told,
A tale of tears, derived from Múbid old,
And thus remembered.—

What a beautiful translation!  It is so true that “All nature fades–the garden’s treasures fall, young bud, and citron ripe–all perish, all.”  

You can find the whole translated epic in Project Gutenberg.  There is also a new illustrated version of the Shahnameh, that has been a labor of love for filmmaker Hamid Rahmanian.

Rustam in Samangan meets the King’s daughter, Tahmineh.

The Atlantic Magazine and CNN have interesting articles explaining the new version of the Shahnameh.

Enjoy!

Justice Sector Support Programs (JSSP) – Legal Systems of the World – Islamic Law

When we talk about the legal systems of the world, there comes to mind Cannon Law (Roman Catholic), Hindu Law, Jewish Law, and Islamic Law (Shariah), to name a few.

In Afghanistan, for example, in the hierarchy of laws, Shariah rules supreme, above the country’s Constitution.  I have never been an “expert” on Islamic Law, but I have found plenty of written resources that have helped me understand the natural tension between the religious law and the secular law.

Here is an interesting compilation of “religious laws in a nutshell” from NYU’s Hauser Global Law School Program.  And here is the Practitioners’ Guide on Islamic Law written by Mr. Hamid Khan for the International Network to Promote the Rule of Law (INPROL), which I highly recommend to anyone who is interested in the Rule of Law.

Justice Sector Support Programs (JSSP) – Legal Systems of the World

It has never ceased to amaze me how many “experts” there are who work in the “justice sector” field, who are woefully unaware as to what the different legal systems are in the international arena.  The problem is that, with JSSP-type programs, not only should the “technical experts” know their subject well, but those who administer the programs (i.e., those involved in the management/operational/business side of the program) need to have an understanding of what the subject matter entails.  It takes a good professional to be willing to understand what the technical side of the project really involves.

As of 2011, the US Department of State relied on the below map to show what the legal systems of the world were.  I always liked the visual simplicity of this map.

Legal Systems of the World

Equally so, the simplicity of the summary of Civil and Common Law systems below helps anyone, lawyer and non-lawyer alike, to understand the significance of the map above.  screenshot_20180126-114224.png

Although a bit old, this State Department guide gives the reader a sense of the scope and breadth of what the US Government has been doing in the JSSP world:  INL Guide to Justice Sector Assistance.   I found this guide quite useful, through the years.

Justice Sector Support Programs – Resources

As an autodidact, I always am looking for material to read, absorb, analyze, and re-purpose for my own benefit.  Maybe it is a result of my life, moving here, there and yonder, having to adapt to new environments, and wanting to learn all about them.

I look forward to studying this amazing and free Manual on Independence, Impartiality and Integrity of Justice: A Thematic Compilation of International Standards, Policies and Best Practice.

The CEELI Institute produced this massive tome, supported by the Bureau of International Narcotics and Law Enforcement Affairs (INL) at the US State Department.  If you have the time to look around, you will find there are many terrific resources for those interested in international development and justice sector support programs.

The Rule of Law – an Australian Perspective

No one seems to have the definitive answer to what “Rule of Law” (ROL) actually means.  There are all sorts of interpretations derived from interpretations of that term of art.   I have found that, depending on one’s background, education, and life experiences, there are some variations of what we might define as “ROL”.  After all, totalitarian states have been ruled by law, whether they might be draconian laws or laws never to be applied to the rulers.  Here in the US I find that -more and more- many a lawyer and legal professional mention that laws are being applied to the “little” people, and that the powerful and rich get away with circumventing the law.  Alas, the way I see it, and because we are flawed human beings, as long as we can talk about this and not end up in a torture chamber or dark cell, we are alright.

From a philosophical perspective, though, I am intrigued by what other societies’ perceptions are about the ROL.  Below is the ROL pyramid that the Rule of Law Institute of Australia has developed:

Screenshot_20180125-095254

As I perused the site, I came across The Rule of Law:  Its State of Health in Australiawritten by University of Sydney Professor Kevin Lindgren, where he points out in his introduction key concepts as to the meaning of the ROL:

The expression signifies not a legal rule, but more generally rule by “law” as distinct from rule by power, free of legal constraint, whether by a democratically elected government, a tyrant or otherwise. So, the ideal signifies that the institutions of the state, and in particular, the individuals and bodies that are invested with power by the state, should be subject to the law rather than above it.

There are narrow and broad meanings of the rule of law. According to the narrow meaning, the rule of law is not concerned with whether a law is good or bad, but only with whether the law is applied equally to all. According to the broader meaning, the ideal embraces human rights standards. On any reckoning, both the rule of law and human rights standards should be respected, observed and protected. The only question is the semantic one of whether we properly treat the former concept as encompassing the latter.

It is quite fitting to recall that these “human rights” that involve the broader meaning of the ROL were drafted by representatives of nations of different legal and cultural backgrounds and became the Universal Declaration of Human Rights, which was proclaimed by the United Nations General Assembly almost 70 years ago, on December 10th, 1948, as a common standard for the world.

SIGAR’s Advice on Program Implementation.

The Special Inspector General for Afghanistan Reconstruction (SIGAR) produced an audit of a Department of Defense (DOD) $635 million program in Afghanistan -the Task Force for Business and Stability Operations (TFBSO), which yields some self-evident and interesting points:

Taking the following actions might improve such an entity’s ability to implement programming and achieve results:

• Define the entity’s mission, scope, and objectives in clear and measureable terms.
• Authorize the entity for longer than 1-year intervals to reduce uncertainty about its future and allow it time to plan ahead for its projects.
• Direct the entity to:

o Develop contract planning policies that emphasize the importance of understanding host-country or local dynamics and obtaining buy-in from all stakeholders before executing a project;
o Develop and implement action plans to minimize the award of  oncompetitive and sole-source contracts;
o Develop and implement action plans to ensure that its staff has adequate training and experience in developing contract requirements and providing contract oversight;
o Work with a single primary contract administration office when developing performance work statements to ensure consistency in drafting requirements;
o Develop management systems to track project metrics, civilian travel, and government-furnished equipment;
o Develop and implement a document retention policy; and
o Develop monitoring, evaluation, and sustainment plans for all projects so that their economic impacts can be accurately measured and sustained, and if necessary, assets can be transferred to an enduring partner.

SIGAR mentions that DOD was given the opportunity to comment on the audit.  Something that struck me was SIGAR’s comment to DOD’s comment, which -in my experience- is the crux of development aid or foreign assistance (emphasis in bold below is mine):

It is important to understand the difference between projects that met or partially met their contractual deliverables and projects that actually met or partially met their program objectives.  DOD is correct in observing that this report finds the contracts directly supporting 16 TFBSO projects generally met their contract deliverables and that contracts directly supporting 12 projects partially met their contract deliverables (or in one case, met them after significant delay). However, just because some TFBSO contractors met their contract deliverables in whole or in part does not necessarily mean that the projects they supported had successful or sustainable outcomes. For example, there are several documented cases where TFBSO contractors completed construction and equipment of a facility, but TFBSO was unable to locate a private company able to operate and maintain it, leading that facility to fall into a state of disuse or disrepair.  Furthermore, as we note in the report, because TFBSO did not consistently track outcomes data, such as the jobs created and government revenues generated by their projects, TFBSO was generally unable to demonstrate whether its projects met its overall objectives to “reduce violence, enhance stability, and support economic normalcy in Afghanistan.”

At the end of the day, what worries me is that many in government and the private sector voice these concerns; however,  not often are the solutions offered taken into account, or, worse still, they are quickly forgotten.  In my own experience, these concerns and suggestions have been made for decades.  I have a theory -which I will try to articulate later- as to why we seem to reinvent the wheel…

 

State’s OIG’s Work Plans.

You may be as quirky as I am, and enjoy knowing what the State Department’s Office of the Inspector General (OIG) plans on auditing/inspecting this year and the next (Work Plans for 2018 and 2019 from the State Department’s OIG).  I have found that, through the years of reading audits and reports from the various US Government OIGs I have learnt a lot, especially how the US Government manages its contracts and personnel, as well as what are some of the contractors’ weaknesses and strengths.

For example, I have often marveled at how bad all parties to a contract can be with poor record keeping, despite knowing that, invariably, a government contract will be audited down the road.  It is amazing to me that we do not make an extra effort to ensure that records are easily accessible.  When working on rule of law programs in other countries, we stress the importance of accountability and transparency, with good record keeping being essential to fight corruption.