They write about the processes being engaged to determine the criteria (often focusing on seemingly irrational process such as the use of irrelevant “anchor points” to determine judgmental criteria). The behavioral economics scientists also note at an even more basic level, that it is important to identify the participants in any decision-making group that is formed to determine the criteria. For instance, who determines criteria for identifying the economic health of a country or the level of social equity or prosperity in the country (leading some behavioral economists to challenge the use of GNP as a primary criterion)? Who is left out of the discussion and decision-making process, and, in turn, what values and perspectives are ignored during this process?
We can probe even deeper: what is the rationale for the decisions that are made and what biases operate in establishing this rationale? What are the vested interests that operate among those establishing the criteria? Doesn’t the rationale that is being used have a major influence on the criteria being established and determination of a program’s success based on these criteria? Do not the vested interests, values and perspectives of those at the table have a major impact on the assessment of outcomes? Is evidence ever gathered and interpreted in a neutral manner?
Though she is focusing on numbers, a recommendation made by Deborah Stone (2021, p. 45) is equally valid when applied to words and narratives:
The best way I know to find out whether a measure is valid is to travel back to the moment of creation and nose around. Who was in the room where it happened? Who was asking the questions and what did they think to ask about? Who got to say what “counts as” the thing being counted?
As the chaos theorist had previously noted, the initial conditions surrounding any process or event plays a long-term determinative role. Those of us operating in the world of psychology have also suggested that early life experiences tend to hang around throughout one’ life as ghosts and goblins that are only partially exorcised by psychotherapy, reality (ego functions) or societal norms and regulations (super-ego functions).
Where is the Expert Sitting?
What about the challenges faced by those gathering information and serving as experts? Are not some people and perspectives absent from the table? If return-on-investment is identified as a key criterion for determining the success of coaching programs, then how is “investment” defined and what does “return” mean? Are both terms defined primarily in financial terms? If this is the case, then are some outcomes being devalued or even ignored? Are there important investments other than money that must be taken into account? It is not just a matter of expanding “investment” to include time spent and facilities engaged, it is also the investment of hope and the price paid by loss or regret. How do we take these into account?
If we reframe the criteria and speak of “return-on-expectations” we may be bringing more people to the table, but at the same time we may be making assessment even more difficult and increasing the intrusion of biases and preconceptions. The world gets messier or at least the mess that is already there becomes more apparent. What are we going to do about this challenge and what would a process of determining criteria for establishing evidence look like when many people are invited to the table—bringing with them diverse perspectives and values? Many behavioral economists propose that this diversity brings more creativity to the table (Page, 2011; Johansson, 2004; Kahneman, 2011). The key question is: do we need creativity when we are trying to build the foundation for evidence-based coaching? Are clarity and consensus more important?
Download Article 1K Club
