Developing evidence-based principles: An Exemplar
By Nora F. Murphy Johnson, PhD
Developing evidence-based principles is an emerging approach that is in contrast to the more traditional method of developing evidence-based best practices. Evidence-based practices and evidence-based principles make different assumptions. Best practice models assume that there is a best way to do things, regardless of context. Effective principles provide guidance for effective action in the face of complexity (Patton, 2010). They assume that while elements of the work are shared, there will necessarily be adaptations across settings and contexts.
People ask me, “How do you develop effective and evidence-based guiding principles?” There are many starting places, ending places, and paths in between. Here is one example.
The Setting
The Otto Bremer Foundation’s Youth Homelessness Initiative was a commitment by the Otto Bremer Foundation to fund six agencies (three emergency shelters, two youth opportunity drop-in centers, and one street outreach organization) in support of collaboration and improved outcomes for homeless youth at the systems level. This approach recognized that youth homelessness is a systems-level problem that cannot be addressed adequately by any one organization and that many homeless youths will access services from multiple agencies to meet their needs.
The Foundation and six agencies engaged in a collaborative principles-driven developmental evaluation to develop guiding principles to drive the systems change.
The Collaborative Process
The Otto Bremer Foundation convened leaders of the six organizations monthly for reflective practice. This group spent two years learning together about how they could improve outcomes for youth experiencing homelessness. What follows is a brief timeline of engagement and principles-development.
Summer 2012. Leaders of the six organizations collaboratively identified the nine shared principles based on a comparison of the six organizations’ existing missions, visions, values, and/or philosophy statements. The Collaboration sought to explore to what extent these principles were actually being implemented in practice. And, if implemented, explore to what extent the principles were meaningful and effective in meeting the needs of homeless youth.
October 2012. Up to this point, the collaborative of leaders, hereafter called “the group,” had been meeting with Michael Quinn Patton. Enter me, a very nervous graduate student! In October, the Collaboration and I had our first meeting to discuss the potential benefits and challenges of working with a graduate student on dissertation research. Ultimately, the Collaboration voted to work with me and approval was granted by The Otto Bremer Foundation’s Board of Trustees. My dissertation committee and the University of Minnesota’s Institutional Review Board gave permission for me to engage in this effort as my dissertation research. [See the published dissertation here: http://bit.ly/NMJ_Dissertation.]
November 2012. A multiple case study was selected as the primary method in this particular context because it provided a way to gather data and conduct analysis across various programs and settings. Sampling criteria were collaboratively developed, and leaders were asked to nominate youth who met those criteria. The down and dirty of the sampling criteria — the young people were successful by their own definition, and they had engaged with three or more of the agencies.
December 2012. We collaboratively identified fourteen young people Taken together; these fourteen young people represented the diversity of the population of youth the organizations served in relationship to race, gender, ethnicity, parenting status, foster care placements, and juvenile justice involvement.
January 2013. The group reviewed the interview protocols I drafted. There was a discussion about the type of data the interview protocol might yield, as well as how the questions might be interpreted by potential interviewees. One question was added, several were removed, and two were modified. The down and dirty of the interview protocol — young were invited to tell their stories in a way that made sense to them. They defined homelessness and what it meant to them. They chose which barriers, supports, strengths, and challenges to share.
I began conducting interviews in places that were selected and known by youth as being safe and non-judgmental. I made sure to schedule the interviews when a mental health practitioner was present.
February 2013. The group reviewed de-identified transcripts from the first few interviews so we could be sure the interview protocol was yielding useful information. We determined that it was and did not make changes.
March 2013. I wrote one young person’s story [These are called case studies in a research setting. This term feels disrespectful and inadequate. I chose to use the term case story]. The group read the completed story and worked in teams to code for evidence of principles. We discussed what we did and did not see in the data. We saw what principles look like in real life. The “evidence” we were collecting and the possibilities these principles would create felt more tangible.
April 2013. I wrote seven of the fourteen stories between the March and April meetings. In April, group members broke into small teams of three to four people. Each team was assigned three case stories to read. They were given instructions and asked to conduct a cross-case analysis. The group could see where their understandings of the principles converged or diverged. They were able to make connections across the principles, their work, and the young peoples’ experiences.
May 2013. I wrote the remaining case stories and completed an independent analysis of several of the principles between the April and May meetings. Group members reviewed my write-up of one principle — trusting youth/adult relationships. We discussed where we felt there was good evidence, weak evidence, reinforced learning, and surprising outcomes.
June 2013. I wrote draft summaries of the evidence I found — or didn’t — for all nine principles by the June meeting. Group members had time to read all of the principles and draft evidence descriptions. We focused our discussion and art-based activity on surfacing interrelationships between principles.
July 2013-August 2013. The Collaboration gave feedback on the findings and conclusions. I wrote the formal dissertation, and they wrote a more use-friendly overview of the process and findings.
September 2013. The Collaboration finalized principle statements and wrote a one-paragraph explanation of and introduction to the principles.
October 2013. The group drafted a dissemination package for use by agencies. The package explains each principle, its research base, and practice implications.
November 2013. The Collaboration reviews the final draft of the dissemination package
December 2013. The Collaboration officially adopts revised principles based on the dissertation research and accompanying literature review.
January 2014-Now. The group members use the principles in different ways to impact work within their own agencies and to impact collaborations with other agencies across various sectors.
Learn More
Curious to learn more about this process and the outcomes? We have more on our resource page. Check it out. https://www.inspire-to-change.org/resources
