Evaluation, Consulting, Research

MacCallum Ross is a collaborative and agile team of professionals with extensive knowledge and first-hand experience in the education, nonprofit and public sectors. Our objective is to provide a deep level of understanding about the issues that matter most to you. To this end, we help articulate your questions, design innovative and tailor-made strategies to answer these questions, and deliver customized, evidence-based, sustainable solutions.

 

Capabilities

Complex problem solving.  We approach problems by working with you to develop a clear and concise understanding of the exact issue. From there we focus on obtaining the data needed for decision-making. While our capabilities fall, broadly, into the following four categories, take a look at some of our case studies to see our approach at work.

All evaluation requires a strong measurement strategy. What you measure and how you measure it determines the quality of the insights you obtain. We start from a clear understanding of what you want to gain from your evaluation. We develop an evaluation strategy and the measurement tools to be used.  Providing content knowledge and clearly communicating results are our forte.

While evaluation is sometimes viewed as an added source of work, taking time away from essential daily activities, it shouldn’t be. We develop evaluation strategies that focus on building staff capacity while incorporating evaluation activities into daily routines. We believe that good evaluations enhance stakeholder involvement, promote skills that are beneficial to the organization and facilitate organizational learning. Engaging stakeholders in the process supports that learning and leads to better use of evaluation results.

Demonstrating successful outcomes often drives the need for evaluation, but answers to other concerns — what are the program’s particular strengths and weaknesses? is it being implemented properly? — are sometimes needed as well. What’s do you want to learn from your evaluation?   We work closely with clients to develop a concise understanding of what they need to gain from an evaluation and this first, important step allows us to then develop the best evaluation and measurement strategy.

Often an organization will have a single data point — perhaps an old participant satisfaction or exit survey — that they have collected over time.  We specialize in developing evaluation strategies that build on the data an organization currently has allowing our clients to build on that knowledge. We’ve also worked with organizations that have collected a large amount of data over the years but have never examined it. Mining that information and for answers to your questions is our specialization.

In the absence of the right information to answer your questions, we can help design valid and reliable data collections tools along with a successful evaluation strategy. Among the tools we’ve designed for evaluations we’ve implemented include focus groups, surveys, interviews, performance assessments, content knowledge tests and patron tracking.

Since we believe that the purpose of an evaluation is to help you understand how well your organization is achieving its goals, to help you make strategic decisions and to take informed action, we place a premium on clear communication, not only of result but during every step along the way.

read more

We work with organizations to improve their effectiveness by both building capacity and determining the best methods for allocating their resources. Beginning with our own research on peer best practices, we identify opportunities to improve effectiveness and turn those opportunities into lasting improvements through rigorous change management.

We begin with a realistic and thorough assessment of current capabilities and resources, placing your organization in the context of your peers to create a detailed understanding of how you can be more effective. We then evaluate your organization’s structure and governance model, and implement a plan for change management that takes into account the current state of the organization and supports significant change.

Sometimes the organizations we work with on effectiveness are large. We were asked by a state higher education agency to analyze statewide teacher preparation programs and determine whether they were producing the right number and kinds of qualified teachers. We developed a survey-based analysis of teacher hiring practices, conducted research on current and future supply and demand of teachers and determined alignment of the teacher preparation programs with best practices.

At other times the focus is on one key function in an organization. We evaluated a state higher education accountability system by developing and holding focus groups with all key stakeholders. Our analysis of the qualitative data from these meetings formed the basis of our recommendations for improving the effectiveness of this accountability system.

Building capacity in support functions is another aspect of organizational effectiveness. In working with an educational institution we identified key performance metrics for all of its non-instructional departments. A central feature of this project was the development of a reporting system for the metrics that has proven to be critical for the change management needed to make these departments more effective.

read more

MYTH: Surveys are easy to develop, can answer most any question and provide reliable results. In FACT, they require knowledge and skill to develop, typically address questions concerning attitudes and opinions, and since they rely upon self-reporting, can be unreliable. Because of this, understanding results requires careful analyses and interpretation.

We begin our survey work by spending time coming to understand what you need to know. For one client, we interviewed representatives of all the major stakeholder groups that the survey was intended for. We’ve also spent time researching issues and talking with other organizations that have carried out similar surveys.

From here, we develop and pilot your survey. Results from the pilot in the form of feedback as well as statistical analyses help us to refine survey items . Our goal is to ensure that you will be working with a valid and reliable survey instrument.

Our survey services allow you to be as involved in the process as you would like. While one client hired us just to develop a valid and reliable instrument for their on-going use, we are more typically engaged in every stage of the project – from development, printing and mailing, reminder mailings and data collection. And, of course, our team specializes in (and thrives on) survey data analysis and reporting.

read more

Successful strategic planning requires an objective view of the facts, the genuine engagement of key stakeholders and a disciplined process. We work collaboratively with organizations to develop strategic goals, the detailed actions needed to reach those goals and clear measures of progress to ensure that the strategy is carried out.

Strategic planning ranges from long-term plans for an entire organization to more short-term or focused initiatives. Whatever the scope of the planning, we begin by working with the organization’s leadership to define a detailed planning process, including timelines and responsibilities for deliverables. We then collect relevant information about the organization, its peers and the external environment and use stakeholder input to identify strategic issues.

A long-term strategic plan requires significant time and effort and a detailed process is needed to complete the plan successfully. As part of developing a yearlong planning process for a liberal arts college, we held a one-day workshop for the leadership of the college. It provided key features of the entire planning process, methods for collecting and analyzing internal and external data, approaches to engaging stakeholders and details on how to develop strategic goals and action plans.

In our strategic planning work we are occasionally asked to help out in a more focused way with a strategic planning process already underway. This was the case when we were asked to develop a business plan for an educational nonprofit to support its efforts to raise funds for its national expansion.

Strategic plans are often at risk because insufficient attention is given to implementation. We have worked with organizations that needed additional support to be able to implement plans. Our work in this area has ranged from a US division of an international organization with a large, complex strategic plan to a small consortium of non-profits with a simple strategic plan and limited revenue.

read more

Evaluation & Measurement

All evaluation requires a strong measurement strategy. What you measure and how you measure it determines the quality of the insights you obtain. We start from a clear understanding of what you want to gain from your evaluation. We develop an evaluation strategy and the measurement tools to be used.  Providing content knowledge and clearly communicating results are our forte.

While evaluation is sometimes viewed as an added source of work, taking time away from essential daily activities, it shouldn’t be. We develop evaluation strategies that focus on building staff capacity while incorporating evaluation activities into daily routines. We believe that good evaluations enhance stakeholder involvement, promote skills that are beneficial to the organization and facilitate organizational learning. Engaging stakeholders in the process supports that learning and leads to better use of evaluation results.

Demonstrating successful outcomes often drives the need for evaluation, but answers to other concerns — what are the program’s particular strengths and weaknesses? is it being implemented properly? — are sometimes needed as well. What’s do you want to learn from your evaluation?   We work closely with clients to develop a concise understanding of what they need to gain from an evaluation and this first, important step allows us to then develop the best evaluation and measurement strategy.

Often an organization will have a single data point — perhaps an old participant satisfaction or exit survey — that they have collected over time.  We specialize in developing evaluation strategies that build on the data an organization currently has allowing our clients to build on that knowledge. We’ve also worked with organizations that have collected a large amount of data over the years but have never examined it. Mining that information and for answers to your questions is our specialization.

In the absence of the right information to answer your questions, we can help design valid and reliable data collections tools along with a successful evaluation strategy. Among the tools we’ve designed for evaluations we’ve implemented include focus groups, surveys, interviews, performance assessments, content knowledge tests and patron tracking.

Since we believe that the purpose of an evaluation is to help you understand how well your organization is achieving its goals, to help you make strategic decisions and to take informed action, we place a premium on clear communication, not only of result but during every step along the way.

Organizational Effectiveness

We work with organizations to improve their effectiveness by both building capacity and determining the best methods for allocating their resources. Beginning with our own research on peer best practices, we identify opportunities to improve effectiveness and turn those opportunities into lasting improvements through rigorous change management.

We begin with a realistic and thorough assessment of current capabilities and resources, placing your organization in the context of your peers to create a detailed understanding of how you can be more effective. We then evaluate your organization’s structure and governance model, and implement a plan for change management that takes into account the current state of the organization and supports significant change.

Sometimes the organizations we work with on effectiveness are large. We were asked by a state higher education agency to analyze statewide teacher preparation programs and determine whether they were producing the right number and kinds of qualified teachers. We developed a survey-based analysis of teacher hiring practices, conducted research on current and future supply and demand of teachers and determined alignment of the teacher preparation programs with best practices.

At other times the focus is on one key function in an organization. We evaluated a state higher education accountability system by developing and holding focus groups with all key stakeholders. Our analysis of the qualitative data from these meetings formed the basis of our recommendations for improving the effectiveness of this accountability system.

Building capacity in support functions is another aspect of organizational effectiveness. In working with an educational institution we identified key performance metrics for all of its non-instructional departments. A central feature of this project was the development of a reporting system for the metrics that has proven to be critical for the change management needed to make these departments more effective.

Survey Research

MYTH: Surveys are easy to develop, can answer most any question and provide reliable results. In FACT, they require knowledge and skill to develop, typically address questions concerning attitudes and opinions, and since they rely upon self-reporting, can be unreliable. Because of this, understanding results requires careful analyses and interpretation.

We begin our survey work by spending time coming to understand what you need to know. For one client, we interviewed representatives of all the major stakeholder groups that the survey was intended for. We’ve also spent time researching issues and talking with other organizations that have carried out similar surveys.

From here, we develop and pilot your survey. Results from the pilot in the form of feedback as well as statistical analyses help us to refine survey items . Our goal is to ensure that you will be working with a valid and reliable survey instrument.

Our survey services allow you to be as involved in the process as you would like. While one client hired us just to develop a valid and reliable instrument for their on-going use, we are more typically engaged in every stage of the project – from development, printing and mailing, reminder mailings and data collection. And, of course, our team specializes in (and thrives on) survey data analysis and reporting.

Strategic Planning

Successful strategic planning requires an objective view of the facts, the genuine engagement of key stakeholders and a disciplined process. We work collaboratively with organizations to develop strategic goals, the detailed actions needed to reach those goals and clear measures of progress to ensure that the strategy is carried out.

Strategic planning ranges from long-term plans for an entire organization to more short-term or focused initiatives. Whatever the scope of the planning, we begin by working with the organization’s leadership to define a detailed planning process, including timelines and responsibilities for deliverables. We then collect relevant information about the organization, its peers and the external environment and use stakeholder input to identify strategic issues.

A long-term strategic plan requires significant time and effort and a detailed process is needed to complete the plan successfully. As part of developing a yearlong planning process for a liberal arts college, we held a one-day workshop for the leadership of the college. It provided key features of the entire planning process, methods for collecting and analyzing internal and external data, approaches to engaging stakeholders and details on how to develop strategic goals and action plans.

In our strategic planning work we are occasionally asked to help out in a more focused way with a strategic planning process already underway. This was the case when we were asked to develop a business plan for an educational nonprofit to support its efforts to raise funds for its national expansion.

Strategic plans are often at risk because insufficient attention is given to implementation. We have worked with organizations that needed additional support to be able to implement plans. Our work in this area has ranged from a US division of an international organization with a large, complex strategic plan to a small consortium of non-profits with a simple strategic plan and limited revenue.

 

Case Studies