Assessing Your Tutoring Program Outline: Part 2

In our last blog post on assessment, we discussed operational objectives and student surveys. In this companion post, we’ll delve into student learning outcomes (SLOs), direct and indirect measures to assess SLOs, and program review. 

If you manage or direct a collegiate tutoring program, each of these will likely be asked of you at some point. Writing clear student learning outcomes and determining appropriate direct and measures to assess those outcomes will be critical to your success when reporting for program review or to your institution’s regional accrediting committee. 

What are they learning?

When assessing your tutoring program, you need to not only care about whether students are satisfied and value your services, but you also need to be able to show if they’re learning--and what they’re learning. Using course final grades is not an ideal measure for student learning in tutoring, since you can’t prove that tutoring was the key factor in the student earning their grade. However, if you identify specific student learning outcomes for students who attend tutoring, you can measure that. The key is to remember  that you want to develop student learning outcomes that can be measured and correlated directly to the work done in tutoring sessions. 

When writing student learning outcomes (SLOs), you’ll want to make sure that you’re considering all of the students impacted by your program--that means writing outcomes for both students who attend tutoring and your tutors. These SLOs should reflect what you expect students to learn by participating in tutoring - not necessarily what course content they’ll learn, but what skills or learning strategies they learn through tutoring. Consider what students should learn or “be able to” do at the conclusion of one or more tutoring sessions. To ensure you’re writing effective SLOs, you’ll want to ensure that you’re including behavior or actions that are both measurable and observable; if you can’t measure or observe the learning, then you don’t want to include it as an SLO. 

For students being tutored, consider SLOs that measure metacognition, learning strategies, self-efficacy, and other observable skills they could demonstrate in a tutoring session--or that they could self-report or reflect on in a survey. For your tutors, think about the skills or competencies that you want them to be learning and developing as tutors. One way to write tutor SLOs is to align them with things like the CAS Standards or NACE competencies. Additionally, you can think about including an SLO that illustrates the impact or importance of being a tutor. 

For both types of SLOs, your language should be clear and concise. Best practices recommends that SLOs include appropriate verbs from Bloom’s Taxonomy, across any of the six levels of understanding. In general, you should avoid using verbs and words like “understand,” “appreciate,” “think about,” and “demonstrate ability to,” as these are not easily measurable and don’t focus the assessment on actual learning taking place. It can be helpful to keep the ABCD model in mind when drafting your SLOs: Audience, Behavior, Condition, and Degree. Who is doing the learning? What do you expect them to learn or be able to do? Under what conditions will this learning occur? To what degree will they learn; that is, what is the acceptable standard to meet? 

A helpful formula for writing SLOs is “Students will [action verb] [something/observable behavior/action].” For instance, “As a result of attending four or more tutoring sessions, students will be able to articulate their preferred learning methods to the tutor.” This behavior can be observed and reported on by the tutor, and “articulate” is an action verb from the “Understand” level of Bloom’s Taxonomy. 

How are they learning? Direct measures

Once you determine what you think students and tutors should be learning through tutoring, you’ll need to determine how you’ll measure it. To ensure a comprehensive assessment of your program, we recommend including both direct and indirect measures for assessing your student learning outcomes. 

Direct measures allow you to measure learning by assessing samples of students’ actual work. These measures can include exam questions, papers evaluated with a rubric, presentations, portfolios, and observations of students. Direct measures are incredibly helpful for assessing student learning outcomes because they capture what students actually can do. Overall, direct measures are the best type of measure for assessing student learning outcomes, as they provide the most accurate representation of a student’s learning. You can use direct measures for assessing your SLOs for both students being tutored and the tutors--though the specific measures you use for each should vary by SLO. For evaluating your “student” SLOs, you can employ a variety of measures in tutoring sessions that won’t disrupt the work being done between tutor and student. 

One effective way to measure student SLOs is to incorporate an observation into the session. This observation can be as formal or informal as you like, so long as the learning is documented. Tutors could conduct observations during sessions, reporting on what the student does during the session. Or, a professional staff member could observe the session and make their own observations about the student’s learning. 

Using an observation as a direct measure for your tutor SLOs can also be effective. If you are the program coordinator or tutor supervisor, consider observing the tutors yourself to document behavior that aligns with your tutor SLOs. You could also include seasoned tutors in this process and have them observe new tutors as part of their own leadership development. 

Direct measures can be a little more difficult to integrate when assessing tutoring versus in a traditional classroom setting, but there are certainly ways to ensure effective, appropriate measurements as part of your assessment process. Dr. Karen Agee and Dr. Jan Norton present some of the challenges of identifying direct measurements of tutoring in their CRLA white paper on assessment, as well as outlining several quantitative and quantitative measures that can be effectively used for assessment. 

How are they learning? Indirect measures

In addition to direct measures, incorporating indirect measures into your assessment process will provide a comprehensive review of student learning. Unlike direct measures, indirect measures imply student learning, typically through self-reported information or reflection. These measures don’t necessarily demonstrate student learning, but they provide a less concrete view and implication of student learning. By gathering self-reported data from students themselves, you can use this information to draw conclusions about what students learned as a result of attending tutoring, as well as what your tutors learned as a result of being tutors. 

Possible indirect measures include student surveys, session evaluations, self-assessments, and even student retention data as discussed in our previous post. You can also combine SLO questions together with student satisfaction questions, as described in our last blog post. You can include specific questions that relate to your student learning outcomes in your regular session evaluation, or end-of-term student surveys, to gather this self-reported data from students. These can be objective, multiple-choice questions as well as more qualitative, open-ended questions. When reviewing the responses, you’ll want to aggregate and report on each question in isolation, so you can tie the responses directly to each specific SLO. 

For measuring your tutor SLOs, a great way to gather information from your tutors is through a tutor self-assessment. This indirect measure allows you to touch on a variety of questions that you may want to ask your tutors--ranging from administrative concerns to tutor training needs to addressing SLOs. As mentioned in our last blog post, the timing of this assessment is important so you can gather meaningful data from tutors, but you also want to be consistent in the timing of delivery. You can certainly employ a self-assessment more than once throughout the year, especially if you want to compare tutors’ responses and perceptions from fall to spring semester to note any change within the year. 

When incorporating indirect measures to assess student learning, it’s important to remember that they are most helpful to gauge perceptions that can then provide insight about a student’s learning, rather than illustrate the learning itself. Combining both direct and indirect measures will help you to provide the clearest, most compelling picture of student learning within the context of your tutoring program.

Next steps

Regardless of the type of measures you choose for your assessment, it’s crucial to make sure each measure is specific and appropriate for the corresponding learning outcome. You also want to be cautious of having too many measures. Sometimes, less is more. Strive for quality over quantity; having fewer measures that ask meaningful questions and provide insightful data is better trying to develop more measures that provide you less productive information. 

Developing a clear process for conducting assessment is also key. Determine a multi-year assessment cycle so you know when you’re delivering each measure; if you don’t want to assess each measure every semester, determine a cycle for when each measure will be administered and assessed. This will help provide consistent, longitudinal data that you can use when you need to conduct your institutional program review, if your school has non-academic units complete review as they do with academic programs. Many regional accrediting bodies are pushing for administrative program review, so familiarizing yourself with what’s involved in program review and what may be expected of you at your next accreditation site visit can help you as you build out or update your assessment process. 

Consider creating an annual report that includes your assessment data for that year and provide this to your administration. Gathering, analyzing, and reporting on your assessment data regularly will ensure that you always know the efficacy of your own program and you’ll be ready for whatever program review or accreditation requirements get thrown your way. Assessment isn’t particularly easy and it’s often time-consuming, but you’ll be the keeper of the facts and data, so you’ll be in charge of the narrative. Whether the audience is your own campus, your school’s leadership, prospective families, or an accrediting body, you and your team are the best ones to tell your program’s story. Let the data gathered help you tell that story. Let the students and your tutors help you tell their story. 

Curious how Knack can help you enhance your data collection processes to unlock new insights into your tutoring program? Fill out the form below, and a member of our team will be in touch.