Misconceptions about Social and Emotional Learning – Misconception One, Part Two

Misconception #1: “You can’t measure social and emotional skills”

Part 2: Reliability and How to Select an SEL Assessment

Main Author:  Cristina Anguiano-Carrasco, Ph.D
Supporting Author: Dana Murano, Ph.D
ACT Center for Social, Emotional, and Academic Learning

This post is the second in a series about misconceptions we often hear about social and emotional learning (SEL). In each post, we tackle a common confusion about SEL and attempt to clarify some misconceptions. In our last post, we addressed the misconception that social and emotional skills can’t be measured by discussing operational definitions and examining different sources of validity evidence. This post will focus on the second corner stone of psychometrics: Reliability.

 

Precision and Consistency: Reliability

While validity is about accuracy, reliability is about precision. Reliability refers to the consistency with which we are measuring a construct. Think of it like a bowler that always hits a strike, Every. Single. Time! We can gather reliability information over time (i.e., test-retest reliability), and across items (i.e., internal consistency). Test-retest reliability answers the question “If I take the same assessment twice, would I get the same results?” So, if I measure my teamwork skills today and again tomorrow, I should get very similar results both times (if nothing has happened in my life during the last 24 hours to increase or decrease my teamwork skills). That is test-retest reliability.

Internal consistency reliability is the extent to which the items (questions) in an assessment are related. Although we expect a certain level of variability across items to be able to capture all the different aspects of a skill, all items must have something in common: the skills we are measuring. Therefore, when measuring teamwork, a response to the item “I’m kind to others” should be consistent with a response to the item “I like to contribute to the work of my team”, which should be consistent with the response to the item “I enjoy working with others”, and so on. A reliable assessment is one that you can trust, knowing that whatever is being measured is being measured with precision, like a good focused picture where you can clearly see what’s in it.

Putting Validity and Reliability Together to Choose an Assessment of SEL

Putting together both pillars of psychometrics, a good assessment is one that is valid (it accurately measures the construct) and reliable (it measures the construct with precision).

So now that you know the most important characteristics of a scientifically sound assessment, you are now ready to search for a social and emotional skills assessment. This can be a bit overwhelming, so a few tips are provided below.

  1. What do you want to measure? Why do you want to measure it?

Before you start comparing and contrasting assessments, it is important that you have a clear idea of what information you want to get from the instrument. What do you want to measure? And why are you measuring it? For example, a teacher may want to know how effective their social skills training was for their classroom, while a counselor may want to know if a student’s difficulty socializing with others may require an intervention. The teacher will get the information needed by using an assessment that measures all aspects of social skills (broader definition of the construct) with an acceptable level of reliability. However, the counselor will be better served by an assessment that provides very accurate measurement of specificaspects of the skill (e.g., the aspects that severely interfere in the student’s daily life).

  1. What is the assessment measuring?

The next step will be to look at the description of the assessment, definitions of the skills, and technical documentation of each assessment you are considering. As discussed in our first post, the operational definition is what allows us to measure the skill (e.g., teamwork), and each construct can have multiple definitions. Therefore, your first step will be to examine the definitions of the constructs to ensure that the assessment intends to measure the construct you are interested in. In the SEL world jingle-jangle fallacies are common; sometimes two skills are different but have the same name (jingle fallacies), whereas other times two skills have different names but are actually the same thing (jangle fallacies). The point is not to get too hung up on the names. Use definitions instead.

  1. Find validity and reliability evidence

When considering the use of an assessment we encourage you to look into the technical documentation and find data-based evidence of accuracy and precision. One example is the ACT® Tessera® technical manual, where you will find detailed definitions and all the research behind the assessment. Will the assessment provide you with an accurate and precise picture of the construct?

Easier said than done, right? Unless you have spent a few hours in higher ed learning about psychometrics, it can be overwhelming… but fear not, we are here to help! Below we provide a list of warning signs classified on caution level to help you differentiate a scientifically valid assessment from fluff.

High level warnings

  • Definitions of what is being measured are not provided.
  • The questions and/or the content does not make sense (e.g., What do pizza toppings have to do with teamwork?).
  • Technical documentation or data related to the assessment properties cannot be found.

If you see all of these warning signs, you probably want to keep looking for a more scientifically grounded assessment.

Medium level warnings

  • Definitions are provided but are not clear, combine several skills into one, or use unnecessarily complex words (supercalifragilisticexpiali-what-now?).
  • Studies and/or data are mentioned but cannot be found.
  • There are case studies or testimonials available, but no data are presented.

If you see some of these warnings, but not all of them, you may want to dig a little deeper to get a better sense of the scientific foundation of the assessment. However, there’s a good chance this is not the assessment for you.

Low level warnings

  • The definition provided does not match perfectly with the skill you want to measure (but what does perfect really mean?).
  • Data is available but results are not clear or are very difficult to interpret.
  • Some studies are presented but they are not published in scientific journals.

Psychometric results can be difficult to simplify, and peer reviewed publications can take years! Check on the credentials of those developing the assessment or contact the assessment owner and ask for clarification before making your final decision.

Now you have the confidence that social and emotional skills can be measured, and all the tools in your belt to begin your search for an assessment that fits your needs. The team at ACT SEL is happy to chat with you about our products and the research behind them.

Start your search here:

And check back next week when we dig into the teachability of social and emotional skills.

Cristina Anguiano-Carrasco, PhD

Cristina Anguiano-Carrasco, PhD

Cristina Anguiano-Carrasco is a senior research scientist in the Center for Social, Emotional, and Academic Learning (SEAL). Her research focusses on the measurement, assessment development, and associated response biases of Social and Emotional skills.