Web-based situational judgment tests (SJTs) present an outdated technology, which has been used by businesses for decades with no avail. Sadly, a handful of for-profit companies have started marketing them as a "novel solution". In such tests, applicants are shown hypothetical real-life situations in a video or written prompt and asked to indicate how they would respond. These tests are claimed to examine “non-cognitive” abilities like problem solving, decision-making and interpersonal skills. But, in our opinion, the evidence against the use of such tests is overwhelming for reasons we'll discuss below:
- Danger #1: Hypothetical Questions Lead to Hypothetical Applicants
Asking hypothetical questions, generally leads the applicants to provide socially acceptable responses and those with higher socioeconomic status normally do better on such tests as a result. This is called the “context principle”. Our behaviors are modified by the context of our environment. Therefore, asking applicants in a test setting how they would react to some hypothetical situation might not be of any value because the applicants know they are being tested and they know that their response has to be a response that brings a favorable outcome, such as being accepted to their program of choice. This means that the applicants might not necessarily act in the same way under real-life situations when there are no supervisions and test pressures. This is the same reason that someone who may be shy in one setting might be fully outgoing in another setting.
- Danger #2: SJTs Cause Cultural, Gender, & Socioeconomic Bias
Judging whether a response to a usually very delicate or stressful imagined scenario is appropriate can vary across cultures. These tests are singularly guided by accepted western cultural norms. This can pose significant challenges to non-native applicants or those who are immersed in another culture. The diverse cultural make up of the United States, the United Kingdom and Canada, for example, means that this is significant issue for applicants. For example, the New York Medical College (NYMC) reported that underrepresented minority applicants scored lower on such a computer-based situational judgment test compared to other applicants and that males scored lower than females creating a gender bias. Furthermore, the multiple mini interview (MMI), which is an offline SJT has been found to cause both socioeconomic bias and gender bias.
- Danger #3: SJTs are Highly Susceptible to Coaching
Related to this is a caveat present in all tests that claim to test non-cognitive skills, the applicant provides the answer they think the reviewers want to hear. Because these tests claim to be less vulnerable to subjective opinions of the interviewer means that an applicant can be more efficiently coached on the correct answer. One study showed that with appropriate coaching applicants can significantly increase their scores by 23-27%! Moreover, applicants who come from more advantaged socioeconomic backgrounds, on average, have stronger cognitive and non-cognitive skills than those from lower socioeconomic strata because of their economically-biased childhood experiences. It is a common opinion that SJTs may discriminate against socio-economically disadvantaged applicants that lacked the opportunity to develop or refine their non-cognitive skills as a result of their lesser social, economic, and cultural capital.
- Danger #4: SJTs Assume Certain Individuals are Born with Professionalism While Other Are Not
In a way, it seems that SJTs assume that people are born with certain personal and professional characteristics when claiming that they are "immune to test preparation". This is fundamentally incorrect; given that all personal and professional behaviors are learned behavior, it is a dangerous claim that can amount to discrimination against certain groups of individuals.
- Danger #5: SJTs are NOT a Valid Measure of Future On-the-job Behavior
SJTs have not been validated to correlate with actual on-the-job behavior and the best correlation found to date has been self reported by the creators of a for-profit company selling such products. In their pilot study, which was funded by public funds, the company founders report a mere low correlation of r = 0.3–0.5 between the computer-based SJT and test scores in future medical licensing examinations (the OSCE). Firstly, in our opinion, the study suffers from the law of small numbers and confidence over doubt bias because of its small sample sizes. Secondly, note again that a) the test is merely a predictor of future test performance not future on-the-job performance and that b) the correlation is weak at best and able to explain up to a maximum of 25% of the variance between the two variables. Imagine going to a doctor who claimed to be able to make a correct diagnosis only 25% of the time!
Here's what to do instead.
To your success,
Your friends at SortSmart
SortSmart Candidate Selection Inc.
