8/6/19: test prep


Malcolm Gladwell has a podcast, “Revisionist History,” and I admire it very much. I have listened to every episode and delighted in the stories he has discovered and so clearly relishes telling. The one with the harmonica guy? Priceless! There’s an episode about how basketball players should shoot free throws underhand but refuse to do so out of some perverse sense of pride even when they are more successful when they do it. I remember listening to that episode while riding around New Zealand on a motorcycle and cackling insanely in my helmet as Gladwell got himself worked into a lather shouting about how Shaquille O’Neal could have been the greatest player of all time if he had just make a smart choice instead of a stupid one. (It does boggle the mind.)

Alas, Mr. Gladwell recently devoted two episodes to a topic with which I am all too familiar and, as much as it pains me to say this, he got it all wrong. For posterity, here’s everything I know about “standardized” tests, including a clarification of Mr. Gladwell’s misconceptions.

Gladwell took issue with the LSAT, the law school admission test. Specifically, he spent two episodes building an argument that the LSAT is a flawed assessment because it is time-constrained to an extent that “tortoises” have no chance at scoring well while the “hares” are able to race through the exam straight to a 180 and then right on into Harvard. To bolster his argument, he turns to the world of chess and to Hikaru Nakamura, a world-class chess player in general but specifically pretty much unbeatable at something called “bullet” chess, in which the entire game is limited to one minute of play. Unfortunately for Nakamura, the gold standard for chess wizardry is classical chess, which allows 165 minutes (almost 3 hours) for a game. Nakamura is ranked 11th in the world in classical chess, which is nothing to be ashamed of, but Gladwell gets him to admit the “arbitrary” decision to prize classical chess victories over those made in bullet chess has robbed Nakamura of a shot at of fame and fortune.

Gladwell takes this back to the world of the LSAT (and presumably other similar exams), arguing that time constraints are foolish as they crown different champions/change scores as the time allowed contracts or expands. The LSAT, in his view, rewards not the right answer, but the person who can get to it the fastest. And, since being fast requires different skills from being slow (cf bullet chess versus classical), slow-but-smart students are unfairly disadvantaged by this exam. What makes that especially vexing, in Gladwell’s opinion, is that the skills required to be a successful lawyer are the tortoise skills: being able to read closely and carefully and to take months to ponder and consider the various elements of a case. He speaks to former Supreme Court clerks to confirm this point, and concludes that the LSAT is, by definition, failing to identify the best future lawyers because it rewards exactly the wrong skills that are required for career success.

It’s a neat little argument and cleverly researched, but…it’s wrong.


Forget what the Law School Admission Council would tell you: the value of the LSAT is not that it can identify who will be a good lawyer (they do not claim that) or be predictive of first-year law school success (they do claim this). The value of the LSAT is that it can identify the people who are best at taking the LSAT.

The same is true of every other exam which has the answers on the page, which is any multiple-choice assessment. If the answer is there in front of you, hidden within three or four other, possible, plausible options, then the highest score goes to the person who can most quickly and efficiently ferret that answer out. Does that automatically mean that these tests reward hares? Maybe they have a leg up, but I would submit, based on more than 10 years of working in the test prep industry, that the single most important factor in a standardized test score is preparation.

And no, I’m not talking about having to hire a $500/hour tutor, but if that gives you peace of mind and you can afford it, then go for it. I’m talking about treating the test like a problem to be solved, or even like a game. Think about it: basketball is a timed game and each team has to follow a set of rules while maximizing the performance they can get from their players. Are the best basketball players the fastest runners? Nope: running speed might help, but the best players are the ones who can execute under pressure, and who can also see the whole court and visualize a play before it unfolds. Nakamura confirms this is true for chess: the longer the game, the more the ability to think ahead is valued, while the shorter the game, the more important intuition becomes. Have a minute? You can use it to think. Have 10 seconds? You must simply act. Few people are equally good at both, but as with most things in life, lawyering will require both thinking and acting, and hopefully one finds the career path that leverages his or her innate strength or preference. And the best shot one has at getting there is preparation, also known as hard work.

While I don’t take issue with Gladwell saying the LSAT isn’t fair, he’s saying it for the wrong reasons. As much as I am NOT a fan of the notion that “standardized” tests can say anything about us, other than how well we can take a “standardized” test, I will point out that doing well on these tests requires long-term, deep thinking as well as short-term quick action. And that’s not fair because that’s not what school is designed to teach—which is why test prep is a thing that matters. As far as I can tell, Gladwell “prepped” by meeting with John Katzman (a rather remarkable man who, among other things, was a cofounder of The Princeton Review) and while I am sure that was entertaining, a conversation with Katzman is not going to move the needle on the logical reasoning section.

I won’t go too far into the weeds on test-taking strategies (see below if you’re dying to know more) so suffice it to say they exist, they work, and the score you get will reflect exactly how well you mastered the getting of it. The LSAT is a test of how efficiently and effectively a test-taker can deploy their critical thinking skills on test day…and how carefully and thoughtfully they can execute those same skills in the practically unlimited time leading up to that day. In other words, it’s a test for all manner of animals…and we should absolutely put it in the garbage with all the other stupid constructs that purport to rank us each against the other.

As Gladwell’s fanciest lawyer friend pointed out, the cases the Supreme Court takes are there precisely because they are complicated and difficult and plenty of smart, experienced people have looked at the facts and drawn different conclusions. In other words, the answers are not on the page. The basket is moving, the value of a free throw in question—the chess players cannot see the same board. Games can be played because they have specific and clear rules; life doesn’t. Want to identify the best and the brightest, the ones who can succeed at this complicated, messy, thinking and acting thing we’re all doing all the time? Train them all to be that way.

A throw-away bit in one episode is Gladwell comparing the elite universities in America to those in Canada and making the point that one major difference between them is not in the quality of the students but the quantity. American relies on exclusion while Canada, quite sensibly, models inclusion. That’s why I’d vote for a unified theory of education reform that focuses on training people to think, not just hoping that having to take tests will accomplish that goal for them. Tests are silly artificial constructs, scavenger hunts for the brain: let’s not waste time getting better at playing those kinds of games. While I think his tongue was firmly in cheek when he suggested Gladwell’s Unified Theory for how to improve education was to move to a “Don’t ask, don’t tell” policy vis a vis scores and schools, I think there’s something there…if you flip that on its head. DO ask; DO tell—the best assessment of anyone’s ability to think and to act is to talk to them and to listen to what they say in return. Open the doors, start the conversation, invite the people in—don’t build dumb time-wasting walls to keep them out.

Finally, in case you have a test coming up, here’s how to do well on the LSAT—or any other test which has the answers on the page. (The ANSWERS are on the PAGE!

  1. Learn the rules of the test: what types of questions does it have? How much time will you have? What kind of score do you want to get? Use these inputs to make a plan for how you will tackle the test.
  2. Practice. I cannot emphasize this enough: studies just simply taking a test more than once will improve your score. Intentional and focused practice—the same kind of practice basketball or chess players do—will make you a better test taker. Remember, the answer is on the page: you need to practice finding it. There are only so many question types that are in existence—you can get better at knowing where the answers hide. Study the questions you answered incorrectly. If you can understand WHY you got a question wrong, you can understand what you need to do to get it right. (Or…you can decide not to bother to try and focus your energy on questions you know you can answer.)
  3. Get a book from the library and learn the strategies test prep companies espouse—they’re pretty much all the same and they’re pretty much all straight-forward and logical. Mostly, they’re about how to have that plan I mention above: take the test, the same way you would play a game.
  4. Get it over with and get on with your life. Like so many things in life (oh, the paperwork), figuring out how to take a test is probably going to be a necessarily evil you must face at some point in your life, either for yourself or with your children. Just remember this: a score says nothing about a person or their potential or what the future may hold: those are things each person gets to decide for their own self.

PS OMG I’m sorry I just worked in test prep for SO LONG: I must note here that success in law school does, in part, depend on one’s ability to think like a lawyer—to look at information in certain ways, to ask certain questions. The LSAT purports to measure this ability, hence why it can show a correlation to first-year law school grades. HOWEVER, I will submit to you that perhaps the connection there is really just that anyone who can train themselves to do well on the test can also train themselves to do well in law school…in other words, garbage. Toss it out and make a smarter decision, no matter how much it might sting someone’s pride.

Comments

  1. Fascinating! I also loved the version TAL did on this story about underhanded free throws. Oh the PRIDE!!

    ReplyDelete
    Replies
    1. pride & fear are two of my mortal sins--they go hand & hand and i'm trying very hard to stop inviting them places. granny shots for us all!

      Delete

Post a Comment