SMART Goals and Anecdotal Evidence

A reader who works as a teacher wrote to me the other day:

I thought of your book once again this week, in an all day meeting of our school leadership team.  We spent the morning confirming that there is much more to educating young people than just generating marks, and that just focusing on numeric measures was the wrong way to go.  And then we spent the afternoon talking about nothing but numbers.  Stupid numbers.  Like the TDSB Director's vision to decrease the number of students needing Special Ed support by 50%.  (Maybe they just need extra vitamins.)  And stupid comparisons of unrelated data, like the achievement of one group of 14-year-olds vs the achievement of an entirely different group of 14-year-olds a year later.  Actually understanding statistics has become a curse!

The Toronto District School Board (TDSB) comment got me curious, so I looked on the web and found their Years of Action: 2013-2017 report.  This document promises 36 separate actions, resulting in 102 different outcomes - most of which have numeric measurements attached.  Wow.  102 targets that the TDSB can be directly scored on.

Such targets are often referred to as SMART goals - goals that are (S)pecific, (M)easurable, (A)cheivable, (R)elevant, and (T)imelined.  Since The Value Crisis talks about how that which cannot be measured is dismissed in this age of scientific thinking, I started to wonder whether that also means that unmeasurable goals therefore 'lack validity' in modern society.

I'm guessing there are three reasons for making a goal measurable.  The first is so that you can track progress and see if you are doing the right things.  The second is so that you will know when you have achieved your goal and can celebrate accordingly.  The third (more cynical reason) is for promises made in the public sphere, so that you can prove to others that, despite appearances, your target has been reached, according to objective measure.  (This might be where classic manipulations of statistics come into play.)

But what if your goal can't be measured so easily?  What if your aim is to be a better parent or to be happier?  The experts would say that you need some way of measuring this.  I say that's nonsense.  If you didn't need a measurement to identify the need for the goal in the first place, you don't need a measurement to know if you are experiencing success.

(I tried to come up with a new M-word for qualitative SMART goals.  So far, the best that I've come up with is "Motivational":  If you are motivated to continue what you're doing, you are likely moving towards the goal that you desired in the first place.  Please make better suggestions!)

So if you're not measuring your progress by some quantifying means, what can you use for feedback on your goal?  Well, you might go with anecdotal evidence.  <Insert dramatic organ chord.>  Anecdotal evidence!  The mere use of the term immediately raises the hackles on statisticians' necks.  Well, I'm here as the Devil's Advocate to suggest that anecdotal evidence still has plenty of value when drawing general conclusions.  On the other hand, the scientific method, when applied inappropriately, can actually the lower value of information.

I'll illustrate this with an old story (an anecdote!) of three professors traveling on a train through a foreign land.  The historian, looking out the window and seeing a black sheep grazing alone on the hillside, says: "How interesting!  The sheep of this land are black."  The statistician looks out and says: "Well, to be more correct, what we know is that in this part of the country there is one sheep that is black."  The philosopher, without looking up from his book, immediately corrects him: "Actually all we can say for certain is that in this specific area there is a sheep which is black on one side."

Had I been in the next seat, I might have added my own quip: "Excuse me, gentlemen, but I believe the only conclusion you can draw is that, looking out the window of this train, an object was observed that looked like what you know of as a sheep, whose wool (at least what you could see) appeared to be very dark in colour."  The historian's statement could clearly be misleading.  The statistician tried to make a more accurate (and thus, presumably, more valuable) statement.  However, further refinements reduced what began as a simple observation to a meaningless statement, of no practical value to anyone.  So where does anecdotal evidence cross the line from being a scientific menace to being meaningful and useful information?

As a scientist, I am fully aware of the abuse heaped upon anecdotal evidence - and rightly so, when it is used to 'prove' things that can only be proved by more rigorous analysis.  On the flip side, one might argue that almost everything we know, as individuals, comes from anecdotal evidence.  We know that last year's vacation was enjoyable or that last night's movie was boring because that was how we experienced them.  We know that flying insects with yellow-and-black-striped bodies can inflict pain, either through personal history or from the anecdotes of others.  That might not be 100% true every time, but it is certainly very useful information.  Indeed, at one time, the entire store of mankind's knowledge was anecdotal and communicated through story-telling.  You may say that it was naive and imperfect, but if I were stranded in the wilderness, I would take the knowledge of our Stone Age ancestors over that of your average physicist any day of the week.

The very things that make anecdotal evidence suspect - such as our innate propensity to focus on exceptional events (confirmation bias) can have their own value.  For example, one adult, telling a child that they are clever enough to do well in school, can alter that child's self-esteem, willingness to work hard, and ultimately improve their academic performance, even if many other adults are providing only discouragement.  (I heard about that happening once, so it must be true. ;-)  Even medical researchers have to concede the validity of the placebo effect, in which measurable outcomes can be altered by our own beliefs in anecdotal evidence.

So where am I going with all this?

Our brains are wired to make use of anecdotal evidence, and while we sometimes get it wrong, I'll wager that it is (more often than not) useful and valuable information.  A single anecdote, such as being out of breath after climbing some stairs, can suggest to us that perhaps we are out of shape and need to exercise more.  If a few more pieces of breathless anecdotal evidence confirm this, we might set a goal to improve our physical fitness.  If we start being more active because of this and find that we can now take the same set of stairs two steps at a time with ease, then we there is no need for science to measure whether or not we have improved our muscle tone or cardiac output.  It is sufficient and rewarding enough for us to feel we have succeeded in what we set out to do and be happier for the effort.

Similarly, in life, if it is our goal to be happier (and who wouldn't want that?), then we don't need measurable 'SMART' goals to tell us if we are doing the right things.  As for my teacher friend, while you might want quantifiable studies to alter policies for the entire board, that does not change the fact that in a specific classroom, for a specific child, anecdotal evidence is the best proof available that what you are doing is working, or not.  So why does number-based policy so often trump anecdotal evidence for individual circumstances?

P.S.  If classroom marks were the truest indicator of value and potential, I wouldn't be here.