What the Best College Teachers Do: A Review of a Vexing Book

What the Best College Teachers Do is not a bad book. It is engaging and reasonably well-written. The topic is both evergreen and timely, and certainly of interest to college teachers at the very least (as well as to people who rate college quality and to people who use those ratings to decide where to go to school). My issue with this book is that it is incapable of answering the question it sets out for itself.

A problem of comparison


The book is based primarily on extensive research by the author, Ken Bain, and his colleagues. The appendix spells out in detail how they identified good college teachers (a combination of student evaluations, examples of student work, department examinations, etc.) and how they collected information about those gifted individuals (interviews, taped class sessions, course materials, etc.). They analyzed these data to determine what these best college teachers did.

Even assuming that (a) their methods successfully identified superior teachers, and (b) they collected the right information about those teachers' practices, this is only half of a study. Without even looking at their data, I can easily rattle off some things all these teachers had in common:

1. They were all human beings.
2. They were all taller than 17 inches.
3. They all spoke English, at least to some degree (the study was conducted in the USA).
4. Most were either male or female.

Commonalities are not limited to attributes of the teachers, but also to what they do in the classroom:

5. Most showed up to at least half of the class periods for a given course.

6. None of them habitually sat, silent and unmoving, at the front of the classroom for the duration of class.
7. They did not assign arbitrary grades to their students (e.g., by rolling dice).
8. Very few spoke entirely in blank verse.

While these statements are almost certainly true of good college teachers, they do not distinguish the good teachers from the bad ones. Since Bain and colleagues did not include a comparison group of bad teachers, we cannot know if their findings distinguish the good teachers from the bad ones.

Science -- like teaching -- requires training


A good test of teaching ability should pick out all the good teachers. It should also pick out only the good teachers. (A somewhat different cut of the issues is to consider test reliability and test validity). What the Best College Teachers Do focuses entirely on the first issue. As my reductio ad absurdum above shows, having only half of a good test is not a test that is 50% right; it's a useless test.


It's unfortunate that Bain and his colleagues failed in this basic and fundamental aspect of scientific inquiry. Although Bain is now the director of the Center for Teaching Excellence at New York University, he was trained as a historian. This comes out in the discussion of the study methods: "Like any good historians who might employ oral history research techniques, we subsequently sought corroborating evidence, usually in the form of something on paper..." (p. 187).

I would hope that any good historian doing comparative work would know to include a comparison group, but designing a scientific study of human behavior is hard. Even psychologists screw it up. And that's the focus of our training, whereas historians are mostly learning things other than experimental design (I assume).

Circular Definitions

Of course, failing to include a control group is not the only way to ruin a study.You can also make it circular.

Chapter 3 focuses on how excellent teachers prepare for their courses:

At the core of most professors' ideas about teaching is a focus on what the teacher does rather than on what the students are supposed to learn. In that standard conception, teaching is something that instructors do to students, usually by delivering truths about the discipline. It is what some writers call a 'transmission model.' ... 

In contrast, the best educators thought of teaching as anything they might do to help and encourage students to learn. Teaching is engaging students, engineering an environment in which they learn.

Here is what the appendix says about how the teachers were chosen for inclusion in the study:

All candidates entered the study on probation until we had sufficient evidence that their approaches fostered remarkable learning. Ultimately, the judgment to include someone in the study was based on careful consideration of his or her learning objectives, success in helping students achieve those objectives, and ability to stimulate students to have highly positive attitudes toward their studies.

It seems that perhaps teachers were included as being "excellent teachers" if they focused on student learning and on motivating students. The researchers then "found" that excellent teachers focus on student learning and on motivating students.

Vagueness and Ambiguity


Or maybe not. I'm still not entirely sure what it means to -- in the first quote -- focus on "what the teacher does" than on "what the students are supposed to learn." For instance, Bain poses the following thought problem on page 52:

"How will I help students who have difficulty understanding the questions and using  evidence and reason to answer them."

Is that focusing on what the teacher does or focusing on what the students are supposed to learn? How can we tell? By what metric?

My confusion here may merely mark me was one of those people expecting "a simple list of do's and don'ts" who are "greatly disappointed." Bain adds (p. 15), "The ideas here require careful and sophisticated thinking, deep professional learning, and often fundamental conceptual shifts." That's fine. But if there is no metric I can use to find out whether I'm following these best practices or not, what good does this book do me?

(Also, without knowing what exactly Bain means by these vague statements, there is no way to ensure that his study wasn't circular, as described in the previous section. I gave only one example, but the general problem is clear: Bain defined great teachers by one set of criteria and then analyzed their behavior in order to extract a second set of criteria. If both sets of criteria are loosely and vaguely defined, there's no way even in principle to know whether he isn't just measuring the same thing both times.)

Credible Reviews

So if we don't trust Bain's study, is there anything else in the book worth reading? Maybe. What the Best College Teachers Do is not myopically focused on Bain's own research. He reviews the literature, citing the conclusions from other studies of teaching quality, broadening the scope of the framework outlined in the book. However, this raises its own problem.

In writing a review, the reviewer is supposed to survey the literature, find all the relevant research, determine what the best research is, and then synthesize everything into a coherent whole (or at least, into something as coherent as the current state of the field allows). The reviewer generally does not describe the studies in sufficient detail to allow the reader to evaluate them directly; only a brief overview is provided, with a focus on the conclusions.

If you trust the reviewer, this is fine. That's why reviews from the most respected researchers in the field are typically highly valued, so much so that publishers and editors often solicit reviews from these researchers. Obviously, a review of the latest research on underwater basket weaving by a fifth-grader would not be so highly prized, because (a) we don't believe the fifth-grader did a particularly thorough review, and (b) we don't trust the fifth-grader's ability to sort the wheat from the chaff -- that is, identify which studies are flawed and which are to be believed.

Bain is clearly very smart. He has clearly read a lot. But I do not trust his ability to read scientific literature critically. The only evidence I have of his abilities is in the design of his own study, which is deeply flawed, as described above. If he can't design a study, why should I trust his analysis of other people's studies?

Building a better mousetrap


Criticizing a study is easy, but it's not much of a critique if you can't identify what a better study would look like. Clearly from my discussion above, I would want (a) clear criteria for defining good teaching, (b) clearly-defined measures of teacher behavior, and (c) a group of good teachers and a group of bad teachers for comparison, and probably a group of average teachers as well (otherwise, any differences between good and bad teachers could be driven by bad habits of the bad teachers rather than good habits of the good teachers).

After a set of behaviors that are typical of good teachers -- and which are less frequent or absent in average or bad teachers -- has been identified, one would then identify a new group of good, average, and bad teachers and replicate the results. (The risk is otherwise is one of over-fitting the data: the differences you found between good teachers and the rest were just the result of random chance. This actually happens quite a lot more than many people realize.)

At the end of this process, we should have a set of behaviors that really are particular to the best teachers, assuming that the criteria we used to define good teachers are valid (not an assumption to be taken lightly).

Becoming a good teacher

Whether or not this information would be of any use to those aspiring to be good teachers is unclear. To find out that, we'd actually need to do a controlled study, assigning one set of teachers to emulate this behavior and another set to emulate behavior typical of average teachers. Ideally, we'd find that the first group ended up teaching better. I'm unsure whether that's particularly likely to happen, for a number of reasons.

First, consider Bain's summary of the habits of the best teachers (summarizing, with some direct quotations, from pps. 15-20):

1. Outstanding teachers know their subjects extremely well.
2. Exceptional teachers treat their lectures, discussion sections, problem-based sessions, and other elements of teaching as serious intellectual endeavors as intellectually demanding and important as their research and scholarship.
3. They avoid objectives that are arbitrarily tied to the course and favor those that embody the kind of thinking and acting expected for life.
4. The best teachers try to create an environment in which people learn by confronting intriguing, beautiful, or important problems, authentic tasks that will challenge them to grapple with ideas, rethink their assumptions, and examine their mental models of reality.
5. Highly effective teachers tend to reflect a strong trust in students.
6. They have some systematic program to assess their own efforts and to make appropriate changes.

Much of this list looks like a combination of intelligence and discipline. That is clearly true for #1, and probably true for #2 and #3. To the extent that #4 is hard to do, it probably takes intelligence. And #6 is just a good idea, more likely to occur to smart people and only pulled off by disciplined people. I'm not sure what #5 really means.

If the key to being a good teacher is to be smart and disciplined, this news will be of little help to teachers who are neither (though it may be helpful to people who are trying to select good teachers). In other words, even if we determine what makes a good teacher, than doesn't mean we can make good teachers.

The best teachers

Of course, even if the strategies that good teachers use are ones you can use yourself, that doesn't mean you can use them correctly.

There is an old parable about two young women. One was exceptionally beautiful. She used to sit at her window and gaze out over the field, looking forlorn and sighing with melancholy. Villagers passing by would stop and stare, struck by her heavenly beauty. One such villager was another young woman, who was the opposite of beautiful. Nonetheless, on seeing this example, she went home, sat at her own window, gazed out over the field and sighed. Someone walked by, saw her, and promptly vomited.

Objectification of female beauty and strange fetishization of melancholy aside, the point of this parable is that just because something works for someone else doesn't mean it'll work for you. When I think about the very best teachers I've known, one thing that stands out is how idiosyncratic their methods and abilities have been. One is a high-energy lecturer who runs and jumps during his lectures (yes, math lectures), who is somehow able to turn linear algebra into a discussion class. Another, in contrast, faded into the background. He rarely lectured, preferring to have students work (in groups or individually) on carefully-crafted questions. A third is a gifted lecturer and the master of the anecdote. While others use funny anecdotes merely to keep a lecture lively, when he uses an anecdote, it is because it illustrates the point at hand better than anything else. Over at the law school, there are a number of revered professors famous for their tendency to humiliate students. This humiliation serves a purpose: to show the students how much they have to learn. The students, rather than being alienated, strive to win their professors' approval.

These methods work for each, but I can't imagine them swapping styles round-robin. Their teaching styles are outgrowths of their personalities. Many are high-risk strategies, which if they fail, fail disastrously (don't humiliate your students unless you have the right kind of charisma first).

Are there strategies that will work for everyone? Is there a way of determining which strategies will work for you, with your unique set of strengths and weaknesses? I'd love to find out. But it won't be by reading What the Best College Teachers Do.

New York Times, You Can't Handle the Truth.

Earlier today I wrote about the research behind an opinion article at the New York Times. When I looked at the sources cited, I was unable to find any information supporting the claims made in the article. In fact, what I found directly contradicted those claims. I finished by saying that while I was willing to believe these claims, I'd like to know what data support them. In passing, I mentioned that I had submitted an abbreviated version of this analysis as a comment on the Times website.

That comment was not published. I figured maybe there had been a computer error, so I submitted another one later in the day. That one was also not published. Finally, at 6:13pm, I submitted an innocuous and useless comment under an assumed name:

I agree with Pat N. It's nice to hear from someone who has some optimism (@ Dr. Q).

This comment was published almost immediately.


The Times states that "comments are moderated and generally will be posted if they are on-topic and not abusive."Since the moderators didn't publish the comment, we can conclude one of two things:

1) Discussion of the empirical claims made in a New York Times article is not "on topic."
2) Pointing out a mistake made in a New York Times article is a kind of abuse.

Do students at selective schools really study less?

*Updated with More Analysis*


So says Philip Babcock in today's New York Times. He claims:

Full-time college students in the 1960s studies 24 hours per week, on average, and their counterparts today study 14 hours per week. The 10-hour decline is visible for students from all demographic groups and of all cognitive abilities, in every major and at every type of college.

The claim that this is true for "every type of college" is important because he wants to conclude that schools have lowered their standards. The alternative is that there are more, low-quality schools now, or that some schools have massively lowered their standards. These are both potentially problems -- and are probably real -- but are not quite the same problem as all schools everywhere lowering their standards.

So it's important to show that individual schools have lowered their standards, and that this is true for the selective schools as well as the not-selective schools. The article links to this study by Babcock. This study analyzes a series of surveys of student study habits from the 1960s to the 2000s, and thus seems to be the basis of his argument, and in fact the introduction contains almost the identical statement that I have quoted above. Nonetheless, despite these strong conclusions, the data that would support them appear to be missing.

SAT scores and size are not available in the early years, so study time by college selectivity is not reported. 

He goes on to say that he can look at selectivity in the more recent surveys: specifically matched 1988-2003 surveys. These do show a decrease in study time from on the order of 1-2 hours for high-, medium- and low-selectivity schools (I cannot find how selectivity was defined). Whether this is even statistically significant is unclear, as he does not report any statistics or confidence intervals. In any case, it is not a 10 hour difference.

What Babcock might have meant, and more problems with the data

It is possible that when Babcock was saying that the decrease in study time was true of all types of schools, he meant that when you look at all types of schools in 2003/4, students at all levels report studying less than the average student reported in 1961. The problem is that, for all we know, the schools in his sample were more selective in 1961 than they were in 2003/4.

Moreover, the is something worrisome about his selectivity data. Whenever analyzing data, many researchers like to do what is called a "sanity check": they make sure that the data contain results that are known to be true. If you were looking at a study of different types of athletes, you might make sure that the jockeys are shorter than the basketball players, lighter than the football players and chew less tobacco than the baseball players. If you find any of these things do not hold, you might go back and make sure there isn't a type-o somewhere in your data-entry process.

I worry that Babcock's data fail the sanity check. Specifically, look at the number of hours studies according to selectivity of school in 2003:

highly selective: 13.47 hours
middle:               14.68 hours
non-selective:     16.49 hours

Note that this effect is larger than the decline in number of hours studied between 1988 and 2003, so in terms of this dataset, this is a large effect (again, I cannot tell if it is significant, because the relevant statistical information is not provided) and it's not in the direction one would think. I will admit that it is possible that students at highly selective schools really do study less than the folks at JuCo, but that conflicts heavily with my pretty extensive anecdotal database. So either a) the world is very different from how I thought it was -- in which case, I want more evidence than just this survey -- b) Babcock has defined selectivity incorrectly, or c) there is something wrong with these data.

One last worrisome fact

I considered the possibility that the data Babcock was quoting were in a different paper. The only other paper on Babcock's website that looked promising was this American Enterprise Institute report. This is not a research paper, but rather summarizes research. Specifically, according to footnote #2, it summarizes the research in the paper I just discussed. Strangely, this paper does have a graph (Figure 4) breaking down study habits of students in the 1960s based on selectivity of the school they are attending: the very data he states do not exist in the later paper.

I'm not really sure what to make of that, and have nothing further to say on the topic. At the very least, I would be hesitant to use those graphs as evidence to support the general claim that study habits have changed even at the selective schools, since it's unclear where the data case from, or if in fact they even exist (to be clear: it's Babcock who says they don't exist, not me).

Conclusion

To summarize, there seems to be very little evidence to support Babcock's conclusion that study time has decreased even at selective schools by 10 hours from the 1960s to modern day. That is, he has a survey from 1961 in which students studied 25 hrs/week, two surveys in the 1980s in which students studied 17 hours/week, and two surveys in the 2000s in which students studied 14-15 hrs/week, but these surveys are all based on different types of students at different schools, so it's hard to make any strong conclusions. If I compared the weight of football places from Oberlin in 1930 and Ohio State in 2005, I'd find a great increase in weight, but in fact the weight of football players at Oberlin probably has not increased much over that time period.

Moreover, there are aspects of these data that deserve some skepticism. When report to people who went to selective schools that these data suggest students at such schools study 13 hrs/week, the response is usually something like, "Do you mean per day?"

Finally, since no statistics were run, it's quite possible that none of the results in this study are significant.

I want to be clear that I'm not saying that Babcock's claims aren't true. I'm just saying that it's not clear he has any evidence to support them (which is not to say I think it's a bad study: it was a good study to have done and clearly took a lot of work, but I find it at best suggestive of future avenues of research and certainly not conclusive).

New Grad School Rankings Don't Pass the Smell Test

The more I look at the new graduate school rankings, the more deeply confused I am. Just after publishing my last post, it suddenly dawned on me that something was seriously wrong with the number of publications per faculty data. Looking again at the Harvard data, the spreadsheet claims 2.5 publications per faculty for the time period 2000-2006. I think this is supposed to be per faculty per year, though the it's not entirely clear. As will be shown below, there's no way that number can be correct.

First, though, here's what the report says about how the number was calculated:

Data from the Thompson Reuters (formerly Institute for Scientific Information) list of publications were used to construct this variable. It is the average over seven years, 2000-2006, of the number of articles for each allocated faculty member divided by the total number of faculty allocated to the program. Data were obtained by matching faculty lists supplied by the programs to Thompson Reuters and cover publications extending back to 1981. For multi-authored articles, a publication is awarded for each author on the paper who is also on a faculty list. 

For computer science, refereed papers from conferences were used as well as articles. Data from résumés submitted by the humanities faculty were also used to construct this variable. They are made up of two measures: the number of published books and the number of articles published during the period 1986 to 2006 that were listed on the résumé. The calculated measure was the sum of five times the number of books plus the number of articles for each allocated faculty member divided by the faculty allocated to the program. In computing the allocated faculty to the program, only the allocations of the faculty who submitted résumés were added to get the allocation.

The actual data

I took a quick look through the CVs of a reasonable subset of faculty who were at Harvard during that time period. Here are their approximate publications per year (modulo any counting errors on my part -- I was scanning quickly). I should note that some faculty list book chapters separately on their CVs, but some do not. If we want to exclude book chapters, some of these numbers would go down, but only slightly.

Caramazza 10.8
*Hauser 13.6
Carey 4.7
Nakayama 5.9
Schacter 14.6
Kosslyn 10.3
Spelke 7.7
Snedeker 1.1
Wegner 2.3
Gilbert 4.0


One thing that pops out is that people doing work involving adult vision (Caramazza, Nakayama, Kosslyn) publish a lot more than developmental folk (Carey, Spelke, Snedeker). The other thing is that publication rates are very high (except for my fabulous advisor, who was not a fast publisher in her early days, but has been picking up speed since 2006, and Wegner, who for some reason in 2000-2002 didn't publish any papers).

What on Earth is going on? I have a couple hypotheses. First, I know the report used weights when calculating composite scores for the rankings, so perhaps 2.5 reflects a weighted number, not an actual number of publications. That would make sense except that nothing I've found in the spreadsheet itself, the description of variables, or the methodology PDF supports that view.

Another possibility is that above I accounted for only about 1/4-1/3 of the faculty. Perhaps I'm over-counting power publishers. Perhaps. But unless the people I left off this list weren't publishing at all, it would be very hard to get an average of 2.5 publications per faculty per year. And I know I excluded some other power publishers (Cavanagh was around then, for instance).

A possible explanation?

The best explanation I can think of is that the report actually is including a bunch of faculty who didn't publish at all. This is further supported by the fact that the report claims that only 78% of Harvard faculty had outside grants, whereas I'm pretty sure all professors in our department -- except perhaps brand new ones who are still on start-up funds -- have (multiple) outside grants.

But there are other faculty in our department who are not professors and do not (typically) do (much) research -- and thus do not publish or have outside grants. Right now our department lists 2 "lecturers" and 4 "college fellows." They're typically on short appointments (I think about 2 years). They're not tenure track, they don't have labs, they don't advise graduate students, and I'm not even sure they have offices. So in terms of ranking a graduate program, they're largely irrelevant. (Which isn't a slight against them -- I know two of the current fellows, and they're awesome folk.)

So of 33 listed faculty this year, 6 are not professors with labs and thus are publishing at much lowered rates (if at all) and don't have outside grants. That puts us in the ballpark of the grant data in the report (82%). I'm not sure if it's enough to explain the discrepancy in publication rates, but it certainly would get us closer.

Again, it is true that the lecturers and fellows are listed as faculty, and the report would be in within its rights to count them ... but not if the report wants to measure the right thing. The report is purporting to measure the quality and quantity of the research put out by the department, so counting non-research faculty is misleading at best.

Conclusion

Between this post and the last, I've found some serious problems in the National Academies' graduate school rankings report. Several of the easiest-to-quantify measures they include simply don't pass the smell test. They are either measuring the wrong thing, or they're complete bullshit. Either way, it's a problem.

(Or, as I said before, the numbers given have been transformed in some peculiar, undocumented way. Which I suppose would mean at least they were measuring the right thing, though reporting misleading numbers is still a problem.)
------
*Before anyone makes any Hauser jokes, he was on the faculty, so he would have been included in the National Academies' report, which is what we're discussing here. In any case, removing him would not drastically change the publications-per-faculty rate.

The Best Graduate Programs in Psychology

UPDATE * The report discussed below is even more problematic than I thought.

The National Academies' just published an assessment of U.S. graduate research programs. Rather than compiling a single ranking, they rank programs in a number of different ways -- and also published data on the variables used to calculate those different rankings -- so you can sort the data however you like. Another aspect to like is that the methodology recognizes uncertainty and measurement error, so they actually estimate an upper-bound and lower-bound on all of the rankings (what they call the 5th and 95th percentile ranking, respectively).

Ranked, Based on Research

So how do the data come out? Here are the top programs in terms of "research activity" (using the 5th percentile rankings):

1. University of Wisconsin-Madison, Psychology
2. Harvard University, Psychology
3. Princeton University, Psychology
4. San Diego State University-UC, Clinical Psychology
5. University of Rochester, Social-Personality Psychology
6. Stanford University, Psychology
7. University of Rochester, Brain & Cognitive Sciences
8. University of Pittsburgh-Pittsburgh Campus, Psychology
9. University of Colorado at Boulder, Psychology
10. Brown University, Cognitive and Linguistic Sciences: Cognitive Sciences

Yes, it's annoying that some schools have multiple psychology departments and thus each is ranked separately, leading to some apples v. oranges comparisons (e.g., vision researchers publish much faster than developmental researchers, partly because their data is orders of magnitude faster/easier to collect; a department with disproportionate numbers of vision researchers is going to have an advantage).

What is nice is that these numbers can be broken down in terms of the component variables. Here are rankings in terms of publications per faculty per year and citations per publication:

Publications per faculty per year


1. State University of New York at Albany, Biopsychology
2. University of Wisconsin-Madison, Psychology
3. Syracuse University Main Campus, Clinical Psychology
4. San Diego State University-UC, Clinical Psychology
5. Harvard University, Psychology
6. University of Pittsburgh-Pittsburgh Campus, Psychology
7. University of Rochester, Social-Personality Psychology
8. Florida State University, Psychology
9. University of Colorado-Boulder, Psychology
10. State University of New York-Albany, Clinical Psychology

Average Citations per Faculty


1. University of Wisconsin-Madison, Psychology
2. Harvard University, Psychology
3. San Diego State University-UC, Clinical Psychology
4. Princeton University, Psychology
5. University of Rochester, Social_Personality Psychology
6. Johns Hopkins University, Psychological and Brain Sciences
7. University of Pittsburgh-Pittsburgh Campus, Psychology
8. University of Colorado-Boulder, Psychology
9. Yale University, Psychology
10. Duke University, Psychology

So what seems to be going on is that there are a lot of schools on the first list which publish large numbers of papers that nobody cites. If you combine the two lists in order to get average number of citations per year per faculty, here are the rankings. I'm including numbers this time so you can see the distance between the top few and the others. The #1 program doubles the rate of citations of the #10 program.

Average Citations per Faculty per Year


1. University of Wisconsin-Madison, Psychology - 13.4
2. Harvard University, Psychology - 12.7
3. San Diego State University-UC, Clinical Psychology - 11.0
4. Princeton University, Psychology - 10.6
5. University of Rochester, Social-Personality Psychology - 10.6
6. Johns Hopkins University, Psychological and Brain Sciences - 8.8
7. University of Pittsburgh-Pittsburgh Campus, Psychology - 8.3
8. University of Colorado-Boulder, Psychology - 8.0
9. Yale University, Psychology - 7.5
10. Duke University, Psychology - 6.9

The biggest surprise for me on these lists is that University of Pittsburg is on it (it's not a program I hear about often) and that Stanford is not.

Student Support

Never mind about the research, how do the students do? It's hard to say, partly because the variables measured aren't necessarily the ones I would measure, and partly because I don't believe their data. The student support & outcomes composite is build out of:

Percent of first year students with full financial support
Average completion rate within 6 years
Median time to degree
Percent of students with academic plans
Program collects data about post-graduation employment

That final variable is something that would only be included by the data-will-save-us-all crowd; it doesn't seem to have any direct relationship to student support or outcome. The fact that they focus on first year funding only is odd. I think it's huge that my program guarantees 5 years -- and for 3 of those we don't have to teach. Similarly, one might care whether funding is tied to faculty or given to the students directly. Or whether there are grants to attend conferences, mini-grants to do research not supported by your advisor, etc.

But leaving aside whether they measured the right things, did they even measure what they measured correctly? The number that concerns me is "percent of students with academic plans," which is defined in terms of the percentage that have lined up either a faculty position or a post-doctoral fellowship by graduations, and which is probably the most important variable of those they list in terms of measuring the success of a research program.

They find that no school has a rate of over 55% (Princeton). Harvard is at 26%. To put none to fine a point on it, hat's absurd. Occasionally our department sends out a list of who's graduating and what they are doing next. Unfortunately, I haven't saved any of them, but typically all but 1 or 2 people are continuing on to academic positions (there's often someone who is doing consulting instead, and occasionally someone who just doesn't have a job lined up yet). So the number should be closer to 90-95% -- not just at Harvard, but presumably at peer institutions.

This makes me worried about their other numbers. In any case, since the "student support" ranking is so heavily dependent on this particular variable, and that variable is clearly measured incorrectly, I don't think there's much point in looking at the "student support" ranking closely.

Tenure, a dull roar

Slate ran an unfortunate, bizarre piece on tenure last week. FemaleScienceProfessor has a good take-down.  Among problems, it repeats the claim that the average tenured professor costs the average university around $11,000,000 across his/her career -- a number that is either misleading, miscalculated, or (most likely) an outright lie. But, as FemaleScienceProfessor points out, tenure itself costs next to nothing, so anyone who says eliminating tenure will save money really means cutting professor salaries will save money but doesn't want to be on the record saying so.

If this seems like deja vu, it is. I just wrote a post about a similarly confused feature in the New York Times. That post is still worth reading (imho).

Which raises the question of why tenure is under attack. I have two guesses: 1) it's a way of ignoring the progressive defunding of public universities, or 2) part of the broader war on science. There are possibly a few people who genuinely think tenure is a bad idea, but not because eliminating it will save money (it won't), because it'll soften the publish-or-perish ethos (yes, the claim has been made), or because it'll refocus universities on teaching (absurd, irrelevant, and beside the point). Which leaves concerns about an inflexible workforce and the occasional dead-weight professor, but that's not on my list of top ten problems in education, and I don't think it should be on anyone else's -- there are bigger fish to fry.

Sounds of Silence

My lament that, with regards to discussion of education reform, a trace of small liberal arts colleges has disappeared into the ether appears to have, itself, disappeared into the ether. Seriously, readers, I expected some response to that one. There are parts of my post even I disagree with.

No tenure, no way!

The New York Times is carrying an interesting but misguided discussion of tenure today. As usual, the first commentator warns that without tenure, academic freedom will die:

As at-will employees, adjunct faculty members can face dismissal or nonrenewal when students, parents, community members, administrators, or politicians are offended at what they say. If you can be fired tomorrow, you do not really have academic freedom. Self-censorship often results. 

Mark Taylor of Columbia replies, essentially, "oh yah?"

To those who say the abolition of tenure will make faculty reluctant to be demanding with students or express controversial views, I respond that in almost 40 years of teaching, I have not known a single person who has been more willing to speak out after tenure than before.

Instead, tenure induces stasis, a point to which Richard Vedder, an economist at Ohio University, agrees:

The fact is that tenured faculty members often use their power to stifle innovation and change.

Money

You might, reading through these discussions, almost think that universities have been slowly doing weakening the tenure system because they want to increase diversity, promote a flexible workforce, and reduce the power of crabby old professors. Maybe some administrators do feel that way. But lurking behind all of this discussion is money. Here's Taylor:

If you take the current average salary of an associate professor and assume this tenured faculty member remains an associate professor for five years and then becomes a full professor for 30 years, the total cost of salary and benefits alone is $12,198,578 at a private institution and $9,992,888 at a public institution.

I'm not sure where he's getting these numbers. The numbers at Harvard for the same period is $6,320,500 for salary alone. Assuming benefits cost as much as the salary alone gets us up to our $12,000,000, but that's for Harvard, not the average university. Perhaps Taylor is assuming the professor starts today and includes inflation in future salaries, but 35 years of inflation is a lot. I'm using present-day numbers and assuming real salaries remain constant.

In any case, money seems to be the real factor, mentioned by more or less all the contributors. Here's Vedder:

My academic department recently granted tenure to a young assistant professor. In so doing, it created a financial liability of over two million dollars, because it committed the institution to providing the individual lifetime employment. With nearly double digit unemployment and universities furloughing and laying off personnel, is tenure a luxury we can still afford?

Adrianna Kezar of USC notes that non-tenured faculty are often not given offices or supplies, which presumably also saves the university money.

Professors make choices, too.

So universities save a lot of money by eliminating tenure. And certainly universities need to find savings where they can. What none of the contributors to the discussion acknowledge, beyond an oblique aside by Vedder, is that tenure has a financial value to professors as well as universities. Removing tenure in a sense is a pay cut, and both present and potential academics will respond to that pay cut.

Becoming a professor is not a wise financial decision. The starting salary of a lawyer leaving a top law school is greater than what most PhDs from the same schools will make at the height of their careers should they stay in academia. And lawyers' salaries, as I'm often reminded, can be similarly dwarfed by people with no graduate education that go straight into finance.

Most of us who nonetheless go into academia do so because we love it. The point is that we have options. Making the university system less attractive will mean fewer people will want to go into it. It's really that simple.

Sell off Harvard Medical School!

Writing in the Chronicle of Higher Education, Andrew Hacker and Claudia Dreifus contend

Colleges are taking on too many roles and doing none of them well. They are staffed by casts of thousands and dedicated to everything from esoteric research to vocational training—and have lost track of their basic mission to challenge the minds of young people... Spin off medical schools, research centers, and institutes... For people who want to do research, plenty of other places exist—the Brookings Institution, the Rand Corporation, the Howard Hughes Medical Institute—all of which do excellent work without university ties.

Never mind that Howard Hughes is intimately tied to the present university system, let's say we're in favor: sell off Harvard Medical School, Harvard Law School, Harvard Kennedy School, etc., until all that's left is the College. That'd make it what? -- Wellesley + men? (This question is meant to be snarky, but not anti-Wellesley, for which I have the utmost respect, as will be clear in the rest of the post).


It's the money, stupid.


The blogosphere has been rising to the defense of the research university, with posters and commenters focusing on the (alleged) claim that universities use research dollars to fund the loss-leading undergraduate programs. Here's Mike the Mad Biologist:

[O]n a federal grant, usually somewhere between 30-40% of the total grant award doesn't go to the researcher for research costs (salaries, supplies, etc.), but to the institution. Now, some of that money is spent on actual administrative costs, but the rest goes to the university*. So if the university spins off $50 million, or $100, or, in the case of the University of Iowa, $169,175,021 of NIH funding alone (never mind other government sources), that's tens of millions of dollars that have to be recovered. Since I've called for more of a research institute model, I'm not opposed to spinning off research institutes. But I have no idea how universities that receive a lot of research dollars will make up the revenue shortfall.

There's an easy way of answering the question: write to any of the numerous, high-calliber exclusively-undergraduate institutions that makes the American education system so interesting: Wellesley, Swarthmore, Amherst, Grinnell, Oberlin, etc. For the last 150-200 years, such schools have focused on teaching, and teaching caliber is weighted heavily in tenure decisions. I had phenomenal professors. To name a few, Arlene Forman could have taught a turnip to speak Russian, and Jim Walsh delivered spellbinding lectures despite unpromising subject material (e.g., linear algebra). People who had never even attended Ron DiCenzo's classes nonetheless raved about the vicarious experience. 

Research University vs. Liberal Arts College

I loved the small liberal arts college experience and wouldn't have traded it for anything. But I have friends who feel the same way about the large research university: the inspirational presence of movers and shakers in the research world, they feel, is irreplaceable. I'm skeptical, but the great thing about the American education system is that it provides both options, something that many (all?) other countries lack. The only distressing thing is that so many students -- along with the Chronicle of Higher Education and essentially every blogger I read and all their commenters -- seem completely unaware that an alternative to the research university exists.

America has research-only institutes. It has undergraduate-only schools. And it has that fabulous hybrid institution: the research university. Arguing that we need to start founding undergraduate-only schools is like saying America really needs subways. Maybe we need more subways (I think we do!), but claiming they don't exist is just ignant, and it's an insult to the ones that exist and the people who made them possible.

Caveat emptor: Is academia a pyramid scheme?

That's the question on the blogs this week (see here and here). The question arises because each professor will have some number of students during their career (10-20 is common among the faculty I know), whereas the number of professorships increases very slowly. So the number of PhDs being produced far exceeds the number of academic positions.

As pointed out elsewhere, this neglects the fact that many PhD students have no intention of going into academia. Even so, it's clear the system is set up to produce more graduates who want academic jobs than there are jobs available. Prodigal Academic wonders if that's any different from any profession -- generally, there are more people who want the best jobs than there are best jobs to go around. Unlike PA, who doesn't think there's a problem, Citation Needed thinks most people entering graduate school aren't aware of how unlikely it is that they will get a tenure-track job, partly because it isn't in the schools' interest to mention this.

It depends

I largely agree with these fine posts, but I think they overgeneralize. Not all PhD programs are the same. Different fields vary wildly in terms of number of students produced, the likelihood of getting an industry job, etc., and also in terms of the caliber of the program. For instance, nearly every graduate of the psych program at Harvard goes on to get a tenure-track job. A sizable percentage get tenure-track jobs at the top institutions (Harvard, Yale, UChicago, etc.).

On the other hand, at even highly-respected but lower-ranked schools, getting a tenure-track job seems to be the exception. Here I have less personal experience, but a friend from Harvard who is a post-doc at a well-known state school was surprised to discover basically none of the students in that program expected to get an academic job. I've heard similar stories from a few other places.

A common problem

This isn't unique to academia. Many people believe lawyers earn a lot of money. Much fuss is made in the New York Times about how starting salary at a major law first is around $170,000/year (or was, prior to the Great Recession). While basically anyone who graduates from the top three law schools who wants such a job can get one (some go into lower-paying public-interest or public-service work), at most law schools, few if any graduates land such jobs and most lawyers never earn anywhere near that money. As a first approximation, nobody who graduates from law school lands a big firm job, just as, as a first approximation, nobody with a PhD gets a tenure track job at a top research institution.

From my vantage point, the problem is that media (newspapers, movies, etc.) fixate on the prosperous tip of the iceberg. Newspapers do this because their target audience (rather, the target audience of many of the advertisers in newspapers) are people who themselves graduated from Harvard or Yale and for whom getting a tenure-track job or being partner at a major law firm is a reasonably common achievement. Movies and television shows do this for the same reason everyone is beautiful and rich on the screen -- nobody ever said Hollywood was realistic.

This is fine as it goes, but can get people into trouble when they don't realize (a) that the media is presenting the outliers, not the norm, and/or (b) just where their own school/program fits into the grand scheme of things. As Citation Needed points out, it's not necessarily in the interest of less successful schools to warn incoming students that their chances of a job are poor. And, particularly in the realm of undergraduate education, there are certainly there are schools who cynically accept students knowing that their degree is so worthless that the students will almost certainly default on their loans.

What to do

Obviously the real onus is on the student (caveat emptor) to make sure they know what their chances of getting the job they want are prior to matriculating -- and this is true for every degree, not just PhDs. For most schools -- undergraduate and particularly graduate -- you can get data on how graduates fare in the marketplace. This can help determine not only which school to go to but whether it's worth going to school at all (it may not be). But to the extent it is in society's interest that people aren't wasting time and money (often as not, taxpayer money), it is worth considering how, as a society, we can make sure that not only is the information available, but people know that it's available and where to get it.

Do professors teach?

Luis Von Ahn has an excellent discussion on his blog about the teaching/research balance at major research universities. The comments are worth a read as well, especially this one and Von Ahn's response.

What does a professor do all day?

Readers of this blog will remember Dick Morris's strange claim that professors don't do anything except teach -- it's not even clear he thinks they have to prepare for class or grade papers. This raised a considerable backlash on the Web, in which many pointed out that teaching is, for many professors, only one pursuit (and often not the main one).

Around the same time, but apparently independently, a professor of psycholinguistics, Gerry Altmann, listed how he had been spending his time. In the space of 2.5 weeks, sent out 18 manuscripts for review (he's a journal editor), wrote 51 action letters (telling authors what decisions had been made), reviewed 7 NIH grants (interesting, since he works in the UK), and visited collaborators in Philly to discuss a new project (presumably part -- but not all -- of the 3677 miles he reports having flown).

Science in the classroom

Science Magazine recently profiled a new website that links up scientists and classroom teachers in order to improve science education. It looks like a great project, and definitely something needed.

Making science relevant

One thing about being a scientist interested in how people think and how different groups of people think differently is that you constantly notice differences in how scientists and non-scientists think differently.

For instance, scientists like evidence. You think how you parent your children affects how they turn out? Maybe. It's a testable question. (For my position on the question and on the evidence, read here.) One mark of a great researcher is the ability to spot untested assumptions (one of my favorite examples being Marc Hauser's work on language evolution). Of course, some scientists are less confined by evidence than others, and my avowedly non-scientist wife is as empirical-minded as they come. But it seems to be generally true (though I admit I don't know any well-controlled studies).

Where am I going with this?

Does making science relevant help science education?

In the last issue of Science Magazine, Hulleman and Harackiewicz point out that

Many educators and funding agencies share the belief that making education relevant to students’ lives will increase engagement and learning. However, little empirical evidence supports the specific role of relevance in promoting optimal educational outcomes, and most evidence that does exist is anecdotal or correlational.

I smell an experiment. It was surprisingly simple: students in a high school system in the Midwest were randomly assigned to write essays (as many as 8 during the semester, with an average of 4.7 essays/student) that either just summarized what they were learning in science or tried to apply what they learned to their own lives. The students who wrote essays relating science to their lives got better science grades and reported more interest in science at the end.

Motivation or depth of processing?

The authors discuss this in terms of motivation: students who see the relevance of science are more interested in it. (They also seem to imply that it improves confidence.) I'm interested in understanding the mechanism better (a professor in my department complains that Science articles are necessarily too short to give necessary experimental detail and theoretical motivation): were these students more interested because of the personal relevance per se, or was it simply that thinking about relevance required investing in and reinterpreting the material. After all, science isn't just a list of facts (or shouldn't be, anyway). Facts are boring; interpretations are what make science science.

But that is, as they say, academic. As the authors point out, this was a relatively simple method that improved grades and interest in science. Assuming, of course, that it replicates, this is a valuable contribution.

And it suggests that making science relevant improves education outcomes. If that seemed obvious from the get-go, it's worth remembering how many obvious truths have turned out to be wrong. Occasionally proving the obvious is an occupation hazard, but still worth the effort.

Are college professors underworked?

According to Dick Morris, I've joined a cushy profession. Professors don't teach very much, which makes college expensive. He argues that by requiring faculty to work harder "approximating the work week the rest of us find normal" and holding down some administrative costs, the tuition can be cut in half!

Comments on The Choice sum up the reaction -- mainly, that strong opinions are easy to have if you have no clue what you are talking about. Most have focused on the ridiculous claim that faculty don't work very hard, presumably due to Morris's odd belief that the only time professors spend working is time spent in the classroom. Morris would presumably cringe at the claim that the only time he spends working is the time he is physically typing out an article.

Well, maybe not Morris. There's no evidence in this article, at least, that he spend any time doing research. But most faculty spend a lot of time doing research, preparing for class, grading, sitting on committees, meeting with students, etc. When I find one who works less than 50 hours a week, I'll ask her secret.

There are also some funny numbers. Morris argues faculty typically teach 5 courses per year, spending 18-20 hours in the classroom per week. If they were to teach 8 courses, they'd spend 24 hours in class per week. Increasing the number of courses by 60% seems to only increase hours by 20%-33%. Sounds like profitability through magical thinking.

There is one point that Morris could have made, though: some universities could be made cheaper by having faculty do no research and less preparation for class. This wouldn't necessarily be an ideal situation, but it would be cheaper. The question is whether it's worth the cost.

Lean Times come to the World's Richest University

Academia is traditionally a good place to wait out recessions. Not so much this year. Harvard has posted a list of cost-cutting measures. Notice in particular that the number of PhD students being admitted has been reduced (no word about masters or professional school students...but then masters and professional school students pay tuition).

How much do professors get paid?

The American Association of University Professors recently released a report on the financial situation of professors. One interesting datum apparently gleaned from the report is a ranking of universities by full professor salaries. I have heard it said that Harvard pays below market because it pays in prestige, but that doesn't jive with its industry-leading $192,600/year (keep in mind this is average for full professor, which is rarely achieved before one's 40s at best).

One interesting fact noted shown in figure 2 of the report itself is that while, yes, PhDs do earn less than professional degrees (law, business, medicine, etc.), the difference is, on average, not nearly so large as one might expect. In 2007, the average PhD made around $95,000, while the average professional school graduate earned about $115,000 (both numbers are down, incidentally, from 1997).

That said, the ceilings are probably different. The average full professor at Harvard -- arguably the pinnacle of the profession for someone with a PhD -- as already said makes just under $200,000/year...or about the same as the typical big-firm lawyer a couple years out of law school (though perhaps not this year).

Should Universities Have Standards?

This is the question asked by the Bologna Process, an alliance of some higher education authorities. The question itself is a bit of Bologna, since, at least in the United States, there is an accreditation process that ensures some minimal standards for higher education. For those who believe that is a low bar, keep in mind that some institutions fail to reach even that standard (cf Michael "Heckuva-Job" Brown's law school).

The Bologna Process have something more aggressive in mind: "quality assurance" and "easily readable and comparable degrees." As described in a recent New York Times article, this involves establishing "what students must learn" rather than simply "defining degrees by the courses taken or the credits earned":

“Go to a university catalog and look at the degree requirements for a particular discipline,” Mr. Adelman [education policy expert] said. “It says something like, ‘You take Anthropology 101, then Anthro 207, then you have a choice of Anthro 310, 311, or 312. We require the following courses, and you’ve got to have 42 credits.’ That means absolutely nothing.”

The new approach, he said, would detail specific skills to be learned: “If you’re majoring in chemistry, here is what I expect you to learn in terms of laboratory skills, theoretical knowledge, applications, the intersection of chemistry with other sciences, and broader questions of environment and forensics.”

The idea, as I understand it, is to help prospective students choose the best schools and to help prospective employers evaluate applicants. Although I recognize this is a problem in need of a solution, I just don't see how this is supposed to work.

Imagine somebody wanted to set up standards for college football teams, in order to allow prospects to better compare potential schools and also to allow pro football scouts better evaluate college talent. You can define football fundamentals and even develop a test for them, but if you want to evaluate both Michael Oher and the left tackle at the local community college by the same standards, they will be either trivially easy or so steep as to "fail" the vast majority of college football players.

Academics has the same issues. Even leaving aside the fact that different colleges attract students of different abilities, the average student at one state school I know puts in about 5-6 hours of studying per week. At other schools, that's the number of hours per course. The amount you are expected to learn is vastly different.

It's also not clear how you would deal with emerging disciplines. Just looking at my own corner of academia, not long ago, few undergraduate schools had neuroscience courses. A handful of schools (Brown & Johns Hopkins being obvious examples) have "cognitive science" degrees. My alma mater had psychology, neuroscience, and "biopsychology" -- a blend of the two former. MIT has one department cellular, systems and computational neuroscience, along with cognitive psychology (clinical and social psychology are absent). In contrast, some years back Harvard had the now-disbanded Department of Social Relations, consisting of social psychology, social anthropology and sociology.

How can we define one set of standards that would apply to all those different departments? Perhaps it is exactly this multitude of department structures that so frustrates the folks at the Bologna Process, but I'm not sure there is an alternative. The late 20th century saw incredible growth and turmoil in the social and neurosciences, and nobody is quite sure what is the appropriate way of carving up the subject matter. Both the Harvard and MIT strategies have something to recommend them, but they are polar opposites.

In the end, I'm just not convinced there is a problem. Ultimately, what employers care about is not the quality of the employee's education, but the quality of his/her work product. So maybe that's what we should be evaluating.

Getting in to Graduate School

It's standard dogma that when the economy is bad, people go back to school. Although it doesn't appear to be major news yet, a number of schools are reporting an increase in applications (here and here, but see also here).

Despite an increase in applications, it is very possible fewer people will actually go to graduate school. This recession may be unique.

There are two problems. First, masters, MD and JD programs are very expensive, and students typically require loans. I shouldn't have to elaborate on why this might present a difficulty for the prospective graduate student right now.

Second, universities are cutting the number of students they are admitting. I don't have systematic numbers, but I know that the Harvard Graduate School of Arts and Sciences is reducing the number of students admitted for PhD programs. If the richest university in the country is slashing enrollment, I don't think I'm going out on too far a limb in assuming others are as well. Large private universities are depending on their endowments (i.e., the stock market) to cover operating expenses, and students are expensive. State schools are dependent on government financing, which is also drying up.

It is obvious why PhD students at a school like Harvard are an expense: instead of paying tuition, they are paid a salary by the school. I don't know if the enrollment cut will hit the professional schools. It is well-known that undergraduate programs are typically run at a short-term loss (tuition does not cover expenses), with the school figuring they'll make up the difference in future alumni donations. I do not know, but suspect, that the same is true for the professional schools. That said, the only schools at Harvard right now that don't seem to have a hiring freeze are the Law and Medical schools.

As I said, this is not being widely reported, and I do not have numbers for the industry as a whole. Hopefully I am wrong, because such a trend would be bad. During a recession, more people suddenly have time for school. When the recovery comes, it meets a better-educated and more capable workforce, (presumably) further fueling the recovery. This time, the opposite may happen.

Politicians who blame universities for their own failings

There has been some flack in Congress lately about the cost of tuition as the famous private universities. The primary senator behind the movement is Charles Grassley, a Republican of Iowa.

Tuition, room, board and fees at a university like Harvard are now hovering around $50,000. In a recent press release, Grassley writes that:

"The Congressional Research Service reports that, on the basis of mean household income of a household in the bottom fifth of the population, the price of college in 2005 was over 70 percent of the household's income."

Is there an easy solution that won't cost me anything?

Luckily, according to Grassley, there is: make schools spend more of their endowments. He notes that endowments are an awful lot like the funds of private foundations:

"In the 1960s, Congressman Wright Patman issued a series of reports, one of which included recommendations to limit foundation life to 25 years..."

So, colleges should only last for 25 years?

I'll give Grassley the benefit of the doubt and assume that's not where he was going with that quote (though there is no evidence to the contrary in the original document). In any case, he has a bigger problem: most schools don't have large endowments. Just 75 universities control 71% of all endowment assets. So that means the other 29% must cover the remaining 2,543 accredited four-year institutions in the US.

It's not clear just how well that will work.

Grassley also has a big conceptual problem. Universities invest their endowments and use the interest for operating costs. Costs go up each year. Inflation is one reason. Increasing numbers of students is another. Libraries must expand to incorporate new books. New departments (microbiology, ecological studies, Arabic, etc.) must be founded, and rarely are old ones (history, physics, literature) abandoned. A stagnant endowment is death to such a university.

Again, if your university has an endowment. So what about the rest?

What about the University of Iowa?

The flagship public university in Grassley's state is the University of Iowa. As a relatively wealthy public university, it has an endowment of almost $1,000,000,000. That seems like a lot, but it also has over 30,000 students, which gives the university an endowment of just over $32,000 per student.

That sounds like a lot, only if the university spends it all this year, after which there won't be any money for next year. Grassley wants universities to spend at least 5% of their endowment each year -- even in bad years in which they get less than 5% interest on the endowment.

That comes out to $1,615 per student per year. The in-state tuition this year (not counting room and board) appears to be $6,554. So that kind of spending will make a dent.

Except that schools already rely heavily on their endowments. That's why they have them. I couldn't find numbers for the University of Iowa, but I have heard that many school already spend over 4% of their endowment yearly. So let's assume the University of Iowa spends 4%. That means Grassley is calling for his home state university to spend an extra $323 per year from its endowment -- or, about the cost of 3-4 college textbooks.

And remember, this is one of the country's richest universities. The vast majority of schools have far, far smaller endowments, if they have one at all.

Right on the fact. Wrong on the reason.

Governments have been steadily cutting funding for publicly-funded universities. I'm fortunate to be at an endowment-rich university, but my father is at a public institution in Michigan. The budget crisis there has caused steep cutbacks in funding ... and thus steep increases in tuition.

This is a story I've heard repeated at many universities across the country.

It's nice to be able to go to Harvard. But only about 2,000 students each year are accepted. I'm sure they'd appreciate the help with tuition, but it's not going to affect most Americans. The university tuition crisis is in the public universities, which have small endowments, if any.

But I suppose Grassley doesn't want to talk about that.