Day 14: Grades, graduation rates, and actual learning
A series following my new class: The Economics of Higher Education
For our last day of reading and discussion, we’re talking about the mechanics of school and how they influence our overall product. Today we’re talking about grades.
The TL;DR: Grade inflation is happening. It is the main reason why college graduation rates have increased. It probably makes students work less hard in school. But actual learning in college is still important to employers. So yes, we are in a bad equilibrium right now.
Today’s readings
Denning, J. T., Eide, E. R., Mumford, K. J., Patterson, R. W., & Warnick, M. (2022). Why have college completion rates increased? American Economic Journal: Applied Economics, 14(3), 1-29.
Bowden, A. B., Rodriguez, V., & Weingarten, Z. (2023). The unintended consequences of academic leniency. Working Paper. (Skip the model section if you’re pressed for time.)
Arteaga, C. (2018). The effect of human capital on earnings: Evidence from a reform at Colombia's top university. Journal of Public Economics, 157, 212-225. (Abstract and Introduction)
Denning, J. T., Eide, E. R., Mumford, K. J., Patterson, R. W., & Warnick, M. (2022). Why have college completion rates increased? American Economic Journal: Applied Economics, 14(3), 1-29.
There are more college graduates in the US workforce now than ever before, in part due to increased rates of going to college AND in part due to increased rates of graduating from college. While college graduation rates fell in the 1970s and 80s, they’ve increased since the 1990s. Why?
This paper says it’s because of grade inflation. Plain and simple. They go through a lot of detailed investigation to rule out as many possible other explanations as they can, and come to the conclusion that grade inflation explains 95% of the increase in college graduation rates. Ok then.
Just how much have graduation rates increased? Graduation rates are increasing in most sectors of the industry. Overall, graduation rates increased from 52.0% in 1990 to 59.7% in 2010.
The authors start by looking at nation wide trends and discuss how those trends should push graduation rates down. For example, enrollment rates have increased. “With a larger fraction of students entering college, there may be more entrants who are relatively less prepared because as more students enter college, they likely come from further down the distribution of student achievement. Therefore, enrollment sector trends are unlikely to explain increases in graduation rates.” Maybe students are learning more in high school so they’re better prepared for college? Not according to NAEP or PISA scores.1 Maybe students are spending more time studying? Nope, studying has decreased. Maybe they’re spending less time working? Nope, hours worked while enrolled doubled from 1970 to 2000. Maybe the price of college has decreased so students can keep taking classes as long as they need to to graduate? Nope, price has increased. Maybe student-faculty ratios have decreased while expenditures on instruction have increased so each students gets more personalized instruction? Nope, student-faculty ratios have increased while expenditures on instruction have decreased.
But these are just broad macro trends. Maybe things look better if you look at microdata from individual students? These authors use microdata from the National Education Longitudinal Study of 1988 (NELS:88) and Education Longitudinal Study of 2002 (ELS:2002) as well as administrative microdata from “a public liberal arts college” (i.e. very likely the United States Military Academy, a.k.a. Westpoint) to look more closely at each of these other possibilities. They use what’s called a Kitagawa or Oaxaca-Blinder decomposition to explore how much of the change in graduation rates is due to changes in observable characteristics (e.g. spending on instruction decreased by $X, which predicts a Y percentage point decrease in graduation rates) and how much is due to changes in returns to those observable characteristics (e.g. the effect of spending $1 on instruction on graduation rates decreased from A to B percentage points). While graduation rates actually increased by 3.77 percentage points, changes in observable characteristics predict that graduation rates should have decreased by 1.92 percentage points.
At the same time, grades have been increasing, across the whole GPA distribution. This plot shows CDFs of first-year GPAs for students at highly selective private schools like the one I work at, but the trends are the same for every type of non-profit school. In 1988, a first year at the 20th percentile would have a GPA of 1.66ish. In 2022, their GPA was 2.0. A student at the 80th percentile would have a GPA of 3.1ish in 1988. In 2022, it would be 3.2ish.
The authors show that this increase in GPA is not due to changes in student characteristics or educational choices by having some fun with omitted variable bias. (Remember that, metrics students?) Here’s the idea: First run a regression of first year GPA on a dummy variable for the 2002 survey. The coefficient on that dummy is equal to the average difference in first year GPAs between the 1988 and 2002 samples. We know that simple regression is missing several relevant variables (e.g. math SAT scores) that are probably correlated with both GPA and maybe the 2002 dummy (e.g. if math SAT scores improved over time). That would cause omitted variable bias. If it has caused omitted variable bias, then including those missing relevant variables should decrease the size of the coefficient on the 2002 dummy, which would tell us that the difference in grades is not due to grade inflation but rather those omitted variables. That’s what the authors test in Table 6. They include controls for student characteristics like test scores, major, and courses taken, and none of those are causing substantial omitted variable bias. “Put another way, equally prepared students in later cohorts from the same zip code, of the same gender and race, with the same initial courses, the same major, and at the same institution have higher first-year GPAs than earlier cohorts.”
And they haven’t even gotten to their most damning evidence. Remember how that paper that studied peer effects at the Air Force Academy had pretty much the ideal scenario to study peer effects? These authors have a similarly ideal scenario to look for grade inflation at West Point. First year students at the USMA must take two required science courses for which the final exams are comprehensive, graded by machine or in teams, and the questions on the test haven’t changed much in 12 years (9 of the 12 tests are exact duplicates). After controlling for the score earned on the test, they find that students who entered school later receive a grade 0.6 GPA points higher than students who entered the year before. “Students with the exact same score on the exact same final exam earned better grades in later years.” Wowza. And that’s at West Point! I shutter to think what the rest of us are doing in less structured environments.
The authors close by returning to the Kitagawa-Oaxaca-Blinder decomposition and adding first-year GPA. They find that changes in first year GPA predict a 3.57 percentage point increase in graduation rates, or 95% of the actual change.
The authors note that the welfare implications of increased graduation rates due to grade inflation are not clear. (First off, our contrarian Bryan Caplan would say that the value of graduating from college is 80% signaling anyway, so who cares what grades you get there. But then again how would that signaling model say the labor market will react to this grade inflation and signal dilution?) It could be bad if this reduces the returns to a college degree because the average skill level of a college graduate falls. (Could that be part of why we saw what we saw on Day 4?) But it’s not clear that would necessarily happen. Sure, maybe reduced standards will reduce student effort and learning because students are no longer afraid of failing out. But maybe students in the middle and lower end of the GPA distribution will now feel like their efforts could really pay off with higher grades so they work harder. Unclear. I wonder if anyone has studied that…
Bowden, A. B., Rodriguez, V., & Weingarten, Z. (2023). The unintended consequences of academic leniency. Working Paper. (Skip the model section if you’re pressed for time.)
This paper looks at academic leniency in high school, not college, but I still think it’s quite relevant to the questions at hand.
In the Fall of 2015, the North Carolina state government changed the way that high schools assign grades. They changed the percentage cutoff for each letter grade, according to the table below. This mechanically will have the effect of increasing the letter grades assigned, even if student and teacher behavior doesn’t change.
Grades should increase mechanically, but student and teacher behavior could influence grades too. Does it? Ooh boy does it.
The top panel of this figure shows numerical scores in 9th grade math at the end of the year before the policy change. Check out the bunching at the grade cutoffs. Students and teachers somehow find a way to just barely bump grades up into the next letter grade band. Now check out the lower panel from the following year, after the policy change. We observe that same bunching, but now at lower scores.
The authors examine the causal effect of the more lenient grading policy on student outcomes by comparing kids who were born just before the kindergarten cutoff date of October 17 (and so just happened to be in 9th grade at the time of the change) to kids born just after October 17 (and so just happened to be in 8th grade at the time of the change). This is called a regression discontinuity analysis. The assumption is that kids’ birthdays are more or less randomly determined and that no mom is going to beg her OB to induce just so her kid can be born before the kindergarten cutoff. Thus kids born just before or just after the cutoff should be pretty similar, and thus one group can serve as the treatment group while the other stands as the comparison. (Of course not everybody goes to kindergarten at their state assigned time, and not everybody proceeds through the school system exactly one grade per year. Don’t worry, the authors use a Fuzzy RD.) We can see that this strategy works because the school entry drops precipitously at the age cutoff date.
The authors find that students under the more lenient system increased their GPAs by 11%, but also increased their absences by 20%.
The increase in GPAs was concentrated in the high-achieving students (an increase of 0.296 for high-achieving students v. an increase of 0.065 for low-achieving students), while the increase in absences was concentrated in the low-achieving students (2.584 additional days absent for low-achieving students v. 0.373 additional days absent for high-achieving students). Grade inflation in this context made achievement and absence gaps worse.
Arteaga, C. (2018). The effect of human capital on earnings: Evidence from a reform at Colombia's top university. Journal of Public Economics, 157, 212-225. (Abstract and Introduction)
In this paper we return to asking the question: what is the point of college? Why do college graduates earn more? Is it because of learning and skill building? Or is it just signaling traits like intelligence, conscientiousness, and conformity?
This paper sets up just about the cleanest test that I can imagine. In 2006, the top university in Colombia (Universidad de Los Andes) reduced the number of courses required for a degree in economics (down 20%) and business (down 11%). Entry into this top university is very competitive and is determined solely by scores on the national standardized high school exit exam, so the quality of students entering the program didn’t change with the change in the content of the program. If an econ or business degree from the top university in Colombia was only valuable because of its signal, wages for graduates after the policy change shouldn’t drop. But if the value of a degree is determined by what students learn, then this policy change should lead to lower wages.
The author compares the difference in wages of econ and business graduates at Los Andes before and after the curriculum change in 2006, and compares that difference to the difference in wages before and after 2006 for graduates of other top programs. Thus, a difference-in-differences design. The author finds that due to the policy change, wages for economics students from Los Andes decreased by 16% while wages for business students decreased by 13%. Why did this happen? The author presents evidence from hiring at the Colombian Central Bank to show that graduates from Los Andes were 17 percentage points less likely to get a job at the Bank (a prestigious job for recent college grads, so I hear2) after the policy change. Thus, the reason why graduates make less money after the policy change is because they do worse in interviews straight out of college.
Human capital and actual learning, for the win.
And that wraps up the reading and discussion portion of the class!
Over the next few posts, I’ll talk about the assignments that the students did for the course, including The Consulting Project for the Enrollment Division, The College Budget Project, and The Policy Memo.
For a final post, I’ll write up what I learned from teaching this course and the big themes I hope you take away about the Economics of Higher Education. Stay tuned!
As I’m sure you’ve heard, the NAEP scores are even worse after COVID. This chart shows scores for 13 year-olds, but I’ve seen the same for 8 and 17 year-olds too.
This is me! Approximately 1 million years ago when I was an RA at the Fed.
The Bowden Rodriguez & Weingarten paper is really interesting to me because of the finding about absences. To what extent could this be picking up on a nationwide trend higher in absenteeism that began in the 2010s (and exploded post-covid)? They don't seem to make an effort to explain why lower academic standards might cause kids to attend school less, but that kind of explanation isn't really what they're set up to do in this study. The whole thing has me so curious given the preponderance of worry about chronic absenteeism in K-12 right now.