Eric Fischer is an ex-Googler, a map geek, and an artist in residence at the Exploratorium who just compiled 68 years worth of San Francisco housing data that he collected by manually reading newspaper archives all the way back to 1948.
Then he made it all freely available on GitHub, presumably as his gift to the world, like one of those billionaires who give it all away to charity because you can't take it with you. If you don’t have the time to analyze 65 megs worth of rent statistics, Fischer set up a quaint Blogspot page to illuminate it all.
Fischer’s methodology was simply to scan through every archived Sunday edition of the San Francisco Chronicle for 68 years, record the price for every advertised housing unit, calculate the median rent year by year, and plot it. (He added Craigslist ads and city data in later decades, as it became available.)
His conclusion: Rents have gone up an average of 6.6 percent a year, every year, since 1956. When adjusted for inflation, that becomes an average of 2.5 percent, and "real rents have quadrupled in effective cost in 60 years."
That's valuable insight in and of itself, but then he transitions into the $3,500/month question: Why do rents go up? Before continuing, it’s worth noting that Fischer is clearly an intimidatingly sharp guy, but no one person's methods or conclusions should be taken as gospel.
Disclaimers aside, Fischer says shortages in the housing stock are indeed the culprit, but since "housing stock has only increased since 1906 while prices have gone both up and down," there has to be more to it than just that. Which is to say, housing shortages are indeed a factor, but they aren't the only factor
He poses a three-variable formula: It’s the margin of unbuilt housing, but it’s also the total wages paid by employers in the city, and the size of the workforce they’re paying out to (this data courtesy of the Bureau of Labor Statistics).
Decrease any one of these figures (the Fischer theory says) and you’ll drive down rents. Decrease all of them and you’ll drive rents down a lot. To push rents back to 1981 prices, for example, it would take "a 53 percent increase in the housing supply, or a 44 percent drop in salaries, or a 51 percent drop in employment."
Fischer admits that this model doesn’t account for every single swing of the dial on housing costs for every single year, but it does seem to account for the overall trend and for the high and low tides of most individual years.
If you’re not convinced, well, he did publish the data, and he welcomes anybody else to crunch the numbers and share their own conclusions.