The latest census data estimates that San Francisco's rise in population since 2010 was 32,000 people, but only 4,200 new housing units have been added in that time. Given that the average household size is 2.3 persons, this means that there are 22,000 more new people than new homes available in the city over the past four years. According to the latest research from local real estate company Paragon Commercial Brokerage, this imbalance has been the biggest factor in the rampant home and rent price increases over the past few years. With the economy booming, new San Franciscans keep arriving despite the fact that the city has no room for more people in its current housing stock. High rents have also meant that the cost of buying an apartment building to rent out has skyrocketed 51% since 2010.
One of the reasons for the housing shortage is the crash of 2008, when the recession caused a huge dip in the number of new projects planned. The construction recovery lagged behind the recovery in the local job market because projects generally take years to build. Building of new units has now started up again, but isn't yet close to meeting the surging new demand.
The high demand and low supply also create extremely low vacancy rates all over the Bay Area. In San Francisco, Pacific Heights is at 1% vacancy and Russian Hill is at 2%. Even the Western half of the city, which traditionally has lower demand, is at only a 3.7% vacancy rate. A slightly higher vacancy rate is SoMa is most likely the result of new buildings being absorbed into the market there rather than any lack of demand.