Zillow.com is a popular website for many homeowners and those who intend to go online to search for specific property valuations. But how accurate can an expert system that is entirely computer generated be without the benefit of actually seeing the property in question, much less the comparables, or even the location?
It was my brother-in-law who first told me about Ziilow’s Zestimate utility when it was new in the summer of 2006. As a real estate appraiser and agent in a major metropolitan area for over 50 years, he was impressed by how close the Zestimate was for a real test, this from a guy who has little use of computers in general. I took this with a grain of salt as our company had put a lot of effort into developing such a utility that uses statistical analysis to determine value.
To understand the problem, it is important to understand the basics of property valuation. Since the savings and loan crisis of the 1980s and 1990s, bank appraisals have been done according to a rigid method called the Uniform Standards of Professional Appraisal Practice (USPAP). In general, the specification calls for comparing three to five similar properties within half a mile and sold within six months. These schemes can be extended when necessary. Differences are then adjusted to arrive at a valuation. The problem was that, given the strict restrictions on which properties the appraiser could use and the small sampling (3-5 comparables), the results tended to be like comparing apples to oranges. And just like the arbitrator’s decision, it may not always be correct, but it is always the bottom line for lenders.
Our utility company takes a different approach when comparing thousands of sales over decades, dating back to 1991. The concept was to eliminate variables by total volume (large sample). Zillow should use something like this because they all have a large sample of properties (72 million nationally as of 2006) even though their algorithms aren’t published.
For our algorithms, differences in the date of a sale are adjusted for area appreciation (and recently depreciation) the location is based on the school district, which is relevant in our market, rather than a restricted area around the property in question that has an unfortunate tendency to cross boundaries between areas of very different values. An appraiser not fully familiar with the area, as is common when ordered by online lenders, might compare properties in different school districts or not adjust for proximity to an undesirable feature, such as a major highway or industrial area, even though the appraisers comparables fall within the prohibited distance of the subject property. A good appraiser will take these adjustments into account, although there is no universal standard other than how a particular geographic feature affects an average of other sales prices for similar properties. USPAP valuations tend to vary by 10-15% from appraiser to appraiser, especially if each uses different comparables. This is one of the weaknesses of small samples.
In general, our utility worked well. Interestingly, considering how much effort went into its creation, we rarely use it. This is because any statistical analysis ignores the really important aesthetic considerations, such as warmth and charm, elegance, style, etc. In addition, the utility does not take into account variables such as mortgage interest rates, the availability of funds (which hinders the current market), and general consumer confidence. When the market is perceived to be appreciating, buyers are willing to pay more, but when it is perceived to be depreciating, buyers tend to insist on further trading to cover any further market erosion. They are also not considered competing properties currently on the market. These tend to have a greater effect on a given sales price than historically comparable ones. All of these factors have a strong influence on how much a given property will sell for. And yet the Zestimate still manages to get a good estimate, at least some of the time.
As of June 13, 2011, Zestimate is in its third algorithm update. But how accurate is it really? After all, a lot can depend on it. Homebuyers often base their offer on a Zestimate and sellers often base their sales price on the same Zestimate. Fortunately, lenders don’t.
As a real estate agent, I am often faced with both buyers and sellers who have placed a great deal of faith in Zestimate over what a municipal appraisal, a real estate agent Competitive Market Analysis (CMA) or even an appraisal suggests. USPAP, especially if it’s what they want to hear.
To find out how accurate the Zestimate really is, at least in our market, we simply compare the actual closed sale price taken directly from our Multiple Listing Service (MLS) to the Zillow Zestimate using last month’s sales data for the counties. . of Albany, Schenectady, Rensselaer and Saratoga which includes the metropolitan areas of Albany, Schenectady, Troy and Saratoga, New York. Of course, each market is different and may not produce the same results. What we found was that Zestimate was surprisingly accurate in predicting the actual sales price with a few notable exceptions. Zillow uses a similar method to judge its own accuracy. However, Zillow’s accuracies are presented as a mean error of actual sales prices for Zestimate. As of their last algorithm update, they claim an error of only 8.5%. ( http://www.zillow.com/blog/research/2011/06/14/upgrading-the-zestimate ) What is not stated is how far the Zestimate is when it is turned off. Let’s just say at this point that no appraiser, municipal appraiser, or real estate agent is likely to miss the mark as much as Zillow misses when it misses.
Moving on to the numbers:
In terms of average error, what we found was a staggering 2.8% error (overvalue) across all price ranges. In general, the lower price ranges saw a Zestimate that was higher than actual sales prices, while in the higher price range, the Zestimate tended to undervalue properties.
Price range – Zestimate – Percentage error
0 to $100K 113.85% 13.85% OVER
$100 to $200K 104.13% 4.13% OVER
$200 to $300K 98.28% 1.72% OFF
$300 to $400K 94.42% 5.58% OFF
$400 to $500K 87.22% 12.78% LESS
$500K and up 85.30% 14.70% LESS
As you can see in the table above, the undervalue in the higher price ranges eliminates the overvalue in the lower price ranges. Gold Zestimate was overvalued 47% of the time; undervalued 53% about half the time.
However, when the Zestimate failed, it missed by a mile. The biggest mistake was an understatement of $221,329 or a 61% understatement. The largest overvalue was $137,850 or a 44% overvalue. This did not happen often, as noted. The graph below shows a large central tendency toward precision and a larger error toward either extreme.
Bottom line: If you have to be right, use an experienced and qualified USPAP appraiser who knows the area. If the intended use of the appraisal is not critical, Zillow is ideal for an excellent approximate appraisal allowing for the possibility of material error.
