Too often, desired conclusions skew analysis to produce self-fulfilling results. While keeping faulty data from contributing to sampling error is simply a matter of objectivity and basic arithmetic, more subtle are the subjective factors that yield non-random samples. In the case of the Silicon Valley survey, the psychological disposition of the respondents is a significant distorting factor.
At the February Oregon Wine Symposium in Portland, Silicon Valley Bank principal, Rob McMillan, gave a preview of the bank’s 2012 State of the Industry Report, projecting Oregon industry growth of between 7% and 11%.
In presenting his data, Rob noted that 80 of some 400 Oregon wineries had responded. This corresponds to a confidence level of 95% (a generally standard target for researchers) with a confidence interval of plus or minus 10%. Thus, with 80 of 400 wineries reporting, if the sample were random, there is a 95% certainty that the range of growth will be between -1% and 19% (plus or minus 10% against the reported average of 9%. This describes industry conditions ranging from stagnation to boomy growth, random indeed.
Adding to the uncertainty is the definition of sales growth. Is one speaking of unit growth or revenue growth? Unit growth may well be accompanied by a drop in revenue (“We lose a dollar on every sale but we make it up on the volume.”) Or, the data may just be GIGO (Garbage In / Garbage Out) as with the 2011 Oregon Winery and Vineyard Report covering 2010.
In that report, industry revenue was reported at $252M on 1.93M cases, which works out to $131 per case. In 2009, the respective figures were $202M and 1.66M cases or, $122 per case. In the environment of rampant discounting that existed in 2010, a growth in case value simply was not possible.
Expressed as a story problem, if in 2009, I sold 10 cases of $600 wine for total revenue of $6000, I would have had to sell $7500 of wine to achieve the 25% reported revenue growth of 2010. Average industry discounting of only 10% below 2009 pricing (not much $50 Oregon Pinot Noir was going at frontline pricing or at least going very fast), would have required a nearly 40% increase in case sales to achieve 25% dollar growth.
If one’s winery is failing, it is highly unlikely that receiving a business conditions survey in the mail is going to be very appealing. Who would want to be reminded, question by question, of how bad things are? Even those making ends meet don’t particularly want to see what others are presumably doing at the top of the scale. Rather, surveys such as Silicon Valley’s tend to be overly optimistic as, even anonymously, people want the world to know how well they are doing. Moreover, as the Silicon Valley survey gives the option of identifying one’s winery, there is even more incentive for the haves to trumpet their success.
Sociology also plays a role in sample error. Electoral surveys that are as statistically well-constructed as possible have two inherent sociological flaws. First, opinionated people are more likely to pick up a telemarketing call during dinner and thus don’t necessarily represent the deliberating voter. Second, as the preponderance of calls go to land lines, the results are slanted to an older, more traditional demographic.
Similarly, surveys of customer satisfaction are more likely to be completed by people either very happy or unhappy with their purchases. In the case of fashion or durable goods purchases such as automobiles, buyers may rationally cite the reasons for their decision but, in the end, people buy to satisfy the complex Gestalt of ego.
Survey responses tend to attract a non-random sample by their very nature: those who perceive something to gain by responding, those more desirous of having their opinion heard or those inclined to boast of their success or distinguish a particular prowess. These factors coupled with conclusion driven analysis and poor sampling methods yield only a superficial view, like the 50 mile setting on a car’s GPS, a level of detail anyone familiar with a map of the United States already knows but doesn’t provide sufficient direction for where one wants to go.
Rob McMillan said:
Bill –
Thanks for musing me.
I agree with you conceptually but your conclusion is askew. I agree using surveys is a tricky proposition. The way questions are formed, the responses people are allowed, and the analytics that follow all can lead to a conclusion that is other than true. And then there is the issue of unanticipated changes in future conditions that can make a forecast wrong.
That said, Silicon Valley Bank uses many cooberating sources; external, proprietary, and in some cases from confidential industry relationships to derive its conclusions in the Annual State of the Industry Report.
You mentioned the expected growth rate of 7%-11% that I referenced at the Oregon Wine Symposium as one conclusion. I am not a fan of surveying anyone to make a conclusion of growth rates because its not something that the respondent controls. It’s devoid of any analytics of market conditions and supply. You will be happy to know that we didn’t use survey information to come up with the growth forecast. In fact the respondent average has varied wildly from our predictions and we’ve still nailed the forecast on growth now 3 years running.
To give you an example of what I’ve observed: When we forecast the growth rate the year after the market collapse, we predicted “flat sales growth” (Zero percent) for the fine wine business. Another survey run at the same time used average of respondent data, and in a speech the expectations were announced for sales growth of more than 10% for the industry. What was picked up in the press was “Wine Sales to Top 10%.” Sales came in flat at year end as we predicted. The following year, we came out with a prediction of 11%-15% sales growth and that same survey from the other group concluded sales growth would be flat. Growth came in around 13% in the fine wine space depending on which data source you wanted to cite.
In those cases, using perceptions of the masses to get any indication of real growth was a useless exercise because they had no control over the outcome. Respondents tend to make their guesses based on the current conditions as a starting point. Where I do find survey useful is when people can control outcomes as in the case of grape purchases for instance. If producers say they expect to buy more fruit, they are likely to enter into grape contracts to do just that.
Hope that helps you and your readers. You owe me a glass of wine for making me type all this before going home for an adult beverage.
Note to readers: Bill Hatcher is one of the most intelligent and creative people I know, and one of my absolute favorite people in the Oregon wine market. He’s just misled at times.
William Hatcher said:
Rob, I will begin with a comment to readers. The SVB Annual Survey is an indispensable tool for A to Z’s strategic planning process. You encapsulate the critical drivers of the industry in a fashion that is as germane to a niche region such as Oregon as it is to California, one hundred times the size.
My post was meant in no way to diminish the report but only to cite the survey portion as a phenomenon–like electoral polls–where the psychological bias of the respondents subverts the objective intentions of the surveyor. I am glad to read that you take the respondents with the proverbial grain of salt and independently develop your forecast. In any case, that forecast is but a small part of the annual report; more overarching is the presciently invaluable research that cogently delineates market and logistical trends.
For the glass of wine, I look forward to it…as if we would stop at a glass.
Rob McMillan said:
Bill –
I will take you up on your offer to share multiple glasses on my next visit!