Accuracy claim of survey in question
A Juneau-based professional survey manager this week questioned the stated accuracy of a recent Haines Borough facilities survey. The $9,000 contract for the survey was awarded to a member of the borough’s facilities committee without a bid.
Haines-based business consultant Lenise Henderson Fontenot reported that the results of the mail-in survey she conducted this fall carried “a 95 percent confidence level with a five percent margin of error.”
But Robert Koenitzer, a survey manager for the McDowell Group, said such statistical measurements are not applied to mail-in surveys.
A margin of error designation is used on scientific, randomized surveys, which excludes mail and web surveys, Koenitzer said. “You actually can’t put a margin of error on a mail survey because it’s not a statistically random sample.”
A mail-in survey, Koenitzer explained, is self-reporting and carries a “self-fulfillment bias.” Results of such surveys can’t be projected with as much accuracy as surveys that randomly sample the entire population, he said.
A confidence level of 95 percent with a five percent margin of error means one can be 95 percent certain that if all Haines residents were actually surveyed, the results would fall within plus or minus five percent of the sample results, Koenitzer said.
“I can’t make a judgment on how representative (the mail survey) is of the community as a whole. I’m not going to say that I endorse the results fully, but I’m not going to completely dismiss them either,” Koenitzer said.
The survey was sent to 1,525 post office boxes in Haines; 335 were returned.
Henderson Fontenot is a member of the Facilities Master Plan Steering Committee, which asked for the survey. Manager Mark Earnest said he chose to give Henderson Fontenot the job and saw no conflict of interest between her position on the committee and her receipt of the work.
The manager can spend up to $10,000 without assembly approval.
“That was my decision to hire her. The steering committee had no role in the selection, the recommendation, or even discussion of that point. So what we arranged for, just to avoid any, you know, remote appearance of any conflict was that she would basically remove herself from any discussion about that work or actually just not even be involved from a committee standpoint during the discussion and deliberation of this portion of the report,” Earnest said.
Henderson Fontenot, owner of the Professional Development Company, has more than 20 years of experience in business consulting and a marketing degree, but had never previously conducted a mail survey.
Asked why he didn’t seek proposals for the job, Earnest said he wanted to save money and hire someone local with previous knowledge of Haines.
“We looked at it from two perspectives. One was cost, and the second one was, ‘Wouldn’t it be nice to have someone here who knows the community, who has knowledge and connection to the community and an interest in the outcome?’ I mean, not an interest in seeing it go in a certain direction, but just in terms of the commitment,” Earnest said.
Koenitzer estimated the McDowell Group would charge $10,000 to $15,000 for a phone survey of 250 people, depending on length and complexity. A mail survey with an announcement card and two mailings would run about $10,000 to $12,000 and include fielding, data processing, analysis, written reporting and a presentation.
Because of her inexperience in the area, Henderson Fontenot took to the Internet to research how mail surveys are conducted. She ended up using a website, surveysystem.com to help her, she said.
Koenitzer said survey websites aren’t as reliable or accurate as professional services. The science of designing surveys is complicated, and software bought or downloaded off the Internet can’t compare to hiring trained professionals to perform a scientific survey. “You get what you pay for,” he said.
“We literally have decades of experience between our 13-14 staff. We’ve found things that do and don’t work, and we’re still learning things all the time...If it’s just a lay person, even an educated lay person, not in that field, there’s definitely pitfalls that make it difficult” to create a good survey, Koenitzer said.
Regarding Henderson Fontenot’s survey, Koenitzer had several small issues, such as the inclusion of a confidence level and margin of error. He also thought the writing in areas was awkward, and that Henderson Fontenot’s interpretation of the data in some areas was not the same as his would be.
Koenitzer acknowledged the percentage of surveys returned was “very impressive,” and Henderson Fontenot’s work was extremely professional.
“That’s a decent sample. They took pretty good care. They double-checked their data entry and that shows a pretty good attention to detail,” he said.
Koenitzer said from a statistical standpoint, phone surveys are best because the sample is more reliably representative of the larger population. Phone surveys are also faster than mail surveys. Mail surveys, however, are cheaper and usually give more people an opportunity to respond.
The McDowell Group conducted a randomized phone survey in 2011 for use in the borough’s comprehensive plan.
Earnest said Henderson Fontenot would present alongside McCool Carlson and Green architect Jason Gamache during the Facilities Master Plan Steering Committee meeting Nov. 26.