A contentious debate over short-term rentals has unraveled over the past five months as the New Orleans City Council considers new restrictions. Recently, the City Planning Commission recommended an overhaul to the law that would prohibit people in residential neighborhoods from using an entire property as a short-term rental.

Short-term rental owners were not pleased.

“Your staff report, however thorough, currently lacks both evidence and understanding,” said Eric Bay at a public hearing earlier this month. Bay is the President of the Alliance for Neighborhood Prosperity, which advocates for short-term rental operators. “However, our UNO economic impact study was reported with factual data.”

Bay was citing a study released in June by the Hospitality Research Center at the University of New Orleans. The study was funded by the Alliance for Neighborhood Prosperity.

It concluded that 595,000 people stayed in short-term rentals in New Orleans in 2017, generating nearly $900 million in economic impact and supporting or creating 10,200 jobs. These numbers have been consistently repeated at public hearings and community meetings by short-term rental proponents.

The study caught the eye of one commission member.

“This seems like a large number,” Commissioner Kathleen Lunn said to the City Planning Commission staff at a recent public hearing. “That would be something reliable?”

The answer is no, according to economists who spoke with The Lens.

The University of New Orleans, a public university, has refused to release any of the underlying data or the full list of data sources it used for the study. The methodology the researchers used to reach their conclusions is also largely unknown. And the university hasn’t released the results of its 2017 New Orleans Visitor Profile — a broader tourism study that served as the foundation of the short-term rental report.

Economists weigh in

“This report is an extremely poor excuse for an economic analysis,” said Jay Brinkmann, the former chief economist at the Mortgage Bankers Association. “If you wanted to put a report together that would maximize the potential perceived benefits of short term rentals, this was the way to do it. … The benefits are overstated, and they have ignored the costs.”

Dr. John Crompton, a professor at Texas A&M University’s Department of Recreation, Park and Tourism Sciences, said that there was “no evidence” that the study’s conclusions were accurate. “It doesn’t include how they got from A to B,” he said. “Without the details you can’t make any sensible judgment.”

Responding to public records requests, the Hospitality Research Center said it was keeping the underpinnings of the study — the data, the data sources, and the 2017 visitor profile — hidden from public view because releasing them would compromise their intellectual property.

After repeated requests, Williams and a UNO spokesman would not grant The Lens an interview for this story.

As a public body, UNO is subject to Louisiana’s public records law, said Alysson Mills, an attorney with the Fishman Haygood law firm who specializes in media and First Amendment law.

”Our goal is to show the importance of STRs to the New Orleans Economy relative to other types of Accommodations like hotel.”—Jim Brubaker, Alliance for Neighborhood Prosperity, in an email to UNO

“Any materials used by a public body in the performance of its business are public records and must be made available for inspection,” she said. “There are exceptions to the Act, but they are rare.”

The university did, however, provide emails between the Alliance for Neighborhood Prosperity and Hospitality Research Center employees. In those emails, the Alliance explicitly laid out what it wanted the study to find.

“Our goal is to show the importance of STRs to the New Orleans Economy relative to other types of Accommodations like hotels,” wrote Jim Brubaker of the Alliance to Maria Ortiz, the UNO researcher who prepared the short-term rental study. “You know the data better than we do, so we defer to you on how you think the analysis should be best structured to reveal this kind of information.”

In a later email, he provided a detailed list of what he wanted the study to say.

“These visitors made a substantial contribution to the New Orleans metro area economy,” read one of the requested conclusions.

“However, if it will take a long time to generate all of these measures by market segment then we’d be happy to first receive just one measure by market segment that tells an impressive story,” wrote Brubaker. “We look to your [recommendation] on which measure would be the quickest to generate and tell an impressive story.”

The Alliance for Neighborhood Prosperity seems to have remained active throughout the four weeks it took to create the study. The contract between the group and UNO specifies that the Alliance would “collaborate with the HRC in answering questions that may arise in the conceptual, methodological, or procedural stages of the study.”

”To suggest that these findings are unsound is to question nationwide best practices when conducting economic impact analyses.”—Eric Bay, Alliance for Neighborhood Prosperity

This appears to be common practice for the Hospitality Research Center. In at least nine study summaries available online, the client is also listed as a collaborator.

Overall, the research center’s opaque data and methodology, as well as the apparent influence its funders have in their process, point to blanket issues with the way it conducts research, economists said.

“Whenever [researchers] don’t produce the details of their study, you have to be suspicious,” said Crompton. “You have to ask, why?”

In an emailed statement, Bay said that the Alliance for Neighborhood Prosperity “unequivocally” stands behind the study. He added that the Hospitality Research Center is the “industry standard” for hospitality research.

“To suggest that these findings are unsound is to question nationwide best practices when conducting economic impact analyses,” he said.

Crompton doesn’t object to his last point.

“Most economic impact studies are commissioned to legitimize a political position rather than to search for economic truth,” he wrote in a 2006 article published in the Journal of Travel Research. “Clearly, there is a dilemma.”

Studies play role in policy debates

The Hospitality Research Center is a collaboration between UNO’s Division of Business and Economic Research and the School of Hotel, Restaurant and Tourism Administration.

Its studies are largely funded by hospitality industry groups and government agencies that promote tourism. They are often used to demonstrate the importance of tourism in Louisiana’s economy or to justify public funding for projects related to tourism. The Alliance for Neighborhood Prosperity paid $7,229 for the short-term rental study, but the research center has charged up to half a million dollars for a single report.

Among its biggest clients is New Orleans & Company (formerly the New Orleans Convention and Visitors Bureau), which has paid UNO more than $2 million for various studies since 2005, according to the resume of Dr. John Williams, the director of the Hospitality Research Center. The research center has historically compiled an annual New Orleans visitor profile on their behalf. Last year, New Orleans & Company decided to end that tradition and hired a different research group, D.K. Shifflet & Associates, to produce the profile.

The Hospitality Research Center’s reports on tourism have played a role in debates over major policy decisions, from hosting major sporting events to the way hotel taxes are divided between New Orleans city government and the tourism industry. Often, their conclusions are cited by funders, news publications, or even the city government, typically without a link to the actual studies.

These studies can be difficult to track down. UNO doesn’t keep a compiled list of them on its website. The 2017 visitor profile that served as the basis of the short-term rental study was never released to the public at all.

“The Hospitality Research Center conducts rigorous and comprehensive in-depth studies,” Williams said in an emailed statement. “The Center is widely recognized for the veracity of its research results.”

“This is more of a poll than a study”

Breonne DeDecker, the program manager at the Jane Place Neighborhood Sustainability Initiative, was skeptical when the research center’s short-term rental study came out in June. Jane Place has been an outspoken opponent of the proliferation of short-term rentals in New Orleans.

“It was really unclear how they got their information and how it was analyzed,” she said.

According to the study, the conclusions were based on a survey of 319 people staying in short-term rentals in New Orleans.

Along with the surveys, “additional sources available to the center were included in the analysis to balance for any biases inherent in data collection methodologies.” The report doesn’t enumerate the additional sources, but it says that they included data purchased from TNS, a global market research company, as well as “intercept survey data” collected by the Hospitality Research Center.

”It was really unclear how they got their information and how it was analyzed.”—Breonne DeDecker, Jane Place Neighborhood Sustainability Initiative

DeDecker submitted a public records request for the survey instrument, the additional data sources, and all drafts or notes that preceded the final study.

“There are no drafts or notes,” responded University compliance officer Doug Mittelstaedt. “Solely produced was the final report.”

The University also did not share the survey instrument used to collect the 319 responses that served as the basis of the study.

“There are none,” said Mittelstaedt.

Mittelstaedt explained to The Lens that the short-term rental survey instrument doesn’t exist because the survey was actually conducted in 2017, before they were hired to produce the short-term rental study, as part of the annual visitor profile. He said that the research center “extracted material” from that study to perform the short-term rental study.

When asked for the data from the 2017 survey, Mittelstaedt said, “No data from the previous study was used. The only information that was used was from the final written results of the study.  No databases were utilized in the study on short term rentals.”

When asked for a copy of the final written results of the previous study, Mittelstaedt refused again, saying “I’m sorry, but we cannot provide a copy of that study. Our position is that the study is our intellectual property.”

As for the study’s “additional sources,” the University refused to share that information as well. Mittelstaedt said that raw data from “public reports” created by “entities” was inserted into the university’s methodology. “No raw data exists nor any listing of raw data at the University,” he wrote.

”All information for the study was input directly into our methodology tool. That tool comprises our methodology and is intellectual property that must be protected.”—John Williams, Hospitality Research Center, responding to public records request

Responding to DeDecker’s requests, Williams said, “All information for the study was input directly into our methodology tool. That tool comprises our methodology and is intellectual property that must be protected.”

It’s unclear what methodology tool Williams was referring to. But whatever he was referencing, Mills says she didn’t have enough information to know for sure whether it would be exempt from public records law.

“The Louisiana Legislature and Louisiana courts have said, loud and clear, that the Act must be construed broadly, in favor of disclosure,” she said.

There is a provision in the law that protects research from disclosure. But that exemption only lasts until the research is published or publicly released. The short-term rental study was published and publicly released by the Alliance for Neighborhood Prosperity. While UNO did not publish its 2017 visitor profile, Williams provided its key findings to a NOLA.com reporter, Mittelstaedt confirmed. Ultimately, Mills says that the burden is on UNO “to prove the exception unambiguously applies.”

”The position that this is somehow proprietary is simply nonsense.”—John Crompton, Texas A&M University

In any case, the economists who spoke to The Lens were unconvinced that the methodology tool used in the short-term rental study has much proprietary value.

“The position that this is somehow proprietary is simply nonsense,” Crompton said. “This is common stuff. You have a sample, you multiply it up, and depending on how you collect the data you weight it or you don’t. There’s nothing proprietary about that.”

Crompton has spent much of his career researching inaccuracies, biases, and deceptions in tourism industry economic studies and has published dozens of peer-reviewed articles on the subject. When asked whether the UNO study shared similar mistakes to those he’s seen over his career, Crompton said there wasn’t enough information in the report to know.

“They don’t tell me anything about how they collected this stuff or the processes they used,” he said. “The devil’s in the details with these things.”

The Hospitality Research Center seems to confirm Crompton’s reasoning. Its contract with the Alliance for Neighborhood Prosperity says that “the quality of the analysis and final study will be contingent upon the quality of surveys collected and the accuracy of visitor counts.”

But from the limited amount of information Crompton did have to work with, he saw some red flags.

He said it was “strange” that the study vaguely refers to additional, unnamed sources. He also pointed to a particular phrase in the methodology section: “Data included in this report was edited for accuracy and logical consistency.”

“I just don’t know what that means,” Crompton said. “If it means they took out extreme outliers, that’s usual. But I don’t know what it does mean.”

He also took issue with the small sample sizes used throughout the study. The study makes conclusions using as few as 17 survey responses. In one graphic, the study lists the countries of residence for international visitors based on only 39 responses. (Both include disclaimers about the small sample sizes.)

“You can’t say anything sensible from 39 international visitors,” Crompton said.

Overall, however, the process was too cryptic for Crompton to make many conclusions.

“Somebody somewhere did some analyses to get these numbers,” he said. “Until they provide a study with the details of how they collected the data, how they multiplied it up to the population, what multipliers they used, and what assumptions they based it on, there’s nothing for me to say.”

The specifics of the research center’s process are vital to judging its accuracy because although the arithmetic is fairly simple, there are a myriad of decisions the research center had to make along the way that could have drastically changed the results. Just getting an accurate number of total visitors, for example, is more difficult than it may seem.

The Hospitality Research Center’s 2017 visitor profile, which was apparently completed in May, concluded that 10.98 million people visited New Orleans in 2017. But two weeks later, a different study, produced by the firm D.K. Shifflet & Associates for the Convention and Visitors Bureau, found that there were 17.74 million visitors in 2017. The second study also found that visitors spent about $1.19 billion more than what the Hospitality Research Center reported.

Brinkmann brought up many of the same issues as Crompton.

”This is not something that would show up in an academic journal.”—Jay Brinkmann, economist

“This is more of a poll than a study,” he said of the short-term rental study. “This is not something that would show up in an academic journal. The problems I’m raising now are the most basic outline of how an academic peer might look at this and dismiss it.”

He said that the distinction between peer-reviewed academic articles and industry-funded studies is crucial. But that distinction can be difficult to discern for people outside of academia, especially when the original studies aren’t easily accessible.

In an article Crompton co-authored for the Journal of Travel Research, he warned of “hidden” processes that can impact the outcomes of economic impact reports.

“They are internal process and procedural decisions made by researchers that most lay audiences are likely to consider esoteric, arcane, and mundane, and to view with disinterest,” it says. “They are frequently invisible, because they are rarely mentioned in reports.”

DeDecker noted that when Jane Place released its own study on short-term rentals in March, it sent all the underlying data to every news publication they spoke with.

“Y’all have access to our data so you can go through and prove our work. That’s not a standard we’re seeing from UNO,” she said.

Keep the good, hold the bad and the ugly

Critics say that the lack of data transparency is only one half of what makes the short-term rental report inadequate. They say that the study only sought to highlight the positive impacts of short-term rentals, not to provide a comprehensive picture of the net effects.

“The underlying premise of this entire study basically treats short-term rentals as if they existed in a vacuum,” said local real estate developer Bill Sawicki. “It treats every short-term rental as if it was a building that was built from scratch for the explicit purpose of a short-term rental on a piece of land that couldn’t otherwise be used.”

”The underlying premise of this entire study basically treats short-term rentals as if they existed in a vacuum.”—Bill Sawicki, real estate developer

Brinkmann agreed. “Even if the data were correct, this still wouldn’t tell us anything,” he said. “[The study] presupposes that if you did away with short term rentals, all of those economic benefits would disappear. In other words, people would not show up at all,” he said. “Life is not that simple.”

He argued that for the most part, visitors would simply choose to stay in traditional hotels if short-term rentals didn’t exist. So why didn’t the research center ask that question?

“My guess is that the Alliance for Neighborhood Prosperity does not want the answer to that question exposed,” Brinkmann said. “The most charitable explanation is that it may not have occurred to the UNO people to ask it, rather than that they were swayed by the views of the financial supporters of the study.”

Brinkmann said that once you find the marginal benefit of short-term rentals, you need to ask about the costs.

“What are we spending to get it? What are we spending in terms of lost affordable housing? What are we spending in terms of destruction to neighborhood quality?” he said. “That would have been an economic study, but nobody’s done that. And I’m not sure they have the capacity within the hotel school to do any kind of analysis like that.”

In another article published in the Journal of Travel Research, “Economic Impact Studies: Instruments for Political Shenanigans?” Crompton concludes:

“Reviewing the stream of mischievous studies masquerading under the rubric of economic impact, one is reminded of Macbeth’s lament in Act V, Scene V: ‘It is a tale told by an idiot, full of sound and fury, signifying nothing.’

“However, the tales are not told by idiots; they are, for the most part, told by knowledgeable people who recognize that the general public and elected officials (audiences they are targeting) are frequently hopelessly deficient in terms of their level of economic literacy.”