The more data and back testing that is used
to establish correlations and investment factors, the less reliable the
research is—or so claims Research Affiliates.
The recent proliferation of data available
to analysts and academics has led to the emergence of a huge number of “factors”
that the researchers claim can influence investment returns. But the way in
which these factors are calculated is highly questionable, according to Jason
Hsu, vice-chairman and co-founder of Research Affiliates.
“The newer anomalies [factors] are most likely results of data mining.” —Jason Hsu, Research AffiliatesIn a recent paper, Hsu said he and his
colleagues had become “deeply
distrustful” of the increasing number of investment factors being proclaimed by
Citing work by Cam Harvey, a professor at Duke University’s Fuqua
School of Business in North Carolina, Hsu said 316 investment factors had been identified
and published in top journals. However, Research Affiliates’ own “factor
robustness” investigations had identified “only a handful of persistent, investable sources of
Traditionally, statistically significant results are
indicated by a “t-statistic” of greater than 2. However, Hsu argued that the
increasing amount of research—and accelerating pace of new publications—meant that
the bar for statistically significant findings should be raised.
“When one runs a back test to assess a signal that
is, in fact, uncorrelated with future returns, the probability of observing a t-stat
greater than 2 is 2.5%,” Hsu wrote. “However, when thousands upon thousands of
such back tests are conducted, the probability of seeing a t-stat greater than 2
starts to approach 100%.”
Professor Harvey concluded that, despite the
hundreds of factors identified by dozens of professors and analysts, the main
factors achieving a higher t-statistic were still the “old classics” of value,
low beta, and momentum.
“The newer anomalies are most likely results of
data mining,” Hsu wrote.
“As we add to our research team and thus the number
of back tests that we perform in aggregate, we recognize that our ‘false
discovery’ rate also increases meaningfully,” he added. In many cases, Hsu
said, Research Affiliates only considers results to be significant with a
t-statistic of 4 or higher.
In his paper, Hsu also argued that factor investing
was insufficient on its own to construct a properly diversified portfolio. Many
different combinations of asset classes could be used to gain exposure to a
particular factor, if other considerations such as price were ignored, he said.
“In order to create a portfolio with the
appropriate exposures at an attractive price, we also need to understand the
valuation levels at which the different assets trade,” Hsu said. “Factor-based
investing and its complement, asset-class-based investing are, in our mind,
incomplete descriptions of the world without each other.”
a blog post last month, AQR’s Cliff Asness argued that the idea of biased
back tests was inaccurate—but his work focused on the factors that Hsu, Harvey,
and Research Affiliates also showed to be the most correlated to investment
Jason Hsu’s paper, “The Whole Story: Factors +
Asset Classes”, can be downloaded from
Research Affiliates’ website.
Share, Deconstructed & Asness
to Haters: Fama-French Weren’t Wrong. You Are