ICI Viewpoints

Part of a series of ICI Viewpoints about problems in the IMF’s analysis of the asset management industry.

In this ICI Viewpoints series, we’ve examined the wide range of data errors, inconsistencies, results that don’t bear statistical scrutiny, and misinterpretations in the International Monetary Fund’s most recent Global Financial Stability Report (GFSR)—specifically, the chapter on “The Asset Management Industry and Financial Stability.” Those problems primarily involved poor understanding of funds and their investors. We didn’t need advanced statistical methods to uncover them.

This installment is different: we delve into some of the IMF’s empirical results. The GFSR summarizes econometric evidence that the IMF interprets as indicating that mutual funds could destabilize bond markets in emerging countries, perhaps through a “first-mover advantage.” The IMF draws upon this evidence and other factors to state, “Even simple investment funds such as mutual funds can pose financial stability risks.”

Regrettably, as with many of the IMF’s other claims in the April report, its suggestion that mutual funds could destabilize emerging bond markets doesn’t hold up to scrutiny. The GFSR confuses correlation with causation, reports weak econometric findings as if they were statistically significant, and uses strained economic assumptions to skew its results toward supporting its conclusions.

What Does It Take to Conclude That Fund Actions Could Destabilize Markets?

We agree with the IMF when it argues that “for mutual funds to have a destabilizing effect, fund trades must first, at least in the aggregate, have an impact on [securities] prices.” In other words, if funds are to destabilize bond markets, bond prices must fall by economically and statistically large amounts when funds sell bonds into the market.

But we disagree when the GFSR goes on to argue that “the literature suggests the existence of price pressures related to mutual fund flows” and states that the IMF’s analysis “is consistent with mutual fund flows affecting asset returns in smaller, less liquid markets,” notably emerging bond markets.

Our reading is that the literature is, in fact, quite mixed on whether fund flows in the aggregate cause significant pressures in stock and bond markets. A number of studies, for instance, have concluded that the evidence does not support the IMF’s conclusion (see the discussion in Collins and Plantier 2014). We aren’t convinced that the IMF’s empirical results have shifted the weight of that evidence.

What Is the GFSR’s Evidence That Fund Actions Affect Securities Prices?

The IMF presents its key findings in Table 3.2 of the GFSR, reproduced below. It tested the links between fund flows and returns using weekly data for four categories of U.S.-domiciled funds (all-equity funds, all-bond funds, high-yield bond funds, and municipal bond funds, from 2007 to 2014). It also examines whether emerging-market stock and bond returns are affected or predicted by flows to funds domiciled in the United States and elsewhere.

See a larger version of this table.

For all six fund categories, the IMF finds that “surprise flows have significant impact on returns” in at least part of the period examined. A second test—a vector autoregression—finds that “flows help predict returns” in three of six categories, according to the IMF table.

Since those findings contradict what we’ve found, we decided to look closer.

Confusing Correlation with Causation

The IMF study does appear to show that fund flows and securities prices are correlated. That’s not a new finding—plenty of research has shown that fund flows and returns tend to move together.

But correlation is not causation. Consider what scientists call the “rooster syndrome”—the rooster crows; the sun comes up; but no one should assume that the crowing rooster caused the sunrise. As biologist Stephen Jay Gould wrote in 1996, “The invalid assumption that correlation implies cause is probably among the two or three most serious and common errors of human reasoning.”

The IMF seems to acknowledge this. In a footnote (footnote 21, page 106), the GFSR states that “The evidence on contemporaneous price effects does not conclusively prove that fund flows drive returns. For example, fund flows and returns could both be driven by news.” In other words, a third factor—new information, such as reports on the strength of the economies or the likelihood of changes to monetary policy—could cause changes in both fund flows and returns.

In fact, because securities prices and fund flows vary daily, we are skeptical that analysis based on weekly data, such as the IMF’s, can provide any evidence on whether fund flows affect returns. Consider two cases:

1. Investors redeem from funds on Monday; securities prices fall on Tuesday.
2. Securities prices fall on Tuesday; investors redeem from funds on Wednesday.

The first case would support the possibility of destabilizing fund flows; the second case would not. But with weekly data, these two cases are indistinguishable: in both cases, weekly data will show fund outflows and a drop in securities prices. 

Misleading Interpretations of Weak Statistical Results

Table 3.2 invites readers to believe that the IMF’s statistical results are stronger than they actually are.

In the 12 cases presented (two tests on six categories of funds), the table answers “Yes” nine times to the question of whether fund flows affect or help predict market returns. The message: fund flows typically affect, or at least help predict, market returns.

Once again, however, the IMF hedges in the fine print. On the first test (reported in Table 3.2 in the line labelled “Surprise flows have a significant impact on returns”) for the broad categories of U.S. equity and bond funds, the claimed causation occurs in only three of the eight years tested. And for high-yield bond funds, the table’s footnote explains that “for the entire sample period, the results are not significant.” (The IMF feels those results still deserve a “Yes” because “subperiod” estimates are statistically significant—but the result “declines steadily over time.”) As might be expected, this kind of approach—where one sifts selectively through the data—is frowned upon by economists because it biases statistical tests in favor of one’s conclusions.

The reported results of the second test—shown in the last row of Table 3.2—are even more misleading. Here, the IMF suggests that, for all U.S. bond funds and U.S. municipal bond funds, fund flows help predict market returns. But it qualifies these results (reported in the table as “Yes***”), noting they are “not robust to all specifications.” In other words, minor changes in the model produced results that varied and weren’t always statistically significant. A more balanced approach would have specified these cases in the table as “Inconclusive”—or even “No.”

The IMF might argue that identifying results as “Yes***” alerts readers that those results are inconclusive. Many readers, however, will not be well enough versed in statistics to understand that “not robust to all specifications” means “inconclusive.” Moreover, even those well versed in statistics will have difficulty interpreting the GFSR’s results and footnotes in Table 3.2. One reason is that the GFSR fails to follow the standard practice in economic research of reporting coefficients, standard errors, or other common statistical measures (such as R-squared) that readers could use to judge the validity of the conclusions.

Diving Deep into the Data

In the last line of Table 3.2, the only unqualified “Yes” on whether “Flows help predict returns” is for emerging-market (EM) bond funds. That caught our attention, because in an ICI Global paper, we used exactly the same weekly data from EPFR Global and the virtually the same statistical approach, and found just the opposite—namely, that that funds’ purchases of EM bonds (or “flows” in IMF terminology) did not have a persistent effect on future returns on those bonds. So we dove into the data to check whether the IMF had found something that we had missed.

What we found confirmed our earlier results: there is very little, if any, evidence that funds’ net purchases of securities help predict future returns in emerging markets. The IMF examined fund flows to 23 different emerging countries. Using vector autoregressions (the same statistical technique used by the IMF), we found no statistical evidence that funds’ net purchases had an overall effect on future bond returns in 20 of those countries after 26 weeks (see the first column of Table 1 below). In two countries (Ukraine and Sri Lanka), we found a positive and statistically significant effect of funds’ net purchases of that country’s bonds on future returns of those bonds after 26 weeks. In the case of China, we found a significant effect at 26 weeks—but it was negative. Taken at face value, this would indicate that fund flows to China cause Chinese bond prices to fall—the opposite of the effect that the IMF suggests.

We looked at effects over shorter time periods—eight and four weeks. We did see evidence that fund flows had a statistically significant impact on returns for three countries over eight weeks, and for eight countries over four weeks. But these effects were both transitory and economically small—raising questions about the robust “Yes” that the IMF assigned them in its Table 3.2. (We have plotted the results of our tests—the “impulse responses” of asset returns to fund flows—for all 23 countries on our website.)

Table 1: Estimated Effect of Fund Net Purchases on Returns Across 23 EM Countries

Source: Investment Company Institute calculations, based on J.P. Morgan and EPFR Global data

If the effect at the country level is not statistically significant in two-thirds or more of the cases, how did the IMF conclude that fund flows cause instability in emerging bond markets? The answer, found in the annex to the asset management chapter, is that the IMF assumed that all 23 countries follow the same statistical model.

For this assumption to be true, all 23 countries must have, in effect, the same responses to economic and financial shocks on average. In that case, fund flows to Brazil and returns on Brazilian bonds should on average follow essentially the same pattern as flows and returns in Bulgaria, Chile, Egypt, Indonesia, Lebanon, Malaysia, South Africa, and Ukraine—among others. Given the vast differences in these countries—in their economies, market conditions, political situations, demographics, and other characteristics—this assumption seems highly questionable.

When we “relax” this assumption—when, in other words, we do not force the statistical models to be identical across all of these countries—we get the results detailed above: the effect at the country level is not statistically significant in two-thirds or more of the cases.

Our analysis indicates that the IMF’s conclusion that “the evidence is consistent with mutual fund flows affecting asset returns in smaller, less liquid markets” is no longer valid. Put differently, the IMF’s empirical results change dramatically, and are no longer statistically significant, when reasonable changes are made to the underlying assumptions.

Other Posts in This Series: 

• The IMF Is Entitled to Its Opinion, but Not to Its Own Facts
• The IMF Quietly Changes Its Data, but Not Its Views
• The IMF on Asset Management: The Perils of Inexperience
• The IMF on Asset Management: Which Herd to Follow?
• The IMF on Asset Management: Sorting the Retail and Institutional Investor “Herds”
• The IMF on Asset Management: Handle Empirical Results with Care