In surveying some of my favorite blogs recently, I have come upon something that hadn’t previously occurred to me, but could potentially alter how I invest. That is the problem with back testing Investing Strategies. Greenbackd posted an interesting starter piece on this subject called Walking the Walk, that led me back to the original blog from Aswath Damodaran called Transaction Costs and beating the Market. I have often thought there were practical problems with back testing, but I had not tried to articulate them until I read these posts. Both are excellent and worth reading. Damodaran, who is a Finance professor at NYU, and an author of Investment Fables (which I own), writes about the many ways to beat the market in general terms and then goes on to say, “Most of these beat-the-market approaches, and especially the well researched ones, are backed up by evidence from back testing, where the approach is tried on historical data and found to deliver “excess returns”.

But what is back testing? It is a process where one takes an idea like investing in low price to book stocks. You formulate a method, for example, that you will buy only price to book stocks in the lowest quintile (20%) on the last trading day of the month and rebalance your porfolio quarterly. Then you test this idea in past period of time. These are usually tested over longer periods. In my post last month about Ben Graham’s Stock Selection Criteria, Oppenheimer back tests the strategy from 1974-1981. In the post on Buying Low Price to Book stocks, Ibbotson back tested from 1967-1984.

I have used these and other academic studies to form the basis of an investment strategy that underpines this website. In fact, it was some of this same back tested research that sparked the idea of 80/20 investing, which I am still developing. But I am not an academic, I am a small time investor. And like many of you, I am really only interested in real returns. So I am keenly aware of some of  the practical limitations of back testing. When I began my research on Graham, years ago, I was puzzled by something. Graham had developed a set of investing criteria. And someone set up a fund that set out to capitalize on this method. The studies show that it should have worked. But it didn’t. The fund had to disband a few years later. Why? What was the disconnect?

So what is the Problem with Back Testing? It turns out there are several Problems for the practical investor.

1) Friction- Friction are the Brokerage Commissions, taxes and bid-ask spread issues that Damodaran referred to. Broker commissions and taxes you can calculate, but because many of the value investing approaches studied deal with small cap stocks that are often illiquid, the difference between the bid for a stock, the amount an investor is offering to buy the stock at, and the ask price, that is the price the seller of that stocks is asking for, can often be quite large, sometimes 10% or more. This cannot be easily calculated in an academic study.

2) Buying on an arbirtrary time frame- For academic research you must attempt measure results over a given time , with specific criteria, but the best deals will not always occur on the end of the month, or once a year.  Rebalancing at mechancial intervals, may also limit your profits, by telling you exit a position too early or too late.

3) Survivor bias- Taleb speaks about this as  a general problem with the investing community in general, and economists in particular. Is the back testing incorporating the number of companies that fit the strategy but failed or when bankrupt?

4) Portfolio limits- If you have a limited investing portfolio such as $5,000, you cannot know going into a year knowing how many opportunities you will have. With such a small portfolio you cannot possibly invest in every opportunity that arises. But that is the very assumption you supposed accept. If 40 opportunities arise, you buy all forty, and if five arise then buy only five. In what proportions? Do you leave everything else in cash?

Are their any solutions to these problems? Some. But not neccessarily solutions that you will be happy with.

1) Friction- There are several ways of dealing with the friction issues. Some I have written about before. For brokerage commissions, select a good discount broker or one that doesn’t charge fees for maintaining a certain balance. Tradeking, my primary broker, charges just $4.95 a trade. Taxes can be controlled by  investing in tax advantaged accounts like ROTH or traditional IRA’s. But bid-ask spreads are a little trickier.  My favorite strategy is to sell on good news. I have noticed that the volume on thinly traded stocks tends to go up when there is good news and investors are trying to buy the equity. Since value investing is also a contrarian strategy, good news is often a catalyst for a price rise and a good opportunity to exit a position. Another tactic is simply patience. Set up flags when a stocks price reaches a certain price. Then put in a limit order. And wait.

2) Buy when opportunities present themselves. Don’t be afraid of cash. It is a better alternative than losing money. I think Graham’s idea of holding for a 50% gain or two years is a good starting place for parameters to frame your investment horizon. But you shouldn’t completely exit a position just because you have reached a 50% gain, if your analysis suggests  a larger upside. Perhaps, in that instance you take a portion of your profits off the table. Two years should be the outside to hold an investment that is not performing. Maybe. If you discover a much better investing opportunity, two years could be too long.

3) Surviving Survivor bias. Look for tools to minimize the chances of a company failing. The most important thing is preservation of capital or as Buffett says, “Rule number #1 is don’t lose money. Rule #2 is don’t forget Rule #1. ” These tools may include the Altman Z score, the Piotroski F score and other more simple financial metrics such as the acid test.

4) Portfolio- Decide how many companies you are willing to invest in. Invest in great investment opportunties when they arise. Keep a Focused Investment strategy. It is possible that if you chose to invest in 8 companies, that after a price drop, one of the companies you have already invested in is the best value, and you should add to your position.

This does not mean a simple approach to investing cannot work. But I am more suspicious of mechanical investing that I was before.

Leave a Reply