[#!: full-line executeEmbeds, phraseValueList = (("fout fout)("backtrack backtrack)) [#!: path_insertIn_fHand (link d_webWork 'fin Head_one.html') fout ; Howell - 1872-2020 SP500 index, ratio of opening price to semi-log detrended price [#!: path_insertIn_fHand (link d_webWork 'fin Head_two.html') fout ; [#!: path_executeEmbedsInsertIn_fHand (link d_webWork 'Menu.html') phraseValueList ; [#!: path_executeEmbedsInsertIn_fHand (link d_webWork 'Menu projects.html') phraseValueList ; [#!: path_insertIn_fHand (link d_webWork 'normalStatus.html' ) fout ;

1872-2020 SP500 index : a [time, mind]-bending perspective

"... History never repeats itself, but it rhymes. ..."

Table of Contents


"Key [results, comments]"

Standard & Poor's index of stock market prices for 500 large capitalisation companies (SP500) from 1872 through 2020 is shown in the semi-log plot below. Two "eyeballed" linear trend lines are shown for the periods [1872-1926, 1926-2020].



The ratio of the historical SP500 price to its semi-log trend price for the same date is plotted in the graph below. The price scaling is according to Fibonnacci ratios, as commonly used by many stock market technical analysts, which I often find to be helpful or at least interesting, as it so often provides convenient matches with data series.



The real fun starts when I rather [arbitrarily, haphazardly] copy-pasted sections of the graph from the 1950-2020 timeframe, and shuffled them to lie over sections of the 1872-1960 section of the original graph. Here is one possible result :



Wow! Even knowing that the [eyes, mind] often see patterns that aren't really there (as per random noise), one can basically re-create the essence of the 1872-1960 timeframe simply by copying ONLY 4 chunks of the 1950-2020 time-frame!! Of course, there is nothing new about this - TradingView members are often comparing current market squiggles to the past, over different timescales. I would actually be surprised if the aqbove graph hasn't already been done hundreds of times before. [System_T, Amjad Farooq, TexasWestCapital, perhaps Harry Dent] and others are examples of recent pattern-matching for the SP500, with their comparisons of the 2020 crash to other time periods. But my overlays in the graph above above did not involve [re-scaling, rotating] or other transformation of the time segments, transparencies], so that is [noteworthy, simple, pure]. Scale is important, even if only to confirm the applicability of multi-scalar processes.

You will notice that I was too cowardly to fill in the 1890-1900 section with an overlay, perhaps from ~1970-1980. That would be equivalent to a forecast of sorts, but what I learned from commodity chemical market research is that "Forecasting is a particularly hazardous occupation, especially if you try to apply it to the future". I also feel that it is "dangerous" to have only one forecast, just as it is dangerous to "believe" only one scientific theory, even if it is an overwhelmingly dominant mainstream consensus science theory-cum-cult-cum-religion. I try to force a "multiple conflicting hypothesis" attitude on myself, even if I have to find a really [defunct, dishonest, abandoned] theory. Otherwise, I often find that I become a tool of the theory, rather than the theory being a tool to be used if and when [desired, needed, appropriate]. Most of all, a theory should NOT become an impediment to [thinking, progress], as is far too often the case.

So maybe YOU can generate forecasts, not necessarily with the transparency overlays.

Play with the [time, mind]-bending perspective yourself

While you probably don't have the gimp (GNU image manipulation program) installed on your computer (yet), it is available for free (I think on MS Windows as well, not just Linux?). With gimp, you will be able to work with my .xcf format file SP500 time-section transparencies. If you are new to gimp, be prepared to lose a lot of hair and gain a lot of wrinkles - it's not the the easiest learning curve, but it is powerful (and cheap!).

You can move these transparencies around in gimp (but not in the image above, which is a jpg format). Furthermore, you can do your own models by copy-paste of sections of a graph, and moving them around. Of course, with the [spreadsheet, .dot text] files, you can impose your wown data transformations as well.

Deja Vu all over again (Yogi Berra) : - This "chopping up a time series" and omitting sections to recreat a time seeries with a model is eerily familiar : Perhaps this section is saying something important about an over-reliance of modern approaches on [modelling, statistics], at the expense of simple pattern-matching? And perhaps it is just me who has this problem...

Furthermore, a key lesson may be what I have long felt for [physics, astronomy, climate, etc] series : the real challenge isn't prediction of the next steps in a time series, but to predict [when, to which state] a system will change in the future.

Ratio of actual to semi-log detrended data : [advantages, disadvantages]

Time greatly distorts the measuring sticks for [money, economics, finance, markets], so it is very important to convert the data before making comparisons across time. This web-page uses a simple semi-log conversion of the S&P500 index opening prices from 1872 to 2020. While other basis such as adjustments with [Producer, Consumer,]-Price Indexes or populations might be better in some ways, the current approach has the advantage that it does not [distort, polute] the data at an early stage, and it is very [easy, clear] to double-check results.

For sure, one would normally do a simple linear regression to generate the [xStart, yStart, slope] parameters for the trend lines for [1872-1926, 1926-2020], however, at this preliminary stage it is more important for me to look at the data and to avoid facetious levels of [accuracy, certainty, self-delusion]. The eye is often much better than ordinary analysis.

line This "semi-log detrended" format has many advantages :

Some disadvantages of the format are :

Future potential work

It's unlikely that I will get much time for this, but here are some ideas that could be interesting.

Comparison of [TradingView, Yahoo finance] data

It's always important to use several sources of data, where possible. [errors, fraud, deception, incomplete disclosure, mis-understood] data treatments aren't rare in science (I don't know about [economics, markets]). But more important is that I may not have understood the [data, analysis, modelling, reality] or it's limitations well enough. Here I merely compare two sources of the same data, which brings some confidence.

TradingView versus yahoo finance - the data series are essentially the same, as no black curve excursions above, below the blue yahoo curve are seen.


TradingView versus 2 times yahoo finance - by multiplying the yahoo data by 2, it is much easier to see the essentially perfect similarity of the data series. This helps as the first comparison above leaves me thinking that maybe something is missing.


[data, software] cart [description, links]


While most results are provided in sections above, links to data [spreadsheets, text files] and software [???, source code] are listed below along with brief comments. A full listing of files (including other SP500 web-pages) can be seen via this Directory's listing. Hopefully this will help those who want to do something different, as the programs etc mayhelp with [learning, debugging].

Data :
Software :

17Sep2020 initial posting
[#!: full-line executeEmbeds, phraseValueList = (("fout fout)("backtrack backtrack)) [#!: path_executeEmbedsInsertIn_fHand (link d_webWork 'fin Footer.html') phraseValueList ;