#] #] ********************* #] "$d_PuetzUWS"'0_PuetzUWS PSOmodel notes.txt' - # www.BillHowell.ca 01Mar2019 initial # view in text editor, using constant-width font (eg courier), tabWidth = 3 #48************************************************48 #24************************24 # Table of Contents, generate with : # $ grep "^#]" "$d_PuetzUWS"'0_PuetzUWS PSOmodel notes.txt' | sed "s/^#\]/ /" # ********************* "$d_PuetzUWS"'0_PuetzUWS PSOmodel notes.txt' - +-----+ References : Market-related [file, dir]s : Setup, ToDos : +-----+ 09Jul2022 emto Puetz : TradingView.com - PuetzUWS SPX [time, price] multiFractal mirrors, 150y 1872-2022 19Jan2021 specific numbering of cycles, phase angles 18Jan2021 gnuplot, multi-fractals and fractional order calculus, sort_fixAngles_save_allBest 17Jan2021 ~04:00 Another run : wee hours of 18Jan morning 17Jan2021 PSO_optimize - Optimize, baby! 17Jan2021 Bug hunt in PSO_optimize 15Jan2021 use my simple structures - tooproblems with mixed-p variables!! 15Jan2021 Bug hunt PSO_optimize 13Jan2021 adjustments to code 13Jan2021 continue [symbol redefs, LOCAL, NONLOCAL] add extra code 10Jan2021 Improved PuetzUWS results I also have to run much larger iterations of PSO (overnight) 10Jan2021 Fourier series analysis 09Jan2021 PSO to fit SP500 with PuetzUWS (cont'd) 08Jan2021 PuetzUWS_refncParams - PSO to fit SP500 with PuetzUWS 07Jan2021 Puetz_UWS_optimisation - PSO to fit SP500 with PuetzUWS 06Jan2021 Linear regression fits of [DJIA 1872-1926, SP500 1926-2020] 07Jan2021 Fibonacci series : 06Jan2021 Linear regression fits of [DJIA 1872-1926, SP500 1926-2020] 04Jan2021 what is WebPlotDigitizer? I seem to have used it? #24************************24 #] +-----+ #] References : Puetz, Stephen J., Prokoph, Andreas, Borchardt, Glenn, Mason 2014, Evidence of Synchronous, Decadal to Billion Year Cycles in Geological, Genetic, and Astronomical Events: Chaos, Solitons & Fractals, v. 62–63, p. 55-75 [ http://dx.doi.org/10.1016/j.chaos.2014.04.001]. Puetz SJ, Prokoph A, Borchardt G (2015). Evaluating alternatives to the Milankovitch Theory. Journal of Statistical Planning and Inference, in press, uncorrected proof. http://www.sciencedirect.com/science/article/pii/S0378375815001901 Puetz SJ and Borchardt G (2015). Quasi-periodic fractal patterns in geomagnetic reversals, geological activity, and astronomical events. Chaos, Solitons Fract (2015) 81:246-270. http://dx.doi.org/10.1016/j.chaos.2015.09.029 Stephen J. Puetz, Glenn Borchardt 2011 "Universal Cycle Theory : Neomechanics of the hierarchically infinite universe" http://www.OutskirtsPress.com ISBN: 978-1-4327-8133-0 http://www.amazon.com/dp/1432781332 Stephen J. Puetz 2009 "The Unified Cycle Theory: How cycles dominate the structure of the universe and influence life on Earth" http://www.OutskirtsPress.com Denver Colorado USA, ISBN: 978-1-4327-1216-7 http://www.amazon.com/dp/1432712160 http://www.uct-news.com/ #24************************24 #] Market-related [file, dir]s : Market data related : "$d_mktData"'[, detrend, digitize graphs, rawCsv, semi-log, SLregress, TV Fibonacci mirror, unixDate, yrFraction]/' "$d_mkts"'Fischer, David 1996 The Great Wave/0_Fischer notes.txt' also, many other [data, analysis] in "$d_[mkts, Qroot]" "$d_SysMaint"'digitizelt graph from image/digitizelt - produce graphical data from image.txt' PuetzUWS [time, price] multifractals "$d_Qmkts"'PuetzUWS calc [time, price] fractals.ndf modelling as continuous systems : "$d_mkts"'PE Schiller forward vs 10yr Tbills/' "$d_PuetzUWS"'0_PuetzUWS PSOmodel notes.txt' #24************************24 #] Setup, ToDos : -18Jan2021 Something is wrong with my setup of the thetaNweightLs, maybe : y get rid of : factors_normal_convertTo_PSO factors_PSO_convertTo_normal Puetz => specific numbering of cycles (I corrected) Puetz => phase angles never change but I was using time lags (should be only for later tuning) L1 norm?!? very short MAXITER? actual [solar, lunar, geomag, human] data resonant oscillator - effects between periods? (multifractal, fractional order calculus, RecurrentNN) frequency splitters (non-linear) in electronics greater lags than accommodated by -PI <= thetaN <= PI (use lags, not phases) use 2014 HUWS fractional order calculus + RecurrentNN I can't find the correct 2014 reference with [labmda, thetaN] numbers for cycles -19Jan2021 y best approach by far is pattern match! do this for David Fischer's data fractional order calculus - Cabuto -20Jan2021 unshackled time lags - phase is NOT enough! especially with [multually influencing, fractional order calculus, etc] #] +-----+ #24************************24 #08********08 #] ??Jul2022 #08********08 #] ??Jul2022 #08********08 #] ??Jul2022 #08********08 #] ??Jul2022 #08********08 #] ??Jul2022 #08********08 #] 09Jul2022 emto Puetz : TradingView.com - PuetzUWS SPX [time, price] multiFractal mirrors, 150y 1872-2022 see emto Puetz as of this date. I hope that this email isn't a repeat, but I can't check due to problems with the Evolution email program that I use (invisible folders - the emails are usually still there, but a real pain to make visible again, meaning I have been too lazy to develop a script to do so). My memory is now terrible, so I can't rely on it either. As always, I appreciate your newsletter. This time it reminded me to send an email regarding my 19Jun2022 primitive implementation of your UWS in the TradingView.com system. TradingView provides near-real-time updating of market data at different timescales, and a huge slection of market symbols (including Federal Reserve time series) to select from. The implementation is for a "DETRENDED SPX (Standard & Poors 500 index, with DJIA proxy prior to ?1946? or something, when the SPX was created). The detrended view is NOT a stand-alone, but intended to be used to complement a semi-log plot. : My Pine Script (easily viewable coding) : my TradingView "home" page : https://www.tradingview.com/u/Bill_Howell/ select the "Scripts" tab : https://www.tradingview.com/u/Bill_Howell/#published-scripts click the "Script" : PuetzUWS [time, price] multiFractal mirrors, SPX 1872-2020 My "live" chart : https://www.tradingview.com/chart/Dpe2yXH0/ Unfortunetly, I have NO idea if others actually see the chart! (probably paying members only?) You can easily click the timescale of interest : 1D5D 1M 3M 6M 1Y 5Y All My normal semi-log plot of SPX : https://www.tradingview.com/chart/Dpe2yXH0/ I always us 3 other charts set up for normal viewing when I am following the markets (each chart with ~8 symbols). I don't invest, being far too good at losing money, but I did trade to learn, and learned that I lose money that way too. My initial posting was removed : I keep putting in links to references etc which isn't allowed, and documentation is a pain, so I just ask that reader search for my webPages. Therefore, the current version provides almost no description, thereby keeping me out of TradngView jail. To date, only 5 people have "liked" the script, versus hundreds or thousands for popular scripts. TradingView notes : you have to become a member to view, but a freee level of membership is available you might have to be a subscribe to use the Pine Script on a chart? setting up the chart is [time-consuming, tricky for newbies], but at least I've provided market [symbol, title]s at the top of the graph you can hide any of the symbol plots easily, which helps either clarity, or multi-symbol comparison. Plot 2 years older than the current "live" version on TradingView : http://www.BillHowell.ca/economics, markets/SP500/PuetzUWS/Howell - PuetzUWS SPX [time, price] multiFractal mirrors, 150y 1872-2022.png This contrasts in appearance a version that I did May-Sep2020 in the Q'Nial programming language, but without the PuetzUWS [time, price] multi-fractals. : PriceFractal comments : the priceFractals seem spookily accurate, even over a the last 150 years (at least to me) users, including day-traders, don't have to keep hand-crafting fractals with very [simple, easy, fast] TradingView tools. Most will though, as they use Fibonnacis unfortunatly, I did not build a tool to easily view priceFactal for visual comparison to normal semi-log prices over the timescale being viewed. Normal prices change along a horizontal priceFractal. But that would be a pain program in Pine Script. TimeFractal comments : Does the timeFractal say something? The eye can be deceiving, and we see patternas that may not be there, but to me there is a significant clarity the timeFractal is mangled, which is frustrating. Most of the time it worked well, but trench warfare to get the priceFractal working ended up corrupting the timeFractal, so many "spikes" (vertical bars) are missing at most timescales. I used periodicities from your ?2011? book, rather than try to sift through your recent papers. the "base-time" for all timeFractals was set to ?1929?, and it may have been shifted a bit. In realityy coding isn't good enough to set it up properly. General comments : both [time, price]Fractals scale with the timescale being viewed - after all, that's a key property of fractals!! no half-UWS for [time, price]Fractals - I felt that this would overly [clutter, carpet-bomb] the charts. At some point excessive grids become meaningless and cloud the bird's eye view. I didn't try to build a regression-style model of the SPX,s you often show in your newsletter. At this stage, I more interesting in time "spikes" (so-called because by priority interest is in spiking neural networks). PuetzUW similar to Fibonnaci [0.318, 0.618]. Everybody uses Fibonacci series, so in a way they would be more comfortable with that. I feel it's better not to confuse readers - just stick with PuetzUWS as I think in the end it's better. I have zero intention of going further with TradingView - it's very good at what it was intended to do, a royal pain of work-around for even primitive programming outside that context. ********************* Howell webPages - very [incomplete, full of errors, links don't work] : I have two webSite directories related to your work. In addition to webPages, the directories include [image, note, etc]s. http://www.BillHowell.ca/economics, markets/SP500/PuetzUWS/Howell - PuetzUWS [time, price] multifractal of detrended SPX 1871-2020.html http://www.BillHowell.ca/Projects - mini/Puetz & Borchardt/Howell - possible phenomenological origins of Puetz Universal Wave Series (UWS).html http://www.BillHowell.ca/Projects - mini/Puetz & Borchardt/Howell - preliminary frequency analysis, Fourier series as origins of Puetz UWS.html There are too many errors in this, but the issue is to look into how stable the UWS series may be from a dynamic point of view (eg frequency [split, scavenge, harmonic, difference, etc]). http://www.BillHowell.ca/Projects - mini/Puetz & Borchardt/Howell - Quick introduction to Puetz Universal Wave Series (UWS) fractal time.html http://www.BillHowell.ca/Projects - mini/Puetz & Borchardt/Howell - References related to Puetz [H]UWS.html 08********08 #] 19Jan2021 specific numbering of cycles, phase angles Puetz => specific numbering of cycles (I corrected) Puetz => phase angles never change but I was accidentally using time lags time lags should be only for later tuning I purchased digitizel for converting a graphical image to data - seems to work wonderfully! 53 $Cdn? $ find "$d_SysMaint" -maxdepth 4 -type f -name "*" | grep --invert-match "z_Old\|z_Archive" | tr \\n \\0 | xargs -0 -IFILE grep --with-filename --line-number "image analysis" "FILE" 08********08 #] 18Jan2021 gnuplot, multi-fractals and fractional order calculus, sort_fixAngles_save_allBest +-----+ 20:38 search "multi-fractals and fractional order calculus" https://www.brown.edu/academics/applied-mathematics/sites/brown.edu.academics.applied-mathematics/files/uploads/11%20EssentialsofFractionalCalculus.pdf >> looks good https://www.hindawi.com/journals/mpe/2010/375858/ Review Article | Open Access Volume 2010 |Article ID 375858 | https://doi.org/10.1155/2010/375858 Ricardo Enrique Gutiérrez, João Maurício Rosário, José Tenreiro Machado, "Fractional Order Calculus: Basic Concepts and Engineering Applications", Mathematical Problems in Engineering, vol. 2010, Article ID 375858, 19 pages, 2010. https://doi.org/10.1155/2010/375858 Fractional Order Calculus: Basic Concepts and Engineering Applications Ricardo Enrique Gutiérrez,1 João Maurício Rosário,1 and José Tenreiro Machado2 1Department of Mechanical Engineering, UNICAMP, 13083-970 Campinas, Brazil 2Department of Electrical Engineering, ISEP, 4200-072 Oporto, Portugal Academic Editor: Katica R. (Stevanovic) Hedrih Received07 Oct 2009 Revised19 Feb 2010 Accepted02 Mar 2010 Published12 May 2010 https://www.academia.edu/29158054/Fractional_calculus_and_continuous_time_finance Fractional calculus and continuous-time finance Finance, 2004 Rudolf Gorenflo Enrico Scalas +-----+ 10:34 gnuplot top fifteen on 3 separate graphs OK, mistake or two or three on every line of code today!!??? I removed from 'test03 DUWS_2011 cycles 26-35.ndf errParam results.dat' : 210117_14h04m13s_0 447.256 3.00481 -0.0456053 1.032 1.3912 -0.583475 0.8527 -0.762975 0.79287 1.31149 1.48436 0.1 0.12 0.14 0.16 0.18 0.2 0.22 0.24 0.26 0.28 >> flatliner, I failed to retain the thetaNweightL for : SP500 1870-2030 y-log detrend yEst_L1L2UWS_L3DUWS_UWS.png Something is wrong with my setup of the thetaNweightLs. Probably just get rid of : factors_normal_convertTo_PSO factors_PSO_convertTo_normal OR try L1 norm?!? very short MAXITER? actual [solar, lunar, geomag, human] data resonant oscillator - effects between periods? (multifractal, fractional order calculus, RecurrentNN) frequency splitters (non-linear) in electronics greater lags than accommodated by -PI <= thetaN <= PI (use lags, not phases) 2014 HUWS +-----+ 07:38 sort_fixAngles_save_allBest IS OP p_tstNumErrParam "210117_14h04m13s_0 "210117_21h17m25s_28 "210117_21h17m25s_793 "210117_21h17m25s_1158 "210117_21h17m25s_1069 "210118_07h24m24s_379 "210117_18h31m58s_0 "210117_21h17m25s_1161 "210118_07h24m24s_1831 "210118_07h24m24s_1502 "210117_21h17m25s_448 "210118_07h24m24s_24 "210118_07h24m24s_177 "210117_21h17m25s_973 "210118_07h24m24s_1253 "210117_14h41m40s_0 > transferred to test03_update 08********08 #] 17Jan2021 ~04:00 Another run : wee hours of 18Jan morning IRang_L IRang_R optrPhr n_particles E_CUTOFF MAXV MAXITER momentum max_runs n_params n_seeds 0. 1. UWS_L2error 2000 5 0.03 100 0.9 1 20 15 seedParamL 0.47823 0.384063 0.992743 0.449189 0.164248 0.882236 0.221416 0.498833 0.907139 0.455139 0.135713 0.603614 0.87857 0.799227 0.12619 0.0355598 0.20873 0.184278 0.236243 0.599387 0.01 0.607013 0.012 0.513251 0.014 0.27449 0.016 0.294658 0.018 0.892139 0.02 0.361255 0.022 0.321672 0.024 0.0776857 0.026 0.273011 0.028 0.455435 08********08 #] 17Jan2021 PSO_optimize - Optimize, baby! Initial run loses seed benefits!!!! In PSO_optimize, I moved pbestx := xx ; After seedParamL % n_pa E_C MAXV MAXI mom max_runs ; % PSO_bitL := 50 5 0.003 2000 0.9 1 ; %nolearn- MAXV too low?, more iterations no help ; PSO_optimize begin time: 210117 16h13m44s iter : 100, 5 best errors : 102922. 125966. 143185. 148289. 157124. iter : 200, 5 best errors : 102922. 125966. 143185. 148289. 157124. iter : 300, 5 best errors : 102922. 125966. 143185. 148289. 157124. iter : 400, 5 best errors : 102922. 125966. 143185. 148289. 157124. iter : 500, 5 best errors : 102922. 125966. 143185. 148289. 157124. iter : 600, 5 best errors : 102922. 125966. 143185. 148289. 157124. iter : 700, 5 best errors : 102922. 125966. 143185. 148289. 157124. iter : 800, 5 best errors : 102922. 125966. 143185. 148289. 157124. iter : 900, 5 best errors : 102922. 125966. 143185. 148289. 157124. iter : 1000, 5 best errors : 102922. 125966. 143185. 148289. 157124. iter : 1100, 5 best errors : 102922. 125966. 143185. 148289. 157124. iter : 1200, 5 best errors : 102922. 125966. 143185. 148289. 157124. iter : 1300, 5 best errors : 102922. 125966. 143185. 148289. 157124. iter : 1400, 5 best errors : 102922. 125966. 143185. 148289. 157124. iter : 1500, 5 best errors : 102922. 125966. 143185. 148289. 157124. iter : 1600, 5 best errors : 102922. 125966. 143185. 148289. 157124. iter : 1700, 5 best errors : 102922. 125966. 143185. 148289. 157124. iter : 1800, 5 best errors : 102922. 125966. 143185. 148289. 157124. iter : 1900, 5 best errors : 102922. 125966. 143185. 148289. 157124. iter : 2000, 5 best errors : 102922. 125966. 143185. 148289. 157124. >> no learning at all! PSO_bitL := 100 5 0.3 500 0.9 1 ; %MAXV too low?, more iterations no help ; ******************************** IRang_L IRang_R optrPhr n_particles E_CUTOFF MAXV MAXITER momentum max_runs n_params n_seeds 0. 1. UWS_L2error 100 5 0.3 300 0.9 1 20 2 seedParamL 0.47823 0.384063 0.992743 0.449189 0.164248 0.882236 0.221416 0.498833 0.907139 0.455139 0.135713 0.603614 0.87857 0.799227 0.12619 0.0355598 0.20873 0.184278 0.236243 0.599387 0.01 0.607013 0.012 0.513251 0.014 0.27449 0.016 0.294658 0.018 0.892139 0.02 0.361255 0.022 0.321672 0.024 0.0776857 0.026 0.273011 0.028 0.455435 PSO_optimize begin time: 210117 17h51m28s iter : 100, 5 best errors : 88698.3 91716.5 120911. 128040. 141985. iter : 200, 5 best errors : 88698.3 91716.5 120911. 128040. 141985. iter : 300, 5 best errors : 88698.3 91716.5 120911. 128040. 141985. ******************************** IRang_L IRang_R optrPhr n_particles E_CUTOFF MAXV MAXITER momentum max_runs n_params n_seeds 0. 1. UWS_L2error 300 5 0.03 100 0.9 1 20 2 seedParamL 0.47823 0.384063 0.992743 0.449189 0.164248 0.882236 0.221416 0.498833 0.907139 0.455139 0.135713 0.603614 0.87857 0.799227 0.12619 0.0355598 0.20873 0.184278 0.236243 0.599387 0.01 0.607013 0.012 0.513251 0.014 0.27449 0.016 0.294658 0.018 0.892139 0.02 0.361255 0.022 0.321672 0.024 0.0776857 0.026 0.273011 0.028 0.455435 PSO_optimize begin time: 210117 18h13m41s iter : 100, 5 best errors : 67758.6 80892.3 92778.6 95583.2 104211. >> Best so far! ******************************** IRang_L IRang_R optrPhr n_particles E_CUTOFF MAXV MAXITER momentum max_runs n_params n_seeds 0. 1. UWS_L2error 1300 5 0.03 100 0.9 1 20 2 seedParamL +------------------------------------------------------------------------------------------------------------- |0.0260734 0.0782202 0.23466 0.703982 2.11194 6.33584 19.0075 57.0225 171.067 513.203 0.1 0.12 0.14 0.16 0.18 +------------------------------------------------------------------------------------------------------------- -----------------------+ 0.2 0.22 0.24 0.26 0.28| -----------------------+ WOW! Some awesome results! dateStampID error thetaN_0 thetaN_1 thetaN_2 thetaN_3 thetaN_4 thetaN_5 thetaN_6 thetaN_7 thetaN_8 thetaN_9 weight_0 weight_1 weight_2 weight_3 weight_4 weight_5 weight_6 weight_7 weight_8 weight_9 210117_14h04m13s_0 447.256 3.00481 6.23759 1.032 1.3912 5.69972 0.852709 5.52022 0.792877 1.31149 1.48436 0.1 0.12 0.14 0.16 0.18 0.2 0.22 0.24 0.26 0.28 210117_21h17m25s_28 37867.2 4.51245 5.83358 1.33096 2.1116 2.91302 3.84082 5.12034 0.911931 3.91352 3.62827 4.24409 1.3979 0.449732 1.96575 1.96089 0.722515 3.9285 3.08567 0.144096 0.786693 210117_21h17m25s_793 57842.9 2.80277 1.95373 4.18282 0.0345537 6.12543 5.79999 1.66976 1.04231 2.97606 3.97397 0.828293 6.18678 0.188399 1.08609 3.75701 4.59449 1.876 0.800365 0.193722 1.37238 210117_21h17m25s_1158 58945.6 1.58014 0.884505 1.60911 2.67587 5.84875 4.31386 4.60188 0.346865 0.846051 0.230068 3.9227 2.14477 0.376663 0.459357 6.48311 0.5865 2.28683 3.06703 3.62057 1.31547 210117_21h17m25s_1069 67129.5 2.28816 0.0331639 4.94947 4.70646 2.58201 4.80404 3.80478 6.20917 5.10961 1.32584 5.81382 2.8479 0.140784 2.10233 0.0382526 1.22354 4.99342 4.59099 2.46379 1.40142 210117_21h17m25s_1161 67873.1 6.25206 2.47914 4.16038 0.310264 2.96191 1.77157 3.95345 1.91531 2.24913 5.02748 1.37238 4.12885 0.281774 2.38808 4.3212 4.2015 1.33985 5.18164 7.59347 4.55414 210117_21h17m25s_448 71074.4 5.937 0.985077 0.158281 4.95502 1.52851 2.99257 0.905237 3.46983 6.03939 0.0182423 1.48421 2.5357 2.10253 3.07351 5.74967 1.38316 0.902206 3.07999 0.297366 8.96619 210117_21h17m25s_973 74372.7 3.6547 1.98012 3.55578 4.02084 1.23034 5.12672 2.88095 4.03569 0.036778 0.964945 0.39295 3.22333 2.15316 3.18743 0.0136679 6.83745 1.80423 4.15656 4.60617 3.92033 210117_14h41m40s_0 75276.2 5.60638 4.30815 1.05178 5.93223 5.7052 2.78357 0.439461 4.725 2.08662 3.74944 3.92568 4.46018 3.24654 1.64911 0.753218 3.70802 5.48839 1.71715 4.01024 6.09636 210117_15h46m16s_0 75748.4 4.44546 4.26065 1.57608 1.9799 5.77002 4.50565 1.0045 4.41267 5.10673 5.35129 1.39249 1.73518 6.8368 3.42938 3.94908 3.46275 2.30092 0.100362 1.84127 0.804073 210117_14h41m40s_1 78012.1 1.52421 1.78671 3.02501 4.82877 2.11589 2.45996 3.5936 0.897004 3.5802 5.02628 0.730051 1.58827 0.594133 1.10166 1.13484 4.65112 2.83383 7.9396 3.23507 3.29602 210117_21h17m25s_234 80765.1 4.17141 4.18515 5.67195 4.71566 4.3834 3.33123 3.10731 1.39784 2.72147 3.52453 1.46092 2.22065 4.88255 0.766433 2.31076 5.261 6.38479 0.370836 2.34024 2.81432 >> I need to get the auto-sort working - too much to go through >> I need the gnuplot routine up & running 08********08 #] 17Jan2021 Bug hunt in PSO_optimize +-----+ 12:54 ?.. Error := apply Optrphr Xx | [ , I_particle ] -->[nextv] ?op_parameter >> Re-try in break mode -->[nextv] xx|[,i_particle] 0.478231 0.992744 0.164248 0.221416 0.907139 0.135713 0.878571 0.12619 0.20873 0.236243 0.01 0.012 0.014 0.016 0.018 0.02 0.022 0.024 0.026 0.028 -->[nextv] apply optrPhr xx|[,i_particle] ?op_parameter -->[nextv] apply optrPhr [xx|[,i_particle]] ?op_parameter -->[nextv] optrPhr UWS_L2error -->[nextv] apply optrPhr (xx|[,i_particle]) ?op_parameter -->[nextv] [xx|[,i_particle]] +------------------------------------------------------------------------------------------------------------- |0.478231 0.992744 0.164248 0.221416 0.907139 0.135713 0.878571 0.12619 0.20873 0.236243 0.01 0.012 0.014 0.01 +------------------------------------------------------------------------------------------------------------- ------------------------------------+ 6 0.018 0.02 0.022 0.024 0.026 0.028| ------------------------------------+ -->[nextv] pxxx := xx|[,i_particle] 0.478231 0.992744 0.164248 0.221416 0.907139 0.135713 0.878571 0.12619 0.20873 0.236243 0.01 0.012 0.014 0.016 0.018 0.02 0.022 0.024 0.026 0.028 -->[nextv] apply optrPhr pxxx ?op_parameter >> Wow! this is a new problem for me! haven't seen this before? link d_Qtest 'PuetzUWS- tests.ndf' UWS_L2error_test >> this has already tested fine, with apply?!?? -->[nextv] apply "UWS_L2error (0.478231 0.992744 0.164248 0.221416 0.907139 0.135713 0.878571 0.12619 0.20873 0.236243 0.01 0.012 0.014 0.016 0.018 0.02 0.022 0.024 0.026 0.028) ?op_parameter -->[nextv] apply "UWS_L2error (solitary (0.478231 0.992744 0.164248 0.221416 0.907139 0.135713 0.878571 0.12619 0.20873 0.236243 0.01 0.012 0.014 0.016 0.018 0.02 0.022 0.024 0.026 0.028)) ?op_parameter WOW! this is even more fucked up thatn I thought! : -->[nextv] UWS_L2error xx|[,i_particle] ?op_parameter UWS_L2error (xx|[,i_particle]) Chicken & egg by changing apply optrPhr' to 'UWS_L2error (xx|[,i_particle]) ' have to loaddefs twice Maybe due to 1 n_thetaNweight vector instead of list? How do I kill the table dimensions? STUPID ME!!! Problem was an undefined variable several optrs down, away from the "apply" qnial> PuetzUWS_optimize PSO_optimize begin time: 210117 14h03m36s iter : 100, 5 best errors : 447.256 137957. 152146. 300413. 461835. iter : 200, 5 best errors : 447.256 137957. 152146. 300413. 461835. +----------------+-+ |run finished! # |0| +----------------+-+ pbest@gbest : 447.256 >> OK,ks like I'm back in business!! +-----+ 10:25 something ewas lost in PSO_optimize? $ diff "$d_Qndfs""optimize - particle swarm.ndf" "$d_Qndfs""z_Archive/210116 21h06m42s optimize - particle swarm.ndf" >> looks OK? Just run again, see why calcs aren't being done! (some error) Ah hah! seedParamL is completely screwed up!! -->[nextv] seedParamL +------------------------------------------------------------------------------------------------------------- |0.478231 0.478231 0.478231 0.478231 0.478231 0.478231 0.478231 0.478231 0.478231 0.478231 0.300481 0.300481 0 +------------------------------------------------------------------------------------------------------------- |?not a number ?not a number ?not a number ?not a number ?not a number ?not a number ?not a number ?not a numb +------------------------------------------------------------------------------------------------------------- |?not a number ?not a number ?not a number ?not a number ?not a number ?not a number ?not a number ?not a numb +------------------------------------------------------------------------------------------------------------- |0.878571 0.878571 0.878571 0.878571 0.878571 0.878571 0.878571 0.878571 0.878571 0.878571 0.552022 0.552022 0 +------------------------------------------------------------------------------------------------------------- ... >> try again, in PuetzUWS_optimize : ?.. Seedparaml := transpose EACH tonumber Allerrparamtable | [ Rowsselect , 2 + ( tell ( 2 * N_thetan ) ) ] -->[nextv] 3.00481 ?not a number ?not a number 5.52022 0.792877 1.31149 1.48436 0.1 0.12 0.14 0.16 0.18 0.2 0.22 0.24 0.26 0.28 ?not a number ?not a number ?not a number -->[nextv] allErrParamTable +------------------+----+-----------+-------------------------------------+------------------------+---------- |210113_11h08m43s_3|9999|3.004813173|6.237592147 1.031999831 1.391197495 |5.699720254 0.852709200|5.52022415 +------------------+----+-----------+-------------------------------------+------------------------+---------- -+-----------+------------+-----------+---+----+----+----+----+---+----+----+----+----+-+-----------+-----+ 6|0.792877167| 1.311489940|1.484360865|0.1|0.12|0.14|0.16|0.18|0.2|0.22|0.24|0.26|0.28| |dateStampID|error| -+-----------+------------+-----------+---+----+----+----+----+---+----+----+----+----+-+-----------+-----+ >> This is OK OH SHIT!!! there were spaces? in 'test03 DUWS_2011 cycles 26-35.ndf errParam results.dat'!!! >> Re-run : -->[nextv] allErrParamTable_seedIndxL 210113_11h08m43s_3 -->[nextv] 3.00481 6.23759 1.032 1.3912 5.69972 0.852709 5.52022 0.792877 1.31149 1.48436 0.1 0.12 0.14 0.16 0.18 0.2 0.22 0.24 0.26 0.28 Problems was that I converted to solitary (dummy) : Change : +.....+ factorTable := (2 * n_thetaN) (gage shape rowsSelect) reshape [factors_normal_convertTo_PSO] ; +.....+ To : +.....+ factorTable := (2 * n_thetaN) (gage shape rowsSelect) reshape factors_normal_convertTo_PSO ; +.....+ +-----+ 08:12 lists_sortupCullOn1st - is a PROBLEM in PSO_optimize I don't want the cull, just the ranking setup.ndf I modified lists_sortupCullOn1st (simple) : IF flag_debug THEN write 'loading lists_sortupOn1st' ; ENDIF ; # lists_sortupOn1st IS OP arrayOfLists - EACH [sortup,cull] set of lists based on first list # sort by "up" (lexicographic comparison - alphabetic for strings) ; # - code from symsExtractTranslate, uses "grade" ; lists_sortupOn1st IS OP arrayOfLists { LOCAL A addressListCulled addressListSorted culledLists sortedLists ; A := first arrayOfLists ; %sort by "up" (lexicographic comparison - alphabetic for strings) based on first List ; addressListSorted := gradeup A ; sortedLists := addressListSorted EACHRIGHT choose arrayOfLists ; } PSO_optimize : transpose added : pbestxSorted := transpose mix iL_sortupParams ; +-----+ 06:55 testing initially in : link d_Qroot 'Working file.ndf' UWS_result_test IS { NONLOCAL xDataL yDataL xDataL_std yDataL_std n_thetaN PSO_thetaNweightL lambdaL factors_normal_convertTo_PSO factors_PSO_convertTo_normal UWS_result_testRes ; xDataL yDataL := xDataL_std yDataL_std ; write (link 'n_thetaN : ' (string (n_thetaN := 10))) ; write (PSO_thetaNweightL := get_test03 "thetaNweightL) ; write (link 'lambdaL : ' (numList_to_str (lambdaL := get_test03 "lambdaL))) ; write (factors_normal_convertTo_PSO := get_test03 "factors_normal_convertTo_PSO) ; write (factors_PSO_convertTo_normal := get_test03 "factors_PSO_convertTo_normal) ; write (UWS_result_testRes := UWS_result PSO_thetaNweightL) ; } # this works UWS_L2error_test IS { LOCAL i_particle error ; NONLOCAL xDataL yDataL xDataL_std yDataL_std xx n_thetaN PSO_thetaNweightL lambdaL factors_normal_convertTo_PSO factors_PSO_convertTo_normal UWS_result_testRes ; UWS_result_test ; write (link 'UWS_L2error_testRes : ' (string (UWS_L2error_testRes := UWS_L2error PSO_thetaNweightL))) ; write (link 'optrPhr = ' (string (optrPhr := "UWS_L2error))) ; write (link 'PSO_thetaNweightL = ' (numList_to_str (PSO_thetaNweightL := get_test03 "thetaNweightL))) ; write 'xx : ' ; write (xx := get_test03 "xx) ; write (link 'i_particle = ' (string (i_particle := 0))) ; write (link 'error = ' (string (error := apply optrPhr xx|[,i_particle]))) ; } # works - key issue is : error := apply optrPhr xx|[,i_particle] ; # if that works, then why does PSO_optimize fail? : >> maybe because xx|[,i_particle] is a verticle vector? check xx|[,0] >> nope, just a list >> Must be a problem within PSO_optimize, [pbest, gbest] could cause errors >> Previously, I put : pbest := n_particles reshape 1e6 ; % each particle's best error? ; >> now change also to : gbest := n_particles reshape 0 ; % global best particle index ; >> need to break trace step-by-step ?.. IF ( Vx @ ( I_param I_particle ) > Maxv ) THEN Vx @ ( I_param I_particle ) := Maxv ; (...) -->[nextv] ?.. ( Vx @ ( I_param I_particle ) > Maxv ) -->[nextv] ?.. Vx @ ( I_param I_particle ) > Maxv -->[nextv] ?.. I_param I_particle -->[nextv] 0 0 o o ?L >> YIKES!! why the ?L -->[nextv] tell_params 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 -->[nextv] i_param 0 -->[nextv] maxv 0.3 -->[nextv] vx@(i_param i_particle) 0.116869 -->[nextv] vx@(i_param i_particle) > MAXV o >> So where did the ?L come from? not obvious... 08********08 16Jan2021 +-----+ Olde code # Olde version - nas helpful (pun intended) # s0_h IS OP s_lab - help, retrieve short description for a label # equivalent to sg_ s0_h IS OP s_lab { LOCAL s_Lab_Indx ; IF (isfault (s_Lab_Indx := find_Howell s_Lab s0_@0)) THEN fault (link 'Error : s0_h (structure component help), unknown s_Lab : ' (string s_Lab)) ELSE s0_@@(2 s_Lab_Indx) ENDIF } # 16Jan2021 this test depends on PuetzUWS load, UWSstd_define_initial s0_h_test IS { helpL := 'list of PuetzUWS parameters, in PSO coordiantes, for use in the PSO optimisation' 'reduces particles velocities as swarm runs' 'Error : s0_h (structure component help), unknown s_Lab : mometum' 'Error : s0_h (structure component help), unknown s_Lab : n_weight' ; s0_helpL := EACH s0_h "PSO_paramL "momentum "mometum "n_weight ; IF (= helpL s0_helpL) THEN write 's0_h_test is OK' ; ELSE write 's0_h_test failed : ' ; write (helpL EACHBOTH = s0_helpL) ; write (picture (transpose mix helpL s0_helpL)) ; ENDIF ; } 08********08 #] 15Jan2021 use my simple structures - tooproblems with mixed-p variables!! setup.ndf - add_ISOP_debug IS OP path qnial> add_ISOP_debug (link d_Qndfs 'Structure.ndf') +-----+ Olde code trash test log (structure sho that, backups over time) % ; % output log entry to keep track of key parameters for the various tests ; p_log := link d_Puetz fnameTest ' setup.log' ; fout := open p_log "a ; PSO_pic := post 'PSO_paramL : ' (picture (9 2 reshape "optrPhr optrPhr "n_particles n_particles "E_CUTOFF E_CUTOFF "MAXV MAXV "MAXITER MAXITER "momentum momentum "max_runs max_runs "seedIndxL seedIndxL "seedParamL seedParamL ) ) ; SP500_pic := post 'SP500paramL : ' (picture SP500paramL) (3 2 reshape "UWS_verPhr UWS_verPhr "UWS_rowIL UWS_rowIL "UWS_colIL UWS_colIL ) '' ; UWS_pic := post 'SP500paramL : ' (picture SP500paramL) (3 2 reshape "UWS_verPhr UWS_verPhr "UWS_rowIL UWS_rowIL "UWS_colIL UWS_colIL ) ; writeL := '************************************' (link 'PuetzUWS_optimize log, ' timestamp_DDMMMYYYY_HMS) (link 'fnameTest : ' fnameTest) '' (PSO_pic SP500_pic UWS_pic) '' ; write post writeL ; % fout EACHRIGHT writefile writeL ; close fout ; 08********08 #] 15Jan2021 Bug hunt PSO_optimize +-----+ ?.. Error := apply Optrphr Xx | [ , I_particle ] -->[nextv] ?op_parameter ?.. IF ( Error < Pbest @ I_particle ) THEN Pbest @ I_particle := Error ; (...) -->[nextv] ?.. ( Error < Pbest @ I_particle ) -->[nextv] ?.. Error < Pbest @ I_particle -->[nextv] optrPhr UWS_L2error -->[nextv] i_particle 0 -->[nextv] xx 0.00414971 0.448169 0.384064 0.961273 0.123186 0.0124491 0.911016 0.44919 0.536883 0.385413 0.0373473 0.119586 0.882235 0.72909 0.810901 0.112042 0.193461 0.498833 0.880408 0.00925989 0.336126 0.476912 0.455139 0.52156 0.851008 1.00838 0.210305 0.603614 0.940779 0.672059 3.02514 0.645433 0.799226 0.597586 0.632906 9.07541 0.600288 0.0355598 0.653786 0.175224 27.2262 0.794918 0.184277 0.141942 0.623868 81.6788 0.892283 0.599387 0.898681 0.127729 0.01 0.727828 0.607013 0.0684692 0.761302 0.012 0.997294 0.513251 0.216841 0.445877 0.014 0.274902 0.27449 0.349186 0.769667 0.016 0.974492 0.294658 0.31737 0.0419098 0.018 0.725584 0.892139 0.175834 0.249515 0.02 0.851333 0.361255 0.610579 0.997389 0.022 0.878701 0.321672 0.342426 0.145985 0.024 0.0817563 0.0776857 0.664121 0.875166 0.026 0.391936 0.273011 0.500378 0.845397 0.028 0.760484 0.455435 0.492228 0.867938 -->[nextv] xx|[,i_particle] 0.00414971 0.0124491 0.0373473 0.112042 0.336126 1.00838 3.02514 9.07541 27.2262 81.6788 0.01 0.012 0.014 0.01 6 0.018 0.02 0.022 0.024 0.026 0.028 -->[nextv] gage shape xx|[,i_particle] 20 +-----+ >> I can't see why the error ?op_parameter -->[nextv] optrPhr UWS_L2error -->[nextv] isphrase optrphr l -->[nextv] apply optrPhr xx|[,i_particle] ?op_parameter -->[nextv] apply optrPhr [xx|[,i_particle]] ?op_parameter -->[nextv] a := xx|[,i_particle] 0.00414971 0.0124491 0.0373473 0.112042 0.336126 1.00838 3.02514 9.07541 27.2262 81.6788 0.01 0.012 0.014 0.016 0.018 0.02 0.022 0.024 0.026 0.028 -->[nextv] apply optrPhr a ?op_parameter Maybe error is from next optr : UWS_result params Try : -->[nextv] apply "UWS_result xx|[,i_particle] >> OK - this is ALL ?op_parameter go to stepv debug mode, but it doesn't get there apply "UWS_result xx|[,i_particle] Calcs are STILL going past MAXITER!? flag_break UWS_result UWS_result IS OP PSO_thetaNweightL qnial> PSO_thetaNweight_test := 0.00414971 0.0124491 0.0373473 0.112042 0.336126 1.00838 3.02514 9.07541 27.2262 81.6788 0.01 0.012 0.014 0.016 0.018 0.02 0.022 0.024 0.026 0.028 0.00414971 0.0124491 0.0373473 0.112042 0.336126 1.00838 3.02514 9.07541 27.2262 81.6788 0.01 0.012 0.014 0.016 0.018 0.02 0.022 0.024 0.026 0.028 qnial> UWS_result PSO_thetaNweight_test ------------------------------------------------------------- Break debug loop: enter debug commands, expressions or type: resume to exit debug loop executes the indicated debug command current call stack : uws_result ------------------------------------------------------------- -->[stepv] nextv ?.. Normal_thetanweightl := Factors_pso_convertto_normal * Pso_thetanweightl -->[nextv] ?no_value ?no_value ?no_value ?no_value ?no_value ?no_value ?no_value ?no_value ?no_value ?no_value ?no_value ?no_value ?no_value ?no_value ?no_value ?no_value ?no_value ?no_value ?no_value ?no_value ?.. % -->[nextv] Factors_pso_convertto_normal ?no_value >> that explains it. Factors_pso_convertto_normal is supposed to be global? PuetzUWS_optimize does that - but Factors_pso_convertto_normal isn't even in the code!! 08********08 #] 13Jan2021 adjustments to code OUCH!! I was integrating link d_Puetz 'PuetzUWS- tests.ndf' into my normal code That's a no-no - the data in tests has to be stable!! 20 (gage shape rowsSelect) reshape factors_normal_convertTo_PSO factorTable := transpose ((2 * n_thetaN) (gage shape rowsSelect) reshape factors_normal_convertTo_PSO) factorTable := ((2 * n_thetaN) (gage shape rowsSelect) reshape factors_normal_convertTo_PSO) seedParamL := factorTable EACHBOTH * (transpose seedParamL) +-----+ olde code # PuetzUWS_testSetup IS - # 13Jan2021 I made global [SP500paramL, UWSparamL, PSO_paramL] to allow setup use for : # while working on other parts of the program # quick PuetzUWS_testSetup edit loops to adjust optimization parameters # avuse of other test %seedParamL := rows allErrParamTable|[rowsSelect,] ; %seedParamL := TWIG tonumber (2 EACHRIGHT drop seedParamL) ; %seedParamL := factors_normal_convertTo_PSO EACHRIGHT * seedParamL ; n_rowsSelect := gage shape allErrParamTable_seedIndxL ; rows_seed cols_seed := gage shape allErrParamTable_seedIndxL ; n_seeds n_params := [gage shape, gage shape first] seedParamL ; 08********08 #] 13Jan2021 continue [symbol redefs, LOCAL, NONLOCAL] add extra code $ d_Puetz="/media/bill/SWAPPER/Website - raw/economics, markets/SP500/PuetzUWS/" $ diff "$d_Puetz""test03 DUWS_2011 cycles 26-35.ndf" "$d_Puetz""test03 DUWS_2011 cycles 26-35.ndf2" +-----+ Olde code # (link `" (str_replace_subStr ' ' '_' timestamp_YYMMDD_HMS) '_') EACHRIGHT link (EACH string tell 10) # olde code % gbest is short for "global best for all particles" from all [iteration, run]s ; i_gbest := find [min, pass] errorL ; paramTable := paramL|[,i_gbest] ; % pbest is short for "global best for each particle" from all [iteration, run]s ; paramTable := link thetaNtable weightL ; strL_n_thetaN := EACH string (tell n_thetaN) ; params_titles := link ( ((n_thetaN reshape ['thetaN_']) EACHBOTH link strL_n_thetaN) ((n_thetaN reshape ['weight_']) EACHBOTH link strL_n_thetaN) ) ; Change : +.....+ % 13Jan2021 I'm removing thetaNweightL etc. Given seedParamL, I only really need lambdaL ; % thetaNweightL - Particle Swarm Optimisation cycle [thetaNL,weightL] parameters ; % (not the [controls, lambda]) ; UWS_verPhr UWS_rowIL UWS_colIL UWS_weightL := UWSparamL ; UWS_versn := table_remove_colTitles execute UWS_verPhr ; % [xCol, yCol] are provided "as seen" in UWS_table, not the "raw" for after removing [row, col] titles ; lambdaL thetaNL := cols UWS_versn|[UWS_rowIL - 1, UWS_colIL - 1] ; % note that necessarily (= n_lambda n_thetaN n_weight) ; n_thetaN := gage shape thetaNL ; % UWS_error needs as global ; % these calculations are the inverse of those in UWS_error ; % note the order: thetaNL then UWS_weightL ; thetaNweightL := link (thetaNL / 2 / PI) (UWS_weightL / 10) ; % UWS_error requires lambdaL n_thetaN dataCols xDataL yDataL ; % but PSO_optimize doesn't use the "normal" form of these variables ; % Rather than pass-through variables, they are set as global for direct access by UWS_error ; +.....+ To : +.....+ +.....+ 08********08 #] 10Jan2021 Improved PuetzUWS results I also have to run much larger iterations of PSO (overnight) Now that DUWS works far better (some coding errors corrected), redo UWS L1&L2. # PSO Control variables as required. These are varied for tests ; # b = bug in code, # E_CUTOFF MAXITER max_runs # PSO_controlL := n_part MAXV PSOweight # PSO_controlL_std := 5 0.02 100.0 2000 0.9 2 ; # [done, current] down this list # PuetzUWS_optimize_test 5 0.02 0.3 2000 0.9 1 oops - MAXITER way too large for debugging! # PuetzUWS_optimize_test 20 20 0.3 1000 0.9 1 still a bit too slow for debug # PuetzUWS_optimize_test 20 20 0.3 200 0.9 1 for a fast "bug-hunt" run, passable results. # PuetzUWS_optimize_test 100 5 0.3 2000 0.9 1 Now hunting for a better solution # PuetzUWS_optimize_test 500 5 0.3 2000 0.9 1 too many [particle, iteration]s? ~8 hour run # This run may have converged very early. I stopped it prematurely # It appears that all particles converge on a common solution fairly quickly. Perhaps it is better to use : # smaller [particle populations, iterations] # do multiple runs # PSO_optimize # comment out to get greater diversity of solutions : seed .25374 # eliminate the initial PSO_params_std : # IF (~= null PSO_paramL) THEN xx|[,0] := PSO_paramL ; ENDIF ; # PuetzUWS_optimize_test 30 5 0.3 200 0.9 10 08********08 #] 10Jan2021 Fourier series analysis Do I have old code - for solar activity Ephemeris modelling? $ find "$d_Qndfs" -maxdepth 4 -type f -name "*" | grep --invert-match "z_Old\|z_Archive" | tr \\n \\0 | xargs -0 -IFILE grep --with-filename --line-number "Fourier" "FILE" /media/bill/PROJECTS/Qnial/MY_NDFS/PuetzUWS - Universal Waves Series.ndf:11:[[standard, Fast] Fourier, wavelet] series fits as comparisons /media/bill/PROJECTS/Qnial/MY_NDFS/PuetzUWS - Universal Waves Series.ndf:65:loaddefs link d_Qndfs 'Fourier transform.ndf' >> Sheesh, embarassing. I just did that check a couple of days ago! /media/bill/SWAPPER/Climate static /usr/share/scilab/ NYET - get back to UWS - I had made graphing errors, the double-wave series works much better now! +-----+ Fourier series file references outside of d_Qndfs : $ find "$d_SWAPPER" -maxdepth 4 -type f -name "*" | grep --invert-match "Website - raw\|z_Old\|z_Archive" | tr \\n \\0 | xargs -0 -IFILE grep --with-filename --line-number "Fourier" "FILE" >> pertinent finds only : /media/bill/SWAPPER/Steve - modelling/170309 CUDA build.txt:44: The FFT is a divide-and-conquer algorithm for efficiently computing discrete Fourier transforms of complex or real-valued data sets. It is one of the most important and widely used numerical algorithms in computational physics and general signal processing. The cuFFT library provides a simple interface for computing FFTs on an NVIDIA GPU, which allows users to quickly leverage the floating-point power and parallelism of the GPU in a highly optimized and tested FFT library. /media/bill/SWAPPER/Steve - modelling/170309 CUDA build.txt:49: The FFT is a divide-and-conquer algorithm for efficiently computing discrete Fourier transforms of complex or real-valued data sets. It is one of the most important and widely used numerical algorithms in computational physics and general signal processing. The cuFFT library provides a simple interface for computing FFTs on an NVIDIA GPU, which allows users to quickly leverage the floating-point power and parallelism of the GPU in a highly optimized and tested FFT library. Binary file /media/bill/SWAPPER/Languages/NeuroSolutions GettingStartedManual.PDF matches /media/bill/SWAPPER/Languages/Fall holidays 2011-Sep-14 through 31/110914 Yaskell email Many thanks.html:19:
When you read though these you'll start to see what is meant by it all.  I ... [big, long] paragraph trimmed ... the better scientists who are coming - and to whom as children or youths, I wish to reach w/ this book. 
Binary file /media/bill/SWAPPER/Climate static/Friends of Science/Holding for Process/__Holding for process, 100617, afjacobs@telus.net, v1, Arrhenius and the Greenhouse.pdf matches Binary file /media/bill/SWAPPER/Climate static/Friends of Science/Holding for Process/__Holding for process, 100617, afjacobs@telus.net, v0, Arrhenius and the Greenhouse.pdf matches Binary file /media/bill/SWAPPER/Climate static/Friends of Science/Holding for Process/__Holding for process, 090319, afjacobs@telusplanet.net, v1, no name matches Binary file /media/bill/SWAPPER/Climate static/Friends of Science/Holding for Process/__Holding for process, 090319, afjacobs@telusplanet.net, v3, no name matches /media/bill/SWAPPER/Climate static/0_solar notes.txt:59: Le and Wang, 2003 - Fourier analysis of sunspot numbers over the past 400 years gave average periods of 53 and 101 years. /media/bill/SWAPPER/Climate static/Datasets/Charvatova solar inertial motion & activity/_Charvatova - solar inertial motion & activity.html:40:correspondance of Fourier and power spectrum periodicities / frequencies for SIM /media/bill/SWAPPER/backup permanent/Pandemics, Kp index, sunspots/0_Pandemics, Kp, sunspot notes.txt:998:The expected number of deaths is based on fitting weekly mortality reports for the previous 5 years (omitting epidemic weeks) to the following equation by a least squares Fourier Regression Model: >> no actual computer programming code +-----+ QNial programs outside of d_Qndfs : $ find "$d_SWAPPER" -maxdepth 4 -type f -name "*.ndf" | grep --invert-match "Website - raw/Software programming & code/Qnial\|z_Old\|z_old\|z_Archive\|backup permanent/Qnial programming language/" In my ToDs list now. I'll copy subDir/*.ndf into d_Qndfs 08********08 #] 09Jan2021 PSO to fit SP500 with PuetzUWS (cont'd) # csvTable_readFrm_path IS OP path n_cols - read a csv table from file (tab-cols, apostrophe-txt) , 2018.97123 , ?tokens left: 1446 <***> , 2019.06027 , ?tokens left: 1447 <***> , 2019.13836 , ?tokens left: 1 >> Not working .. # p_data := link d_Puetz 'SP500 1870-2030 y-log detrend.dat' csvTable_readFrm_path p_data +-----+ olde code IF flag_debug THEN write 'loading PuetzUWS_optimize_check_PSO_controlL_test' ; ENDIF ; # PuetzUWS_optimize_check_PSO_controlL_test IS - check the PSO control parameters # 08Jan2021 won't work - output standards are for non-standard inputs PuetzUWS_optimize_check_PSO_controlL_test IS { LOCAL p_data p_out PSO_controlL_double_check ; NONLOCAL UWScycleVersion_std UWSoptrPhr_std UWS_2011_test n_UWScycles UWS_rowIL_std lambdaL_std thetaNL_std weightL_std n_params_std params_std lambdaL_stdPSO thetaNL_stdPSO weightL_stdPSO PARAMS_STDPSO IRang_L_std IRang_R_std n_particles_std tell_particles_std E_CUTOFF_std MAXV_std MAXITER_std PSOweight_std max_runs_std PSO_controlL_std vx_std xx_std gbest_std pbest_std x_start_std x_end_std x_step_std xList_std ; p_data := link d_Qtest 'fit_linearRegress - tests SP500 1872-2020 TradingView download.dat' ; p_out := link d_Qtest 'fit_linearRegress - tests SP500 1872-2020 PSO_controlL output.txt' ; PSO_controlL_double_check := 0.02 0.1 2000 0.9 2 (0.992744 0.164248 0.221416 0.907139 0.135713 0.878571 0.12619 0.20873 0.236243) (0.1 0.12 0.14 0.16 0.18 0.2 0.22 0.24 0.26) '/media/bill/ramdisk/PSO_optimize output.txt' ; % SP500paramL := optrPhr xCol yCol p_data p_result p_gnuPlot ; % UWSparamL := UWSversnPhr UWS_rowIL UWS_colIL UWS_weightL ; IF (= PSO_controlL_std PSO_controlL_double_check) THEN write 'PuetzUWS_plotTrans_test01 : OK' ; ELSE write 'PuetzUWS_plotTrans_test01 : ERROR' ; ENDIF ; } # loaddefs link d_Qtest 'PuetzUWS- tests.ndf' 08********08 #] 08Jan2021 PuetzUWS_refncParams - PSO to fit SP500 with PuetzUWS +-----+ Olde code link d_Qndfs 'PuetzUWS - Universal Waves Series.ndf' # loaddefs link d_Qndfs 'Puetz - Universal Waves Series.ndf' >> For this, I have simple_globals_test in link d_Qtest 'PuetzUWS- tests.ndf' IF flag_debug THEN write 'loading PuetzUWS_refncParams' ; ENDIF ; PuetzUWS_refncParams IS OP UWSoptrPhr UWS_rowIndxL UWS_colIndxL - setup reference [weightL, lambdaL, thetaNL] as global (NONLOCAL) variables # PuetzUWS_refncParams IS OP UWS_phr UWS_rowIndxL UWS_colIndxL - # setup reference [weightL, lambdaL, thetaNL] as global (NONLOCAL) variables # 18Apr2020 initial, 28Dec2020 revamp, 02Jan2021 generalize # example weights for each cycle are arbitrary -> see Puetz & Borchard 2011 p467 # UWS_phr = ["UWS_2011, "UWS_2014], but usually "UWS_2011 PuetzUWS_refncParams IS OP UWS_phr UWS_rowIndxL UWS_colIndxL { LOCAL UWS_model ; NONLOCAL lambdaRefL thetaNRefL weightRefL ; weightRefL := 1 + ((tell (gage shape UWS_rowIndxL)) * 0.2) ; UWS_model := execute UWS_phr ; lambdaRefL thetaNRefL := cols UWS_model|[UWS_rowIndxL,UWS_colIndxL] ; } # UWS_2011|[2 28 29 30 31 32 33 34 35,1 3] # args for d_webRawe, typically (x_start x_end) = (1870 2030) UWStype y[start,end,inc] UWS_series UWScols dir fBaseName SP500_PuetzUWS 1870 2030 0.1 "UWS_2011 (1 3) d_Puetz fBaseNameUWS SP500_PuetzUWS 1870 2030 0.1 "UWS_2011 (2 4) d_Puetz fBaseNameDUWS # args for d_Qtest - just test the regular series, see PuetzUWS_test_all below UWStype y[start,end,inc] UWS_series UWScols dir fBaseName SP500_PuetzUWS 1870 2030 0.1 "UWS_2011_test (1 3) d_Qtest fBaseUWS_test # p_datDUWS := link dir 'PuetzDUWS 1870-2030 test01.dat' ; # fBaseNameUWS := 'PuetzUWS model for SP500 1870-2030 non-fitted' ; fBaseNameDUWS := 'PuetzDUWS model for SP500 1870-2030 non-fitted' ; fBaseNameUWS_and_DUWS := 'Puetz [,D]UWS models for SP500 1870-2030 non-fitted' ; fBaseUWS_test := 'test- PuetzUWS model for SP500 1870-2030 non-fitted' ; fBaseDUWS_test := 'test- PuetzDUWS model for SP500 1870-2030 non-fitted' ; fBaseUWS_and_DUWS_test := 'test- Puetz [,D]UWS models for SP500 1870-2030 non-fitted' ; # args for d_webRawe UWStype y[start,end,inc] UWS_series UWScols dir fBaseNameUWS_and_DUWS SP500_PuetzUWS_and_DUWS 1870 2030 0.1 "UWS_2011 (2 4) d_Puetz 'Puetz [,D]UWS models for SP500 1870-2030 non-fitted' # args for d_Qtest UWStype y[start,end,inc] UWS_series UWScols dir fBaseUWS_and_DUWS_test SP500_PuetzUWS_and_DUWS 1870 2030 0.1 "UWS_2011_test (2 4) d_Qtest 'test- Puetz [,D]UWS models for SP500 1870-2030 non-fitted' # Puetz_UWS_optimisation - flag_test forces output, but does NOT launch PSO : IF flag_test THEN write E_CUTOFF MAXV MAXITER PSOweight max_runs paramsGiven p_out ; ENDIF ; qnial> Puetz_UWS_optimisation_test 27 28 29 30 31 32 33 34 35 >> OK # run test - intermediate outputs are tables too large to paste here -->[nextv] 24352.4 >> maybe set E_CUTOFF := 1000 in Puetz_UWS_optimisation to see how that goes # typical settings : xDataL -> time (years), but could be a different basis than for "time-series analysis" yDataL -> given (data) values for each of xDataL. With SP500, there ily a single output variable UWS_indxL := 27 + (tell (35 - 27 + 1)) ; p_data := link d_webRawe 'economics, markets/SP500/multi-fractal/SP500 1872-2020 TradingView download.dat' ; # p_data := link d_Qtest 'test- SP500 1872-2020 TradingView download.dat' # dataTable := table_remove_colTitles tabTable_readFrm_path p_data PSO_controlL := n_particles E_CUTOFF MAXV MAXITER PSOweight p_out ; 08********08 #] 07Jan2021 Puetz_UWS_optimisation - PSO to fit SP500 with PuetzUWS link d_Qndfs 'PuetzUWS - Universal Waves Series.ndf' # Puetz_UWS_optimisation IS OP UWS_phr UWS_cols UWS_indxL p_data p_out - # adapt PuetzUWS cycle [weight, phase angle]s to fit the SP500 seemi-log detrended series 1872-2020 link d_Qtest 'PuetzUWS- tests.ndf' big [LMDE, Dell 64] computer freeze (probably didn't reboot after updatess) 08********08 #] 06Jan2021 Linear regression fits of [DJIA 1872-1926, SP500 1926-2020] I had to massively revamp my code : link d_Qndfs 'fit_linearRegress.ndf' linearRegressSingle (SLR) is needed, not MLR, which is good because the MLR is now broken $ gnuplot "$d_webRawe""economics, markets/SP500/PuetzUWS/SP500 1870-2030 y-log.plt" $ gnuplot "$d_webRawe""economics, markets/SP500/PuetzUWS/SP500 1870-2030 y-log detrend.plt" #] 07Jan2021 Fibonacci series : https://moneyminiblog.com/investing/fibonacci-sequence/ Note from Kalen: There are four main Fibonacci retracements: 23.6% – Shallow retracements, flags (short continuation patterns) occur here 38.2% – Moderate retracements 50% – This retracement is not a ratio from the Fibonacci sequence, this number is found from a Dow Theory that states that on average retracement are half of their prior move 61.8% – The golden retracement based upon the Golden Ratio https://www.smithsonianmag.com/science-nature/fibonacci-sequence-stock-market-180974487/ If you divide each number in the sequence by its predecessor (except for 1 divided by 0), then as you move toward higher numbers, the result converges on the constant phi, or approximately 1.61803, otherwise known as the golden ratio. I commonly see : 0.236, 0.382, 0.5, 0.724, 1., 1.382, 2., 2.618, 4.23729 qnial> fib := 0 1 1 2 3 5 8 13 21 34 55 89 0 1 1 2 3 5 8 13 21 34 55 89 qnial> front fib / (rest fib) 0. 1. 0.5 0.666667 0.6 0.625 0.615385 0.619048 0.617647 0.618182 0.617978 gold := 1.61803 qnial> (2 dropright fib) / (2 drop fib) 0. 0.5 0.333333 0.4 0.375 0.384615 0.380952 0.382353 0.381818 0.382022 qnial> (3 dropright fib) / (3 drop fib) 0. 0.333333 0.2 0.25 0.230769 0.238095 0.235294 0.236364 0.235955 qnial> (4 dropright fib) / (4 drop fib) 0. 0.2 0.125 0.153846 0.142857 0.147059 0.145455 0.146067 qnial> (3 dropright fib) / (3 drop fib) 0. 0.333333 0.2 0.25 0.230769 0.238095 0.235294 0.236364 0.235955 qnial> (5 dropright fib) / (5 drop fib) 0. 0.125 0.0769231 0.0952381 0.0882353 0.0909091 0.0898876 qnial> 1 / 0.618 0.382 0.236 0.090 1.61812 2.6178 4.23729 11.1111 get the last one more accurately qnial> fib := 0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 qnial> (5 dropright fib) / (5 drop fib) 0. 0.125 0.0769231 0.0952381 0.0882353 0.0909091 0.0898876 0.0902778 0.0901288 0.0901857 qnial> 1 / 0.09015 11.0926 or 11.1 to the accuracy I did it 08********08 #] 06Jan2021 Linear regression fits of [DJIA 1872-1926, SP500 1926-2020] Yesterday, I got the multipleLinearRegress code of Smillie to work (again, as I have done so, from Smillie and others, a few times in the past) Now to do semi-log fits of the markets, followed by down-shifting to 0 (zero) average over 1872-2020 from : link d_webRawe 'economics, markets/SP500/PuetzUWS/SP500 1872-2020 TradingView, 1928-2020 yahoo finance.ods' /media/bill/SWAPPER/Website - raw/economics, markets/SP500/PuetzUWS/SP500 1872-2020 TradingView, 1872-1926 DJIA, 1926-2020 SP500.dat year,1872-2020 SPCFD:SPX index (source https://www.tradingview.com/chart/0Nbdx1FG/ download : long [uptrends are compressed downtrends are expanded] in %),ratio of opening price/semi-log trend,1871-1926 ratios : [xStart yStart semiLogSlope] = [1871.08 6.09 0.000140925],1926-2020 ratios : [xStart yStart semiLogSlope] = [1926.25 6.20 0.0289587 ], >> OK, I played with gnuplot and data to start to get ready Multiple linear regression fits : [1872-1926, 1926-2020] put data in log format in file of [year, log(SP500 1872-1926), log(SP500 1872-1926)] - just add to spreadsheet!!! set : qnial> pinn := link d_webRawe '' qnial> pout := link d_webRawe '' qnial> dirArchive := link d_webRawe '' from : link d_Qndfs 'matrix operations - symbolic & real-valued.ndf' : multipleLinearRegress IS OP xColIndxL yColIndx pinn pout dirArchive qnial> multipleLinearRegress [0] yColIndx pinn pout dirArchive see link d_Qtest 'multiple linear regression - tests.ndf' +-----+ $ gnuplot "$d_webRawe""economics, markets/SP500/PuetzUWS/SP500 1870-2030 y-log.plt" Y F8:F421 X : D8:D421 Linear type : leave blank stats : TRUE 08********08 #] 04Jan2021 what is WebPlotDigitizer? I seem to have used it? +-----+ linear regression of log[DJIA 1872-1926, SP500 1926-2020] I created optrs in link d_Qndfs 'PuetzUWS - Universal Waves Series.ndf' section : linear regression of log[DJIA 1872-1926, SP500 1926-2020] Interpolation.ndf - has regression (based on Smillie, with data structures) $ find "$d_Qndfs" -maxdepth 4 -type f -name "*.ndf" | tr \\n \\0 | xargs -0 -IFILE grep --with-filename --line-number "regression" "FILE" add_ISOP_debug (link d_Qndfs 'multiple linear regression - NN41 Smillie Uof Alberta.NDF') add_ISOP_debug (link d_Qndfs 'matrix_multiply.ndf') +-----+ search "Libre Calc and where is the regression analysis toolpack?" https://forum.openoffice.org/en/forum/viewtopic.php?f=9&t=83005 Re: Is there an analysis toolpak for OpenOffice Calc Postby jrkrideau » Wed Apr 20, 2016 12:46 pm SamRose wrote: Hello. I'm currently taking a statistics class and for one of the assignments, if I were using Microsoft Excel, I would need the Analysis Toolpak. I'm working pretty exclusively with OpenOffice programs, so I wanted to know if there was something similar in OpenOffice Calc? There are at least two extensions listed as statistical add-ons for Calc. R4Calc - R Statistics for OOoCalc and Statistical Data Analyser for OO Calc. Just go to the extensions site and type statistics into the search box. I have no experience of either but assuming R4Calc does what it says it does, I'd suggest trying it as the underlying R programming will be okay. However I consider using a spreadsheet for statistics to be very poor practice. See for example Spreadsheet errors. Note the example describes Excel and LibreOffice, but AOO Calc did the same the last time I checked. Unless the course demands the use of spreadsheets, I'd suggest using an statistical program. Have a look at R . It is a real stats system and even gives the right answers. Not exactly really easy to use at first but much better than a spreadsheet. You might want to have a look at RCommander, an R GUi which probably do as much or more than Excel's Analysis Toolpak with the added bonus it will do it correctly. +-----+ search "https://forum.openoffice.org/en/forum/viewtopic.php?f=9&t=83005" https://www.howtogeek.com/281537/how-to-install-and-manage-extensions-in-libreoffice/ Lori Kaufman @howtogeek November 23, 2016, 10:24am EDT To begin, if you’re going to install an extension for a specific LibreOffice program, open that program. Otherwise, you can open any of the programs. We’re going to use Writer for our example, so we’ll open Writer. Then, go to Tools > Extension Manager. To browse and download extensions for LibreOffice, visit the LibreOffice Extension Center. http://extensions.libreoffice.org/extension-center You can also access the Extension Center by clicking the “Get more extensions online” link on the Extension Manager dialog box. >> I can't find the regression analysis package - may not be there anymore? Use my own QNial program and try the R stat system which I downloaded # enddoc