#] #] ********************* #] "$d_Qtest"'fit_linearRegress/0_fit_linearRegress notes.txt' - ??? # www.BillHowell.ca 31Jul2022 initial # view in text editor, using constant-width font (eg courier), tabWidth = 3 #48************************************************48 #24************************24 # Table of Contents, generate with : # $ grep "^#]" "$d_Qtest"'fit_linearRegress/0_fit_linearRegress notes.txt' | sed "s/^#\] / /" # #24************************24 # Setup, ToDos, #08********08 #] ??Aug2022 #08********08 #] ??Aug2022 #08********08 #] ??Aug2022 #08********08 #] ??Aug2022 #08********08 #] 02Aug2022 regressions - Revamp [one, ALL] semi-log [data, plot]s 'stock index [semi-log, detrend] data processing.ndf' revampONE 'DAX40 Jan1970-Jul2022 TradingView' >> oops, need to add `# to tbls in regression file outputs >> OK, done # qnial> revampONE 'INDIA50 Mar2013-Jul2022 TradingView' # qnial> revampONE 'NI225 Jan1984-Jul2022 TradingView' # qnial> revampONE 'SP500 1872-1940 [TV,yahoo,article]' # qnial> revampONE 'SP500 1926-2022 [TV,yahoo,article]' # qnial> revampONE 'SP500 1938-2022 [TV,yahoo,article]' # qnial> revampONE 'UKX100 Apr1988-Jul2022 TradingView' Whew! had trouble with changes - reverted back to tbl_writeTabSeparated_padd >> now works OK +-----+ olde code % output_table := EACH phrase output_table ; % nyet - need tabs fout EACHRIGHT writefile (rows picture output_table) ; % writefile fout '# xObs yObs yEstObs %ResdObs xTrn yTrn yEstTrn %ResTrn yDetrend' ; #08********08 #] 01Aug2022 yDetrend calcs in residual_linearSingle, problematic transforms question is what is appropriate? I am lost... >> see revamped approach : "$d_Qtest"'fit_linearRegress/0_fit_linearRegress notes.txt' cannot handle complexity of gnuplot of detrend in : "$d_Qndfs"'economics, markets/stock index [semi-log, detrend] data processing.ndf' >> remove code (see below), when ALL symbols have been processed semi-manually, use : "$d_Qndfs"'economics, markets/220730 Puetz2011 detrended international indexes, gnuplot specs.ndf' >> which in turn calls : "$d_Qndfs"'gnuplot template, subFiles [create, insert].ndf' #08********08 #] 01Aug2022 yDetrend calcs in residual_linearSingle, problematic transforms question is what is appropriate? I am lost... I ripped apart optrs to include detrend etc - tests no longer work! one-at-a-time tests via : "$d_Qndfs"'economics, markets/stock index [semi-log, detrend] data processing.ndf' +-----+ # single dataset at a time +--+ # qnial> revampONE 'DAX40 Jan1970-Jul2022 TradingView' qnial> revampONE 'DAX40 Jan1970-Jul2022 TradingView' rm: cannot remove '/home/bill/SG6/WebSite/economics, markets/SP500/PuetzUWS intnl stocks/yrFrac DAX40 Jan1970-Jul2022 TradingView.txt': No such file or directory sh: /home/bill/SG6/WebSite/economics, markets/SP500/PuetzUWS intnl stocks/semi-log DAX40 Jan1970-Jul2022 TradingView.plt: No such file or directory cat: '/home/bill/SG6/WebSite/economics, markets/SP500/PuetzUWS intnl stocks/semi-log 0_gnuplot template.txt': No such file or directory "/home/bill/SG6/WebSite/economics, markets/SP500/PuetzUWS intnl stocks/semi-log DAX40 Jan1970-Jul2022 TradingView.plt", line 0: Cannot open script file '/home/bill/SG6/WebSite/economics, markets/SP500/PuetzUWS intnl stocks/semi-log DAX40 Jan1970-Ju >> wrong d_work??... NASTY problem - NASTY problem - grep gets incorrect environmental variable : see "$d_Qroot""help - [develop, debug, error list, etc]/0_QNial special problems.txt" Try again : qnial> revampONE 'DAX40 Jan1970-Jul2022 TradingView' *** freeit: ignoring release of -3, code = -3 *** out of range blockptr set output "/media/bill/Dell2/Website - raw/economics, markets/SP500/semi-log DAX40 Jan1970-Jul2022 TradingView.png" ^ "/home/bill/SG6/web/economics, markets/SP500/PuetzUWS intnl stocks/semi-log DAX40 Jan1970-Jul2022 TradingView.plt", line 21: cannot open file; output not changed system error: No such file or directory >> crashes again (gnuplot, I think) did create : semi-log DAX40 Jan1970-Jul2022 TradingView regression fit.txt semi-log DAX40 Jan1970-Jul2022 TradingView.plt apparent problems : # The following are in transformed coordinates (eg log) : # xObsMin xObsMax yObsMin yObsMin yDetrendMin yDetrendMax # ?no_value ?no_value ?no_value ?no_value ?no_value ?no_value Table of residuals appears as single column of chrs!!??!! deactivate gnuplot in revampONE of : 'stock index [semi-log, detrend] data processing.ndf' bye, cold boot qnial : loaddefs link d_Qndfs 'economics, markets/stock index [semi-log, detrend] data processing.ndf' qnial> revampONE 'DAX40 Jan1970-Jul2022 TradingView' *** freeit: ignoring release of -3, code = -3 *** out of range blockptr >> OK, not a gnuplot problem, it is my use of 'picture' I'm lost - where is read-in of data? "$d_Qndfs"'economics, markets/stock index [semi-log, detrend] data processing.ndf' fit_linearRegressSingle_TradingView tbl_raw := tbl_remove_colTitles pinn_read_tabTable pinn ; create loaddef file for tests!! pinn := link d_work 'yrFrac ' symName_dateRange '.txt' ; >> copy yrFrac file 'yrFrac SP500 1926-2022 [TV,yahoo,article].txt' to "$d_Qtest"'fit_linearRegress/yrFrac SP500 1926-2022 [TV,yahoo,article] loadable.ndf' +--+ # qnial> revampONE 'INDIA50 Mar2013-Jul2022 TradingView' # qnial> revampONE 'NI225 Jan1984-Jul2022 TradingView' # qnial> revampONE 'SP500 1872-1940 [TV,yahoo,article]' # qnial> revampONE 'SP500 1926-2022 [TV,yahoo,article]' # qnial> revampONE 'SP500 1938-2022 [TV,yahoo,article]' # qnial> revampONE 'UKX100 Apr1988-Jul2022 TradingView' +-----+ olde code fit_linearRegressSingle : this still might be good!! : IF (= "log yOptrPhr) THEN relStdErr := 10 power StdErrEst ; ELSE relStdErr := StdErrEst ; ENDIF ; IF (= "log yOptrPhr) THEN yEstTran := 10 EACHRIGHT power yEst ; ENDIF ; % tally first second constant ???, why not just tally Y? ; % close file so I can get good formatting with tbl_writeTabSeparated_padd ; % tbl_writeTabSeparated_padd output_table povr ; # loaddefs link d_Qndfs 'fit_linearRegress.ndf' IF flag_debug THEN write 'loading residual_linearSingle' ; ENDIF ; #] residual_linearSingle IS OP constant Coeff relStdDevTrn #] xObs yObs xOptrPhr yOptrPhr - #] calc linear regression estimate (Yest) for given [xObs, Yobs] #] show [model, stat, residual]s #] [constant, slope, relStdDev, etc] are for "transformed" coordinates!! # IF flag_break THEN BREAK ; ENDIF ; residual_linearSingle IS OP xObs yObs yEst yRes pct_resid yDetrend { LOCAL fout oldFormat pct_resid relStdDevObs yEst yRes ; xTran yTran := xObs yObs ; IF (= "log xOptrPhr) THEN xTran := 10 EACHRIGHT power xObs ; ENDIF ; yEstTrn := constant + (Coeff * xTrn) ; % using "normal" (data, Obs) formats [yObs, yEst] ; IF (= "log yOptrPhr) THEN yEst := 10 EACHRIGHT power yEstTrn ; ENDIF ; yRes := yObs - yEst ; pct_resid := yRes / yObs * 100 ; relStdDevObs := sqrt (sum square (pct_resid / 100) / ((gage shape pct_resid) - 1)) ; % yDetrend := log (yObsTrans / yEstTrans) ; % wrong, incomplete ; % yDetrend := (yObsTrans / yEstTrans - 1) / relStdDev ; % even worse ; yDetrend := log (yEstTrn / yTrn / relStdDevTrn) ; } # 26Jul2022 keep stats in transformed (eg log) co-ordinates!? residual_linearSingle IS OP constant Coeff xObs yObs xOptrPhr yOptrPhr { LOCAL fout oldFormat yEst xObsTrans yObsTrans yEstTrans yResTrans pct_resid yDetrend ; yEst := constant + (Coeff * xObs) ; xObsTrans yObsTrans yEstTrans := xObs yObs yEst ; IF (= "log xOptrPhr) THEN xObsTrans := 10 EACHRIGHT power xObs ; ENDIF ; IF (= "log yOptrPhr) THEN yObsTrans := 10 EACHRIGHT power yObs ; ENDIF ; IF (= "log yOptrPhr) THEN yEstTrans := 10 EACHRIGHT power yEst ; ENDIF ; yResTrans := yObsTrans - yEstTrans ; pct_resid := yResTrans / yObsTrans * 100 ; relStdDev := sqrt (sum square (pct_resid / 100) / ((gage shape pct_resid) - 1)) ; % ; % yDetrend := log (yObsTrans / yEstTrans) ; % wrong, incomplete ; % yDetrend := (yObsTrans / yEstTrans - 1) / relStdDev ; % even worse ; yDetrend := log (yObs / yEst / relStdDev ; output_table := transpose mix xObsTrans yObsTrans yEstTrans yResTrans pct_resid yDetrend ; output_table := tbl_add_colTitles output_table ("xObs "yObs "yEst "yResids "pct_resid "yDetrend) ; (min xObsTrans) (max xObsTrans) (min yObsTrans) (max yObsTrans) (min yDetrend) (max yDetrend) output_table } # 18Jul2022 olde code : output_table := titlesRows hitch (("xObs "yObs "yEst "yResiduals "pct_resid) EACHBOTH hitch ( xObs yObs yEst yResiduals pct_resid)) ; output_table := transpose mix output_table ; residual_linearSingle IS OP fout titlesRows constant Coeff xObs yObs % important for copy-paste of yEst for subsequent [spreadSheet, loaddef] : ; fout EACHRIGHT writefile '' '' 'yEst' ; fout EACHRIGHT writefile (EACH string yEstTrans) ; # I don't fully understand the log transforms, but I use : IF (= "log yOptrPhr) THEN relStdErr := 10 power StdErrorEst ; ELSE relStdErr := StdErrorEst ; ENDIF ; +-----+ olde code - cannot habndle complexity of gnuplot of detrend in : "$d_Qndfs"'economics, markets/stock index [semi-log, detrend] data processing.ndf' IF flag_debug THEN write 'loading plot_detrend' ; ENDIF ; #] plot_detrend IS OP strPuetzUWS yrStrtEndP yrUWSzero - PeutzUWS, generate gnuplot text for tSpikes # 25Jul2022 initial # IF flag_break THEN BREAK ; ENDIF ; plot_detrend IS OP strUWS yZero yrP { LOCAL cmd finn fout i line pPlt pSpikes pTmp pTmplt strDateRange tLine tSpike tSpikeLine tSpike_lines tSpikeL tSpikeLL yrStrt yEnd ; NONLOCAL d_work ; % ; % [calc, create] a gnuplot subfile of tSpikeLL ; tSpikeLL := strUWS_yrZero_yrP_calc_tSpikeLL IS OP strUWS yrZero yrP ; % assign line widths for each tSpikeL ; widthLL := % ; % create a file of horizontal PuetzUWS yDetrend gridlines ; % ; % create a file of plot args ; cmd := link 'gnuplot "' d_work 'detrend 0_gnuplot StockMkt Indices.plt"' ; cmd := link 'gnuplot "' pPlt '" ' ; % write cmd ; host cmd ; } # plot_detrend 'PuetzUWS2011' (1871.219 2022.133) 1929.75 # plot_detrend 'PuetzUWS2014' (1871.219 2022.133) 1929.75 # pinn := '/media/bill/Dell2/Website - raw/economics, markets/SP500/PuetzUWS intnl stocks/gnuplot tSpike 1871-1871 PuetzUWS2011.txt' # pth_typ_exists pinn "pinn # pinn_read_strL pinn # enddoc