9Feb2021-12h58m : this file was generated by "$d_bin""0_list Howells [bash scripts, QNial operators].sh" find "/media/bill/Dell2/PROJECTS/Qnial/MY_NDFS/" -name "*.ndf" | grep --invert-match "z_Old" | grep --invert-match "z_Archive" | sort | tr \\n \\0 | xargs -0 -IFILE grep "#\] " "FILE" | sed "s/#] /\t/" >>"/media/bill/Dell2/PROJECTS/Qnial/MY_NDFS/0_list of Howells QNial [ndf, optr]s.txt" ********************* loaddefs link d_Qndfs 'txtFile - replace coded segments.ndf' - ********************* loaddefs link d_Qndfs 'type_check.ndf' - Routines for implementing checking of data types ********************* loaddefs link d_Qndfs 'USB workspace loads.ndf' ********************* loaddefs link d_Qndfs 'virus - clamscan output filtering of viruses found.ndf' ********************* loaddefs link d_Qndfs 'windows [open, close, ID, title], [get,set,move] cart [posn,size].ndf' window [open, close, ID, title] : winIDorTitle_get_winID IS OP winIDorTitle - returns a list of winIDs having the name requested winIDorTitle_get_winTitle IS OP winIDorTitle - winIDorTitle_isDead IS OP winIDorTitle - winID_delay_lastStarted IS OP seconders - time delay to make sure most recent window opening is captured winIDorTitle_getAll_winIDs IS OP winIDorTitle - returns a list (NOT a string!) winIDorTitle_close IS OP winIDorTitle - winIDorTitle_activate IS OP winIDorTitle - check window activation from Qnial winTitle_change IS OP winID winTitle - (doesn't work) window [get,set,move] cart [posn,size] : winIDorTitleD_get_winPosnSize IS OP winIDorTitle - winIDorTitle_siz_mov IS OP winIDorTitle winPosn winSize - winIDorTitle_raiseFocus IS OP winIDorTitle - ********************* loaddefs link d_Qndfs 'windows system for startup.ndf' **************************** Coding : fname_for_title IS OP fname - winSizeMove IS OP appType winID winPosn winSize wait_hhmmss - (wmctrl) get_winConfig IS OP appLabel - window configurations, default general setup appOpen IS OP app d_name f_name fileOpen IS OP appOpen d_name f_name - not yet defined ********************* loaddefs link d_Qndfs 'audio file concatenation - sox.ndf' - join wma audio files f_ins := list of Chapters for the ACM Facility Safety presentation wav_conv2 IS OP f_voice - single file tranlsation .wma -> .wav audio file format wav_all2 IS OP f_ins - translate full list of audio files p_ins := list of audio file to catenate ********************* loaddefs link d_Qndfs 'audio - music, microphone.ndf' audio_play IS OP path volume_pct - play an audio file audio_crop IS OP pinn pout t_start t_end - crop an audio file audio_start IS OP typer sub_path fname volume_pct duration music_start IS OP args - play an audio file in preset music directory voice_start IS OP args - plays a microphone-recorded voice file in preset directory ********************* loaddefs link d_Qndfs 'base64_pre_clean for mime.ndf' - remove CR from text files so that MIME decoding can take place is_blankLin IS OP Str - checks if line is all [char_tab, char_space, char_carriagereturn, fault_noexpr] isempty_Lin IS OP Lin - check if line is null pre-clean designated file (open expression, no optr defn) ********************* loaddefs link d_Qndfs 'base64_pre_clean.ndf' - routine to remove CR from text files so that MIME decoding can take place base64 -d - open wexression, no formal optr defn ********************* loaddefs link d_Qndfs 'Capitalize first letters.ndf' cap_first IS OP str - cap_each IS OP str - small_all IS OP str - ********************* loaddefs link d_Qndfs 'Carve up textFile.ndf' carveUp_textFile IS OP webPage webSite - saves byte code of files in ASCII text, constant line lengths? ********************* loaddefs link d_Qndfs 'Cash-Karp-Runge-Kutta integrate ODEs with accuracy monitoring.ndf - adapted rkck Cash-Karp-Runge-Kutta step used by rkqs, rkqs integrate one step of ODEs with accuracy monitoring derivs IS EXTERNAL operation ; derivs_e := f_Cash_Karp_Runge_Kutta ; - sign IS OP a b { IF b >= 0.0 THEN abs a ELSE opposite abs a ENDIF } - rkck IS OP x y dx dydx deriv_name - rkqs IS OP x y dx dydx eps yscal deriv_name - odeint IS OP x1 x2 ystart dx1 dx_min dxsav eps kmax deriv_name - ********************* loaddefs link d_Qndfs 'conference guides - header.ndf' ********************* loaddefs link d_Qndfs 'conference guides - update inserts.ndf' extract_title IS OP str - given a string, if ':@file-insert@:' appears, extract the next chr_blanks-surrounded subString update_webPage IS OP webPage - update webPage using appropriate headers update_webSite IS OP webSite - update webSite using appropriate headers update_webGuide IS - create a listing of all html filepaths for a [flat, 2 layer] website ********************* loaddefs link d_Qndfs 'copyover - remove empty directories.ndf' copyover_clean IS - ********************* loaddefs link d_Qndfs 'diff_Howell.ndf' - diff that crawls along text ignoring newlines and reports only parts of text that differ diff_Howell IS OP path1 path2 - diff that crawls along text ignoring newlines and reports only parts of text that differ. Re-arranged text requires more than this! ********************* loaddefs link d_Qndfs 'diff unicode.ndf' diff_unicode IS OP fname1 fname2 - USELESS!!! - diff works with unicode!!! what a waste of programming time. ********************* loaddefs link d_Qndfs 'directed graphs.ndf' get_Neighbours_in_from_Edges IS OP edges - get_Adjacency_from_Edges IS OP edges - matrix_degreeIn IS OP neighbor_list - cp IS cell_write_parenthesis_aligned theta1_t1_expanded_string - ********************* loaddefs link d_Qndfs 'economics, markets/chloralkali forecast.ndf' - USA market ********************* loaddefs link d_Qndfs 'economics, markets/chloralkali forecast.ndf' - USA market ff IS forecast US_Data "Cl2Cons (solitary "IIP) - gg IS Bostonian US_Data "Cl2Cons (solitary "Cl2Pr) 250000. - ********************* loaddefs link d_Qndfs 'economics, markets/Fischer header.ndf' Fischer_workFlow_define IS OP yearEndPoints i_dat - res ipsa loquitor Fig_0_01_process IS - England consumable prices 1201-1993 Fig_0_02_process IS - Price of grain in England 1201-1960 Fig_5_01_process IS - prices in ancient Babylon 1850-1600 BC Fig_5_02_process IS - ancient Greece [barley, olive oil] 450-150 BC ********************* loaddefs link d_Qndfs 'economics, markets/Fischer workFlow.ndf' +-----+ Fischer great price waves - revolutions versus equilibria Fischer_regressions IS OP xColIndx yColIndx selCol endPoints pinn pStats pResults - Fischer_sortAddlog IS OP f_digitized_dat - Fischer_initialize IS OP fnameBase yearEndPoints i_dat - ********************* loaddefs link d_Qndfs 'economics, markets/Puetz test03 DUWS_2011 cycles 26-35.ndf' Insights test03 initial structure definition - updated by "DUWS_2011 cycles 26-35.ndf" test03_define_initial IS - initial definition of test03_struct, not repeated later to avoid data loss +-----+ test03 avatar operator expressions get_test03 IS OP s_lab - get component of structure put_test03 IS OP s_lab s_value - [put,change] component of structure of test03 help_test03 IS OP s_lab - help, retrieve short description for a label of test03 write_test03 IS - write test03_struct, one format for showing a 2D data structure (table) +-----+ Run setups [test03_define_initial, test03_update] ********************* loaddefs link d_Qndfs 'economics, markets/test03 DUWS_2011 cycles 26-35.ndf update.ndf' test03_update IS - routine primarily for [simple, rapid] changes to optimisation parameters test03_check_PSOcontrols IS - routine primarily for [simple, rapid] changes to optimisation parameters ********************* loaddefs link d_Qndfs 'economics, markets/PuetzUWS big data.ndf' +-----+ tables of cycle parameters UWS_2011 IS - Universal Waves Series table of constants (time periods), 2011, [1] pp565-68 UWS_2014 - Universal Waves Series table of constants (time periods), 2014 reference [3] comparison to stock market cycles 2009 Puetz book chapters : 28.68d 86.04d 258.11d 2.12y 6.36y 19y 57y 172y 515y 1545y 4636y 2014 Puetz, Stephen J., Prokoph, Andreas, Borchardt, Glenn, Mason "Evidence of Synchronous, Decadal to Billion Year Cycles in Geological, Genetic, and Astronomical Events" UWS alternative variables [lambda, thetaN, weight, etc] - for testing and debugging +-----+ Big data - problematic for normal viewing of structures xDataL_std - standard year list for running models 1870-2020, (post US Civil War) yDataL_std - standard SP500 data for standard years 1870-2020, (post US Civil War) DJIA proxy prior to 1926 UWS_testRes_std - for "UWS_result PSO_thetaNweightL" ********************* loaddefs link d_Qndfs 'economics, markets/PuetzUWS header.ndf' +-----+ PuetzUWS standard structure definition - default for globals when not otherwise specified? UWSstd_define_initial IS - define the standard structure for PuetzUWS system useful [as default, for testing] get_UWSstd IS OP s_lab - get component of structure put_UWSstd IS OP s_lab s_value - [put,change] component of structure of UWSstd help_UWSstd IS OP s_lab - help, retrieve short description for a label of UWSstd write_UWSstd IS - write UWSstd_struct, one format for showing a 2D data structure (table) ********************* loaddefs link d_Qndfs 'economics, markets/PuetzUWS - Universal Waves Series.ndf' +-----+ 2-line segment linear regression of log(SP500 [1872-1926 (DJIA based), 1926-2020]) linearReg_log_DJIA_SP500 IS OP p_logDJIA p_logSP500 - +-----+ Plot standard [,D]UWS cycles just to look at them UWS_calc IS OP x_current lambda θn UWSweight - calculate UWS Puetz_UWS_model IS OP UWSoptrPhr UWS_rowIL UWS_colIndxL x_start x_end x_step weightL pout - ratioMaxUnity IS OP x x unityFactor - shift the mean to 1.0, unityFactor depends - eg [1.3, 2, 4.5, etc] PuetzUWS_plotTrans IS OP xList weightTrnL thetaNTrnL colLabels p_yCalc p_gnuPlot - [calc, plot] model, with [constant lambdaL, transformed [weightL, thetaNL], given xList] +-----+ Calculate log(SP500) detrended and errors UWS_result_normalCoord IS OP yearLambdaThetaNweight_table - estimate SP500 for all years UWS_result IS OP PSO_thetaNweightL - estimate SP500 for all years +-----+ Puetz-based UWS parameter fitting to "1872-2020 SP500 index semi-log detrended 1871-1926" UWS_L1error IS OP params - calculate total error for current PSO_position (params) UWS_L2error IS OP params - calculate total error for current PSO_position (params) PuetzUWS_optimize IS OP SP500paramL UWSparamL PSO_paramL - adapt PuetzUWS cycle [UWSweight, phase angle]s to fit the SP500 seemi-log detrended series 1872-2020 PuetzUWS_optimize_weightsOnly IS OP SP500paramL UWSparamL PSO_paramL - adapt PuetzUWS cycle [UWSweight, phase angle]s to fit the SP500 seemi-log detrended series 1872-2020 +-----+ [Sort, fixAngles, save] the collection of best results from all trials sort_fixAngles_save_allBest IS OP p_tstNumErrParam - +-----+ Gnuplot particles' SP500 estimates gnuplot_best15_particles IS - gnuplot estimates of SP500 for .plt defined year range, for selected particles HELPER_WCCI2020 IS - generate conference mailto lists, auto-insertion into web-pages, or copy-paste test_inquiry IS - create and send a test email to the daemon daemonEml_to_anonymous IS - forward "Anonymous Receiver Email Daemon" (ARED) emails emailSplit_daemonInbox IS OP p_fldrSrc - for daemon, split inbox into [good, badd] email folders, then process "good" daemon_run IS - continuous, real-time control loop for ARED daemon mime_types_ALL - list of mime types mime_types IS OP pname_inn pname_out - extract a list of mime_types from file mime_IMAGE IS OP finn fout - dump the image in an email folder mime_base64 IS OP finn fout - extract from email folder MIME coding (base64) and expand to unicode mime_from IS OP finn fout - extract MIME coding (base64) in email "From - " and expand to unicode mime_expand IS OP pname_inn pname_out - expand MIME coding [base64,image/png] within an email folder emlAddress_format IS OP toFromCCLines - write_htmlHdr IS OP fout emlSubject emlFrom emlDate emlTo emlCC - write_htmlEnd IS OP fout - emailFolder_saveAs_htmls IS OP p_log d_fldrSrc f_Src d_htmlOut - split email folder into html files # they should have coded, eg : ' From - ' or something! (shitty job) email_count IS OP pname_inn - count the number of emails in a text file # they should have coded, eg : ' From - ' or something! (shitty job) email_remove_massEmail IS OP pname_inn pname_out - remove the mass email if included in reply preprocess_folder IS OP p_folder - [expand mime, remove mass mail] if flagged pstprocess_folder IS OP finn oldEmail_2lines - [expand mime, remove mass mail] if flagged remove_1stWhiteLines IS OP p_inn p_out - empty lines don't give proper email file! emailCounts IS - clean_emlAddress IS OP line - remove ['"] and extra_whitespace from email address list find_emailHdr_items IS OP finn item - get an emailHdr_item (eg (From, Subject, CC]) from p_emlOne email_filter IS - splits single email tempFile emailRead_one IS OP finn oldEmail_2lines - read one email from input folder, expand mime, remove mass email # they should have coded, eg : ' From - ' or something! (shitty job) emailSplit_folder IS - automatically split an email folder according to key phrases in emails emailSplit_stats IS - tally [keyCountsALL, folderCountsALL] in p_stats1 emailSplit_statsManual IS - post folderCountsALL@i (but NOT keyCountsALL@i) in p_stats1 emailSplit_archive IS - archive results for QNial "email - extract, sort, cull addresses from text.ndf" ********************* loaddefs link d_Qndfs 'file_ops.ndf' +-----+ file [exist, size, dateTimeStamp], [extract, change] names path_exists IS OP typer path - returns true if a [directory, path] exists path_extract_fname IS OP path - extract the directory from a path path_extract_dir IS OP path - extract the directory from a path path_extract_dirFname IS OP path - extract the [directory, fname] from a path path_extract_dirBaseExtn IS OP path - extract [dir, base, extension] from a path-file name path_retrieve_subDirFname IS OP path dirBase - returns (subDir fname) from a path path_size IS OP path - returns the filesize path_dateTimeStamp IS OP path - return the dateTimeStamp of the file path_change_extension IS OP path fileExtension_new - changes fileExtension path_delete IS OP path - delete a file +-----+ [backup, restore] cart [path, dir]s path_backupDatedToSameDir IS OP path - backup dated file in same directory (normally don't use) path_backupDated_delete IS OP path - rename a file with date precursor (normally don't use) path_backupDatedTo_dir IS OP path dirBackup - backup dated version of a file to a (created) FLAT dir path_backupTo_dir IS OP path dirBackup - backup file to a (created) FLAT dir pathList_backupTo_dir IS OP pathList dirToCreateBackupDir - backup dated files to a FLAT dir dirBackup_restoreTo_paths IS OP flag_fnamesDated d_backup pathList - restore paths from a backup +-----+ [sort, catenate] file pinn_sortupTo_pout IS OP p_inn p_out pinn_sortCatTo_pout IS OP pinnList pout - sort each file in pinnList, concatenate to pout path_findChrsInSet IS OP flag_chrs_include chrs_set fname_in - find [in, NOT in] characters +-----+ replace string in [file, directory] path_removeLeadWhiteLines IS OP p_emlOne p_emlOne_noEmptyLines - required to use emails as attachments str_replaceIn_path IS OP flag_backup strOld strNew path - change strOld to strNew in path str_replaceIn_pathList IS OP flag_backup d_backupRoot strOld strNew pathList change strOld to strNew in pathList, for strPattern, automatic path backups to d_backup beware of possible alternate appearances of the f_Pattern!! use only for filePaths, or other well-constrained strings!! str_replaceIn_dir IS OP flag_backup strOld strNew f_Pattern dir d_backup - change strOld to strNew in dir for strPattern path_change_lines IS OP p_inn p_out linesInn linesOut - [p_inn linesInn] to [p_out, linesOut] sed_file_pretreat IS OP pname - pre-process pname to handle [apo, quote,'&\/\n']) pListFiles_findCountsBY_strList IS OP p_listOfPaths strList p_countsByPath - count each of strList hits in each path listed in a file sed_file_postreat IS OP pname_out - translate pname_inn for QNial use mostly [apostrophes, quotes,'&\/\n'], via bash script +-----+ replace words in [file, directory], (sed, grep is too, but maybe just spaces?) word_replaceIn_path IS OP sOld sNew p_inn cList flag_backup - change word throughout file word_replaceIn_dir IS OP sOld sNew dir fExtn flag_backup cList - change word all directory +-----+ Inter-path processing - eg insert a file into another, maybe remove at a future date? path_insertIn_fHand IS OP p_insert fHand - insert p_insert into fHand at the current writefile point +-----+ executables embedded in files (eg html files) - related optrs in other *.ndf' files These are listed in fileops.ndf as they can be generalised for other applications internalLinks_return_backupSubDirFnames IS OP strLeft strRight line - returns backupFnameSubDir for an an internal link, where SubDir goes into d_webSite, 'Website updates.ndf' internalLinks_return_relativePath IS OP strLeft strRight line - convert links to relative also using executeEmbeds when present, 'Website updates.ndf' str_executeEmbeds IS OP line fout backtrack - execute embeds in line '[#!: ' is for full-line embeds; '[#=; ' is for embeds within a line, 'strings.ndf' webPage_update IS OP flag_test pinn pout d_inn d_out - update pinn to d_out using executeEmbeds Although built for webSite maintenance, with adaptation this has far more general applicability. 'Website updates.ndf' +-----+ Process executable embedds in a file (eg html files) pathList_change_headFoot IS OP pathList - change all whole-line-embeds for pathList this is currently d_webRaw specific executeEmbedsGet_pathTemp IS - generate a unique p_temp fname path_executeEmbeds IS OP path phraseValueList - execute embeds in path, [output to, return] p_temp '[#!: ' is for full-line embeds; '[#=; ' ' ;=#]' are for embeds within a line path_executeEmbedsInsertIn_fHand IS OP path phraseValueList - execute embeds, insert in fileHandle +-----+ URL [down, up]-load url_downloadTo_path IS OP URLname d_out fname - command line download +-----+ [read, insert, remove, write] cart [strList, table, UnixCmd] strList_readFrom_path IS OP path - simple read of file as a list (1D array) of strings, one per line strList_writeTo_path IS OP strList path_out - writefile a list (1D array) of strings str_convertTo_numOrStr IS OP str - if str can be made numeric [boolean, integer, real] do so otherwise wrap with chr_apo then reduce double to single chr_apos, used by csvTable_readFrm_path csvTable_readFrm_path IS OP path - read a csv table from file, cells to [apo-str, numeric] table_writeCSVformatTo_path IS OP table path - writefile table to path, [tab-separated cols, apo-text] important - assumes 2D table of ONLY [string, atomic data] types (eg not lists of numbers...) tabTable_readFrm_path IS OP path - read file as a table with tab delineated cols (eg spreadsheet) tabTable_writeTo_path IS OP tabTable path - write table to file, tab delineated wcUnix_readFrom_path IS OP path - read file as a table with tab delineated cols (eg txt from spreadsheet) +-----+ Specialized stuff for 2017 IJCNN Alaska line_charcodes IS OP path line_start line_end - convert ASCII file lines to charcodes over interval rename_files IS [later ... OP dname fname_oldPattern fname_newPattern] - rename a bunch of files pathList_moveEach IS - move selected files listed in text file ********************* loaddefs link d_Qndfs 'fit_linearRegress.ndf' +-----+ Single linear regression residual_linearSingle IS OP fout titlesRows constant Coeff xGiven yObs - linear regression estimate (Yest) for given [xGiven, Yobs], show [statistics, residuals] fit_linearRegressSingle IS OP xColIndx yColIndx pinn pout dirArchive - single independent variable fit_linearRegressSingle_segments IS OP X Y - single independent variable, extract segments of data for regression, returns array +-----+ Multiple linear regression mat_selfInvert IS OP matrix - invert any simple matrix the matrix is made square by self InnerProduct. all components are atomic numbers, no nesting residual_linearMulti IS OP fout Coeff X Y - estimate Y for given X table, show residuals (Y- Yest) fit_linearRegressMulti IS OP xColIndxL yColIndx pinn pout dirArchive - run all multLinReg components multipleLinearEstimate IS OP matCoeff rowTitle X - estimate Y for given X table ********************* loaddefs link d_Qndfs 'forecast.ndf' - routines for doing a simple forecast of market volumes Coding short_stats IS OP A B - returns pair (Coefficients Rsquared) short_reg IS OP Years Short_Stats_data - forecast IS OP DataTable Depend Independs normalization_fn normalization_inverse_fn Baseline_compare_fn Baseline_compare_inverse_fn data_years forecast_years delta_years cumulate IS OP starter lister - used for Bostonian optr Bostonian IS OP DataTable Depend Independs CumStart data_years forecast_years - Bostonian Learning curves - (standard long term price-cumulative volume relationship) ********************* loaddefs link d_Qndfs 'genetic algorithm.ndf' stnormal IS OP x - normal IS - de_normal IS OP mean stddev x - linear IS OP low high fraction - transcribe IS OP gene_transform gene_low gene_high sample - first_less IS OP a b - test_fn IS OP gene - randomize_list IS OP a - find_atom_paths IS OPERATION Array - deepplaceall IS OP C A - gene_initialize IS - adjust IS OP low high distribn stddev random_x val_original - crossover IS OP g1 g2 - mutate IS OP gene - evolve IS - ********************* loaddefs link d_Qndfs 'gnuplot.ndf' gnuplot IS OP title xLabel yLabel cmtLabel xRange yRange axisLogs axisTics imageSize specialFeatureLines ySources p_templt p_gnuplt p_image - auto-generate single [script, png image], from multi-data sources in different dirs gnuplot_multi_xRanges IS OP title xLabel yLabel cmtLabel xRanges yRange ySources scriptTemplate d_out - auto-generate [script, png images], multi-data sources in different dirs ********************* loaddefs link d_Qndfs 'Linear interp data table.ndf' linear_interp IS OP dir_name fin_name fot_name x_ints - glaciation_linearInterpolate IS - ********************* loaddefs link d_Qndfs 'Interpolation.ndf' +-----+ Data preprocessing - interpolation_struct interpolation_setup IS OP struct_name - check_t IS OP struct_name t_start_results t_end_results t_delta_results - interpolate_over_interval IS OP struct_name t_start_results t_end_results t_delta_results t_start_data t_end_data fin_name Code - daily data (22May2020 not used now) Potsdam_dailyKp_To_Day IS - data from www.gfz-potsdam.de/en/kp-index/ NOAA_dailyKp_To_Day IS - data from ftp://ftp.ngdc.noaa.gov/STP/GEOMAGNETIC_DATA/INDICES/KP_AP/ Daily Kp tables to 3hr list (22May2020 - currently used) NOAA_dailyKp_To_3Hr IS - data from ftp://ftp.ngdc.noaa.gov/STP/GEOMAGNETIC_DATA/INDICES/KP_AP/ Potsdam_dailyMusicKp_To_3Hr IS - data from www.gfz-potsdam.de/en/kp-index/ Derived results bin_KpBy3Hr IS - calculate Kp bins from 1 to n 3Hrs of zeroKp year_fraction IS OP year_count - convert text date to year_fracction Kp_avgMonthly IS - calculate Kp monthly average from 3hr time series, no filtering, just raw Kp_3hr_yearFrac IS - re-express datta with year fractions instead of 3hr text dates Kp_dayAvg_write IS OP - update "old" measures, write to file Kp_dayAvg IS - re-express data as Kp daily averages using 3hr data text date data list flu_Doshi - flu data 1941-1975 from Peter Doshi May2008 USApopulation_by_timeFrac IS OP - ********************* loaddefs link d_Qndfs 'Linux signals applications non-window.ndf' Convenient applications signals : send_Thunderbird IS OP winName - find winID of email, send, wait get_ID_QNial IS vlc_kill IS - quit vlc video player list_searchSequential_a IS OP lister a - linear search on sorted search, binary search on nonsorted list_searchBisect_a IS OP lister a - method of bisections for sorted list list_searchBisect_a_test IS - simple test of search algorithms comparison: [sequential time = 6.53, bisect time = 0.25, Linux find 0.01]s bessj0 IS OP x - Bessel function J_0 bessj1 - Bessel function J_1 bessj.c - Bessel function J of general integer order Test of odeint - derivs IS OP x y - Bessel_odeint IS OP x1 x2 dx1 - ********************* loaddefs link d_Qndfs 'math - trigonometry.ndf' +-----+ Given [sin, cos] what is the angle? arc_sin_cos IS OP sin cos - arc_cos_sin IS OP cos sin +-----+ What is the minimu equivalent angle +-PI? reduceAngle IS OP angle - minimize the angle (radians) reduceAngle_test IS - Conversion of coordinates [Cartesian, cylindrical, spherical] RF_convert_crt_to_cyl IS OP vector_cartesian for 0 <= θ < 2*π RF_convert_cyl_to_crt IS OP vector_cylindrical - RF_convert_crt_to_sph IS OP vector_cartesian - for 0 <= θ <= π ; 0 <= φ < 2*π RF_convert_sph_to_crt IS OP vector_spherical RF_convert IS OP vector RF_desired - Time derivatives dp_dt_cartesian IS OP vector_cartesian - dp2_dt2_cartesian IS OP vector_cartesian - dp_dt_cylindrical IS OP vector_cylindrical dp2_dt2_cylindrical IS OP vector_cylindrical dp_dt_spherical IS OP vector_spherical - Convert [text, txtLists] to arrays txt_rows_to_array IS OP list_of_rows - txt_cols_to_array IS OP list_of_cols - vector_from_list IS OP c_r lista vector -> array vector_from_txtList IS OP c_r txtList - scp IS OP str_value - special, convert a txtNum to a format that will work in string_array_plus Reshaping, merging - Howell's stuff merge_2_1 IS OP A - create a single 2D array by "stacking-merging", each above next merge_2_2 IS OP A - create a single 2D array by "stacking-merging" 4 arrays above other split_Howell IS OP shapes B - my version of how to split up an array slice_parse IS OP slices - Howell's replacement of QNial's slice operator '|', returns proper vectors, not just a list!! slice IS OP A slices slice_sym IS OP A_nameStr slices rows_to_array IS OP list_of_rows cols_to_array IS OP list_of_cols vector_from_list IS OP c_r lista vector -> array A blend B - merge axes of B according to axis numbers in A (QNial) A fuse B - array formed from B with the axes reordered according to permutation A pack A - interchange the top two levels of A mix A - merge the first two levels of A A raise B array containing arrays of items of B partitioned along axes so that the result has the first A axes of B A lower B array containing arrays of items of B partitioned along axes so that the items have the last A axes of B A split B arrays of items of B where the axes mentioned in A become the axes of the items of the result A blend B array of valence equal to valence of B plus the valence of an item of B obtained by merging axes of B according to axis numbers in A content A - list of atoms of A in depth-first row-major order flip Numeric arrays - simple optrs Identity_matrix IS OP r - Usually called the "Identity matrix" (I) matrix_equalApprox IS OP A B tolerance - allows for approximately equal comparisons (eg round-off errors etc) txtList_optr IS OP txtList - applies optr to txtList string_array_plus IS OP String_list - string_array_minus IS OP String_list - string_array_times IS OP String_list - string_array_divide IS OP String_list - vector_dot_multiply IS INNER[+,*] EACH mix - this isn't really required as matrix_multiply works with vectors and vector/matrix mixes matrix_multiply IS INNER[+,*] EACH mix - standard matrix multiplication for numeric-only arrays use QNial innerproduct for large arrays, fast process matrix_multiply is good otherwise - reminds us how its done, which ivery important innerproduct A B - special Q'Nial code more efficient than INNER[+,*] EACH mix, matrix multiply - number and symbolic string_array_plus IS OP String_list - A is a vectored list of strings, ??? relies on pervasive property of "+" ??? string_array_times IS OP String_list vector_dot_multiply_symbolic IS - produces result of matrix multiplication on A B , matrix_add_symbolic IS - produces result of matrix addition of A B , matrix_add_symbolic_preCheck IS OP m1 m2 - invokes matrix_add_symbolic AFTER checking that shape m1 = shape m2 matrix_multiply_symbolic_preCheck IS OP m1 m2 - invokes matrix_multiply_symbolic AFTER checking that cols m1 = rows m2 vector_dot_multiply IS - not required as matrix_multiply works with vectors and vector/matrix mixes matrix_multiply IS - standard matrix multiplication for numeric-only arrays remember : (second m1_shape) = (first m2_shape), cols m1 = rows m2 innerproduct A B - special Q'Nial code more efficient than INNER[+,*] EACH mix, matrix inversion and multiple linear regression see link d_Qndfs 'multiple linear regression - NN41 Smillie Uof Alberta.NDF' Labelled arrays - basic operators and transformers array_indexLabelled IS OP Symbol_string shapet - shows array_name@[i,j] symbols for easily following processes! array_numeric_to_indexLabelled IS OP A_name A - indexLabelled array from "A" (old version) Labelled_array - for easily following processes! (old, simpler version of array_indexLabelled) array_indexLabelled IS OP Symbol_string shapet - indexLabelled arrays - shows array_name@[i,j] symbols for easily following processes! array_numeric_to_indexLabelled IS OP A_name A - builds an indexLabelled array from "A", LCM IS OP a b - Least Common Multiple (from Wikipedia, using GCD = Greatest Common Denominator) identity_matrix IS OP r - generate a NUMERIC identity matrix (r = rows,cols) identity_matrix_is IS OP matrix - check to see if a data is an identity matrix identity_matrix_sym IS OP r - identity matrix '0's and '1's are strings, not numbers, suitable for symbolic computations, 13Apr2016 identity_matrix_sym_is IS OP matrix - check that a matrix is an identity_matrix_sym (string 0s & 1s) diag_c IS OP r c - generate a diagonal matrix from a constant diag_c_sym IS OP r c - generate a diagonal matrix from a [constant, variable] string diag_v IS OP V - generate a diagonal matrix from a vector diag_v_sym IS OP V - generate a diagonal matrix from a symbolic vector (strings) diag_v_get_sym IS OP A - generate a column vector from a square matrix diagonal (strings) diag_block_make_sym - construct a square matrix from two input matrices Symbolic array operations - see also 'symbols ndfs' for workings with derivations (different issue) matrix_add_symbolic IS OP A - symbolic matrix addition mm_constant_to_string IS OP c - matrix_add_symbolic_preCheck IS OP A - matrix_symmetric_symbolic IS OP matrix - make a symmetric matrix by adding a materice's transpose matrix_divide_symbolic IS EACHBOTH string_array_plus A_sym B_sym - symbolic addition of A B matrix_divide_symbolic_preCheck IS OP m1 m2 - invokes matrix_divide_symbolic AFTER checking that shape m1 = shape m2 vector_dot_multiply_symbolic - This is the SAME as matrix_multiply_symbolic !! matrix_multiply_symbolic IS INNER[string_array_plus, string_array_times] A_sym B_sym - produces result of matrix multiplication on A B , mms_textWrap_scalar IS OP c_str A_cell_str - ??? matrix_multiply_symbolic_preCheck IS OP A - associative, but not commutative except constants matrix_subtract_symbolic IS - produces result of matrix addition of A B matrix_subtract_symbolic_preCheck IS OP m1 m2 - invokes matrix_add_symbolic AFTER checking that shape m1 = shape m2 Hadamard operators HadamardVector_func_sym IS OP func A V - generalizes HadamardVector_prod_sym HadamardVector_prod_sym IS OP A V - symbolic Hadamard matrix product Aij*Bij, = EACH [row, cols] A B, Hadamard_prod_sym IS OP A B - symbolic Hadamard matrix product Aij*Bij, = EACH [row, cols] A B Kronecker operators Kronecker_prod IS OP A B - Kronecker product (circle with "X" in it) Kronecker_prod_symbolic IS OP A B - Kronecker matrix product Cheng's semi-tensor product operators Semi_tensor_prod IS OP A B - Cheng's semi-tensor product Semi_tensor_swap_matrix_make IS OP M N - Cheng's swap for semi-tensor operations Execute equation - symbolic processing of equation execute_equation IS OP eqn_string - execute equations written in "normal format", by substituting for [+,-,*,/] calc_once IS OP eqn_rows - 09Oct2020 move to a test file!! for a special listing of tests ********************* loaddefs link d_Qndfs 'matrix - symbolic, parenthesis aligned.ndf' states_initialize IS - states_add_primitive IS OP text_state text_first text_second text_third - states_add IS OP text_state text_first text_second text_third - states_decrement_to IS OP parts - text_add_to_states IS OP t_type chr_next - text_extract IS OP cell cell_write_parenthesis_aligned IS OP cell - matrix_multiply_write_parenthesis_aligned IS OP matrix_name - ********************* loaddefs link d_Qndfs 'matrix - symbolic reduction.ndf' parenthesis_check IS OP stringer - 1st STEP!!! Ensures that an expression is "coherent" for latr processing! space_single IS OP stringer - space_no_lead_trail IS OP stringer - opn_exprns_parse IS OP stringer - zero_out IS OP str_opn_exprns_closr - removes zeroed expressions text_chunker IS OP cell - chunk_add IS OP A - chunkit IS OP text - chunk_reduce IS OP change_flag chunklet - op_redundancy - removes redundant operations in string mm_symbolic_reduction IS OP A - ********************* loaddefs link d_Qndfs 'matrix - write with parenthesis aligned.ndf' States nofspace IS OP str - states_initialize IS - states_add - is only for NON-[test, space] = [t, s] coding states_decrement_to IS OP parts - text ops - for mms = matrix_multiply_symbolic text_extract IS OP cell - write_lines_out IS OP indent wl line_out - cell_write_parenthesis_aligned IS OP cell - matrix_multiply_write_parenthesis_aligned IS OP matrix_name - matrix_write_cells_parenthesis_aligned IS OP matrix_name For matrix_write_cells_parenthesis_aligned text_extracter IS OP cell - write_noBlankNulls IS OP wrt_indents str - write_parenthesis_aligned IS OP text indent_level - link d_QNial_mine 'MindCode/arith, trig examples.ndf' *** intg00 - [example, test] integer network setup *** real00 - [example, test] real network setup Code tests - these should be lauched from 'MindCode/MindCode basics.ndf' tests IS - standard checks that operators function as expected link d_QNial_mine 'MindCode/bits to-from [intg, real].ndf' sign_it IS OP spikeTrain - converts a "sign" spike of a train to [+-] integers bits_to_intg IS OP bits - convert intg bits to integer number, high bit on -> negative spikes_to_intg IS OP bits - [s2i], for final output: spike pattern to integer number intg_to_bits IS OP intg_in - integer number to bitWidth code, first bit is sign (+-1) intg_to_spikes IS OP intg_in - [i2s], integer number to 8 neuron spike pattern real numbers - bits_to_real IS OP bits - spikes_to_real IS OP spikes - [s2r], spike pattern to real number real_to_bits IS OP real - real number to bitCode real_to_spikes IS OP real netID - [r2s], integer number to 8 neuron spike pattern link d_QNial_mine 'MindCode/code test/bits to-from [intg, real] tests.ndf' integers_test IS - test correctness of results using known results reals_test IS - test correctness of results using known results link d_QNial_mine 'MindCode/code test/fire tests.ndf' fire_test01 IS - simple first test to see if "fire" operator works as expected link d_QNial_mine 'MindCode/code test/index tests.ndf' integers_test IS - test correctness of results using known results link d_QNial_mine 'MindCode/code test/MLP tests.ndf' *** net - column descriptions, tables are split up for ease of [view, edit]ing etc ****** net sensory (inputs) and motor controls (outputs) intgConstNetIDL - netIDs for integer constant sensory inputs to net realConstNetIDL - netIDs for real constant sensory inputs to net inConstantL - sequence expressions for sensory inputs ****** net_morphFlip IS - [morph, flip]ing parameters 'modL' - neuromodulator state [dopamine, serotonin, noradrenaline, acetocholine] 'wghtL' - weight list, required for classical (non-epi-DNA) neurons ****** net_triggers - ****** net_results IS - very simple example net 'fireL' - fire state of the net output, expressed as spikes (booleans) 'ResStdL' - "standard" result (presumably correct result for comparison against ResCalcL) 'ResCalcL' - results at current time step 'OKL' - check if net produced target result ('ResStdL' = 'ResCalcL') integers_test IS - test correctness of results using known results reals_test IS - test correctness of results using known results link d_QNial_mine 'MindCode/code test/trigger [now, cum, seq, tim, mix] tests.ndf' notes trgr_test IS - test correctness of results, no history, just nowFireL link d_QNial_mine 'MindCode/code test/control structures.ndf' *** Control structures if_exprn_Z80 IS OP netID strtNetID endNetID ifNetID for_loop_Z80 IS OP netID strtNetID endNetID sequenceID while_loop_Z80 IS OP netID strtNetID endNetID sequenceID link d_QNial_mine 'MindCode/code test/data types.ndf' initialize [data types, global variables] setup_8bit IS - currently 8 neuron spike patterns (8 bits = byte, signed) setup_32bit IS - 32-bit neuron spike patterns (8 bits = byte, signed) power_series IS OP bitWidth - produce a power series for conversion of bits to [intg, real, etc] loaddefs link d_QNial_mine 'MindCode/globals initialize.ndf' - link d_QNial_mine 'MindCode/globals initialize.ndf' initGlobals IS - initialise global variables view MindCode global [network, cols] table_write IS OP titles - ONLY for columns of the global MindCode net!!! dataTable_write IS OP titles dataTable - output any table with single-line headers link d_QNial_mine 'MindCode/MindCode basics.ndf' build MindCode global [network, cols] local_to_global_cols IS - based on all sub_nets loadded, adapt local-to-global indexing nmID_to_ID IS OP nmID - given the nmID=[nameL, IDsubL], find the stdID stdInOutTrig IS - build stdOutL list of (each net [input, output, trgrPat] connections) stndNet_build IS - add n_cols of of "standard" cnstNet data to corresponding globalCols of MindCode fire [sensory, netID fire IS OP netID - fire netID (of lastFireL nets), using its optr MindCode_[stop, cycle, load] MindCode_stop IS - output the final results MindCode_cycle IS - fire appropriate neurons, then update the net MindCode_load IS - load all sub-nets proceed to run netMindCode ; tests IS - collection of standard checks that operators function as expected link d_QNial_mine 'MindCode/MLP - multi-layered perceptron.ndf' *** classical NNs perceptron IS OP neuronID - classical continuous neuron, inNets=(synaps, weight) MLP_backprop IS OP netID - basic feedforward MLP training algorithm MLP_orderiv IS OP netID - solid MLP training algorithm link d_QNial_mine 'MindCode/net constructors.ndf' net construction - reduces [error, time] of hand coding (later - separate files for MLPs etc) net_construct IS OP net_specs - build code for net based on net_specs memory - storage of spiking patterns, must address [short, long] term (06Mar2020 - 'histPatL' is all I have) link d_QNial_mine 'MindCode/net finder.ndf' finders - find nets that is available to do a task, amongh those that are able to do it findArith IS OP order type optr - search for a net that can handle [type,op] findExpert IS OP order type op - search for a net that can handle [type,op] taskers - allocate a task to one of nets capable of doing it arithTasker IS OP order type op - search for a net that can handle [type,op] formulators - arithFormulator IS OP problem - translate a [problem, objective, interest, curiosity] to a useable format commanders - drive results cmndrQNial IS OP order - demands made simply via QNial [data, op, transform]s (code) "personnel" (nets) cmndrClient IS OP order - demands made of "clients" (nets) cmndrPredator IS OP order - demands made of "predators" (nets) cmndrPrey IS OP order - demands made of "prey" (nets) addInteger IS OP - create [neuron, connect]s for a neuron cluster that represents an integer integer IS OP - give some short description... standard basic trigger patterns [now, cum, seq, tim]Trgr netIDs pass general data (NOT just a single spike), which can be used in triggers nowTrgr IS OP trgrPat histPat - no history, just (first lastFireL) cumTrgr IS OP trgrPat histPat - [order, timing] not important seqTrgr IS OP trgrPat histPat - sequence important, but not timing timTrgr IS OP trgrPat histPat - [sequence, timing] are important! mixTrgr IS OP trgrPat histPat - mixture of other trgrTyps (each case is special, separate oprtr) link d_QNial_mine 'MindCode/Z80 operator listing.ndf' *** Usage via fire_apply (see "MindCode basics.ndf") fire IS OP netID - fire netID (of nowFireL nets), using its optr *** Logic operators 'and' 'or' 'not' '=' ['~=', 'unequal'] 'diverse' *** Comparison operators (boolean) ['>' , 'gt' ] ['>=' , 'gte'] ['<' , 'lt' ] ['<=' , 'lte'] 'match' 'mate' 'max' 'min' 'up' *** Types operators (boolean), for atomic types 'isboolean' 'ischar' 'isfault' 'isinteger' 'isphrase' 'isreal' 'isstring' *** Arithmetic operators, [integer,real] [inputs, output] where Xxx = [Z80, Pat, ...] Pat = pattern, [unary, binary, multi] pervasive ['sumXxx'] - multi, ['productXxx'] - multi, 'timesXxx' - binary, 'divideXxx'] - binary, 'opp' - pair, produce the negative (* -1) of the single input to netID '*' - multiply inputs of ALL preNetIDs 'recip' - produce the reciprocal (1 /) of the SINGLE input to netID 'exp' - produce the exponent (e^) of the SINGLE input to netID 'ln' - produce the natural logarithm (ln) of the SINGLE input to netID ['^', 'power'] - inNets = [baseNet, exptNet] in that order, raise baseNet to the power *** Trigonometric operators require real inputs 'sin' 'cos' 'tan' 'cosh' 'tanh' 'arcsin' 'arccos' 'arctan' 'arcsinh' 'arccosh' 'arctanh' *** NN-specific basics - sigmoidal, counter, timer, concept neuron 'count' timer IS OP timerNetID - ['dotProdxxx', 'INNER [+, *]'] IS - produces the inner (dot) product of two arrays of arbitrary dimension ********************* loaddefs link d_Qndfs 'Music - radio record, cut, splice.ndf' get_timestamp IS - radio_record IS OP duration - cut_music IS OP filename - rr IS radio_record (0.1*60) - ********************* loaddefs link d_Qndfs 'NEUNET-D-14-00130 tests.ndf' ********************* loaddefs link d_Qndfs 'numberBase conversions.ndf' baseChr_to_decInt IS OP baseChr - decInt_to_baseChr IS OP a - basePre_to_basePwr IS OP a - baseStr_to_basePwr IS OP a - baseStr_to_baseChrs IS OP a - basePre_baseChrs IS OP a - baseStr_to_basePwrSeq IS OP baseStr - wind_dwn_recurPair IS OP a - baseStr_to_decIntUnSigned IS OP baseStr - decInt_to_baseChrs_raw IS OP pwr decInt - decInt_to_baseStr IS OP decInt basePre - decInt_to_baseStr_len IS OP decInt basePre len - baseStr_to_baseStr_len IS OP baseStr basePre len - baseStr_to_decIntSigned_len IS OP baseStr len - baseStr_to_decIntSigned IS OP baseStr - baseStr_to_decIntSigned_time IS - build_hexTable IS - ********************* loaddefs link d_Qndfs 'ODE Cash-Karp-Runge-Kutta integrate ODEs with accuracy monitoring.ndf' derivs IS EXTERNAL operation ; derivs_e := f_Cash_Karp_Runge_Kutta ; - sign IS OP a b { IF b >= 0.0 THEN abs a ELSE opposite abs a ENDIF } - rkck IS OP x y dx dydx deriv_name - used by rkqs rkqs IS OP x y dx dydx eps yscal deriv_name - integrate one step of ODEs with accuracy monitoring odeint IS OP x1 x2 ystart dx1 dx_min dxsav eps kmax deriv_name - ********************* loaddefs link d_Qndfs 'optimize - particle swarm.ndf' +-----+ Code PSO_optimize IS - general Particle Swarm Optimizer ********************* loaddefs link d_Qndfs 'paper reviews - strip out illegal characters.ndf' char_substitute IS OP char_boolean { first (char_boolean sublist chrs_subbers) } - paper_remove_characters IS OP fname_inn fname_out - review_clean IS paper_remove_characters - ********************* loaddefs link d_Qndfs 'PayPal transaction text process.ndf' get_date IS OP texter - get_amount IS OP texter - PayPal_process IS OP f_name - ********************* loaddefs link d_Qndfs 'port GREP_Fortran_to_QNial.ndf' GREP_Fortran_to_QNial IS OP filename_in filename_out - ********************* loaddefs link d_Qndfs 'port Matlab to Nial.ndf' - simple work to help convert Matlab code to Nial Matlab_to_Nial IS OP File - METHOD_AN1 IS 1 ; - METHOD_AN1_CJ IS 2 ; - METHOD_AN2 IS 3 ; - METHOD_AN2_CJ IS 4 ; - METHOD_AN1_LM IS 5 ; - METHOD_AN2_LM IS 6 ; - METHOD_REGRS_CJ IS 7 ; - METHOD_REGRS_LM IS 8 ; - METHOD_SS IS 9 ; - METHOD_SSG IS 10 ; - #C_to_Nial_testfilename IS OP File - Assignment_substitute_test IS OP Record - ********************* loaddefs link d_Qndfs 'port_translate_Fortran_to_QNial.ndf' GREP_Fortran_to_QNial IS OP filename_in filename_out - lf IS loaddefs f 0 ; - g IS translate_Fortran_to_QNial filename_in filename_out ; - ********************* loaddefs link d_Qndfs 'port_translate_Matlab_to_QNial.ndf' seq IS OP a b - output_funcEnd IS - clean_func_declare IS OP func_declare - test_state_line IS OP state_phrase - translate_Matlab_to_QNial IS OP filename_in filename_out - g IS translate_Matlab_to_QNial filename_in filename_out ; - ********************* loaddefs link d_Qndfs 'QNial symbol changes.ndf' - change names of operators, all QNial (not even started work!) QNial_symbol_changes IS OP dir - change ALL occurences of Howell's QNial symbols in directory dir ********************* loaddefs link d_Qndfs 'loaddefs link d_Qndfs 'QNial symbol list, clean.ndf' is_blankstring IS OP Str - towords1 Str - returns the list of STRINGS (not phrases!! like the QNial library) clean_symbol_list IS OP directory filename_in filename_out - ********************* loaddefs link d_Qndfs 'random_lists.ndf' - [[pre-,] number, [pre,,post]-cart[capital, small] chars ********************* loaddefs link d_Qndfs 'readDataFile corona virus global.ndf' - code is [incomplete, in change] dataSets by format type fList_1968_1978 IS - fList_1979_1988 IS - fList_1989_1998 IS - fList_1979_1988 IS - fList_2003_2018 IS - load_dataSet IS OP fLister - unfinished or in change ********************* loaddefs link d_Qndfs 'readDataFile fixed cols suicides USA - old version.ndf' downloadCDC_1968_to_2018 IS - download & unzip 1968-78 "USA CDC-NCHS mortality multiple cause files" download_fList IS - download & unzip failed download sampleData IS OP fname - print 1st 5 lines of a "USA CDC-NCHS mortality multiple cause" file filterDataFile_test IS - extract suicides from "USA CDC-NCHS mortality multiple cause file" filterDataFile IS - extract suicides from "USA CDC-NCHS mortality multiple cause file" gnuplot_daily_suicides_by_year IS - auto-generate png images for each year ********************* loaddefs link d_Qndfs 'readDataFile header.ndf' suicides IS OP yearStart yearEnd gnuplot_daily_suicides_by_year IS - auto-generate png images for each year SP500_multiFractal IS OP yearStart yearEnd - SP500 monthly averages gnuplot_xSpanSet IS OP yearStart yearEnd ySourcesName - auto-generate png image gnuplot_xSpanYrs IS OP yearStart xSpanYrs - auto-generate png image of a set span in years ********************* loaddefs link d_Qndfs 'readDataFile.ndf' - dataFile_sample IS OP fname - print 1st 5 lines of a file filterPosn IS OP - extract data cols selectively from text file with character separator of cols filterCol IS OP - extract data [cols, rows] selectively from text file cull_period IS OP d_wrk pinn pout yearList - pull data for specific time (or other dataType) period ********************* loaddefs link d_Qndfs 'readDataFile suicides USA header.ndf' +-----+ dataSets - [define filePaths, download] of [individual, groups of similar] data URLgroupL - remove those not desired (but keep in the backup "full_groupL_"!!), solitary if single item in groupList ********************* loaddefs link d_Qndfs 'readDataFile suicides USA.ndf' +-----+ data[Set, Group]s specifications download_dataSets IS - download [full, part] of dataSets from USA CDC-NCHS-NVSS group_1968_1978 IS group_1979_1988 IS group_1989_1998 IS group_1999_2002 IS group_2003_2018 IS +-----+ dataSets processing recodeOP IS OP fields - 1968-2002 make all sex 'M' or 'F' selectLine IS OP fields - select whether to include the fields in the output file process_dataGroup IS OP f_groupLer - process a group of similar datasets checks on results - simply compare [official, Howell] number of cases +-----+ [unify, bin, normalize, pretreat, process, gnuplot] ALL dataSets cat_allFiles IS - concatenate all suicide files suicides_by_day_sex IS - bin the number of suicides for each day, using [cumulative, sorted] data suicides_by_month_sex IS - bin the number of suicides for each day, using [cumulative, sorted] data process_allData IS OP - [process, select, gnuplot] ALL unified data +-----+ Instructions : ********************* loaddefs link d_Qndfs 'readFile_Unix.ndf' can read Unix-like (LF or /n) end-of-line text files fin - is the file number of a text file that has already been opened 09Feb2021 - just use Unix find | sed readfile_Unix IS OP fin - convert links to relative ********************* loaddefs link d_QNial_mine 'review move comments.ndf' list of peer review comments by class move_comments IS OP fname - moves simply-coded comments from "Details" note-taking area to correct section of peer review ********************* loaddefs link d_Qndfs 'reviews Schmidhuber - references analysis.ndf' - referenceList_unifyRefs IS OP fin_name fot_name line_start line_end - this combines multi-line references into a single text line for each reference test1 IS referenceList_unifyRefs fp1_0 fp1_1 8 1353 - test1 IS referenceList_unifyRefs fp2_0 fp2_1 1 1665 - authors_extract IS OP fin_name fot_name fut_name line_start line_end - test3 IS authors_extract fp1_2 fp1_3 fp1_4 1 601 - test3 IS authors_extract fp2_2 fp2_3 fp2_4 1 752 - co_authors_refs IS OP singleAuthor_list ref_AuthorList co_authors_refs_fName line_start line_end - test5 IS co_authors_refs fp1_5 fp1_4 fp1_6 1 731 - test5 IS co_authors_refs fp2_5 fp2_4 fp2_6 1 1079 - ********************* loaddefs link d_Qndfs 'reviews - strip out illegal characters.ndf' paper_remove_characters IS OP d_inn d_out fname - ********************* loaddefs link d_Qndfs 'semi-log formula.ndf' year_frac IS OP str_ddmmmyyy - cheap inaccurate estimate of fraction of year that has passed setup_semilog IS OP xStr xEnd yStart yEnd - get_yEnd IS OP xVal - write_summary IS OP testName params dateStrs y_standards - test_SP500_1871_1926 IS - test_SP500_1926_2020 IS - ********************* loaddefs link d_Qndfs 'semi-tensor product - examples and checks.ndf' - ********************* loaddefs link d_Qndfs 'semi-tensor product.ndf' semi_tensor_LCM IS OP A B - provides appropriate dimensions for semi_tensor_prod, order of A B is critical semi_tensor_prod_symbolic IS OP A B - for "symbolic" arrays - uses "indexLabelled" or "numString" arrays semi_tensor_prod IS OP A B - This is for "numeric" arrays, which can be calculated immediately semi_tensor_prod_allSteps IS OP A_name_vals B_name_vals - the idea here is that the user sees the index_Lablled, numStrings, and final results array_to_row_stack IS OP A - Row stacking form of an array swap_matrix_make IS OP M N - swap matrix, this is ONLY [0,1] valued! ("equivalent" to symbolic?) ********************* loaddefs link d_Qndfs 'SP500 Shiller-forward PE ratio and Tbond rates.ndf' model_basic IS OP - ********************* loaddefs link d_Qndfs 'spreadsheet from text preps.ndf' sheet_cleaner IS OP File - cleans up email list from Thunderbird or LibreCalc email_extract_addresses IS OP File - extracts addresses from a Thunderbird email list sheet_clean IS sheet_cleaner '/media/bill/SWAPPER/Qnial/MY_NDFS/spreadsheet - text.in' - #email_extract IS email_extract_addresses '/media/bill/SWAPPER/Qnial/MY_NDFS/email - list of recipients.in' - ********************* loaddefs link d_QNial_mine 'strings.ndf' opL_strings IS host 'grep "^ " "$d_Qndfs""strings.ndf" | sed "s/^#\]/ /" ' # $ grep "^ " "$d_Qndfs""strings.ndf" | sed 's/^#\]/ /' +-------------------+ characters [special, unicode] often used ... : +-------------------+ Date-related [timestamps, strings, waits] : sec_from_timestamp IS hhmmss_to_seconds IS OP hhmmss sec_to_hhmmss IS OP secs hhmmss_elapsed IS OP t_start_sec - elapsed time from a given t_start_sec waitSecs IS OP secs - uses Linux sleep, best to simply use host 'sleep 1s' ; wait_hhmmss IS OP secs - Linux sleep, converts hhmmss to ss waitLocal IS OP hhmmss - Linux sleep, converts hhmmss to ss waitVideo IS OP t_voice - used for video production software day_2digit IS OP day - day of month in 2-digit format month_name IS OP month_number - given 2-digit month number, returns 3 letter month name month_number IS OP month_name - given 3 letter month name , returns month's 2-digit number emlDate_to_YYMMDD IS OP emlDate emlDate_to_YYMMDD_HHMMSS IS OP emlDate timestamp_YYMMDD IS - returns timestamp in obvious format timestamp_YYMMDD_HM IS - returns timestamp in obvious format timestamp_YYMMDD_HMS IS - returns timestamp in obvious format timestamp_to_YYMMDD_HMS IS OP sys_timestamp - format timestamp eg "dayOfWk MMM DD hh:MM:SS MST YYYY" timestamp_DDMMMYYYY IS - returns timestamp in obvious format timestamp_DDMMMYYYY_HMS IS - returns timestamp in obvious format sleep_debug IS OP n_operator fname t_start t_wait - sleeps depending on timestamps +-------------------+ string cleanup str_surroundWithApos IS OP str - convert str to string with apostrophes, or if not a string return str_surroundWithAposAddTab IS OP str - convert str to string with apos & tab, or return input str_to_unicodeList IS OP str - convert a string to a list of unicode characters (not unicode # sequences) str_lastNonWhiteChr_position IS OP Str pos - retrieves the last non-whitespace character str_toUpperCase IS OP str - convert all letters in a string to upper case, STUPID - use toupper! str_toLowerCase IS OP str - convert all letters in a string to lower case, STUPID - use tolower! str_flipCase IS OP str - 18Jun2020 do this later str_capitalizeCase IS OP str - 1st capital letter, rest small, 18Jun2020 do later str_sentenceCase IS OP str - 1st capital letter, rest small, 18Jun2020 do later +-------------------+ find [chrs, sub-strings, etc] : str_findChrsInSet IS OP flag_chrs_include chrs_set str_in - search str_in str_findPositionsOf_subStrQ IS OP Substr Str - returns list of Substr positions in Str, or an error str_findPositionsOf_subStrs IS OP Substrs Str - returns list of Substr positions in Str for EACH Substrs, or an error (23Oct2020 incomplete) +-------------------+ tests of strings chr_in_str IS OP chr Str - simple indication of whether chr is in str subStr_in_str IS OP Substr Str - simple indication of whether a Substr is in str str_in_path IS OP Substr p_name p_hostCmdRslt - boolean indicator that Substr is in file p_name str_isBlank IS OP Str - test that string is entirely "whitespace" (space, tab, CR, NL) str_isWhite IS OP line - tests if a line is all "whitespace" [space, tab, CR, \n] write_testStr IS OP t_name t_input t_standard t_result - means of testing string answers test_comment IS OP t_name t_input t_standard t_result - means of testing string answers write_test_head IS OP t_name t_input t_standard t_result - output header for test? str_eachChrIs_white IS OP Str - boolean list of text - l for whitespace, o for other +-------------------+ string [add, remove, replace] parts : lead_chrs IS OP len chr st - returns : if length < len, returns str, else first len chrs str_prepadWith_chr_toLength IS OP str chr length - prepads str with chrs to get length str_postpadWith_chr_toLength IS OP str chr length - postpads str with chrs to get length str_strip_chr IS OP Str - remove chr from string, standard QNial library str_stripAllSpacesQ IS OP Str - standard QNial library str_stripAllSpaces IS OPERATION Str - removes ALL spaces from a string str_removeExtra_whitespace IS OP Str - removes [leading, multiple-sequence, trailing] whitespace str_remove_subStr IS OP str subStr - removes subStr from str str_repeatNtimesTo_strList IS OP str n - repeats str n times str_replace_subStr IS OP sub_old sub_new str - substitutes sub_new for sub_old in str str_replaceAll_strOld IS OP strOld strNew strFull - strOld is replaced by strNew in strFull str_extractPast_strFront IS OP strFull strFront - extract end of strFull after strFront str_reduceTo_width IS Op str1 sw - cut string to fixed length, or pad end with spaces to make length +-------------------+ string Lists : str_splitBy_str IS OP S1 S2 - splits S2 at each point that S1 occurs, S1 part of next split str_splitBy_char IS OP chr Str - cut a string at each chr, omitting chr # str_splitBy_char `: ' webRaw_links IS - generate fileList with %20 in links (p_html_files_pct20)' str_splitWith_subStr IS OP S1 S2 - splits S2 at each point that S1 occurs, S1 separate part of list str_splitLftRgtTo_midIndxs_StrList IS OP strLft strRgt str - split str, return [indxs, strLst] str_replaceSubTo_strList IS OP S1 S2 S3 - splits S3 at each S1, replace S1 with S2 str_splitByWidthTo_strList IS OP Text Width - cut text into strings of equal length, broken on word boundaries strList_replace_str IS OP strOld strNew strList - replace all strOld by strNew in strList aryList_extractMulplicate_subArys IS OP selectOp strList - find mulplicates in strList by selectOp works with strLists, or any array type (in form of list) - see d_Qroot 'setup.ndf' +-------------------+ Words - are NOT "corrupted" by [search-&-replace, sed, other] replacements (eg QNial operator names) note that the sed -w option does better than my simple coding str_splitWithChrsAllowedTo_words IS OP chrSet str - splits str into words based on allowable chrSet str_splitByChrsNotAllowedTo_words IS OP chrSet str - splits str into words by dis-allowable chrSet booleanList_cutTo_indexLists IS OP booleanList - boolean list cut into alternate [o, l] sublists str_cutTo_wordList IS OP str chrList - returns the list of words in a string, word_findPosnIn_str IS OP wrd str wType cList - returns a list of positions of a word in a string word_replaceIn_str IS OP sOld sNew str chrList - change word throughout string (safe for variables) +-------------------+ convert strings [to, from] other types : atomList_to_str IS OP S A - creates one string from a list of atoms, separating each by S array_atomsTo_strs IS OP A - convert atoms of A to strings (leaves strings as strings) array_leafsTo_strs IS OP A - every atom is converted to a string, to enable symbolic processing array_to_str IS OP A - converts array A to a string, each item separated by one blank (Qnial lib) str_to_phrases IS OP Str - returns the list of phrases of the words in the string. numList_to_str IS OP numList - changes a list of reals to a string, each real followed by a space +-------------------+ HTML-related, or other markup languages adaptable str_extractBetween_StrtEnd IS OP strStrt strEndr strIn - extracts sub-strings bracketed by [strStrt,strEndr] I need an operator for intraPageLinks to TableOfContents (invocable by embedded executable) I need operators for conversions between [QNial, HTML, csv, loaddef] tables +-------------------+ Embedded executables - markup language (eg html) coding in pre-processed text file '[#!: ' is for full-line embeds; '[#=; ' is for embeds within a line a bit like javascript etc, but not "live", just for "compilation" str_executeEmbeds IS OP str phraseValueList - execute embeds in line, return a str '[#!: ' is for full-line embeds; '[#=; ' ' ;=#]' are for embeds within a line % remove [#= = brackets ; ********************* loaddefs link d_Qndfs 'Structure.ndf' ToDos +-----+ Generic structure operators - for use by any [real, avatar] structure sd_ IS OP s_structPhr s_raw - structure definition Alternative initial real structures, required to load [data, optrs] sCompare_ IS OP struct0 struct1 - shows differences, only useful when structures are same +----------------+ Real structure operators - for use by any avatar structure, but just one avatar per real, specialised expressions to hide real structure, eg "put_test03 IS s1_p" +-----+ s0_ - first of several [simple, real] 2D structures, for which avatars are used in apps s0_g IS OP s_lab - get component of structure s0_p IS OP s_lab s_value - [put,change] component of structure s0_p_test IS - test s0_p s0_w IS - write s0_, one format for showing a 2D data structure (table) s0_h IS - help, full list of structure [phrases, explanations] +-----+ s1_ - second of several [simple, real] 2D structures, for which avatars are used in apps s1_g IS OP s_lab - get component of structure s1_p IS OP s_lab s_value - [put,change] component of structure s1_w IS - write s1_, one of many formats for showing a 2D data structure (table) s1_h IS - help, full list of structure [phrases, explanations] +-----+ s2_ - third of several [simple, real] 2D structures, for which avatars are used in apps s2_g IS OP s_lab - get component of structure s2_p IS OP s_lab s_value - [put,change] component of structure s2_w IS - write s2_, one format for showing a 2D data structure (table) s2_h IS - help, full list of structure [phrases, explanations] +-----+ General hierarchical tree structure (not developed fully) s_varsTypesAdd IS OP s_valuePhr s_TypePhr - general (tree) structure SEQ_REST IS OP f_A_Phr f_A_B_Phr A B - my screw-ups with transformers : s_phrList_to_indx IS OP s_TypePhr s_labPhrList - sG_g IS OP s_Phr s_labPhrList - general form of simple s[.]_g for an hierarchical tree framework sG_g_tests IS - sG_p IS OP s_TypePhr s_Phr s_labPhrList s_value - sG_p_tests IS +-----+ Playing around - perhaps for some time in the future sgLab_ IS OP winComp compLabSeq - perhaps for some time in the future +-----+ Older approach - from several years before Aug2017, but I can't remember when struct_get_def IS OP struct_name component_name - get definition of structure struct_get_many IS OP struct_name component_names - get many components of structure sp_ IS OP struct_name component_name new_value - put value into structure struct_help IS OP struct_name component_name - help message for structure struct_get_deep IS OP struct_name component_name component_path - struct_put_deep IS OP struct_name component_name component_path new_value - untested!! ********************* loaddefs link d_Qndfs 'symbolic reduction - examples.ndf' ********************* loaddefs link d_Qndfs 'symbolic reduction.ndf symbols&Identifiers IS - string_test_constant IS OP Str - determine if str is a constant is_zero IS OP str - tests if an expr is zero, within a tolerance of zero_tolerance is_one IS OP Str - tests if an expr is one, within a tolerance of zero_tolerance of IS OP str1 str2 - currently only used for arrays... string MUST be in parenthesis !! paren&whiteSpace IS OP Str - produces boolean list of paren&whiteSpace in a string process_string IS OP Str i - doesn't account for '' -> quote character in string process_paren IS OP Str i - tests if parenthesis are balanced append_block IS OP block_list block_now - [creates, appends] to a block_list paren_chunk IS OP txt - splits text into white-space separated text plus paren-enclosed txt parenTxt_split_by_whiteSpace IS OP Str - same as string_split_by_paren&whiteSpace, except paren-expressions non-separate append_block IS OP block_list block_now - [create, append]s to block_now parse_simple IS OP txt - extracts "zeroth-level" expressions prefix_opertr&num&Var IS OP opertr_positions replaceAll_Typ removeOne_Typ typeLst blockLst infix_opertr&num&Var IS OP typeLst blockLst - for now this simply goes left to right, opList_zero_or_one IS OP typeInParen blockInParen - takes exprn from which parenthesis have been removed mm_paren_prefix IS OP paren_positions replaceAll_Typ removeOne_Typ typeLst blockLst mm_symbolicReduction IS OP typeLst blockLst - condense an expression with constants, where this is easy symbolic_reduction IS OP txt - simplify expressions, ********************* loaddefs link d_Qndfs 'symbolic reduction - tests.ndf' - fon IS { NONLOCAL flag_break_srTest flag_break_symRedn ; flag_break_srTest flag_break_symRedn := ll ; } - foff IS { NONLOCAL flag_break_srTest flag_break_symRedn ; flag_break_srTest flag_break_symRedn := oo ; } - test_varName&Index IS - test_of IS - test_process_string IS - test_process_paren IS - #test_string_split_by_paren&whiteSpace IS - test_string_split_by_paren&whiteSpace IS - test_paren&whiteSpace IS - test_view_type&block IS view_type&block - test_is_constantStr IS - test_is_zero IS - test_is_one IS - test_type_text IS - #test_type_text IS - #test_prefix_opertr&num&Var IS - test_prefix_opertr&num&Var IS - #test_opList_zero_or_one IS - test_opList_zero_or_one IS - test_parenTxt_split_by_whiteSpace IS - #test_infix_opertr&num&Var IS - #test_infix_opertr&num&Var IS - test_arithmeticExprn IS - #test_mm_paren_prefix IS - test_mm_paren_prefix IS - #test_mm_symbolicReduction IS - #test_mm_symbolicReduction IS - test_mm_symbolicReduction IS - test_mm_paren_prefix_del3 IS - test_mm_symbolicReduction_del3 IS - test IS OP selNum - ********************* loaddefs link d_Qndfs 'symbols/symbols translate - Laskar.ndf' symbol_find IS OP sym_in p_sorc f_outr - find one [normal,weird,orphan] sym_in, output [line#, line] symbol_finds IS OP p_tranList p_sorc p_outr - find tranList [normal,weird,orphan], output [line#, line] symParen_contents IS OP sym_in - remove [1st,last] paren match, 1st level processing syms_inParens IS OP symList - to extract symbols RECURSIVELY from parenthesized expressions symExtract_decisionTable IS - logic table of expressions strStd_extract_symParnPosn IS OP str_stdForm - extracts symParns from a "standard format string" symbols_compare IS OP pn_Lucas pn_HFLN - compare a list of extracted symbols to known symbol lists file_extract_symParn IS OP pname_stdForm_inn d_out - extract list of [all, known, unknown] parn symbols symbol_check_POI_format IS OP symChrList - check if POI format is standard strExtract_POIs IS OP str - extract POI symbols from a string symList good_badPOIs IS OP d_inn d_out d_archive fname_inn - extract BAD POI symbols from a file symList_good_fixPOIs IS OP d_inn d_out d_archive fname_inn fname_trn - replace bad POI in a "symList good" file symList_good_badParens IS OP d_inn d_out fname_inn - generate fileList of unbalanced parens in a "symList good" file symList_good_fixParens IS OP d_inn d_out d_archive fname_inn fname_trn - fix parens in a "symList good" file symParen_file_bads IS OP pname_inn d_out - consolidate lines with line#'s, output in standard format sorter IS OP fname archive_move IS delete_temps IS archive_all IS append_all IS sort_andOr_move_all IS do_all IS archive_some IS append_some IS sort_andOr_move_some IS do_some IS sym_stdForm IS OP str - str to standard format for [symbols,expressions], mostly remove [spaces,tabs] sym_stdForm_file IS OP pname_inn d_out - consolidate lines with line#'s, output in standard format for symbols ********************** load [trans_table,transExpr_table] for [HFLN,Luca] sort_transParnLists IS - QNial sorts Parn lists, cannot do expression lists as those must be ordered! ********************* loaddefs link d_Qndfs 'symbols/symbols translate - Laskar.ndf' is_blankstring IS OP Str - towords1 IS OPERATION Str { Strings := EACH pass ( ` match Str cut Str ) } - clean_symbol_list IS OP directory filename_in filename_out - file_remove_extra_lineTypes IS OP pname_inn pname_out - removes lineTypes (single characters at start of line) trnInfo IS OP tblName - convert to "state-based" continuation on sequential lines strOfType_extract_symGenls IS OP strOfType tblType - produces a list of expressions in strOfType strOfType_translate_symGenl IS OP strOfType - [extract, translate] cart sym[Parn,Expr] strStdForm_extractTranslate_symGenl IS OP str_stdForm - dictionary translate-substitute string file_translate IS OP pname_stdForm_inn d_out - [extraction,translation] of [syms,exprs] in Lucas f_derivation file file_translate_debug IS OP pname_inn d_out - generate output of [sym_stdForm, symExtract, symTranslate] to help fix code ********************* loaddefs link d_Qndfs 'Thunderbird - delete binary attachments.ndf' - Month_convert IS OP Month_3name - Process_date IS OP Lin - is_Odd IS OP a { (a mod 2) = 1 } - is_blankLin IS OP Str - isempty_Lin IS OP Lin - clean_text IS OP texter - clean_filename IS OP texter - clean_pathname IS OP texter - get_Fromer IS OP Lin - readBil IS - Process_timestamp IS - Find_attachName IS OP stringer - dash_skip IS - is_attach_hdr IS OP Lin - build_multiLine IS - From_process IS OP Lin - dash_process IS OP Lin - Attach_del IS OP target_foldar - unfold_dirs IS OP path - emailer IS OP target_folders - clen_filenames IS OP target_folder - save_filenames IS OP target_folder - fix_filenames IS OP target_folders - save_dir IS OP target_folder - store_files IS OP target_folders - ********************* loaddefs link d_Qndfs 'Thunderbird email fix.ndf' - second_more IS OP a b { (second a) >= (second b) } - analyze IS OP fin_name fot_name - modify IS OP fin_name fot_name analysis no_print_list - ********************* loaddefs link d_Qndfs 'txtFile compare.ndf' is_similar_wordList IS OP keys_old keys_new - paras_Equal IS OP pname_same pname_indx pname_semi pname_diff - fTxt_get_paras IS OP pname - setup_Wheatland_Fire IS - ********************* loaddefs link d_Qndfs 'txtFile - replace coded segments.ndf' - fTxt_replace_coded IS OP p_inn p_repl code_strt code_end - fTxt_replace_from_fList IS OP p_list p_repl code_strt code_end - update_QNial_setup_description IS - uqsd IS update_QNial_setup_description - ********************* loaddefs link d_Qndfs 'type_check.ndf' - Routines for implementing checking of data types islist IS OP data { IF tally TWIG shape data = 1 THEN l ELSE o ENDIF } - type_check IS OP object_type data - ********************* loaddefs link d_Qndfs 'USB workspace loads.ndf' screen_setup IS OP computer orientation - start_Home IS - start_ToDos IS - start_Neural IS - start_Suspicious IS - start_Jobs IS - start_QNial IS - start_Stalin IS - start_Lucas IS - DesktopSetup IS - desktop_switch IS OP desktopNum - start IS OP projectPhrase DesktopNum - start_std IS - ********************* loaddefs link d_Qndfs '[video, audio] files [crop, convert, blend, synchronize].ndf' video_play IS OP path winPosSiz - play a video file video_crop IS OP pinn pout t_start t_end - crop a video file videoAudio_synchronize IS OP mediaSpecsList - play [audio, video] files in sync videoAudio_cropConvertBlend IS OP mediaSpecsList - [crop, convert, blend] [audio, video] files ********************* loaddefs link d_Qndfs 'video production/video conversion/0_vidConv - common data, operators.ndf' +-----+ Time-related data & operators sec_from_timestamp IS - obvious function, depends on initial date basis sec_from_hhmmss IS OP time_fmt - calculate seconds from reference start time of hhmmss time_from_secs IS OP secs - converts seconds to hhmmss t_fnameFmt IS OP t_colonFmt - splits time provide as string 'hh:mm:ss' into [hh, mm, ss] +-----+ conversion of snippets to ogg file format, high quality audioVideo_crop IS OP pinn pout hhmmss_start hhmmss_end - crop [audio, video] audioVideo_convert IS OP audioVideo_convert IS OP pinn pout - convert [audio, video] codecs videoOnly_convert IS OP pinn pout hhmmss_start hhmmss_end - convert [audio, video] codecs audioOnly_convert IS OP pinn pout a_codec - convert [audio, video] codecs audioVideo_blend IS OP p_videoIn p_audioIn d_videoOut ts_video ts_audio fps t_end - blend [video, audio] streams into ogg file format dir_2_text IS - simply generate a listing of the paths of temporary video snippets (ACM video) +-----+ video concatenation vidList_cat IS OP dCat_name vid_list fot_name - catenate videos in the order of their list appearance vidCat_retime IS OP pCat_name - reset the timeCode in a video (doesn't work?) test_Cat IS OP pCat_name - returns run time of video catenation, as a test ********************* loaddefs link d_Qndfs 'video production/video conversion/0_vidConv - common data, operators.ndf' +-----+ Time-related data & operators sec_from_timestamp IS - obvious function, depends on initial date basis sec_from_hhmmss IS OP time_fmt - calculate seconds from reference start time of hhmmss time_from_secs IS OP secs - converts seconds to hhmmss t_fnameFmt IS OP t_colonFmt - splits time provide as string 'hh:mm:ss' into [hh, mm, ss] +-----+ conversion of snippets to ogg file format, high quality audioVideo_convert IS OP p_videoIn d_videoOut fps ts_start t_end - convert [audio, video] codecs audioVideo_blend IS OP p_videoIn p_audioIn d_videoOut ts_video ts_audio fps t_end - blend [video, audio] streams into ogg file format dir_2_text IS - simply generate a listing of the paths of temporary video snippets (ACM video) +-----+ video concatenation vidList_cat IS OP dCat_name vid_list fot_name - catenate videos in the order of their list appearance vidCat_retime IS OP pCat_name - reset the timeCode in a video (doesn't work?) test_Cat IS OP pCat_name - returns run time of video catenation, as a test ********************* loaddefs link d_Qndfs 'video production/video production/0_vidProdn - common data, operators.ndf' NOTE : many operators have been superceded in d_Qndfs, but they are retained here to avoid extensive re-writes +-----+ window current re-[size, position], background [start,kill] set_wPosnSize_now IS OP wPosn wSize - set window [size, position] for current operations background_start IS - screen-show background, typically at start of video production background_kill IS - kill the background window, typically at end of video production +-----+ Image list operators imageList_add IS OP ID_imRef wPosn_label fname - imageList_remove IS OP ID_imRef - remove an image from the imageList imageList_takeRght IS OP takeRght_num - imageList_kill IS - reset (null) the entire image list image_raiseFocus IS OP ID_image wPosn_label fname - bring an image into view +-----+ Kill [application,window] media_kill IS OP winID - generally kill a window, typically a video window gedit_kill IS OP winID - kill (close) a gedit window (text file) image_kill IS OP winID - kill (close) an image window +-----+ Heading operators head_hide IS - hide, rather than close, the heading to the current scene heading_post IS OP header wait_hhmmss - post the heaing of a [video, scene] data_headings := - headings for Big Data video, as an example head_test IS - headings_test IS - +-----+ reference operators - gives source references for [data, image, video, audio, music, etc] refs_hide IS OP wPosn_label - hides, rather than closes, references. This avoids delay of re-opening reference_post IS OP imagePosn_label refernc - post a refernce to the reference-window data_refs := - refernces for Big Data video, as an example refs_test IS - test a reference setup +-----+ image list operators [start,next] image_start IS OP args - uses an image_pgm to open an image file and [position, size] it image_next IS OP args - uses image viewer program "next image" (rather than "new image") +-----+ audio related - start [audio, music, voice] files, different [volume, directory]s audio_start IS OP typer sub_path fname volume_pct duration - start an audio file music_start IS OP args - start a music file voice_start IS OP args - start a voice file +-----+ video video_start IS OP args - start a video ********************* loaddefs link d_Qndfs 'video production/video production/0_vidProdn - common data, operators.ndf' NOTE : many operators have been superceded in d_Qndfs, but they are retained here to avoid extensive re-writes +-----+ window current re-[size, position], background [start,kill] set_wPosnSize_now IS OP wPosn wSize - set window [size, position] for current operations background_start IS - screen-show background, typically at start of video production background_kill IS - kill the background window, typically at end of video production +-----+ Image list operators imageList_add IS OP ID_imRef wPosn_label fname - imageList_remove IS OP ID_imRef - remove an image from the imageList imageList_takeRght IS OP takeRght_num - imageList_kill IS - reset (null) the entire image list image_raiseFocus IS OP ID_image wPosn_label fname - bring an image into view +-----+ Kill [application,window] media_kill IS OP winID - generally kill a window, typically a video window gedit_kill IS OP winID - kill (close) a gedit window (text file) image_kill IS OP winID - kill (close) an image window +-----+ Heading operators head_hide IS - hide, rather than close, the heading to the current scene heading_post IS OP header wait_hhmmss - post the heaing of a [video, scene] data_headings := - headings for Big Data video, as an example head_test IS - headings_test IS - +-----+ reference operators - gives source references for [data, image, video, audio, music, etc] refs_hide IS OP wPosn_label - hides, rather than closes, references. This avoids delay of re-opening reference_post IS OP imagePosn_label refernc - post a refernce to the reference-window data_refs := - refernces for Big Data video, as an example refs_test IS - test a reference setup +-----+ image list operators [start,next] image_start IS OP args - uses an image_pgm to open an image file and [position, size] it image_next IS OP args - uses image viewer program "next image" (rather than "new image") +-----+ audio related - start [audio, music, voice] files, different [volume, directory]s audio_start IS OP typer sub_path fname volume_pct duration - start an audio file music_start IS OP args - start a music file voice_start IS OP args - start a voice file +-----+ video video_start IS OP args - start a video ********************* loaddefs link d_Qndfs 'virus - clamscan output filtering of viruses found.ndf' clamscan_filter IS OP filepath - ********************* loaddefs link d_Qndfs 'vlc_kill.ndf vlc_kill IS - kill vlc after playing video (close isn't enough) ********************* loaddefs link d_Qndfs 'Website conversions from old formats.ndf' +-----+ Change web-pages' Howell's embeddedExecutables from old to [new, more powerful] version webPage_convertEncoding IS OP webPage - change [file-insert, www.BillHowell, .../, menuHeadFoots] webPage_mailtoFix IS - put back %20 instead of space in mailtos webPage_convert IS OP flag_overwrite webPage webSite - converts Howell's "embedded encoding" webSite_convert IS OP convertOptr - converts Howell's "embedded encoding" and fixes mailtos webPage_convertBodyLinks IS OP webPage webSite - change links in the body of html to a relative format webSite_convertBodyLinks IS - update working webSite (d_webRaw) to "uploadable" version ********************* loaddefs link d_Qndfs 'webSite header.ndf' +-----+ Global variables +-----+ Find all d_webRawe html files related to webSite [URLs, convert, update] webSite_extract_pathsSubDirsFnames IS - all relevant webSite paths to p_[all, html]FileList sorted [fname, all] lists # 29Oct2020 - there are 2729 relevant files (except 15 ^[.]files, 1 ^[ files I can't easily grep-out) excluding "Conference guides" webSite_readpathsSubDirsFnames IS - read stable [path, dir] lists ********************* loaddefs link d_Qndfs 'webSite maintain [menu, header, footer, body] links, TableOfContents.ndf' webSite maintain [menu, header, footer, body] links, TableOfContents +-----+ Change links in the body of an html file to a relative format, exclude 'http' '#' 'mailto:' midIndxsLines_ignoreBads IS OP webPage webSite - remove "bad links" that should not be processed internalLinks_return_relativePath IS OP backtrack strLeft strRight line - convert links to relative +-----+ Update webPages[Raws, Sites, All] webPageRawe_update IS OP flag_backup webPage - update pinn to d_out using executeEmbeds Although built for webSite maintenance, with adaptation this has far more general applicability. webPageSite_update IS OP flag_backup webPage - executeEmbeds of webPageRawe, then update webPageSite Although built for webSite maintenance, with adaptation this has far more general applicability. webAllRawOrSite_update IS OP flag_backup optr_rawOrSite - update webSite to "uploadable" version +-----+ URLs - check and count, helps for debugging webRawLinks_remove_pct20 IS - generate htmlPathLists webURLs_extract IS - extract all link urls from a website [external, internal, menu, pagePosn] this create d_temp files, to be further processed by urls_check urls_check IS OP linkType - create sublists of [internal,xternal] links classed as [fail, OK] check internal with path_exists '-f', externals with curl webSite_link_counts IS - summarize the counts for links [external, internal, menu, tableOfContent]s +-----+ Quick [searches, fixes] for bad webSite links - example [QNial-bash] fixes # previous [subDir, ` problems should now be OK, although sometime in the distant future I have to write link-checks for `# links backup files!! - important safety measure during code development pathList_backupTo_dir htmlPathsSortedByPath (link d_webRawe 'z_Archive/') restore files from backups - important safety measure during code development dirBackup_restoreTo_paths l (link d_webRawe 'z_Archive/201117 17h00m21s backups/') htmlPathsSortedByPath update a webPageRawe after editing - webPageRawe_update & webPageSite_update [find, grep] specific str find specific '!!linkError!!' search allFnamesSortedByFname for an fname [find, grep, sed] to replace a specific str, example : !!linkError!! change strOld to strNew in pathList, for strPattern, automatic path backups to d_backup str_replaceIn_pathList l d_webRawe (link '[#!: path_insertIn_fHand' chr_tab chr_tab chr_tab chr_tab chr_tab '(link d_webWork ' chr_apo 'confHead.html' chr_apo ')' chr_tab 'fout ;') (link '[#!: path_insertIn_fHand (link d_webWork ' chr_apo 'fin Head_one.html' chr_apo ') fout ; ' chr_newline ' Howell : [#=; fname ;= ' chr_newline '[#!: path_insertIn_fHand' chr_tab chr_tab chr_tab chr_tab chr_tab '(link d_webWork ' chr_apo 'fin Head_two.html' chr_apo ') fout ; ') htmlPathsSortedByPath str_replaceIn_pathList o d_webRawe (link '[#!: path_insertIn_fHand' chr_tab chr_tab chr_tab chr_tab chr_tab '(link d_webWork ' chr_apo 'confHead.html' chr_apo ')' chr_tab 'fout ;') (link '[#!: path_insertIn_fHand '(link d_webWork ' chr_apo 'fin Head_one.html' chr_apo ') fout ; ' chr_newline ' Howell : [#=; fname ;= ' chr_newline '[#!: path_insertIn_fHand' chr_tab chr_tab chr_tab chr_tab chr_tab '(link d_webWork ' chr_apo 'fin Head_two.html' chr_apo ') fout ; ') (solitary (link d_webRawe 'Neural nets/Conference guides/Author guide website/Author guide.html')) test for path existance, example failure geany text editor : regular expressions [search, replace] for quick editing of text files these aren't specific to this .ndf file, so they can also be handy examples for your other work normalGuide_header IS OP flag_backup webPage - example of special edits, in this case, webPage headers adds 'normalStatus.html' to end of top-of-file executableEmbeds confGuide_header IS OP flag_backup webPage - example of special edits, in this case, webPage headers +-----+ Run the whole shebang except manual [edits, fixes, etc] webSite_doAll IS - do all webSite[update, url] optrs ********************* loaddefs link d_Qndfs 'Website updates.ndf' +-----+ Change links in the body of an html file to a relative format, exclude 'http' '#' 'mailto:' midIndxsLines_ignoreBads IS OP webPage webSite - remove "bad links" that should not be processed internalLinks_return_backupSubDirFnames IS OP strLeft strRight line - returns backupFnameSubDir for an an internal link, where SubDir goes into d_webSite webPage_convertBodyLinks IS OP webPage webSite - change links in the body of html to a relative format webSite_convertBodyLinks IS - update working webSite (d_webRaw) to "uploadable" version +-----+ Change web-pages' Howell's embeddedExecutables from old to [new, more powerful] version webPage_convertEncoding IS OP webPage - change [file-insert, www.BillHowell, .../, menuHeadFoots] webPage_mailtoFix IS - put back %20 instead of space in mailtos webPage_convert IS OP flag_overwrite webPage webSite - converts Howell's "embedded encoding" webSite_convert IS OP convertOptr - converts Howell's "embedded encoding" and fixes mailtos +-----+ Update web-[pages, site] internalLinks_return_relativePath IS OP strLeft strRight line - convert links to relative also using executeEmbeds when present % remove [#= = brackets ; str_executeEmbeds IS OP line fout backtrack - execute embeds in line '[#!: ' is for full-line embeds; '[#=; ' is for embeds within a line % remove [#= = brackets ; pinn_executeEmbedsTo_pout IS OP flag_test pinn pout d_inn d_out - update pinn to d_out using Although built for webSite maintenance, with adaptation this has far more general applicability. webSite_update IS OP webSite - update working webSite (d_webRaw) to "uploadable" version (d_webSite) ********************* loaddefs link d_Qndfs 'windows - header.ndf' ********************* loaddefs link d_Qndfs 'windows [open, close, ID, title], [get,set,move] cart [posn,size].ndf' window [open, close, ID, title] : winSpecs_allWindows IS - winIDorTitle_get_winID IS OP winIDorTitle - returns a list of winIDs having the name requested winIDorTitle_get_winTitle IS OP winIDorTitle - cannot have multiple windows with a given winIDorTitle!! winIDorTitle_isDead IS OP winIDorTitle - winID_delay_lastStarted IS OP seconders - time delay to make sure most recent window opening is captured winIDorTitle_getAll_winIDs IS OP winIDorTitle - returns a list (NOT a string!) winIDorTitle_close IS OP winIDorTitle - winIDorTitle_activate IS OP winIDorTitle - check window activation from Qnial winIDTitle_change IS OP winID winTitle - (doesn't work) winTitle_change IS OP winID winTitle - old Code? window [get,set,move] cart [posn,size] : winIDorTitle_get_winPosnSize IS OP winIDorTitle - winIDorTitle_siz_mov IS OP winIDorTitle winPosn winSize - winIDorTitle_raiseFocus IS OP winIDorTitle - ********************* loaddefs link d_Qndfs 'windows system for startup.ndf' Coding : fname_for_title IS OP fname - winSizeMove IS OP appType winID winPosn winSize wait_hhmmss - (wmctrl) get_winConfig IS OP appLabel - window configurations, default general setup appOpen IS OP app d_name f_name fileOpen IS OP appOpen d_name f_name - not yet defined ********************* loaddefs link d_Qndfs 'workFlow loop.ndf' workFlow_status IS OP workFlowStr - update the status of all workFlow_steps (like make utilitiy, kind-of) dirFiles_output IS OP workFlowStr workFlowCol - add quotes etc for write, retain symbol for dir workFlow_loop IS OP workFlowPhr - hybrid [human, machine] workFlow assists with [complex projects, forgetting]