Howell's extensions of the Q'Nial programming language
21Mar2024 initial points, this webPage will evolve over [month, year]s...
Finally, after starting with QNial in 1985, I've created a webPage for my most important programming tool, Q'Nial (QNial - Nested Interactive Array Language) of Queen's University in Kingston, Ontario, Canada. Created by Michael Jenkins of Queens and Trenchard More of of
the IBM Scientific Center, Cambridge
I like to describe QNial as the beautiful child of a marriage between the LISP and APL programming languages.
Although my [program, library]s for QNial, as well as [notes, descriptions, etc] of my work have long been on my webSite, these are working text files for myself, and are less well suited descriptions for others that lack easy linking to different [directory, file, program (.ndf)]s. Hence this webPage.
Table of Contents
Major extensions of QNial
Lays (layouts) versus types: [non(Ary), itm, sub]s
lays (layouts) versus types: [non(Ary), itm, sub]s
saveData : like loaddefs for variables, text (unlike save workspace)
[type, layout] checks on-demand only, fault handling
These and many others are my own creations related to "The computability of symbol names" concept.
As a strict definition, by "type" I am often refering to non-nested)arrays built of a single (homogeneous) atomic type (see a listing of atomic types above).
These homogenous atomic arrays are often important in some applications. Types are also very important because they are the lowest data, which guarantee that the array, at that point, has no further "nested" data. This is a stopping point for many array operators. Even then, [string, phrase]s may represent [operator, data] via the [execute, apply] operators, but that does not affect the initial "deep" analysis of an array. For an example of a recursive operator delving into an array, see nry_itm_get_ndxL, which uses nry_itm_ndxInn_get_ndxL_recurse in arrays nested.ndf.
Closely related to types are "layouts" (lay), for [non-homogenous, nested] arrays. ???...I need to add a description and links here...???
[search, pattern, morph]s
symbol acronyms
Within the context of the "Computability of symbols names", I've tried to use a [consistent, short] set of acronyms for sysmbol names, which helps to keep the already-long [operator, vartiable] names to a manageable length (reduce typing on my part, and make it easier to aling program coding). These are described in QNial userWkspSymbol header.ndf, which is greatly out-of-date as of 23Mar2024. No doubt, a great deal of [use, memorisation] will be need to others who may use my libraries, so that they can understand what is being done by operators.
Operators to help make changes to my code towards using the symbol acronyms, are provided in QNial userWkspSymbol.ndf, which is again way-out-of-date.
fractal trees
Not created yet... but it's a priority for neural dendritic trees...
hexagonal arrays (honeycombs) in QNial
Not created yet... but important for many natural systems...
computability of symbol names
An obvious play on Alan Turing's paper "The computatility of numbers", this concept has been slowly under development since ?2008?.
Programming principles
Common [data, operator]s with arbitrary dimensionality
QNial userWkspSymbol [philosophy, grammar, design].txt
QNial userWkspSymbol [philos, grammar, design].txt
QNial-bash hybridization
Although I've long used simple bash capabilities, it took years before I was knowledgable enough to learn [bash, regular expressions, [quote, apostophe] issues, etc] to effectively hybridize QNial and [bash, Unix utilities].
- After long experience I learned that it was not worth the effort to create relatively primitive QNial operators that could never approach the [power, speed] of bash for what it does well, especially with the huge number of options to contend with.
- More generally, in order to use other [language, capability]s, I always ran into to the concrete wall that to get the capabilities of QNial, I would have to re-create QNial in my own code (eg in the C programming language), which would be insane.
- This does mean that the [slow, interpreted] language of QNial is slow, but host calls to programs can largely mitigate that, and it saves me enormous [time, work] when compared to other languages, which generally lack QNial's power.
What really drove my advances was the 2-4 years (2021-2024) when I was constantly re-writing my QNial libraries to implement a more [robust, reliable] set of concepts (see Major extensions of QNial above) as a basis of my current approach to my QNial [symbol name, layout, library, etc]s. During this time, all my QNial libraries were in limbo, so I did extensive bash programming of my [webSite management, file operator, etc] systems to continue working.
pipes, computing without symbols, [object, check, ]s when desired
Unix regular expressions for [alternative, hybrid] processing
Unix-like function options?
Abstracted data, @@pth is the ID
Abstracted data to easily search structures, after they are confirmed by nry_itm_get search.
List of abstract datPhr, and their @@pth for easy reference.
Scope up&down by depth - easy, chop or add to @@pth
KISS - Keep It Simple Stupid
- reduce number of operators, sacrifice [speed, efficiency, memory],
use C etc for very large, slow optrs
23Jun2022 example 'arrays.ndf' I had created 'aryShp' optrs, but this would have required EITHER :
- 2 optrs for one
- complexity of [naming, conditionals, checks, twice as many tests]
- optr apply at as high a level as practical
- aryTop, reduced #optrs, easier to [remember, find], higher level think
- don't type-check at base level, use special optrs for that, greatly [simplify, speed] code
Pragmatic interoperability
QNial, bash, host calls, parallel computing
Tables [QNial, html, csv (spreadsheet)]
I still have a lot of detailed work to do on the very important data issue of table conversions, especially with respect to retaining formating like [font, [text, number] alignments, background colors, etc].
For an example converting QNial tables to html, see the section below QNial workspace, QNial versus Howell [variable, operator, etc]s, for which QNial tables were automatically converted to html format by tblQNial_put_tblHtmOut in the file "$d_Qndfs"'arrays atomic.ndf'.
For csv files from a spreadsheet program, QNial operators are used to convert between [QNial ary, csv file] formats. These operators aren't complete, but are "workable" if you are careful with [apostrophes (apo), quotes] which differ in function for [QNial, csv]. These operators are in the file file_ops.ndf, and use the operators :
- pinn_read_csvTable - read a csv table from file, cells to [apo-str, numeric]
- tbl_writeCSVformatTo_pnew - writefile table to pnew, [tab-separated cols, apo-text]
I have not yet created tests for the above, as for now it's not a priority (it works well enough for me, albeit that's not an excuse).
External webSite scraping
An example of pulling information from a website is winURL yahoo finance news download.ndf. I used this to extract news itmes from Yahoo financial news to provide a basis for analysing markets. It was too much work to maintain, and is probably out of date by now in terms of my changes to all of my libraries since I used it, but it would be easy to [update, extend].
???
MindCode - my long-time-priority neural network project
My priority "hobby" research area since 1988 has been neural networks, with "MindCode" being a persistent, but rarely active, concept of interest. This is described in a collection of themes which I've expanded since ~2022 :
While there is a smattering of QNial coding for various parts of MindCode and releted projects, this is still in its infancy, even if I've had this as a priority-I-don't-work-on-much since the late 1990's - early 2000's.
Somewhat related are the themes :
-
Consciousness - I jumped into this with the excitement surrounding chatGPT post Nov2022 when it as available publicly for free, and I noticed a striking resemblance of the basic Transformer NN architecture to Grossberg's concepts going back at least to ~1970's or 1980'ds.
Given Stephen Grossberg's impressive neural network research [principle, concept]s foundations and far [broad, deep]er non-conscious concepts, this is the only context for consciousness with which I am comfortable. Unlike any other theory that I am aware of, Grossberg's EMERGED from his extremely impressive long-term research on [neuron, [micro, macro]-circuit, psychology, neuroscience] concepts! Grossberg's concepts of consciousness are summarized and compared to other consciousness concepts in the table of What is consciousness: from historical to Grossberg and in [definitions, models] of consciousness.
The only other framework that I like is John Taylor's consciousness, which I see as paritally emergent from Paul Werbos's "advanced control theory" and brain concepts.
However, I've long been less than impressed by much of the other work in the area of consciousness. There are many great ideas by others, but it's the old 1 in 10,000 rule, whereby you have to wade through a lot of [yap, arm waving], and I never can recognize what will ultimately be important. Furthermore, none of it seems to be in a [serious, substantive] framework where data takes precedence over [poetry, flights of fuzzy imagination] (like science fiction). Consciousness also seems to be the intellectual graveyard of many Nobel Laureates (eg [Francis Crick, Gerald Edelmann, ???, and now Roger Penrose]).
Ultimately, I am far more interested in working with Grossberg's concepts in general, rather than just consciousness itself.
-
Transformer Neural Networks - This is the basis of today's Large Language Models (LLM) such as [OpenAi's chatGPT, Google's [Lambda, Bard], Elon Musk's Claude AI, etc]. Vaswani etal of Google produced the foundational "Attention is all you need" paper.
Handy bash tools - for independent use, or called from QNial
As per the section QNial-bash hybridization above, my QNial programming is hybridized integrally with my bash libraries, so when downloading QNial, users must also download by bash (bin) directory. Here is a quick overview of the bash side.
My bash libraries are an incomplete port of my QNial libraries, and although this porting is incomplete, there is afar too much material to describe here. So I mention only a few item below.
- fileops TblOfContents.txt -
this is the largest bash library I have created, and is critical to have [consistent, robust, safe, archived] capabilities.
- linux.sh - by typing "linux" in a terminal,
a host of very handy files listing bash [code, usage] examples are provided. I use this all of the time, as it's impossible for me to remember a huge number of details.
- strings TblOfContents.txt
- given QNial's power with arrays, and the bash fileops library, most string functions are omitted here. This library is mostly for date functions for other bash scripts.
-
webSite TblOfContents library.txt
A more extensive, albeit brief, listing of functions can be seen in my bash directory. Scripts for [audio, encryption, gnuplot, image, video, virus, wget, etc] can be seen in that directory.
Notable sub-directories include :
- backupper - backups to USB hard drives etc
- email scripts - various capabilities
- file conversion utilities
- notably [catdoc - docToText in a directory.sh, pdfToText directory.sh] (other capabilities in fileops directory etc)
- starter - setup of my [workspace, file&apps]s
when booting my computer
- video production
- this is a big topic, having build my own [simple, crude] video production system
Some [fun, simple, handy] capabilities are :
- webPage table of contents
- creates a table of contents for a webPage, with each level indented as required. It is very easy to update a table as work progresses. See the section "htmlHeadings_to_TblOfContents - examples" in the file, which makes use of file_ops.sh as listed above.
Other junk
QNial data [type, layout]s
Types (typ) - There are 6 atomic types in the current QNial version (I didn't compile QNial with complex numbers) :
- boolean [o,l]=[false, true]
- character - strings are merely lists of characters, but they are treated specially
in QNial for output display, and many primitive operators make them easy to use.
- fault -
- integer
- phrase
- real
Layouts (lay) are structures of :
- atomic data types - homogeneous (all same atomic type)
- [bol, chr, flt, int, phr, rel]*[,L,LL,LLL,LA,A,N]
- (boolean, character, fault, integer, phrase, real)
- (basic, Lst, LstLst,LstLstLst, LstAry, Ary, Nst)
- data lays (layouts) - heterogeneous ([atomic type, lay]s can differ - classed by highest lay)
- isLay[ary, itm, lst, mor, nry, nlc, pat, slc, sub]*[,L,LL,LLL,LA,A,N]
- (atomic array, item, morph, nested array, nested slice, pattern, slice, sub-array)
- non-ary : [null, single, verBar, ???]
nry - nested arrays in an interpreted environment :
- are always of "rectangular" shape above the atomic level
- as the language is interpreted (no compiler), hence it's name :
Q'Nial : Queens Nested Interactive Array Language
where "Queen's" refers to Queen's university of Kingston, Ontario, Canada
Atomic versus other data lays : the Grand Divide
A constant problem that I've had over the years is the failure of my operators due to tripping over the easy use of [int, real, phrase]*[atm, lst, ary, nry]. Operators that worked very well for simple programs would make a mess when going to the next stage of complexity, often because atomic data [, lst]s are easy to implement, but trickey to extend.
Transformation of my QNial libraries since 2020 especially, has sought to properly account for the differences in implementation between atm and other "Layouts" (lays), which are as trickey as they are simple. Endless hours of fixes over the years have been un[necessary, fortunate].
Much of the work (to 26Mar2024) has gone into defining operators such as :
- isTyp[bol, chr, flt, int, phr, rel]*[,L,LL,LLL,LA,A,N]
(boolean, character, fault, integer, phrase, real)
(basic, Lst, LstLst,LstLstLst, LstAry, Ary, Nst)
- isLay[ary, itm, mor, nry, nlc, pat, slc, sub]*[,L,LL,LLL,LA,A,N]
(atomic array, item, morph, nested array, nested slice, pattern, slice, sub-array)
- non-ary : [null, single, verBar, ???]
- special operators such as :
- intL_to_str - developed for robust idxA processing
-
-
-
Key ".ndf" files are :
- types [atomic, nested].ndf -
- arrays [atomic, nested].ndf -
- and of course their related test files
-
I have only just begun...
[status, help] : see the menus at the top of each webPage
Unfortunately, the status information in particular is usually badly out of date (it takes time to update!).
Howell's QNial libraries
My core QNial libraries which are loaded on startup (listed in order of loaddef) :
- Howell_setup_ndfs :
-
QNial setup.ndf - define setup of Howell's QNial workspace, including core stuff
-
QNial userWkspSymbol.ndf - displays status of current workspace, lists symbol naming conventions and common acronyms, "The computability of symbol names" concept and the "The computability of symbol names" concept
- Howell_core_ndfs :
-
types atomic.ndf - atomic data [typ, lay, basic test]s
-
types nested.ndf - nested data [lay, basic test]s
-
faults.ndf - fault (error) operators, allow easy "flow-through"
-
boolean.ndf - Howell's operators [basic, use with [lst, ary, nry]
-
arrays atomic.ndf - operators for TOP level only (not nested!)
-
arrays nested.ndf - operators for nested arrays
-
lists.ndf - sorting of list lays
-
strings.ndf - special string-related ops (most ops are ary, nry])
-
file_ops.ndf - [safe, conservative] ops with [dir, fil]s
-
windows.ndf - way out of date, probably won't work so well after all the changes?
-
encryption.ndf - random code ops [alpha, numeric, alpha-num, full]
-
math - [quick, handy] stuff.ndf - a few handy ops
- Howell_core_headers, only a few are mentioned :
- Howell_core_tests provide a large set of tests across my QNial libraries that are used often during majpr changes to [identify, fix] problems. Here are some examples :
There are many other [dir, fil]s in my QNial directory. A [random, rare] selection of interesting ndf [dir, fil]s includes :
QNial workspace, QNial versus Howell [variable, operator, etc]s
The following information is produced at each QNial startup by the operator usr_WkspcSym_status in "$d_Qndfs"'QNial setup.ndf'. In this case, the information is from 21Mar2024, and includes ONLY the core libraries. I have perhaps an equivalent number of operators defined in non-core programs.
Status (words except #freeblocks):
|
free |
lgstblk |
#freeblks |
Wkspc |
stack |
atom |
internal |
|
31513709 |
31461922 |
1581 |
32000000 |
10000 |
10000 |
|
Loaddefed CORE symbol counts [Howell, system] :
|
|
total |
vars |
optrs |
exprns |
trans |
ident
|
|
total |
1255 |
338 |
747 |
88 |
33 |
1
|
|
system |
470 |
16 |
349 |
25 |
32 |
0
|
|
Howell |
785 |
322 |
398 |
63 |
1 |
1
|
The above table only includes QNial system symbols and the 13 CORE Howell library files :
- types, faults, boolean, arrays, lists, strings, file_ops
- QNial setup, QNial userWkspSymbol, windows, encryption, math - [quick, handy] stuff
- (usr_WkspcSym_status, tblSymbolCounts)
[noun, verb] CORE symbols by type
Not all types are shown.
|
\HowellSyms\ |
both |
noun |
verb
|
|
total |
147 |
136 |
11
|
|
noun_verb |
97 |
93 |
4
|
|
Adjctv_Adverb |
16 |
11 |
5
|
All CORE [adjective, adverb] counts, after culling.
Check against Adjctv_Adverb in table above. ...
|
count |
adjective |
adverb
|
|
totals |
11 |
5
|
QNial [download, install, environmental variables]
QNial community links
Official QNial documentation for [reference, beginner]s
I have NOT provided descriptions of the standard QNial capabilities etc, as these are well described (better than I could) in the official documentation :
I should do my own version of these, but this webPage and the context-sensitive webPage menus are probably as far as I will get?