"$d_PROJECTS""2020 WCCI Glasgow/reviews - mine/'/home/bill/PROJECTS/2020 WCCI Glasgow/reviews - mine/20867 r Chevtchenko, Ludermir - Learning from Sparse and Delayed Rewards with a Multilayer Spiking Neural Network.txt'" # www.BillHowell.ca 09Feb2020 updated from a long series of iterations over the years To automatically move comments from the "DETAILS" section to other sections, see # loaddefs '/media/bill/PROJECTS/Qnial/MY_NDFS/review move comments.ndf' +-----+ IJCNN Paper Reviews (As per IEEE-CIS formatting style) IEEE-CIS system does NOT accept greek letters : α β γ δ ε ζ η θ ι κ λ μ ν ξ ο π ρ ς σ τ υ φ χ ψ ω EasyChair does, too. Copy my comment section to authors : From : review writeup To : /media/bill/HOWELL_BASE/Qnial/MY_NDFS/paper reviews - raw review.in Run QNial "clean-up" : paper reviews - strip out illegal characters.ndf /media/bill/HOWELL_BASE/Qnial/MY_NDFS/paper reviews - strip out illegal characters.ndf Paper : '20867 Chevtchenko, Ludermir - Learning from Sparse and Delayed Rewards with a Multilayer Spiking Neural Network' Assigned : '04Feb2020' Due : '23Feb2020' Scale : strong accept, Accept, Weak accept, Neutral, Weak reject, Reject, Strong reject Originality: - WA Significance of topic: - A Technical quality: - A Relevance to IJCNN : - SA Presentation: - SA Overall rating: - A Reviewer's expertise on the topic: Low Most suitable form of presentation: Any Best Paper Award nomination: >> No Please provide a brief and compelling argument supporting (a) your recommendations and (b) the above ratings: >> See my comments to the authors. ********************************************** This reviewer's personal approach, nomenclature examples: p1c1h0.8 = means page 1, column 1 80% of the way down the page (very approximately) C2. = means Comment section number 2 WEAKNESSES (note that actions by the authors are NOT required for the points) '~#^&|–' = '' '' '' '' '' '-' (these are illegal characters for IEEE review submissions) view this file in constant width font (eg courier 10) and tab = 3 spaces ++---------------------------++ ACTIONS REQUESTED OF THE AUTHORS I have no critical changes to request of the authors. RRR NOTE: The points above are the ONLY points (there are NONE!) that I request that the authors address. From here to the end of the review, there is no requirement for the authors to make any changes to the paper, nor is there a need to respond to me about those points. I provide comments that they may consider at their own discretion, and it is NOT my intention that any of the points below have to be addressed (other than those listed above). I am very afraid that authors feel that they are obliged to answer or make changes, which could waste far too much of their time, and several of my comments may be speculative and may not even be correct! *************************** *************************** COMMENTS ONLY - actions by the authors are NOT required for the points listed below, to the end of the review. Perhaps some of these comments will be helpful in some way. (Main paper contributions, positive aspects, observed deficiencies, and suggestions on how to improve them:) PERTINENCE of the paper for IJCNN2020, Glasgow : As per the "Call for Papers" listed at http://www.ijcnn.org/call-for-papers : "... ??? ..." The current paper fit the conference topics very well. SUMMARY COMMENTS III ++---------------------------++ C1. STRENGTHS OF THE PAPER: SSS ++---------------------------++ C2. WEAKNESSES: (again, changes to the paper are not require for these comments) WWW ++---------------------------++ C3. QUESTIONS and COMMENTS: (no need to answer) QQQ +-----+ COMMENTS: CCC ++---------------------------++ C4. DETAILS and GRAMMAR: (again, changes to the paper are not required for these suggestions) This paper is well written and is easily understood, so I have very few suggestions to make. These tend to be stylistic, and may not be better than the authors' original version, so they should only be considered if to the authors' liking. ccc p1c1h0.85 "... First, while animals can quickly adapt and learn new behaviors, training an artificial neural network (ANNs) is time-consuming and typically requires a large collection of examples. ..." >> I've long thought that a major missing component of ALL ANNs is a basisc of a [arge, robust, flexible] base of [DNA, epigentic] evolved [data, functions, ...]. sss p1c1h0.9 "... For instance, while the human brain consumes about 20 watts, the Human Brain Project’s simulation of the cortex is expected to consume 500 MW, which is roughly equivalent to 250 thousand households [5], [6]. ..." p1c2h0.45 "... Instead of computing a global gradient over all of the synapses, SNNs can employ a biologically plausible process called spike-timing-dependent plasticity (STDP). This synaptic plasticity rule only requires each synapse to be aware of corresponding pre- and post- synaptic neurons. ..." >> Great examples that put solid justification for the paper objectives. p2c2h0.9 "... from each of ns sensors in a one-hot configuration. ..." >> one-hot not one-shot? Ok, makes sense. But how is that different from the old term "Winner Take All" (WTA)? qqq p3c1h0.6 Eqns (1), (2) >> Howell : how do these compare to other work? looks conventional, but what am I missing? qqq p3c1h0.95 What is the presynaptic trace "P_"? >> Howell - link this to Zj,i >> Ahh see Fig 3, Eqn (4) - nice illustration qqq p3c2h0.55 "... In other words, input and hidden layers are sparsely connected with Ni × (n+ + n− ) synapses, where n+ and n− are much smaller than Nh . ..." >> very interenting sparsity - what [advantages, challenges] does this present? qqq p3c2h0.6 "... In the experiments presented in this work the weights of these synapses are kept constant (+1 or -1) and are not subject to modulation by reward. ..." >> Nice - makes learning much [simpler, faster]. Here I assume that the fixed weights are betwen the [pre-synaptic, hidden] neurons via Eq (4), whereas the fixed +-1 weights are between the [hidden, post-synaptic] neurons? Will the authors comment on potential for more power from adjusting the weights between the [hidden, post-synaptic] neurons? ...or whether that wouldn't really help much? p3c2h1.0 "... The hidden layer and place neurons are fully connected with plastic synapses, following the update rule from Equation 4, with presynaptic trace P− resetting to −n+. ..." >> I was wrong - Eq (4) is from hidden to place neurons. www p3c2h0.8 "... In the proposed model there are Np neurons, which is much smaller than the number of all possible input states (nns ). ..." p3c2h0.85 change to "... An intuitive and experimentally supported [25] reasoning is that we expect an animal in a labyrinth to better remember a path that leads to successive rewards rather than other available routes. ..." >> It's not clear that the given architecture can remember a path? ccc p4c1h0.15 "... As in the place layer, in order to ensure a single action choice only the neuron with highest potential is allowed to spike at each time step. ..." >> Presumably this rules out "multiple conflicting hypothesis" and may force a "line of pursuit". sss p4c1h0.5 "... Over time, more distant choices also become increasingly likely as the corresponding synaptic weights are successively increased. ..." >> Good description. As is often the case, does this approach work backward from the solution? Normally and agent won't have the solution. But if that is not the case, is random search success the initial reward? But you have the answer then, so what use further iterations except for optimality? -> The authors confirm this in the next paragraph! qqq p4c1h0.65 "... The experiments described in Section IV use a constant 2% baseline. ..." >> This seems quite low? ccc p5c2h0.15 "... This way the target value Yt^Q is computed with an offline copy of the parameters θ t , which is updated every τu steps. The online neural network is trained through gradient descent using batches, sampled from a large memory bank of observed (st, at, r, st+1) transitions. ..." >> Now this approach is really starting to look like Approximate Dynamic Programming with the [actor-critics, gradient descent, NN representations], but without much of the optimality context and without a selection of the right type of ADP for a given problem (eg [HDP, DHP, ADHDP, etc]. Note the {Approximate, Adaptive] Dynamic Programming have also long been applied to discrete problems). However, there is a big advantage to simplicity that is enough for the type of problem being tackled, and getting closer to real-time performance. sss p7c2h0.5 V. Conclusions and Future Work >> Advantages of the authors' approach are well-described. "... For instance, similar states that do not require a distinct action could gradually converge on a single place neuron. ..." >> Nice, simple, probably effective! Makes sense. "... Additionally, evolutionary algorithms such as NEAT [33] have been shown to generate efficient spiking controllers for simple tasks [34]. ..." >> I also think that the neuro-evolution work has very interesting potential. pch0. change to "... ..." pch0. change to "... ..." pch0. change to "... ..." pch0. change to "... ..." ++---------------------------++ C5. REFERENCES (using a quick web search, as opposed to checks using Scopus or standard indexes.) I do not have access to CrossRefs "CrossCheck" via Elsevier's "iThenticate", but I assume that this is done by the editorial board. C5a) Are references and citations in the standard format? +-----+ IJCNN example (IEEE-CIS standard) : [ISNN example (IEEE-CIS standard) : [27] F. Akopyan, J. Sawada, A. Cassidy, R. Alvarez-Icaza, J. Arthur, P. Merolla, N. Imam, Y. Nakamura, P. Datta, G.-J. Nam et al., “Truenorth: Design and tool flow of a 65 mw 1 million neuron programmable neurosynaptic chip,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 34, no. 10, pp.1537–1557, 2015. ... - indicates italicized lettering - References in standard format >> seems OK. I am not sure of the exacts specification for the formatting of ISNN references, and authors seem to have slightly different styles. But the essential information is all there. - References should be sorted by numerical order (= order of citation) >> - Citations in the text are of the form "... [7] ..." (where 7 is the reference number) >> - Citations are numbered by order of appearance in the text. >> +-----+ C5b) Check a few reference details, and if they are legitimate (using a quick web search and personal familiarity with references)? >> By assuming that the editorial board uses iThenticate or tools like that, I no longer do this check. C5c) Is this paper significantly different from previous papers by the same authors? >> C5d) Is this paper novel with respect to the literature? >> In order to assess the novelty of the current paper I only did a superficial search for related papers, not an exhaustive search. Additionally, I do not have CrossCheck, which I assume is being used by the editors. nnn In order to assess the novelty of the current paper I only did a superficial search for related papers, not an exhaustive search. Additionally, I do not have CrossCheck, which I assume is being used by the editors. C5e) Is the relevant literature well represented in breadth and Depth? >> Yes. The authorrs provide both a history and very pertinent recent research related to their paper. It would have been nice to ALSO see the reference for the ?1981 temporal difference learning? paper by Sutton and Barto, which would show a key historical origin with respect to those authors. [1] R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT press, 2018. >> Oops - the authors do refer to Sutton's 1988 paper [28] later on p5c1h0.4. ************************** C6. MATH CHECKS - step-by-step +!!!!!!!!!!!!!!!+ WARNING : The following paragraphs are several different comments that may be applicable to the paper being reviewed. If you can see this message, then I forgot to customize the comments, and this sub-section should be ignored. I normally do detailed step-by-step derivations for one or more parts (Lemmas, Theorems, Corollaries) of a paper for my reviews, which gives me much greater confidence in a paper. Unfortunately, unexpected problems with a project and a deadline prevented me from putting the required time into this (>8 to 12 hours). However, there are no Theorem developments in this paper. OR Here are a very few step-by-step derivations that I had the time to check : OR Step-by-step derivations were done for ??[background, Theorems(1,2,.../Nt), Lemmas(1,2,.../Nl),Corollaries(1,2,.../Nc)], amounting very roughly ???% of the The details of my math check for this paper and be found at : This text document is a record of this reviewer's step-by-step check over parts of the paper. It is too lengthy for a read-through by the authors, but if they want to scrutinize specific parts of my checks, there are provided in detail. My [style,nomenclature] is distracting, but quick and easy to use for a review. IMPORTANT! For proper alignment of math expressions, view this file in a text editor with : - UNICODE characters (most modern text editors have this) - constant width font (eg Courier 10) - tabs of 3 spaces each - ensure that long text lines "wrap" to a new line. Reviewer names and the paper title have been removed from the text file, and the obscure posting of it on my zero-traffic website should provide the necessary privacy (I hope). If the authors want me to remove it after they have a chance to look at it, simply email me (my email address is in the file). !!!!!!!!!!!!!!!! MMM ************************** C7. LIMITATIONS OF THIS REVIEW Reviewer's expertise on the subject: Low NNN http://www.BillHowell.ca *************************** C8. THOUGHTS: (again, changes to the paper are not required for these - in fact changes SHOULD NOT be made!) Here are some long-winded thoughts that are not really relevant to the paper review per se... For interest only, even if that. These are separated from the "COMMENTS" above because they are less relevant to the actual paper. TTT *************************** CONFIDENTIAL COMMENTS for review chair / committee use only: None other than the list of Ratings enddoc