http://www.BillHowell.ca/Neural nets/Paper reviews/240208 conference paper review- math only.txt www.BillHowell.ca 08Feb2024 initial view this section in constant width font (eg Liberation Mono 10 point) and tab = 3 spaces This file consists of notes takern to force myself to go through the details of the paper. I have NOT [assessed, re-derived] any content, as I sometimes do with a small portion of theoretical developments in papers on other topics. #24************************24 # Table of Contents, generate with : # $ grep "^#]" "$d_web"'Neural nets/Paper reviews/240208 conference paper review- math only.txt' | sed "s|^#\]| |" # Nomenclature Gene examples (oldies but goodies) +-----+ interaction model : Dilated Inception: d dilated convolution of x with respect to the kernel k lncRNA attention : protein attention : attention matrix : MultiHead attention consists of h individual heads. MultiHead attention mechanism -> interaction features between [lncRNA, protein] separately final local feature maps: merge [attention, original] feature maps +-----+ global independent feature extractor 1) KmerEmbedding: 2) Deep Neural Nets (DNN) +-----+ C. Classification +-----+ Questions : useful (to me) notes from the authors' paper : +-----+ C8. THOUGHTS: Genetics and NNs Stephen Grossberg 2021 "Conscious Mind, Resonant Brain" 24************************24 Genetics notes 08********08 #] Nomenclature lncRNAs long non-coding RNA endogenous single-stranded polynucleotides with a sequence length >=200 nucleotides that does not encode proteins ncRNA non-coding RNA LPIGLAM LPI prediction based on [global, local] features of lncRNA and protein LPI lncRNA-protein interactions 08********08 #] Gene examples (oldies but goodies) p53 & H19 interplay has major roles in tumorigenesis and metastasis? H19 tumorigenesis, but also crucial to embryonic development one of the first discovered lncRNAs p53 tumor suppressor, represses the H19 gene are mutually counter-regulated : |-> P53 represses the H19 gene |-> H19-derived miR-675 inhibits p53 and p53-dependent protein expression HOTAIR HOX Transcript Antisense Intergenic RNA |-> PRC2 H3K27-methylation |-> LSD1 H3K4-demethylation [7] 08********08 #] +-----+ #] interaction model : +-----+ #] Dilated Inception: d dilated convolution of x with respect to the kernel k K = [k(−c), k(−c+1), ..., k(c)] 1-D convolutional kernel, size 2c + 1 L embed = [l1, l2, ..., ln] input lncRNA sequence of length n d dilated convolution of x with respect to the kernel k (1) (K ∗ Lembed)@t = sum[i=(-c) to c: k(i)·l(t−d·i)] where t ∈ {1, 2, ..., n} d dilation rate. qqq p3c2h0.67 CrossAttention module comprises both [protein, lncRNA] attention, sharing the same weights >> Why do [protein, lncRNA] attention modules share the same weights? +-----+ #] lncRNA attention : Lcnn ∈ R^(L'(L) × dcnn) lncRNA feature map Q(Lmi) ∈ R^(L'(L) × dhead) lncRNA query vector, from passing Lcnn through a linear layer Pcnn ∈ R^(L'(P) × dcnn) protein feature map K(L,i) ∈ R^(L'(P) × dhead) lncRNA key vector, using Pcnn V(L,i) ∈ R^(L'(P) × dhead) lncRNA value vector, using two additional linear layers (2) Q(L,i) = f(Q,i,Lcnn) lncRNA’s queries K(L,i) = f(K,i,Pcnn) lncRNA’s keys V(L,i) = f(V,i,Pcnn) lncRNA’s values where i = 1, 2, ..., h dhead = dcnn/h channel dimension h number of attention heads f = transform(w) * x w weights +-----+ #] protein attention : Pcnn ∈ R^(L'(P) × dcnn)? protein feature map Q(P,i) ∈ R^(L'(P) × dhead) protein query vector, from passing Lcnn through a linear layer K(P,i) ∈ R^(L'(P) × dhead) protein key vector, using Pcnn V(P,i) ∈ R^(L'(P) × dhead) protein value vector These three protein linear layers share weights with the previous three lncRNA linear layers (3) Q(P,i) = f(Q,i,Pcnn) protein’s queries K(P,i) = f(K,i,Lcnn) protein’s keys V(P,i) = f(V,i,Lcnn) protein’s values where f = transform(w) * x +-----+ #] attention matrix : After that, we compute the attention matrix using the softmax function. Then, we multiply each attention matrix with its corresponding value matrix to obtain attention feature maps for each attention head. These attention feature maps are then concatenated along the channel dimension and fed into a linear layer, resulting in the final attention feature map. (4) Attention (Q,K,V) = softmax(Q*K^T / sqrt(dk)) * V where dk dimension of queries and keys +-----+ #] MultiHead attention consists of h individual heads. (5) head(i) = Attention(Q(i), K(i), V(i)) MultiHead attention combines the outputs from all h heads as follows: (6) MultiHead{Q(1...h), K(1...h), V(1...h)} = Concat(head 1, ..., head h) * Wo where Wo ∈ R^(dcnn × dcnn) weight matrix shared for computing the interaction features between lncRNA and protein +-----+ #] MultiHead attention mechanism -> interaction features between [lncRNA, protein] separately (7) Lattention = MultiHead{Q(L,1...h), K(L,1...h), V(L,1...h)} (8) Pattention = MultiHead{Q(P,1...h), K(P,1...h), V(P,1...h)} +-----+ #] final local feature maps: merge [attention, original] feature maps The feature map is downsampled into a final one-dimensional vector representation by applying a maxpooling layer. (9) Llocal = Maxpooling((Lattention + Lcnn) / 2) (10) Plocal = Maxpooling((Pattention + Pcnn) / 2) +-----+ #] +-----+ #] global independent feature extractor #] 1) KmerEmbedding: We use k-mer features to encode lncRNA and protein to capture the global characteristics of sequences. The k-mer features transform variable-length sequences into fixed-length feature vectors. For lncRNA sequences we calculate the corresponding nucleotide frequencies (A, U, G, C) to fully extract features. Then, we take combinations of k = 1, 2, 3, and 4 to obtain a 340-dimensional feature vector. For proteins, based on dipole moment and side chain volume, we divide the 20 amino acids into 7 groups: {Ala, Gly, Vlal}, {Ile, Leu, Phe, Pro}, {Thr, Met, Tyr, Ser}, {His, Asn, Tpr, Gln}, {Arg, Lys}, {Glu, Asp}, {Cys} [27]. We take combinations of k = 1, 2, and 3 to calculate the frequency of protein sequences, resulting in a 399-dimensional feature vector. #] 2) Deep Neural Nets (DNN) As k-mer features already contain higher-level information, we utilize a simple deep neural network to extract features, employing LeakyReLU to prevent the vanishing gradient problem and dropout to address overfitting issues. (11) Lglobal = DNN(1, Lkmer) (12) Pglobal = DNN(2, Pkmer) where DNN 1 and DNN 2 are deep neural networks constructed by stacking multiple layers in the arrangement of dropout layer, fully connected layer, and LeakyReLU activation function. +-----+ #] +-----+ #] C. Classification In this study, lncRNA protein interaction prediction is modeled as a binary task. All the global and local features learned in the previous sections are concatenated and fused into downstream fully connected dense layers. (13) ŷ = δ ([Llocal; Plocal; Lglobal; Pglobal]*Wout + bout) where δ sigmoid activation function [W,b]out learnable parameters ŷ predicted interaction probability To ensure that the predicted interaction probabilities approx- imate the true interaction values, we utilize the binary cross- entropy loss function as our training objective. (14) loss = 1/N * sum[1=1 to N: -[y(i) · log(ŷ(i)) + (1 − y(i))*log (1 − ŷ(i))] where y(i) real label ŷ(i) predicted label N number of training samples 24************************24 #] +-----+ #] Questions : Look at [architecture, function, process, control,...] as well, not just protein coding? >> lncRNA techniques all do some of this, even if unbeknownst to their authors? Extract "program code snippets" to build a programming language from, look for : variable=const if-then-fi looping constructs [for, while]-done, symbolic assignments : eg symVar = symName What if some protein coding seqences AREN'T protein coding sequences : eg, protein coding sequences used as "local addresses" for [t,m]RNA binding? +-----+ #] useful (to me) notes from the authors' paper : p1c1h0.70 "... In recent years, there has been a growing body of evidence highlighting the significant roles played by non-coding RNAs (ncRNAs) in various biological processes [2]. Among these, lncRNAs play key roles in regulating chromatin dynamics, gene expression, growth, differentiation, and development [3], [4]. ..." >> This certainly fits a small part of my long-standing (always unfulfilled) objectives. For instance : 1. HOTAIR (HOX Transcript Antisense Intergenic RNA) stands out as one of the extensively researched lncRNAs, exhibiting overexpression across various cancers such as breast, colorectal, hepatocellular, gastroin- testinal, and non-small cell lung carcinomas [6]. 2. HOTAIR interacts with two pivotal gene silencing factors: PRC2 and LSD1. By recruiting PRC2 and LSD1 to the target gene, HOTAIR inducing gene silencing through H3K27-methylation and H3K4-demethylation [7]. 3. H19 is one of the first discovered lncRNAs and a crucial player in embryonic development and tumorigenesis. The tumor suppressor p53 and H19 are mutually counter-regulated. P53 represses the H19 gene, and the H19-derived miR-675 inhibits p53 and p53-dependent protein expression. The p53-H19 interplay appears to play major roles in tumorigenesis and metastasis [8]. *************************** #] +-----+ #] C8. THOUGHTS: (again, changes to the paper are not required for these - in fact changes SHOULD NOT be made!) Here are some long-winded thoughts that are not really relevant to the paper review per se... For interest only, even if that. These are separated from the "COMMENTS" above because they are less relevant to the actual paper. +-----+ #] Genetics and NNs what if many lncRNAs don't program for proteins, but for function, similar to classical computer programs, but perhaps very different in many ways, with programming concets that we do not yet have? My own priority interest is in looking for potential links between [neuron, brain] function and genetics : [architecture, function, process, operating system, control, plan, optimize, ontogeny, behaviour, consciousness, [pathogen, social] defense, [cooperate, compete], ...] An overly simplistic idea is to separate [DNA, RNA] coding into "protein" and "program", the latter bing possibly one aspect of the ncRNA. Perhaps even some of the [junk, repeating] DNA has role as repeat-code with a functional use. This would possibly be like computer programming, if [function, operator, transformer] calls are not used for loops or arguments? More specifically at this time : - electro-mechanical [real, biological] mechanisms for "transporting" bio-chemical material around (eg microtubules etc) - [gel, bulk] water phase changes - most importantly right now - mechanisms for coprocessing multiple mRNA strands - the [role, processes of] of [program, protein] coding in [DNA, mRNA] - if program coding exists (I assume it must) If the above (what-if, highly speculative) topics were combined with "Free-Flowing Ion Concentrations" (FFIC) framework, well, the complexity skyrockets, but not to the full level that biology handles it? Very [incomplete, point form] : http://www.BillHowell.ca/Neural nets/MindCode/MindCode webPage.html +-----+ #] Stephen Grossberg 2021 "Conscious Mind, Resonant Brain" Also a major idea with little tie in to [my, your] work is Stephen Grossberg's work, as well described in his 2021 huge book "Conscious Mind, Resonant Brain". His is the only [basis, context, framework] that I am aware of that I feel can answer many of my interests. ~620 captioned images are available on the webPage "Grossbergs list of [figure, table]s" as per a link below (posted with Grossberg's permission). Multiple images can be opened at a time, making it much easier to read the book. +--+ http://www.BillHowell.ca/Neural nets/TrNNs_ART/webWork/pMenuTopMenu TrNNs_ART.html#Grossberg http://www.BillHowell.ca/Neural nets/Grossberg/Grossbergs list of [figure, table]s.html http://www.BillHowell.ca/Neural nets/Grossberg/Grossbergs overview.html http://www.BillHowell.ca/Neural nets/Grossberg/Grossbergs [core, fun, strange] concepts.html My webPages have been moved, but links to older pages (retained for now) will still hopefully work, Note that my webPages are p[incomplete, changing as I work]. # enddoc