Please add arXiv:2410.01201

[daedalus]

Were RNNs All We Needed?
https://arxiv.org/abs/2410.01201
https://dhruvmsheth.github.io/projects/gpu_pogramming_curnn/

[BradKML]

How is this even relevant to architecture search?

[daedalus]

its not architecture search in the Neural Architecture Search (NAS) sense but:

They take LSTMs and GRUs and remove gates’ dependence on previous states, drop range restrictions like tanh, and normalize gates so they sum to 1 .

These "minLSTM" and "minGRU" variants become parallelizable RNNs trainable without backpropagation through time (via parallel scan) .

The whole point is to show that drastically simplified, decades-old architectures can still achieve competitive performance against Transformers, Mamba, etc...

So in terms of relevance to architecture search:

If you’re strictly asking "is this a NAS paper?" -> No, not at all.

If you’re asking "why would an architecture search researcher care?" -> Because it broadens the architecture design space. They demonstrate that a very simple, neglected corner of the space (minimal RNNs) is actually competitive. That means NAS search spaces that exclude such designs might be missing surprisingly strong candidates.

Also, their ablations (adding Conv/MLP layers, forget-gate bias changes ) resemble manual micro-search. One could argue this gives a starting point or baseline for automated NAS.

In short: it’s not NAS, but it’s relevant because it reshapes the assumptions about what the "good" regions of the architecture landscape even are. A NAS practitioner could treat minLSTM/minGRU as new strong baselines or as part of the candidate search space.

[BradKML]

Relates to here as well, the issue is that ASI-ARCH promises open/boundless search spaces, and a lot of novel layers are just combinations of basic sub-layers. What other types of layers can we inject anyways? #7

[HZxCzar]

Thanks for all these recommended papers!

I hypothesize that the initial architecture, especially at the very beginning phase of the iteration, largely originated from human paper design. Therefore, the human cognition base has a bigger impact on the root-end architectures, which in turn influences the downstream trend.

Based on this view, I want to enhance cross-region discovery by adding more diverse paper information into the cognition base—like the points you shared here. I'm imagining what if we add papers from other regions, like Physics or Psychology? I've done some tests, and they show the model has the ability to be more creative!

Furthermore, if the downstream architecture isn't creative enough, we could try using a more fundamental root (e.g., RNN or Transformer). However, modifying the entire model is still hard (so we don't let the system build it self #6 ) and I believe memory management would be needed for such a long-horizon task. Alternatively, the cross-region discovery method might also work.