Question on this 30x scaling tool

[BradKML]

1. Assuming this can generate new architectures, can they merge different techniques to see if there are synergistic gains, kinda like the methods of DeepSeek or Qwen iterations? Sometimes certain features are non-compatible and they need a bit of outlining
2. Have you tested out-of-sample (or outside of training data) tasks such that LLMs can be evaluated based on acceleration of more complex emergent properties? For example, check new small models on diverse hard tasks usually reserved for larger models.
3. Can this system do replication studies to prevent flukes? What about understanding how certain advancements work through deep investigations and research? How could that be turned into a separate metric (maybe an ELI5 meta-judge)?
4. Would it be better if this project were PKM-compatible, such that humans can read AI-generated research notes and self-studies? That way it might be able to organize thoughts in more efficient ways.
5. Consider NanoPoor by @VatsaDev (and similar projects), would ASI-Arch be good for small-scale citizen research with less GPU compute?

P.S. The 30x estimation comes from 4000 hr mark when all the exponential optimization has been squeezed out, and the architecture count goes linear. Kinda weird but something worth looking into

[HZxCzar]

Thank you for raising these profound and valuable questions! We respond to each one as follows:

1. Architecture Fusion and Synergistic Effects

We have currently designed a complete model hierarchy during the evolution process. AI has already attempted to integrate Mixture of Experts (MoE) elements into linear attention architectures, which is indeed very interesting! We believe our current framework can achieve multi-technology fusion at the hierarchical level, whether at the high-level design thinking level or specific component combinations.

For some larger-scale fusions (such as those requiring multimodal capabilities, tool usage, etc.), we believe AI's modification scope needs to extend from the hierarchical level to higher levels, such as overall architecture design, data flow processing, and even training paradigms. This will place higher demands on the programming capabilities of existing models and require stronger correctness verification mechanisms to ensure the stability of complex fusions.

2. Out-of-Sample Task Evaluation

How generalizable are the architectures we discovered?

To ensure result accuracy, we adopted three measures:

1. Dataset Generalization Validation: We used Wikipedia test sets during validation to test the model's generalization ability. Results show that our models achieved good performance on out-of-distribution (OOD) test sets, with significantly lower perplexity.

2. Model Scale Generalization Validation: We used different sized models during exploration and validation phases (20M scale for exploration, 340M scale for validation). The change in model scale itself demonstrates our architecture's generalization ability, and experiments prove that our architecture maintains good consistency with small model results even after model scale increases.

3. Sample Scale Generalization Validation: Some test tasks appeared in both exploration and validation phases. For these test tasks, we used same-distribution but different test samples. The excellent results indicate that our architecture indeed has superior performance.

Beyond the above aspects, we believe that in traditional human expert research paradigms, overfitting issues may arise. For example, new researchers tend to adopt benchmarks used by predecessors and surpass previous architectures on these metrics, which in some sense is continuously fitting or even overfitting these datasets. In contrast, our pipeline adopted OOD methods in validation, making it more scientific. We enabled AI to complete research that might take years for humans in thousands of iterations. While following human research patterns, this approach more easily identifies whether scientists continuing to follow existing benchmarks face overfitting risks. We believe our approach actually reveals potential issues in past mechanisms.

3. Reproduction Research and Deep Understanding

1. Reproducibility: Due to randomness in sampling during each iteration, multiple experiments will indeed produce different results. However, this can be addressed by adopting specific algorithms (such as UCT) to balance the development of various exploration directions during evolution, and similar results should be obtainable after high computation time.

2. Understanding Mechanisms: This is an excellent question. We believe AI may have already surpassed humans in certain aspects. AI successfully created architectures, and humans need to learn from them and gain more important experience. Please look forward to our upcoming Part 2 work, where we will summarize and analyze all searched architectures, extract common mechanisms, and explore more feasible universal design principles.

3. Evaluation Metrics: Evaluation metrics determine the direction of evolution, similar to reward mechanisms in reinforcement learning. Single metrics easily lead to overfitting, but if metrics are designed comprehensively enough, they can effectively guide the search.

4. PKM Compatibility

This is an excellent suggestion! We also believe that the experience AI accumulates during exploration is as worthy of attention as the final results. We are planning to collect and organize the experience and insights from AI's exploration process and provide feedback to the research community. In the future, we may establish a library of AI-autonomously generated mini papers, allowing humans to learn AI's research approaches.

5. Small-Scale Citizen Research

This is a very valuable application direction! Projects like NanoPoor represent an important trend toward "democratized AI research," enabling resource-limited researchers to participate in cutting-edge exploration. Our two-stage strategy (20M exploration + 340M validation) aims to save GPU resources and accelerate the exploration process. However, current evolution-related projects basically require thousand-round searches to obtain good structures, and the excellent architectures we discovered were mostly found around the 600th round. Even using 20M small models still requires considerable computational support.

To further lower resource barriers, we are exploring several possible approaches:

1. Alternative Evaluation Methods: Consider using more lightweight evaluation criteria, such as MATH and other synthetic tasks as evaluation benchmarks, avoiding the overhead of large-scale language modeling.

2. Enhanced Analysis Capabilities: Explore stronger research and analysis modes, reducing the number of architectures that need to be tried through more intelligent search strategies.

Actually, computational consumption largely depends on the specific training and testing framework. Our pipeline is not only applicable to linear attention but can also be applied to other tasks after adjustment. If these are tasks with more convenient evaluation (such as theoretical derivation tasks, like some algorithmic problems implemented by AlphaEvolve), the required computational power will be greatly reduced.

6. Linear Growth Phenomenon

This is an excellent question that indeed deserves our further in-depth research. We will also refer to the inspiration you provided and conceive further experiments. Please look forward to our Part 2 results.

Thank you again for the valuable suggestions you provided for our work! This feedback has important guiding significance for our future research directions.

[BradKML]

Thanks for the response @HZxCzar, I got a few more bits of commentary to note down

7. There are existing automated research tools like Sakana's work, as well as Open implementations of AlphaEvolve. Each of them tackles a different layer of the problem, the former focusing on novel ideas over improving performance, while the latter focuses on quantitative optimizations rather than more holistic methods. Since they are complementary, in later versions of ASI-ARCH it would be great to see the suite become more comprehensive. https://github.com/SakanaAI/AI-Scientist-v2 https://github.com/codelion/openevolve/
8. Other than architecture, would you also consider automating the research for new and comprehensive reinforcement learning algorithms? I have seen hierarchical RL, "intuitors", chain-of-thought learning, and others like them on ArXiV. Generic RLVR and GRPO likely carry some bias with them, other than overthinking and overconfidence.
9. To give the whole "democratized science" idea a fairer shake, assume that people can keep their consumer GPU running 24/7 for a week or so, with cache and checkpoint just in case something happens with electric outages. If evolutionary algorithms can be made more effective in discovering new methods, they need to have awareness of computational complexity AND understanding what is highly probable in existing literature. Maybe a Socratic agent swarm with "deep research" can help with this as well?
10. And beyond the layer of algorithmic exploration, what about the tooling and codebase management layer? There are a few tools out there that do the job, and lowering the barrier of entry (not dependent on Anthropic but rather using their standard API). What would community-driven research look like? Paper collections are often too unapproachable to those with less academic exposure, but chatbots and wikis/PKM can help. PKM are very agentic as well, so there is some synergy there. https://github.com/ruvnet/claude-flow https://github.com/SuperClaude-Org/SuperClaude_Framework

Addendum: I remember the following blog on "general purpose technology" and additive growth

• https://growthecon.com/feed/2022/08/09/Additive.html
• https://www.vox.com/future-perfect/23048981/economic-growth-productivity-philippon
• https://www.noahpinion.blog/p/contemplating-a-world-of-linear-growth

[BradKML]

Bonus: ShinkaEvolve improved in DGM + AlphaEvolve/OpenEvolve, and can be combined with AI-Scientist and similar research agents. Will there be an update to this paper for a more well-rounded project (as part 2)? https://github.com/SakanaAI/ShinkaEvolve