idea: Native ARM64 Support and Snapdragon NPU Integration for Windows

[Jasonthefirst]

Problem Statement

While Jan currently runs on Windows devices with Snapdragon Elite chips through emulation, it's not optimized for the ARM64 architecture. This means we're not taking full advantage of the native performance capabilities of these chips. Additionally, the Snapdragon NPU (Neural Processing Unit) remains unutilized, despite its potential to significantly accelerate AI inference.

Feature Idea

1. Implement native ARM64 builds for Windows to eliminate the need for x64 emulation on Snapdragon devices. While the current emulation works adequately, native support would ensure enhanced performance and resource utilization.
2. Add support for Snapdragon NPU acceleration through Qualcomm's Neural Processing SDK.
   This would:
   • Enable hardware-accelerated inference using the dedicated NPU
   • Potentially reduce power consumption compared to CPU/GPU inference
   • Take advantage of Qualcomm's AI optimization tools for improved performance
   • Future-proof Jan for the growing ARM64 Windows ecosystem

The implementation could follow a phased approach:
Phase 1: Native ARM64 Windows build
Phase 2: Snapdragon NPU integration via Qualcomm's SDK
Phase 3: Performance optimization and testing
This would make Jan one of the first local AI assistants to fully support the emerging Windows on ARM ecosystem and might nudge people to change from other programs.

[louis-jan]

I love to have this work. Snapdragon Surface fan here.