bug: Cortex Engine crashes - Doesn't start any local models after the latest update

[Graninius]

Jan version

0.5.4 AppImage

Describe the Bug

I can't start any local models on my machine after the latest update. The previous version worked fine with various models.

Steps to Reproduce

1. Go to Settings
2. Start model (for exmp: tinillama-1.1b)
3. Model failed to start

Screenshots / Logs

2024-10-02T07:05:01.312Z [SPECS]::Version: 0.5.42024-10-02T07:05:01.406Z [SPECS]::CPUs: [{"model":"Intel(R) Core(TM) i7-3740QM CPU @ 2.70GHz","speed":3292,"times":{"user":834730,"nice":9370,"sys":370970,"idle":7985930,"irq":51350}},{"model":"Intel(R) Core(TM) i7-3740QM CPU @ 2.70GHz","speed":3292,"times":{"user":785930,"nice":10240,"sys":295490,"idle":8264430,"irq":34730}},{"model":"Intel(R) Core(TM) i7-3740QM CPU @ 2.70GHz","speed":3292,"times":{"user":950650,"nice":8490,"sys":387840,"idle":7993180,"irq":24150}},{"model":"Intel(R) Core(TM) i7-3740QM CPU @ 2.70GHz","speed":3292,"times":{"user":846580,"nice":5280,"sys":335720,"idle":8020080,"irq":23820}},{"model":"Intel(R) Core(TM) i7-3740QM CPU @ 2.70GHz","speed":3292,"times":{"user":887910,"nice":6730,"sys":342210,"idle":7957790,"irq":26750}},{"model":"Intel(R) Core(TM) i7-3740QM CPU @ 2.70GHz","speed":3292,"times":{"user":848830,"nice":7610,"sys":285660,"idle":8229460,"irq":17600}},{"model":"Intel(R) Core(TM) i7-3740QM CPU @ 2.70GHz","speed":3292,"times":{"user":976110,"nice":4140,"sys":352530,"idle":7941260,"irq":23180}},{"model":"Intel(R) Core(TM) i7-3740QM CPU @ 2.70GHz","speed":3292,"times":{"user":897820,"nice":2650,"sys":292090,"idle":8237660,"irq":17280}}]
2024-10-02T07:05:01.407Z [SPECS]::Machine: x86_64
2024-10-02T07:05:01.407Z [SPECS]::Endianness: LE
2024-10-02T07:05:01.408Z [SPECS]::Parallelism: 8
2024-10-02T07:05:01.408Z [SPECS]::Free Mem: 10204839936
2024-10-02T07:05:01.408Z [SPECS]::Total Mem: 16666611712
2024-10-02T07:05:01.408Z [SPECS]::OS Version: janhq/jan#1 SMP PREEMPT_DYNAMIC Mon Sep 30 19:30:33 UTC 2024
2024-10-02T07:05:01.408Z [SPECS]::OS Platform: linux
2024-10-02T07:05:01.408Z [SPECS]::OS Release: 6.11.1-1-MANJARO
2024-10-02T07:05:01.408Z [CORTEX]::Debug: Adding additional dependencies for @janhq/inference-cortex-extension 1.0.17
2024-10-02T07:05:01.409Z [APP]::{"notify":true,"run_mode":"cpu","nvidia_driver":{"exist":false,"version":""},"cuda":{"exist":false,"version":""},"gpus":[],"gpu_highest_vram":"","gpus_in_use":[""],"is_initial":false,"vulkan":false}
2024-10-02T07:05:10.684Z [CORTEX]::CPU information - 4
2024-10-02T07:05:10.686Z [CORTEX]::Debug: Request to kill cortex
2024-10-02T07:05:10.919Z [CORTEX]::Debug: cortex process is terminated
2024-10-02T07:05:10.920Z [CORTEX]::Debug: Spawning cortex subprocess...
2024-10-02T07:05:10.922Z [CORTEX]::Debug: Cortex engine path: /home/user/jan/extensions/@janhq/inference-cortex-extension/dist/bin/linux-avx
2024-10-02T07:05:10.922Z [CORTEX]::Debug: Spawn cortex at path: /home/user/jan/extensions/@janhq/inference-cortex-extension/dist/bin/cortex-cpp, and args: 1,127.0.0.1,3928
2024-10-02T07:05:10.922Z [CORTEX] PATH: /tmp/.mount_jan1XIC5L:/tmp/.mount_jan1XIC5L/usr/sbin:/home/feltsi/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/var/lib/flatpak/exports/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/opt/rocm/bin:/var/lib/snapd/snap/bin::/home/feltsi/jan/engines/@janhq/inference-cortex-extension/1.0.17:/home/feltsi/jan/extensions/@janhq/inference-cortex-extension/dist/bin/linux-avx
2024-10-02T07:05:11.034Z [CORTEX]::Debug: cortex is ready
2024-10-02T07:05:11.035Z [CORTEX]::Debug: Loading model with params {"cpu_threads":4,"ctx_len":2048,"prompt_template":"<|system|>\n{system_message}<|user|>\n{prompt}<|assistant|>","llama_model_path":"/home/user/jan/models/tinyllama-1.1b/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf","ngl":23,"system_prompt":"<|system|>\n","user_prompt":"<|user|>\n","ai_prompt":"<|assistant|>","model":"tinyllama-1.1b"}
2024-10-02T07:05:11.298Z [CORTEX]::Debug: cortex exited with code: null
2024-10-02T07:05:12.209Z [CORTEX]::Error: Load model failed with error TypeError: fetch failed
2024-10-02T07:05:12.209Z [CORTEX]::Error: TypeError: fetch failed

What is your OS?

• MacOS
• Windows
• Linux

[baraich]

@Graninius, Give a moment, let me try to reproduce the issue

[baraich]

@Graninius, I am able to run the same model (tinillama-1.1b) without any issues

[Graninius]

@baraich It doesn't start any models, and it doesn't matter which model I try

[baraich]

@Graninius, Let me try to replicate on a Linux machine.

[Graninius]

vers.0.5.5 - the same result, can't start models (

[louis-jan]

Hi @Graninius, it's a weird issue, I don't see any clear error message. cc @vansangpfiev for the assist, I think there might be a missing.so file.

[dan-menlo]

@hiento09 Can we see if we have a Manjaro VM that we can test it on? My concern is whether we have not tested AppImage very thoroughly.