jan loads/unloads the LLM model into memory for each query #6446
              
                Unanswered
              
          
                  
                    
                      MRDOCTOROO
                    
                  
                
                  asked this question in
                Get Help
              
            Replies: 0 comments
  
    Sign up for free
    to join this conversation on GitHub.
    Already have an account?
    Sign in to comment
  
        
    
Uh oh!
There was an error while loading. Please reload this page.
-
Because it is unloaded every few seconds when not used, it takes a long time to load the model each time, and the cost is higher. Also, the model list cannot be obtained using the API service. When the model is not started, a whitelist is added and it is started with 0.0.0.0. The model cannot be accessed from outside the local cross-domain.
Beta Was this translation helpful? Give feedback.
All reactions