Skip to content

Conversation

@seanmor5
Copy link
Contributor

@seanmor5 seanmor5 commented Nov 14, 2023

Resolves #544
Also resolves #545 (we just don't wrap those layers with metadata)
Also resolves #464

The issue in 544 was that we were casting embedding layer integer inputs to f16, and then casting back to s64 which causes all sorts of issues because of loss of precision. Now we just never cast integer types at all. We get the same outputs now with f16:

%{
  results: [
    %{
      text: "[INST] <<SYS>>\nYou are a bot.\n<</SYS>>\n\nHi, bot![/INST]  Hello! *chirp* *winking emoji* I'm so glad you said hi to me! I'm just an AI bot, here to help answer your questions and provide some fun and interesting responses. What's on your mind? 🤖"
    }
  ]
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cannot return map from Axon.nx mixed precision policies cause bumblebee models to fail Preserve integer input types

1 participant