-
Notifications
You must be signed in to change notification settings - Fork 489
ChatAsync inconsistently adds the Assistant message to the chat history #1199
Copy link
Copy link
Closed
Description
Summary
When calling ChatSession.ChatAsync the method returns an IAsyncEnumerator which can be used like such:
await foreach (string response in defaultSession.ChatAsync(
new ChatHistory.Message(AuthorRole.User, input),
inferenceParams,
cancellationTokenSource.Token))
{
if (SomeArbituaryCondition())
{
break;
}
stringBuilder.Append(response);
Console.Write(response);
}If the cancellation token is cancelled or if the loop breaks, AddAssistantMessage(assistantMessage); isn't called:
LLamaSharp/LLama/ChatSession.cs
Lines 444 to 459 in e52f5ef
| string assistantMessage = string.Empty; | |
| await foreach ( | |
| string textToken | |
| in ChatAsyncInternal( | |
| prompt, | |
| inferenceParams, | |
| cancellationToken)) | |
| { | |
| assistantMessage += textToken; | |
| yield return textToken; | |
| } | |
| // Add the assistant message to the history | |
| AddAssistantMessage(assistantMessage); | |
| } |
This can be remedied by surrounding the async foreach within a try/finally statement:
StringBuilder assistantMessage = new();
try
{
await foreach (
string textToken
in ChatAsyncInternal(
prompt,
inferenceParams,
cancellationToken))
{
assistantMessage += textToken;
yield return textToken;
}
}
finally
{
// Add the assistant message to the history
AddAssistantMessage(assistantMessage.ToString());
}Also you should switch to using a StringBuilder as adding strings directly is extremely expensive within allocations: https://learn.microsoft.com/en-us/troubleshoot/developer/visualstudio/csharp/language-compilers/string-concatenation
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels