Skip to content

Fixed Issue where Model with Grammar constrains only generates one output in Example #908

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
martindevans merged 1 commit into from
Aug 26, 2024
Merged

Fixed Issue where Model with Grammar constrains only generates one output in Example #908

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 9 additions & 6 deletions LLama.Examples/Examples/GrammarJsonResponse.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,20 +25,23 @@ public static async Task Run()
Console.WriteLine("The executor has been enabled. In this example, the LLM will follow your instructions and always respond in a JSON format. For example, you can input \"Tell me the attributes of a good dish\"");
Console.ForegroundColor = ConsoleColor.White;

using var grammarInstance = grammar.CreateInstance();
var samplingPipeline = new DefaultSamplingPipeline
{
Temperature = 0.6f
};

var inferenceParams = new InferenceParams()
{
SamplingPipeline = new DefaultSamplingPipeline
{
Temperature = 0.6f,
Grammar = grammarInstance
},
SamplingPipeline = samplingPipeline,
AntiPrompts = new List<string> { "Question:", "#", "Question: ", ".\n" },
MaxTokens = 50,
};

while (true)
{
using var grammarInstance = grammar.CreateInstance();
samplingPipeline.Grammar = grammarInstance;

Console.Write("\nQuestion: ");
Console.ForegroundColor = ConsoleColor.Green;
var prompt = Console.ReadLine();
Expand Down