Add Llama 3 instruction template

[Touch-Night]

A simple template

Checklist:

• I have read the Contributing guidelines.

[berkut1]

@Touch-Night sorry, it was my mistake :) Yeah it includes everything. Everything fine, sorry.

P.S I deleted my message, but you still managed to reply :)

[Touch-Night]

Using that template with chat-instruct seems to result in an assistant being at the end of all the messages it sends.

Sorry I lost a -... Now this issue should've been resolved.

[AtanasValkov]

Using that template with chat-instruct seems to result in an assistant being at the end of all the messages it sends.

Sorry I lost a -... Now this issue should've been resolved.

your template is pretty solid, but it doesn't work for multi turn chats. This is the template we need to follow according to Meta.

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

{{ system_prompt }}<|eot_id|><|start_header_id|>user<|end_header_id|>

{{ user_message_1 }}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

{{ model_answer_1 }}<|eot_id|><|start_header_id|>user<|end_header_id|>

{{ user_message_2 }}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

which sadly I'm not too sure how to implement into ooba

[Touch-Night]

but it doesn't work for multi turn chats.

I don't see how. It works great for me. I don't think it's because this template doesn't work for multi turn chats. Is your context window too small?
What prompt structure do you think will this template finally build?

[AtanasValkov]

but it doesn't work for multi turn chats.

I don't see how. It works great for me. I don't think it's because this template doesn't work for multi turn chats. Is your context window too small? What prompt structure do you think will this template finally build?

I load models at 4k context but usually around the 2k mark (depending on base prompt) outputs become identical (this is with varied settings).
From Meta they say that we need to surround each command (even in history) with the start and end header like the structure I sent. But I just don't know if ooba supports such an implementation where the tokens get stored on each new input/output.

currently it outputs like this:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Persona of your AI assistant who does stuff and speaks in a certain way.<|eot_id|><|start_header_id|>user<|end_header_id|>

User: Hello!
Bot: Hi!
User: How are you?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

instead it should be:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Persona of your AI assistant who does stuff and speaks in a certain way.<|eot_id|><|start_header_id|>user<|end_header_id|>

User: Hello!<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Bot: Hi!<|eot_id|><|start_header_id|>user<|end_header_id|>

User: How are you?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

[Touch-Night]

This is an intended prompt design by @oobabooga .
The full prompt does looks like:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

{{Custom_system_prompt}}<|eot_id|><|start_header_id|>user<|end_header_id|>

{{ChatTab.Command for chat-instruct mode}}(continue the dialogue below)

{{CharacterContext&Description}}

{{ParametersTab.Chat.User.Description}}

{{character_name}}: {{character_message_1}}
{{user_name}}: {{user_message_1}}
{{character_name}}: {{character_message_2}}
{{user_name}}: {{user_message_2}}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

{{character_name}}:

It's not a thing that can be done with an instruction template. Try editing chat template instead.
If you want it to be like

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

{{ChatTab.Command for chat-instruct mode}}(act as {{character}}/you are {{character}}...)

{{CharacterContext&Description}}

{{ParametersTab.Chat.User.Description}}

<|eot_id|><|start_header_id|>assistant<|end_header_id|>

{{character_message_1}}<|eot_id|><|start_header_id|>user<|end_header_id|>

{{user_message_1}}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

{{character_message_2}}<|eot_id|><|start_header_id|>user<|end_header_id|>

{{user_message_2}}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

You can try this Chat template (redundant, not tested, may be wrong):

{%- if message[0]['role'] == 'system' -%}
    {%- if message['content'] -%}
        {{- message['content'] + '\n\n' -}}
    {%- endif -%}
    {%- if user_bio -%}
        {{- user_bio + '\n\n' -}}
    {%- endif -%}
{%- else -%}
    {%- if message['role'] == 'user' -%}
        {{- message['content'] + '<|eot_id|>'-}}
    {%- endif -%}
{%- endif -%}
{%- for message in messages[1:-1] %}
    {%- if message['role'] == 'system' -%}
        {%- if message['content'] -%}
            {{- message['content'] + '\n\n' -}}
        {%- endif -%}
        {%- if user_bio -%}
            {{- user_bio + '\n\n' -}}
        {%- endif -%}
    {%- else -%}
        {%- if message['role'] == 'user' -%}
            {{- '<|start_header_id|>user<|end_header_id|>\n\n' + message['content'] + '<|eot_id|>'-}}
        {%- else -%}
            {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' + message['content'] + '<|eot_id|>' -}}
        {%- endif -%}
    {%- endif -%}
{%- endfor -%}
{%- if message[-1]['role'] == 'system' -%}
    {%- if message['content'] -%}
        {{- message['content'] + '\n\n' -}}
    {%- endif -%}
    {%- if user_bio -%}
        {{- user_bio + '\n\n' -}}
    {%- endif -%}
{%- else -%}
    {%- if message['role'] == 'user' -%}
        {{- '<|start_header_id|>user<|end_header_id|>\n\n' + message['content'] -}}
    {%- endif -%}
{%- endif -%}

[AtanasValkov]

This is an intended prompt design by @oobabooga . The full prompt does looks like:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>
{{ChatTab.Command for chat-instruct mode}}(continue the dialogue below)

{{ParametersTab.Chat.User.Description}}

<|eot_id|><|start_header_id|>user<|end_header_id|>
{{user_name}}: {{user_message_1}}
{{character_name}}: {{character_message_1}}
{{user_name}}: {{user_message_2}}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

It's not a thing that can be done with an instruction template. Try editing chat template instead.

You're right. I'll do it ASAP with chat template. Will edit this comment with results.

[AtanasValkov]

You can try this Chat template (redundant, not tested, may be wrong):

This one is the template I use to achieve the meta recommendation, ooba complained a lot about yours and was a bit too long for me to look how to fix. It works like a charm in chat mode. I still need to figure out which generation settings to use since I still encounter repetition errors but I hope this helps anyone looking for what I was looking for.

{%- for message in messages %}
    {%- if message['role'] == 'system' -%}
        {%- if message['content'] -%}
            {{- '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n' + message['content'] + '\n\n' -}}
        {%- endif -%}
        {%- if user_bio -%}
            {{- user_bio + '\n\n' -}}
        {%- endif -%}
    {%- else -%}
        {%- if message['role'] == 'user' -%}
            {{- name1 + ': ' + message['content'] + '<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n' -}}
        {%- else -%}
            {{- name2 + ': ' + message['content'] + '<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n' -}}
        {%- endif -%}
    {%- endif -%}
{%- endfor -%}

[Touch-Night]

{%- for message in messages %}
    {%- if message['role'] == 'system' -%}
        {%- if message['content'] -%}
            {{- '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n' + message['content'] + '\n\n' -}}
        {%- endif -%}
        {%- if user_bio -%}
            {{- user_bio + '\n\n' -}}
        {%- endif -%}
    {%- else -%}
        {%- if message['role'] == 'user' -%}
            {{- name1 + ': ' + message['content'] + '<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n' -}}
        {%- else -%}
            {{- name2 + ': ' + message['content'] + '<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n' -}}
        {%- endif -%}
    {%- endif -%}
{%- endfor -%}

Won't it end up like this?

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

{{CharacterContext&Description}}
{{ChatTab.Command for chat-instruct mode}}(act as {{character}}/you are {{character}}...)

{{ParametersTab.Chat.User.Description}}

<|eot_id|><|start_header_id|>user<|end_header_id|>

{{user_name}}: {{user_message_1}}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

{{character_name}}: {{character_message_1}}<|eot_id|><|start_header_id|>user<|end_header_id|>

{{user_name}}: {{user_message_2}}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

<|eot_id|><|start_header_id|>assistant<|end_header_id|>

This chat template will cause:

1. Redundant <|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n at the beginning of the prompt.
2. Redundant <|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n at the end of the prompt.
3. It is possible to have consecutive two or more messages from the same role, so you can't simply put <|start_header_id|>assistant<|end_header_id|> at the end of user's turn.

[AtanasValkov]

Won't it end up like this?

1. I did say I'm using it in chat mode, not instruct or chat-instruct, so no, I'm not getting the double system in my prompt.
2. Same as statement 1.
3. If you're doing consecutive messages from the same role aren't you already messing with the already strict baked in turn order of the AI training? (I'm not sure on this point since I avoid more than one message per role and I'm not familiar with model training.)

Like I said, it's already working as intended for me. This isn't that clean of a solution anyway, since it should be achievable through instruct mode, how I assume most people use models these days.

[Touch-Night]

Sorry, you said

currently it outputs like this:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Persona of your AI assistant who does stuff and speaks in a certain way.<|eot_id|><|start_header_id|>user<|end_header_id|>

User: Hello! Bot: Hi! User: How are you?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

So I thought you are using chat-instruct mode.

Anyway, one chat template cannot achieve the meta recommendation on chat and chat-instruct mode at the same time. Mine for chat-instruct mode and yours for chat mode.

I'm aware that users should not mess with the turn order, but since it's possible, there will be such user doing that. I want to make sure Llama 3 doesn't wrongly regard a user message as from assistant.

[kalle07]

anny progress ?

if i use llama3 without any Modifications in notpad as (alpaca)
its talks with hisself until the moon goas up ^^

the tts based on Coqui_tts talks in the chat with his sself untill ... ^^

[Touch-Night]

I think this template needs no more changes, it's done. So no progress unless @oobabooga gives his advice on something like should the <|begin_of_text|> be added to the template.
I'm using this template these days with no other setting tweaks, and it's working fine.

[kalle07]

i dont know where i should copy that in oobadoga

i can only use the llama3 template from gpt4all and there i know where i can copy/paste ^^

[berkut1]

What about <|begin_of_text|>? There is an another template https://github.com/mamei16/LLM_Web_search/blob/main/instruction_templates/Llama-3.yaml

[nickpotafiy]

What about <|begin_of_text|>? There is an another template https://github.com/mamei16/LLM_Web_search/blob/main/instruction_templates/Llama-3.yaml

I'm not so sure that bos_token is added automatically. I'm using this template and it seems to be working well. If I omit <|begin_of_text|> it's not present in the debug text when using API, so I'm convinced it needs to be added.

[oobabooga]

I don't like adding the BOS token to instruct templates, as every tokenizer already adds it automatically by default. In fact, I go out of my way to remove the BOS token from instruction templates somewhere in this repository to prevent a double BOS from happening.

Note that Llama-3 chat models already come with a template in the metadata, which gets detected and applied automatically, so it's not necessary to have a dedicated yaml.