Fix duplicated session_id when pipeline is used by multithreads

[irexyc]

Motivation

When using pipeline.batch_infer / pipeline.stream_infer with multiple threads, the default session_ids are all start of zero which will makes batch inference impossible.

from threading import Thread
from lmdeploy import pipeline, GenerationConfig
pipe = pipeline('/mnt/140/InternLM/internlm2-chat-1_8b', log_level='INFO')


def work(ss):
  gen_config = GenerationConfig(ignore_eos=True, max_new_tokens=512)
  for i in range(10):
    for x in pipe.stream_infer('hello', gen_config=gen_config):
        pass


threads = []
for i in range(5):
  t = Thread(target=work, args=(i * 10, ))
  t.start()
  threads.append(t)

for t in threads:
  t.join()

[lvhan028]

Introducing session_ids will make the API hard to understand.
Is there any way we can handle it internally?

[irexyc]

The stream_infer supports batch inputs, we need to match the inputs and outputs and we distinguish it by session_id currently. If we don't introduce the session_ids, we need to change the output like below which i present the ith input in the batch.

                i, out = outputs.get(timeout=0.001)
                if out is None:
                    break
                yield i, out

[lvhan028]

@AllentDan Do you remember our motivation to support batch prompts inference in streaming mode?

[AllentDan]

@AllentDan Do you remember our motivation to support batch prompts inference in streaming mode?

Required by XTuner and community. #636

[AllentDan]

Users tend to input only one prompt in each thread in multithread situations. Shall we provide an infer function for it? The infer function is a synchronized function of async generate. It does not offer batch inference ability. Users can control the session id themselves.

[lvhan028]

@AllentDan Do you remember our motivation to support batch prompts inference in streaming mode?

Required by XTuner and community. #636

It didn't make sense to me.
If users request streaming output for multiple prompts, can't we recommend they use async_stream_infer and pass the prompts one by one?

[AllentDan]

Maybe the user is not capable of using coroutine programming. I recommend they all use the generate function directly if possible.

[lvhan028]

That's not my point
My point is that outputting batch prompt response in streaming mode is probably inappropriate.
Is it supported by vLLM?

[lvhan028]

The root is we don't want users being bothered by session_ids in multithread scenarios.

[lvhan028]

todo item after inner discussion:

• not export session_ids in pipeline API but maintain an internally incremented global session_id
• add index in response structure, indicating the id of the corresponding prompt in the prompt list

[lvhan028]

"self._session_id" is better. After all, it is only one int

[AllentDan]

LGTM

[lvhan028]

May merge main so as to do the test