document and explain output format

[Gldkslfmsd]

hi,

can you please document and explain the output format? Because @pe-trik 's OmniSTEval uses it as one supported output format.
I want to use OmniSTEval for evaluating output of my project without merging my code with simulstream (unless I'll be convinced that it is most reasonable way).

I saw one example here: https://raw.githubusercontent.com/pe-trik/OmniSTEval/refs/heads/main/examples/simulstream_example/simulstream_log.jsonl

{"model_loading_time": 22.951433420181274}
{"id": 0, "metadata": {"wav_name": "//storage/MT/sara/MUSTC/en-de/data/tst-COMMON_wav/ted_1096.wav"}}
{"id": 0, "total_audio_processed": 1.0, "computation_time": 0.4919137954711914, "generated_tokens": [], "deleted_tokens": []}
{"id": 0, "total_audio_processed": 2.0, "computation_time": 0.1842632293701172, "generated_tokens": [], "deleted_tokens": []}
{"id": 0, "total_audio_processed": 3.0, "computation_time": 0.18722200393676758, "generated_tokens": [], "deleted_tokens": []}
{"id": 0, "total_audio_processed": 4.0, "computation_time": 0.19884991645812988, "generated_tokens": [], "deleted_tokens": []}
...
{"id": 0, "total_audio_processed": 14.0, "computation_time": 0.2456223964691162, "generated_tokens": ["\u2581Zur", "\u00fcck"], "deleted_tokens": []}
{"id": 0, "total_audio_processed": 15.0, "computation_time": 0.24962759017944336, "generated_tokens": ["\u2581in", "\u2581New", "\u2581York", ",", "\u2581ich", "\u2581bin"], "deleted_tokens": []}

• is model_loading time really used?
• what is total_audio_processed? is it necessary to have it when the output is empty?
• what if it is a cascade, and we don't know how many audio ASR processed at the time when LLM translated a piece?
• do the generated tokens need to be as BPE tokens? What if a model does not use BPE tokens, but full text?

I would suggest this format for STEval tools. What would you think, @pe-trik and @mgaido91 ?

{ document: "doc1.wav", updates: [{ "emission_time": 1.34343, "text": "prefix", "unconfirmed_text": suffix"}, ...] }
{ document: "doc2.wav", updates: [{ "emission_time": 1.3223213, "text": "prefix", "unconfirmed_text": suffix"}, ...] }

[Gldkslfmsd]

moreover, I don't understand why OmniSTEval needs to have --simulstream_config_file provided. What necessary information is there that is not in output? Can it be passed to output, so that we get rid of one file?

[mgaido91]

My two cents and responses:

• simulstream is meant to be used to generate the outputs, so my advice would be to create a processor within simulstream based on your code. This would enforce that the results you get are compatible and coherent with the way these things are measured in simulstream. If you just build a raw log in your codebase, there is no guarantee that your implementations computes timings in the same way. In addition, in case of bugs in computing the timings (either in your codebase or in simulstream), your results won't be comparable. So I would be strongly against just reproducing the same format in a different repo for the sake of comparability. In addition, simulstream is designed to make the creation of a new processor quite straightforward, there is also a Docker-based option if you want to keep a separated python environment, so I really see no reason to do something different from this.
• model_loading time is not used for latency computations, or other metrics computations at the moment. It can be a useful information to leverage in the future.
• total_audio_processed is "how much audio has been received/processed so far". The reason not to skip the line if the output is empty is that we want to collect the imformation of the computational cost incurred, even though nothing was produced. This is useful to compute stats (e.g. RTF).
• "what if it is a cascade, and we don't know how many audio ASR processed at the time when LLM translated a piece?" well, you should know how much audio has been received so far. You should know how much audio ASR has processed, otherwise how do you compute latency for the whole system?
• "do the generated tokens need to be as BPE tokens? What if a model does not use BPE tokens, but full text?" there are many tokenizer possible. Check the config files for a processor. E.g. use this tokenization config if you output full text.

[Gldkslfmsd]

My two cents and responses:

simulstream is meant to be used to generate the outputs, so my advice would be to create a processor within simulstream based on your code.
This would enforce that the results you get are compatible and coherent with the way these things are measured in simulstream. If you just build a raw log in your codebase, there is no guarantee that your implementations computes timings in the same way.
In addition, in case of bugs in computing the timings (either in your codebase or in simulstream), your results won't be comparable. So I would be strongly against just reproducing the same format in a different repo for the sake of comparability.

OK. I think that the algorithm for simultaneous simulation and collecting timing is so simple so that it can be described in one paragraph, similar to this one https://github.com/ufal/whisper_streaming?tab=readme-ov-file#as-a-module , so that anyone can reimplement it without a bug. Is it? If not, why?

In addition, simulstream is designed to make the creation of a new processor quite straightforward,

OK, but this way has limits. So you support python, not other programming languages. And how about transcripts of human interpreters? Or closed-source API services?

I think that IWSLT sim task should allow comparing systems, and not only models.

there is also a Docker-based option if you want to keep a separated python environment, so I really see no reason to do something different from this.

We don't use docker for experimenting, we don't have it on our big slurm cluster for disk and security reasons. I could have one machine with docker -- no easy parallel compute, and big overhead of setting it. There already are simpler ways.

model_loading time is not used for latency computations, or other metrics computations at the moment. It can be a useful information to leverage in the future.

ok

total_audio_processed is "how much audio has been received/processed so far". The reason not to skip the line if the output is empty is that we want to collect the imformation of the computational cost incurred, even though nothing was produced. This is useful to compute stats (e.g. RTF).

ok

"what if it is a cascade, and we don't know how many audio ASR processed at the time when LLM translated a piece?" well, you should know how much audio has been received so far. You should know how much audio ASR has processed, otherwise how do you compute latency for the whole system?

From the times when output was emitted. There can be edge cases when the user stops sending audio but the system can finish processing previous chunks.

"do the generated tokens need to be as BPE tokens? What if a model does not use BPE tokens, but full text?" there are many tokenizer possible. Check the config files for a processor. E.g. use this tokenization config if you output full text.

OK, but too complicated, imho, the evaluator does not need to be aware of model tokenizer.

Anyway, thanks for discussion.

[mgaido91]

OK. I think that the algorithm for simultaneous simulation and collecting timing is so simple so that it can be described in one paragraph, similar to this one https://github.com/ufal/whisper_streaming?tab=readme-ov-file#as-a-module , so that anyone can reimplement it without a bug. Is it? If not, why?

The world is full of bugs for even simpler things. And besides what is a bug or not, the way the data is collected might contain subtle differences across systems and implementations so in general two different implementations would not be comparable. Ensuring things work in the same way would require extensive integration tests, which is way more complicated that just creating a processor here. You can check, e.g., BLEU scores returned by fairseq and sacrebleu... They would match, even though is ìjust calling the same sacrebleu method, which can be described in one line, not in a paragraph. In general every skilled software engineer would tell you that bug-free code simply does not exist adn I do not see the value in creating a pletora of tools for evaluating the same thing. The only value is to introduce possible issues. If you do not want to adopt simulstream, just do everything outside of it, this would be the safest option to me.

So you support python, not other programming languages. And how about transcripts of human interpreters? Or closed-source API services?

That is not true. If you check the docker example, the communication happens over HTTP. So you can just implement the REST API within the docker container with the programming language you want. I do not understand what does closed-source API services have to do with this, you can just build a processor that calls the service API and that's it.

We don't use docker for experimenting, we don't have it on our big slurm cluster for disk and security reasons. I could have one machine with docker -- no easy parallel compute, and big overhead of setting it. There already are simpler ways.

Docker is a very simple and widespread mechanism to get a reusable env. Anyway, you are not forced to use docker if you do not want. You can implement your processor in python or you can use the HTTP REST interface I mentioned before if you want to e.g. use a different programming language and this HTTP processor can run as a standalone server executed in the way you want. It can also be in a different machine from the one that runs the simulstream server. I hardly see a more flexible thing than an HTTP REST API to implement.

From the times when output was emitted. There can be edge cases when the user stops sending audio but the system can finish processing previous chunks.

When the audio stream is over, the processor gets a "audio is over" flag and can act accordingly. After that, what can a system do? there is no more input, the stream is over. This is similar to e.g. simuleval.

OK, but too complicated, imho, the evaluator does not need to be aware of model tokenizer.

This is not true. In retranslation systems a part of a word can be rewritten or part of a word can be emitted at one stage and at the next iteration a conti nuation of the word can be generated. Now in the logs you have the exaxt definition of what happened, everythiung can be reconstructed and metrics can decide how to assign latency to a word that was half-emitted in one stage or half-rewritten. Not to mention that if we want to switch from word to char-level evaluation with this method we have full control, while if we operate at different levels, we cannot reconstruct what happened, as information needs to be aggregated (e.g. across BPE tokens). The only burden from a user is to have an entry in their processor config, which they would anyway need to specify because the detokenization needs to be carried out sooner (ie. at inference time as you suggest) or later (at evaluation time as done here). It is just a matter of where you do the thing, but the thing has to be done.

[Gldkslfmsd]

From the times when output was emitted. There can be edge cases when the user stops sending audio but the system can finish processing previous chunks.

When the audio stream is over, the processor gets a "audio is over" flag and can act accordingly. After that, what can a system do? there is no more input, the stream is over. This is similar to e.g. simuleval.

Can audio be paused and then continued?

---

OK, thanks. Now I understand the concept, I would also propose a similar one on your place, with a "mediator" server.

So, my remaining concerns are little:

• is there a mode for computational aware and unaware simulation? It can happen that a model is slower than chunk size, so that in comp. aware mode, it naturally processes the audio that arrived by that time.
• I wish there would be a simple demo of processor inference in one script, not like simulstream_inference code with many fancy functions.
• I don't like simulation arguments, like chunk size, in the same config.yaml as model configs. They should be cmdline args because you may need to run the model with various args, and changing a file for each set of args is not nice. So I'd propose simulstream_inference_for_my_processor.

Thanks!

[mgaido91]

Can audio be paused and then continued?

I am not sure I understand what you mean. If you mean: can you stop streaming to the simulstream server (pause) and then start again sending speech data, simulating a continuous stream? then the answer is yes. The server runs the speech processors when it gets the data, so if the data flow is paused, the server just waits for new data (and this is not added in the latency computation of course, that is why in the logs you read total_audio_processed, which may be unrelated with the wall time or anything like that. If I did not understand, please let me know.

is there a mode for computational aware and unaware simulation? It can happen that a model is slower than chunk size, so that in comp. aware mode, it naturally processes the audio that arrived by that time.

Again I am not sure I understand. The way it works is that it runs the speech only after the previous step is completed. So if you run slower than speech chunk size you accumulate a lag (in a demo scenario). For latency computation, this does not affect anything. The amount of speech processed at each iteration is not affected, the computational cost of course is. So you would not see this effect reflected in penalizing the model e.g. in LAAL computation (even though the comp. aware LAAL will be of course high due to the high costs), but you would definitely see this in the RTF.

I wish there would be a simple demo of processor inference in one script, not like simulstream_inference code with many fancy functions.

You can just ignore everything there. You just need to implement your "speech processor". Like in simuleval you had to implement an agent. There is no need to get into other complexities just to use simulstream and extending with your system. I did my best to describe this in the documentation (https://simulstream.readthedocs.io/en/latest/usage.html#customize-with-your-speech-processor) but I am sure it could (should) have been done better. If you have any suggestion regarding how to improve the doc and making easier the adoption, I would be extremely grateful.

I don't like simulation arguments, like chunk size, in the same config.yaml as model configs. They should be cmdline args because you may need to run the model with various args, and changing a file for each set of args is not nice. So I'd propose simulstream_inference_for_my_processor.

I see your point. I do not understand what you mean by simulstream_inference_for_my_processor. However, passing it as command line argument has its downsides too: it implies that every processor would work with speech chunk sizes, while here you have the flexibility to decide for one speech processor to determine the speech chunk size dynamically or based on other model configurations. We can anyway consider revisiting this decision if there is consensus in the community in the next major release (v1.0), as this would be a breaking change.

Thanks for the suggestions.

[Gldkslfmsd]

is there a mode for computational aware and unaware simulation? It can happen that a model is slower than chunk size, so that in comp. aware mode, it naturally processes the audio that arrived by that time.

Again I am not sure I understand. The way it works is that it runs the speech only after the previous step is completed. So if you run slower than speech chunk size you accumulate a lag (in a demo scenario).

So you assume fixed-sized chunk all the time, even for the cost of accumulating lag? I don't think it's realistic. There's a simple way to avoid accumulating lag with a slow system: if update takes longer than min-chunk-size seconds (a parameter), just catch-up by processing longer chunk of audio than min-chunk-size. Then you get better quality, due to longer context, and latency guided flexibly by speed of your hardware.

In other words, there's never too slow system, but sometimes there's unsuitably small chunk-size.

Moreover, sometimes you think you have a system with good RTF<1, but some rare hallucinations make it >1. So you need a way to catch up. And you should be ready that a real-life HW may be shared with other processes, so it suddenly becomes slower.

For latency computation, this does not affect anything. The amount of speech processed at each iteration is not affected, the computational cost of course is. So you would not see this effect reflected in penalizing the model e.g. in LAAL computation (even though the comp. aware LAAL will be of course high due to the high costs), but you would definitely see this in the RTF.

OK, your way is good enough for progress in development.

But I propose a way where simulation output and latency equals output and latency in real-life demo system that never accumulates lag:

• when you do simulation from file, choose whether you want computational aware or unaware simulation.
• Server has only comp. aware mode.
• in unaware, chunk size is always min-chunk-size
• in aware, chunk size is at least min-chunk-size, or longer if update took longer and some audio accumulated
• you get different outputs in aware and unaware simulation. So your output log format and evaluation pipeline must reflect it.
• the same latency metrics (LAAL, YAAL etc.) are applied on logs either from comp. aware or unaware simulation.

See https://github.com/ufal/whisper_streaming/blob/6da90b44b7e50d79695e68166d2a2c7609c75abb/whisper_online.py#L914 , resp. line 940 for comp. aware mode

I wish there would be a simple demo of processor inference in one script, not like simulstream_inference code with many fancy functions.

You can just ignore everything there. You just need to implement your "speech processor". Like in simuleval you had to implement an agent. There is no need to get into other complexities just to use simulstream and extending with your system. I did my best to describe this in the documentation (https://simulstream.readthedocs.io/en/latest/usage.html#customize-with-your-speech-processor) but I am sure it could (should) have been done better. If you have any suggestion regarding how to improve the doc and making easier the adoption, I would be extremely grateful.

I'd like to understand what is inside simulstream_inference when I run the sample:

simulstream_inference --speech-processor-config config/speech_processor.yaml \
    --wav-list-file PATH_TO_TXT_FILE_WITH_A_LIST_OF_WAV_FILES.txt \
    --tgt-lang it --src-lang en --metrics-log-file metrics.jsonl

Of course I can read the code, I tried few minutes and gave up, so many functions and modules. I understand that this project is complex and less readable, therefore I'd like to see a demo code that illustrates simple minimum usage of processor would be nice.

I don't like simulation arguments, like chunk size, in the same config.yaml as model configs. They should be cmdline args because you may need to run the model with various args, and changing a file for each set of args is not nice. So I'd propose simulstream_inference_for_my_processor.

I see your point. I do not understand what you mean by simulstream_inference_for_my_processor. However, passing it as command line argument has its downsides too: it implies that every processor would work with speech chunk sizes, while here you have the flexibility to decide for one speech processor to determine the speech chunk size dynamically or based on other model configurations. We can anyway consider revisiting this decision if there is consensus in the community in the next major release (v1.0), as this would be a breaking change.

This is just question of convenience, not a big deal.

As simulstream_inference_for_my_processor I mean to encourage users to implement an entry point script specifically for their own processor. Then they can specify cmdline parameters only for that processor.

And still, there can be both ways.

Thanks for the suggestions.

[mgaido91]

So you assume fixed-sized chunk all the time, even for the cost of accumulating lag? I don't think it's realistic. There's a simple way to avoid accumulating lag with a slow system: if update takes longer than min-chunk-size seconds (a parameter), just catch-up by processing longer chunk of audio than min-chunk-size. Then you get better quality, due to longer context, and latency guided flexibly by speed of your hardware.

This is something that can be decided in the speech processor. As I said before, the speech processor can decide what speech chunk size to use dynamically, so what you suggest can be done by a speech processor containing this logic.

But I propose a way where simulation output and latency equals output and latency in real-life demo system that never accumulates lag:

Yes, we can consider adding this. Probably, the easiest way to achieve this is to create a wrapper speech processor that controls the speech chunk size with this logic.

I understand that this project is complex and less readable, therefore I'd like to see a demo code that illustrates simple minimum usage of processor would be nice.

We are trying to follow best practices common in open source project to make the code as readable as possible. If you have suggestions on what can be improved to make the code easier to read feel free to open a PR. I do not understand what you mean by demo code for a speech processor. There are several speech processor implementations in the repository that can be used as examples (e.g. the sliding window one, the streamatt one, ...). What do you mean by demo code?