Add Support for Streaming Requests to Completions API

[Bslabe123]

Addresses #15

Changes:

• Setting .api.streaming to true will make all responses streamed (default: False)
• API type config moved from .api to .api.type
• Report now includes stream_metrics section when streaming responses since TPOT, TTFT, and ITL depend on per token latency measurements.

Report snippet from local testing:

  "successes": {
    "count": 33,
    "latency": {
      "request_latency": {
        "mean": 3.31325431142327,
        "min": 1.62129471905064,
        "p10": 1.67609986825846,
        "p50": 2.11507539497688,
        "p90": 5.94717199734878,
        "max": 6.30658466403838
      },
      "normalized_time_per_output_token": {
        "mean": 0.104340420636009,
        "min": 0.0506654599703325,
        "p10": 0.0523781208830769,
        "p50": 0.0670631669655753,
        "p90": 0.189047570470012,
        "max": 0.20343821496898
      },
      "time_per_output_token": {
        "mean": 0.0836929455635872,
        "min": 0.0517028436646797,
        "p10": 0.0530815053513894,
        "p50": 0.0611870964678625,
        "p90": 0.152292036800645,
        "max": 0.17837208439984
      },
      "time_to_first_token": {
        "mean": 0.800974442732916,
        "min": 0.0625283779809251,
        "p10": 0.072068731742911,
        "p50": 0.203539535985328,
        "p90": 2.26959549135063,
        "max": 4.46773961000145
      },
      "inter_token_latency": {
        "mean": 0.0836929455635872,
        "min": 0.000007129972800612,
        "p10": 0.0534287681337446,
        "p50": 0.0591336835059337,
        "p90": 0.084046097996179,
        "max": 0.614475268055685
      }
    },
    "throughput": {
      "input_tokens_per_sec": 643.576644186323,
      "output_tokens_per_sec": 32.544923821416,
      "total_tokens_per_sec": 676.121568007739,
      "requests_per_sec": 1.0238155253639
    },
    "prompt_len": {
      "mean": 628.606060606061,
      "min": 4,
      "p10": 11.4,
      "p50": 364,
      "p90": 2427.6,
      "max": 3836
    },
    "output_len": {
      "mean": 31.7878787878788,
      "min": 30,
      "p10": 31,
      "p50": 32,
      "p90": 32,
      "max": 32
    }
  },

[achandrasekar]

Can we organize it like this?

- successes
  - count
  - latency
    - req latency
    - ntpot
    - ttft
    - itl
    - tpot
  - throughput
    - input tokens / s
    - output tokens / s
    - total tokens / s
    - req / s
  - prompt len
  - output len
- failures
  - count

[achandrasekar]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: achandrasekar, Bslabe123

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [achandrasekar]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment