Support for Scheduling Policy for Search and Compression Jobs

[goynam]

Request

[SEARCH]
We have a usecase where multiple SDEs will be using CLP UI and CLP CLI for quering the CLP archives from S3 storage.

Can we implement a resource manager/scheduling policy where query from one SDE doesnt block the resources for query fired by a different SDE.

Note : Solution should scale for multiple SDEs firing different queries ( grepping on message content ) at same time. We are expecting sub second latencies for search operations.

[COMPRESSION]

Our services hosts are producing raw log files every minute which is ingested into S3 bucket. We have a scheduler running which listens for S3 notification on newly ingested log file and calls CLP compression scheduler for compression. Now, we have more than 50k hosts across different services which are producing GBs of data every minute.

We need a solution where we can scale compression worker jobs to create the archives from raw logs with minimal latencies ( so that near real time logs are available for search)

Possible implementation

DIscussed with Kirk, We can have a simple scheduling policy implementation as a first release which can be improved if we see latency issues.