While writing a CI test, I realize that the following fails when using a dask_distributed_local_core_fraction other than 0 or 1 if the machine has only one core (like the standard GitHub CI compute).
|
n_system_cores = os.cpu_count() |
|
# compute the number of cores to use |
|
n_local_cores = int(args.dask_distributed_local_core_fraction * n_system_cores) |
|
# get the total system memory |
|
total_memory = psutil.virtual_memory().total |
|
# compute the memory per worker |
|
memory_per_worker = ( |
|
total_memory / n_local_cores * args.dask_distributed_local_memory_fraction |
|
) |
Even though this case will hardly ever be an issue for our computations, I suggest to replace int() with numpy.ceil() or raise a specific warning. Currently one just gets a ZeroDevisionError.
While writing a CI test, I realize that the following fails when using a
dask_distributed_local_core_fractionother than0or1if the machine has only one core (like the standard GitHub CI compute).mllam-data-prep/mllam_data_prep/__main__.py
Lines 56 to 64 in a283eff
Even though this case will hardly ever be an issue for our computations, I suggest to replace
int()withnumpy.ceil()or raise a specific warning. Currently one just gets a ZeroDevisionError.