fastai_inference.py is non-deterministic

[dthaler]

Running the inference script on the same input yields non-deterministic results due to randomness in the method used.
Repro:

model_type: "FastAI"
model_local_threshold: 0.5
model_global_threshold: 3
model_path: "./model"
model_name: "model.pkl"
hls_stream_type: "DateRangeHLS"
hls_polling_interval: 60
hls_start_time_pst: "2025-12-27 23:09"
hls_end_time_pst: "2025-12-27 23:10"
hls_hydrophone_id: "rpi_sunset_bay"
upload_to_azure: False
delete_local_wavs: True

Running:

python src/LiveInferenceOrchestrator.py --config ./config/Test/Positive/FastAI_DateRangeHLS_SunsetBay.yml --max_iterations 1

Will report Found about 50% of the time and not found the other 50% of the time.

[akashmjn]

Nice catch! Noticed it too when working on parity tests for #405

Cause

With help from Claude, managed to narrow it down to requiring a 1-line config addition.

config.pad_mode = "zeros-after" # Make deterministic: zeros at end only

Input segments were zero padded up to a fixed 312 frames (~4s) before model(). The fastai databunch being used, splits required padding between left/right sides at random. This is desirable during training, but not inference.

Effect

Variance is quite large

== run 1 ==
Summary: 3/59 segments predicted positive
Detected in segments: [47, 48, 50]

global_confidence: 59.470
global_prediction: 1

*** ORCA DETECTED! ***

== run 2 ==
Summary: 3/59 segments predicted positive
Detected in segments: [42, 48, 50]

global_confidence: 64.633
global_prediction: 1

*** ORCA DETECTED! ***

Fixing it seems to consistently produce higher scores.

== deterministic == 
Summary: 6/59 segments predicted positive
Detected in segments: [35, 36, 42, 47, 49, 50]

global_confidence: 68.727
global_prediction: 1

*** ORCA DETECTED! ***

@dthaler - feel free to make a quick patch for now.

Once #405 merges, this is exactly why it'd be best to port over to cleaner rewritten model inference without fastai altogether.