Re-think use of num_rows and proportion for detection

[Leobouloc]

(port to the forked version)

In the code below, if the target proportion is not 1, the validation functions are called on all the values of the serie that is passed (serie.apply(test_func).sum() ). This should be replaced, at the very least, by a call on num_rows values and ideally should be replaced by some heuristic that tests a limited number of values before testing more as is done when proportion == 1.

def test_col(serie, test_func, proportion=0.9, skipna=True, num_rows=50):
    ''' Tests values of the serie using test_func.
         - skipna : if True indicates that NaNs are not counted as False
         - proportion :  indicates the proportion of values that have to pass the test
    for the serie to be detected as a certain type
    '''
    serie = serie[serie.notnull()]
    ser_len = len(serie)
    if ser_len == 0:
        return False
    if proportion == 1:  # Then try first 1 value, then 5, then all
        for _range in [
            range(0, min(1, ser_len)),
            range(min(1, ser_len), min(5, ser_len)),
            range(min(5, ser_len), min(num_rows, ser_len))
        ]:  # Pour ne pas faire d'opérations inutiles, on commence par 1,
            # puis 5 puis num_rows valeurs
            if all(serie.iloc[_range].apply(test_func)):
                pass
            else:
                return False
        return True
    else:
        return serie.apply(test_func).sum() > proportion * len(serie)