Support Apache Beam >= 2.44.0

[albertvillanova]

Once we find out the root cause of:

• CI tests are broken: SchemaInferenceError #5426

we should revert the temporary pin on apache-beam introduced by:

• Fix CI by temporarily pinning apache-beam < 2.44.0 #5429

[albertvillanova]

Some of the shard files now have 0 number of rows.

We have opened an issue in the Apache Beam repo:

• [Bug]: Some written Parquet shard files have 0 num_rows apache/beam#25041