Follow up work for Iceberg Writing #5989

[malhotrashivam]

Following is a list of follow up tasks for #5989 which should be done as and when needed:

• Add support for writeDataFiles API in python.
• Add support for overwrite API in Java and Python after fully understand the impact of concurrent writes.
• Add support for create+append API which can do both these tasks in a single transaction
• Improve support for adding a single partition value to a table being written out. (Check the TODO in source code for more details).

[malhotrashivam]

The feedback that we received on our community channel (link to slack discussion):

• Partition handling needs some work to support time/date partitions properly as well as the various transform modifiers for the partitions such as day , year , etc from here: https://iceberg.apache.org/spec/#partition-transforms
• Docs could use some updates to pointers a few things that took me by surprise:
  • Needing to specify s3.S3Instructions multiple times, i.e. catalog creation and then table writer creation as an example. (This is more about mising docs since this wasn't on the docs site I was going off of pydoc and missed that the data_instructions were needed.)
  • Clarifying where/when s3.S3Instructions are used vs the configuration supplied in properties would be helpful for folks especially if they need to go to a different provider than AWS.
  • Clarify that custom JAR's are needed to use anything but the rest/glue catalogs in AWS.
  • Clarify that when you pass partition_paths into the writer instance it actually generates new columns based on those paths, and that the columns should be removed from the table before calling write.