Format of 'flist'

[AterLuna]

I want to train network with new dataset.
For training, I tried to modify inpaint.yml file, but I'm not sure about how to set dataset path.
It seems that I have to add new dataset as 'flist' file to DATA_FLIST, but I cannot find how to make appropriate 'flist' files.
Is there any reference for flist file format, or some examples of them?

[TrinhQuocNguyen]

Hi AterLuna,
You have to write code for yourself to generate the flist file. Here is my code:

#!/usr/bin/python

import argparse
import os
from random import shuffle

parser = argparse.ArgumentParser()
parser.add_argument('--folder_path', default='./training_data', type=str,
                    help='The folder path')
parser.add_argument('--train_filename', default='./data_flist/train_shuffled.flist', type=str,
                    help='The output filename.')
parser.add_argument('--validation_filename', default='./data_flist/validation_shuffled.flist', type=str,
                    help='The output filename.')
parser.add_argument('--is_shuffled', default='1', type=int,
                    help='Needed to shuffle')

if __name__ == "__main__":

    args = parser.parse_args()

    # get the list of directories
    dirs = os.listdir(args.folder_path)
    dirs_name_list = []

    # make 2 lists to save file paths
    training_file_names = []
    validation_file_names = []

    # print all directory names
    for dir_item in dirs:
        # modify to full path -> directory
        dir_item = args.folder_path + "/" + dir_item
        # print(dir_item)

        training_folder = os.listdir(dir_item + "/training")
        for training_item in training_folder:
            training_item = dir_item + "/training" + "/" + training_item
            training_file_names.append(training_item)

        validation_folder = os.listdir(dir_item + "/validation")
        for validation_item in validation_folder:
            validation_item = dir_item + "/validation" + "/" + validation_item
            validation_file_names.append(validation_item)
    # print all file paths
    for i in training_file_names:
        print(i)
    for i in validation_file_names:
        print(i)

    # This would print all the files and directories

    # shuffle file names if set
    if args.is_shuffled == 1:
        shuffle(training_file_names)
        shuffle(validation_file_names)

    # make output file if not existed
    if not os.path.exists(args.train_filename):
        os.mknod(args.train_filename)

    if not os.path.exists(args.validation_filename):
        os.mknod(args.validation_filename)

    # write to file
    fo = open(args.train_filename, "w")
    fo.write("\n".join(training_file_names))
    fo.close()

    fo = open(args.validation_filename, "w")
    fo.write("\n".join(validation_file_names))
    fo.close()

    # print process
    print("Written file is: ", args.train_filename, ", is_shuffle: ", args.is_shuffled)

[JiahuiYu]

@TrinhQuocNguyen Thanks for you response. @AterLuna In addition, the format of file lists are attached. You can either use absolute path or relative path for training.

/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00027049.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00017547.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00023248.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00029613.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00007055.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00021404.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00008928.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00003579.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00010811.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00014556.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00015131.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00015634.png
...
...

[AterLuna]

@TrinhQuocNguyen Thank you for sharing your codes. I'll try it with my dataset.
@JiahuiYu Thank you for your example.

[TrinhQuocNguyen]

@AterLuna , Supposed you want to use my code, you need to make a new directory named training_data. In that directory, you make 2 more new directories: training and validation respectively and put your images in those.
@JiahuiYu You are super welcome, thank you for your contribution.

[javanli]

@TrinhQuocNguyen
In your code ,line 41,for validation_item in training_folder:
maybe it should be for validation_item in validation_folder?

[TrinhQuocNguyen]

I have modified a little bit for anyone needs it. Moreover, you may apply "data augmentation" and put them into other folders ( folder1, folder2,...) without being messy 😄
The directory tree should be looked like:

- model_logs
- neuralgym_logs
- training_data
  -- training
    --- <folder1>
    --- <folder2>
    --- .....
  -- validation
    --- <val_folder1>
    --- <val_folder2>
    --- .....
- <this_file.py>

import argparse
import os
from random import shuffle

parser = argparse.ArgumentParser()
parser.add_argument('--folder_path', default='./training_data', type=str,
                    help='The folder path')
parser.add_argument('--train_filename', default='./data_flist/train_shuffled.flist', type=str,
                    help='The train filename.')
parser.add_argument('--validation_filename', default='./data_flist/validation_shuffled.flist', type=str,
                    help='The validation filename.')
parser.add_argument('--is_shuffled', default='1', type=int,
                    help='Needed to be shuffled')

if __name__ == "__main__":

    args = parser.parse_args()

    # get the list of directories and separate them into 2 types: training and validation
    training_dirs = os.listdir(args.folder_path + "/training")
    validation_dirs = os.listdir(args.folder_path + "/validation")

    # make 2 lists to save file paths
    training_file_names = []
    validation_file_names = []

    # append all files into 2 lists
    for training_dir in training_dirs:
        # append each file into the list file names
        training_folder = os.listdir(args.folder_path + "/training" + "/" + training_dir)
        for training_item in training_folder:
            # modify to full path -> directory
            training_item = args.folder_path + "/training" + "/" + training_dir + "/" + training_item
            training_file_names.append(training_item)

    # append all files into 2 lists
    for validation_dir in validation_dirs:
        # append each file into the list file names
        validation_folder = os.listdir(args.folder_path + "/validation" + "/" + validation_dir)
        for validation_item in validation_folder:
            # modify to full path -> directory
            validation_item = args.folder_path + "/validation" + "/" + validation_dir + "/" + validation_item
            validation_file_names.append(validation_item)

    # print all file paths
    for i in training_file_names:
        print(i)
    for i in validation_file_names:
        print(i)

    # shuffle file names if set
    if args.is_shuffled == 1:
        shuffle(training_file_names)
        shuffle(validation_file_names)

    # make output file if not existed
    if not os.path.exists(args.train_filename):
        os.mknod(args.train_filename)

    if not os.path.exists(args.validation_filename):
        os.mknod(args.validation_filename)

    # write to file
    fo = open(args.train_filename, "w")
    fo.write("\n".join(training_file_names))
    fo.close()

    fo = open(args.validation_filename, "w")
    fo.write("\n".join(validation_file_names))
    fo.close()

    # print process
    print("Written file is: ", args.train_filename, ", is_shuffle: ", args.is_shuffled)

[ChiediVenia]

I have modified a little bit for anyone needs it. Moreover, you may apply "data augmentation" and put them into other folders ( folder1, folder2,...) without being messy smile
The directory tree should be looked like:

- model_logs
- neuralgym_logs
- training_data
  -- training
    --- <folder1>
    --- <folder2>
    --- .....
  -- validation
    --- <val_folder1>
    --- <val_folder2>
    --- .....
- <this_file.py>

import argparse
import os
from random import shuffle

parser = argparse.ArgumentParser()
parser.add_argument('--folder_path', default='./training_data', type=str,
                    help='The folder path')
parser.add_argument('--train_filename', default='./data_flist/train_shuffled.flist', type=str,
                    help='The train filename.')
parser.add_argument('--validation_filename', default='./data_flist/validation_shuffled.flist', type=str,
                    help='The validation filename.')
parser.add_argument('--is_shuffled', default='1', type=int,
                    help='Needed to be shuffled')

if __name__ == "__main__":

    args = parser.parse_args()

    # get the list of directories and separate them into 2 types: training and validation
    training_dirs = os.listdir(args.folder_path + "/training")
    validation_dirs = os.listdir(args.folder_path + "/validation")

    # make 2 lists to save file paths
    training_file_names = []
    validation_file_names = []

    # append all files into 2 lists
    for training_dir in training_dirs:
        # append each file into the list file names
        training_folder = os.listdir(args.folder_path + "/training" + "/" + training_dir)
        for training_item in training_folder:
            # modify to full path -> directory
            training_item = args.folder_path + "/training" + "/" + training_dir + "/" + training_item
            training_file_names.append(training_item)

    # append all files into 2 lists
    for validation_dir in validation_dirs:
        # append each file into the list file names
        validation_folder = os.listdir(args.folder_path + "/validation" + "/" + validation_dir)
        for validation_item in validation_folder:
            # modify to full path -> directory
            validation_item = args.folder_path + "/validation" + "/" + validation_dir + "/" + validation_item
            validation_file_names.append(validation_item)

    # print all file paths
    for i in training_file_names:
        print(i)
    for i in validation_file_names:
        print(i)

    # shuffle file names if set
    if args.is_shuffled == 1:
        shuffle(training_file_names)
        shuffle(validation_file_names)

    # make output file if not existed
    if not os.path.exists(args.train_filename):
        os.mknod(args.train_filename)

    if not os.path.exists(args.validation_filename):
        os.mknod(args.validation_filename)

    # write to file
    fo = open(args.train_filename, "w")
    fo.write("\n".join(training_file_names))
    fo.close()

    fo = open(args.validation_filename, "w")
    fo.write("\n".join(validation_file_names))
    fo.close()

    # print process
    print("Written file is: ", args.train_filename, ", is_shuffle: ", args.is_shuffled)

I created the folder tree, I put the images in the folders, I launch the file with your code and I return this:

./training_data/training/folder1/asd.jpg
./training_data/training/folder6/vdikkj.jpeg
./training_data/training/folder4/vdb.jpeg
./training_data/training/folder2/images.jpeg
./training_data/training/folder3/scv.jpeg
./training_data/training/folder5/waq.jpeg
./training_data/training/folder8/123.jpeg
./training_data/training/folder9/das.jpeg
./training_data/training/folder7/index.jpeg
./training_data/validation/val_folder6/vdikkj.jpeg
./training_data/validation/val_folder9/das.jpeg
./training_data/validation/val_folder1/asd.jpg
./training_data/validation/val_folder2/images.jpeg
./training_data/validation/val_folder7/index.jpeg
./training_data/validation/val_folder3/scv.jpeg
./training_data/validation/val_folder5/waq.jpeg
./training_data/validation/val_folder8/123.jpeg
./training_data/validation/val_folder4/vdb.jpeg
Traceback (most recent call last):
File "this_file.py", line 58, in
os.mknod(args.train_filename)
FileNotFoundError: [Errno 2] No such file or directory

Can you help me?please.

[JiahuiYu]

@ChiediVenia Please make sure that your input file exists. Normally it can be fixed by a careful check.