Transitous-mirrored GTFS ZIP archive contains two identically named files

[derhuerst]

Note sure if this is a symptom of an actual underlying problem that should be fixed, or just a random blip that we'll never encounter again. I thought I'll just document this here in case someone stumbles upon something similar.

There are two entries called attributions.txt in the Transitous-processed (and gtfsclean-ed). Also note the modification/creation dates/times.

wget 'https://api.transitous.org/gtfs/jp_tokyo-rail.gtfs.zip' -U derhuerst -O /tmp/jp-tokyo.gtfs.zip
sha256sum /tmp/jp-tokyo.gtfs.zip
# a4afc501bd248f0f8c936d522cdafb25b04d489fd8e92d921332983b0972ac9a  /tmp/jp-tokyo.gtfs.zip
unzip -v /tmp/jp-tokyo.gtfs.zip
# Archive:  /tmp/jp-tokyo.gtfs.zip
#  Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
# --------  ------  ------- ---- ---------- ----- --------  ----
#     3878  Defl:N     1332  66% 00-00-1980 00:00 a0e9a361  agency.txt
#       97  Defl:N       84  13% 00-00-1980 00:00 35117809  feed_info.txt
#   206683  Defl:N    66701  68% 00-00-1980 00:00 2ba7202f  stops.txt
# 17362401  Defl:N  3555012  80% 00-00-1980 00:00 ba6d5194  shapes.txt
#    13013  Defl:N     4640  64% 00-00-1980 00:00 d24b410a  routes.txt
#      257  Defl:N      123  52% 00-00-1980 00:00 5788b045  calendar.txt
#     1136  Defl:N      223  80% 00-00-1980 00:00 9995ecc4  calendar_dates.txt
# 11180504  Defl:N   801214  93% 00-00-1980 00:00 2702f290  trips.txt
# 109804491  Defl:N  8307244  92% 00-00-1980 00:00 eb360951  stop_times.txt
#  1898240  Defl:N   263605  86% 00-00-1980 00:00 12106c04  transfers.txt
#      283  Defl:N      210  26% 00-00-1980 00:00 d71c0b46  attributions.txt
#      300  Defl:N      218  27% 00-00-1980 00:00 bdb2156d  attributions.txt
# 39805428  Defl:N  1995898  95% 00-00-1980 00:00 4769d379  translations.txt
# --------          -------  ---                            -------
# 180276711         14996504  92%                            13 files

wget 'https://mkuran.pl/gtfs/tokyo/rail.zip' -U derhuerst -O /tmp/jp-tokyo2.gtfs.zip
sha256sum /tmp/jp-tokyo2.gtfs.zip
# 562ed6f50524604c0c9c589c55507e9761ff4e35df40c03c97157d4f2a82d784  /tmp/jp-tokyo2.gtfs.zip
unzip -v /tmp/jp-tokyo2.gtfs.zip
# Archive:  /tmp/jp-tokyo2.gtfs.zip
#  Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
# --------  ------  ------- ---- ---------- ----- --------  ----
#     3922  Defl:N     1324  66% 01-26-2026 18:04 f5bbc4d1  agency.txt
#    13567  Defl:N     4586  66% 01-26-2026 18:04 405d02b2  routes.txt
#   216152  Defl:N    72790  66% 01-26-2026 18:04 8162658c  stops.txt
#      262  Defl:N      120  54% 01-26-2026 18:04 52270b66  calendar.txt
#     1188  Defl:N      225  81% 01-26-2026 18:04 fc339c5c  calendar_dates.txt
# 11271432  Defl:N   825917  93% 01-26-2026 18:04 66f76f75  trips.txt
# 120660000  Defl:N 11445351  91% 01-26-2026 18:04 7051b2d2  stop_times.txt
# 27223785  Defl:N  6522763  76% 01-26-2026 18:04 5daa311e  shapes.txt
#      303  Defl:N      218  28% 01-26-2026 18:04 2d20e2f9  attributions.txt
#       99  Defl:N       83  16% 01-26-2026 18:04 83fd8580  feed_info.txt
# 40771922  Defl:N  2103697  95% 01-26-2026 18:04 80ccfaf2  translations.txt
#  1925133  Defl:N   180795  91% 01-26-2026 18:04 dcbfdc98  transfers.txt
# --------          -------  ---                            -------
# 202087765         21157869  90%                            12 files

Maybe even the upstream feed was broken for a brief time @MKuranowski ?

[MKuranowski]

I highly doubt it was an issue on my side, the files are 1) first held on a disk before being compressed [1] and 2) file names are held as dictionary keys [2]. Both of these prevent the same file from being visited twice.

I was able to download the two exact copies of the files, and it simply looks like somewhere in your pipeline you're appending a modified attributions.txt, instead of replacing it.

Note that unzip -p tokyo_rail_transitous.zip attributions.txt prints both files concatenated together, but you can clearly see that one file is the original, and one has the is_operator column removed and the extra is_data_source column moved to the end. I'm absolutely not qualified to know how your pipeline works, but that sounds like gtfsclean (?) removing a pointless is_operator column.

$ curl -o tokyo_rail_mkuran.zip -L 'https://mkuran.pl/gtfs/tokyo/rail.zip'
$ sha256sum tokyo_rail_mkuran.zip
562ed6f50524604c0c9c589c55507e9761ff4e35df40c03c97157d4f2a82d784  tokyo_rail_mkuran.zip
$ unzip -v tokyo_rail_mkuran.zip 
Archive:  tokyo_rail_mkuran.zip
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
    3922  Defl:N     1324  66% 2026-01-26 18:04 f5bbc4d1  agency.txt
   13567  Defl:N     4586  66% 2026-01-26 18:04 405d02b2  routes.txt
  216152  Defl:N    72790  66% 2026-01-26 18:04 8162658c  stops.txt
     262  Defl:N      120  54% 2026-01-26 18:04 52270b66  calendar.txt
    1188  Defl:N      225  81% 2026-01-26 18:04 fc339c5c  calendar_dates.txt
11271432  Defl:N   825917  93% 2026-01-26 18:04 66f76f75  trips.txt
120660000  Defl:N 11445351  91% 2026-01-26 18:04 7051b2d2  stop_times.txt
27223785  Defl:N  6522763  76% 2026-01-26 18:04 5daa311e  shapes.txt
     303  Defl:N      218  28% 2026-01-26 18:04 2d20e2f9  attributions.txt
      99  Defl:N       83  16% 2026-01-26 18:04 83fd8580  feed_info.txt
40771922  Defl:N  2103697  95% 2026-01-26 18:04 80ccfaf2  translations.txt
 1925133  Defl:N   180795  91% 2026-01-26 18:04 dcbfdc98  transfers.txt
--------          -------  ---                            -------
202087765         21157869  90%                            12 files
$ curl -O tokyo_rail_transitous.zip -L 'https://api.transitous.org/gtfs/jp_tokyo-rail.gtfs.zip'
$ sha256sum tokyo_rail_transitous.zip 
a4afc501bd248f0f8c936d522cdafb25b04d489fd8e92d921332983b0972ac9a  tokyo_rail_transitous.zip
$ unzip -v tokyo_rail_transitous.zip 
Archive:  tokyo_rail_transitous.zip
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
    3878  Defl:N     1332  66% 1980-00-00 00:00 a0e9a361  agency.txt
      97  Defl:N       84  13% 1980-00-00 00:00 35117809  feed_info.txt
  206683  Defl:N    66701  68% 1980-00-00 00:00 2ba7202f  stops.txt
17362401  Defl:N  3555012  80% 1980-00-00 00:00 ba6d5194  shapes.txt
   13013  Defl:N     4640  64% 1980-00-00 00:00 d24b410a  routes.txt
     257  Defl:N      123  52% 1980-00-00 00:00 5788b045  calendar.txt
    1136  Defl:N      223  80% 1980-00-00 00:00 9995ecc4  calendar_dates.txt
11180504  Defl:N   801214  93% 1980-00-00 00:00 2702f290  trips.txt
109804491  Defl:N  8307244  92% 1980-00-00 00:00 eb360951  stop_times.txt
 1898240  Defl:N   263605  86% 1980-00-00 00:00 12106c04  transfers.txt
     283  Defl:N      210  26% 1980-00-00 00:00 d71c0b46  attributions.txt
     300  Defl:N      218  27% 1980-00-00 00:00 bdb2156d  attributions.txt
39805428  Defl:N  1995898  95% 1980-00-00 00:00 4769d379  translations.txt
--------          -------  ---                            -------
180276711         14996504  92%                            13 files
$ unzip -p tokyo_rail_mkuran.zip attributions.txt 
attribution_id,organization_name,attribution_url,is_producer,is_operator,is_authority,is_data_source
1,Schedules: Copyright (c) 2019-2025 Akihiko Kusanagi (under MIT license),https://github.com/nagix/mini-tokyo-3d/,0,0,1,1
$ unzip -p tokyo_rail_transitous.zip attributions.txt 
attribution_id,organization_name,is_producer,is_authority,attribution_url,is_data_source
1,Schedules: Copyright (c) 2019-2025 Akihiko Kusanagi (under MIT license),,1,https://github.com/nagix/mini-tokyo-3d/,1
2,GTFS: Mikołaj Kuranowski,1,1,https://github.com/MKuranowski/TokyoGTFS,1
attribution_id,organization_name,attribution_url,is_producer,is_operator,is_authority,is_data_source
1,Schedules: Copyright (c) 2019-2025 Akihiko Kusanagi (under MIT license),https://github.com/nagix/mini-tokyo-3d/,0,0,1,1
2,GTFS: Mikołaj Kuranowski,https://github.com/MKuranowski/TokyoGTFS,1,0,1,1