What are clashes? Why don't I find clashes when listing but push fails?

[georgewf]

Hi, I've run into a problem...
drive version 0.3.6

I'm trying to push files from an ftp server into google drive. All was working for some weeks. Now, it fails with clashes. When I run the command drive clash list there are no clashes.

When I try -fix-clashes it still fails.
drive push -fix-clashes -no-clobber -no-prompt -destination _FTP_clients

Using -ignore-name-clashes does work but I have no idea what is happening. Is that not transferring clashing files?

[odeke-em]

Hello there @georgewf, thank you for reporting this issue and welcome to drive.

So clashes are the presence of at least two files with the same name in the same folder/directory. Clashes only exist remotely on Google Drive because for object uniqueness, it uses fileIds and names are just a way of accessing objects. However, on our filesystems, files are uniquely identifiable by their paths ie name within a directory so we can only have one file with a unique name in one directory.
This means that if you are pulling down more than one file with the same name say 'f' into a directory say 'a', the last pulled in file overwrites and wins, this behavior is racy and unpleasant.

Clashes are caused by a few ways:

• Explicitly creating new files with the same name in the same folder on Google Drive say with the UI.
• drive itself failing to find out whether a file exists or not. This problem is triggered in events of say 503 e.g in these issues drive pull 503 errors trigger future erroneous conflicts #668, Rate Limit errors trigger full update #480 and others.
• drive new to mimick the freedom that the Google Drive UI gives you, since those files are created remotely.

To try to mitigate this issue, could you firstly try explicitly this

$ drive clashes --list
$ drive clashes --fix

Please provide as many descriptions and error responses that drive gives you back so that I can try to see exactly where it is bugging out. Hope this helps out.

[georgewf]

Hi Emmanuel,

My problem is on pushing to google drive. * -fix-clashes* doesn't seem to
do anything like renaming the file on google drive.

When I tried uploading a folder the drive program would report clashes
exist. But when I run drive clashes --list it reports nothing (maybe
because I'm using -destination on the push operation).

So if I added -ignore-name-clashes then new files would be uploaded but
modified files are not uploaded. Actually, with a modified file present on
local side, the drive program would report everything up to date when
that is not correct or is misleading.

To upload a modified file I would have to remove the -no-clobber option but
that deletes from google drive everything not on the local side.

Anyways to work around this I found a better way. I am now launching the
drive program for every new local file as it is received, instead of
running it as a folder upload after the ftp client disconnects. I am
prohibiting the ftp user from deleting or renaming any files. I hope this
way my team can be certain that everything is transferred from the ftp
server to google drive.

So I guess the main point of this is that I don't understand why
-fix-clashes doesn't do anything when pushing to google drive.

On Sun, Jul 3, 2016 at 9:37 PM, Emmanuel T Odeke notifications@github.com
wrote:

Hello there @georgewf https://github.com/georgewf, thank you for
reporting this issue and welcome to drive.

So clashes are the presence of at least two files with the same name in
the same folder/directory. Clashes only exist remotely on Google Drive
because for object uniqueness, it uses fileIds and names are just a way of
accessing objects. However, on our filesystems, files are uniquely
identifiable by their paths ie name within a directory so we can only have
one file with a unique name in one directory.
This means that if you are pulling down more than one file with the same
name say 'f' into a directory say 'a', the last pulled in file overwrites
and wins, this behavior is racy and unpleasant.

Clashes are caused by a few ways:

• Explicitly creating new files with the same name in the same folder
  on Google Drive say with the UI.
• drive itself failing to find out whether a file exists or not. This
  problem is triggered in events of say 503 e.g in these issues drive pull 503 errors trigger future erroneous conflicts #668
  drive pull 503 errors trigger future erroneous conflicts #668, Rate Limit errors trigger full update #480
  Rate Limit errors trigger full update #480 and others.
• drive new to mimick the freedom that the Google Drive UI gives you,
  since those files are created remotely.

To try to mitigate this issue, could you firstly try explicitly this

$ drive clashes --list
$ drive clashes --fix

Please provide as many descriptions and error responses that drive gives
you back so that I can try to see exactly where it is bugging out. Hope
this helps out.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#680 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/ASnY0R5CyrR37xEe-ccofz7DtHOAnkS5ks5qSGPMgaJpZM4JD9W1
.

George Frehner
Analogue Media Technologies Inc.
Toronto: 602 Gordon Baker Road, North York, ON M2H3B4
tel: 647-478-4273 cell: 647-518-8273

Montreal: 716 Golf Road, Verdun, QC H3E1A8
tel: 514-878-TAPE

[odeke-em]

@georgewf I see. Hope this isn't too much of an inconvenience for your team, I hear ye.
So --fix-clashes is not working well for you probably because of this bug #317 in which not all clashes were found before the next operation so sub-optimally. I haven't verified to see if it still exists.
--ignore-name-clashes is a duct tape solution to try to mitigate scenarios where files are overwritten because we don't know which one exactly to push/pull to e.g when you are pushing from a unique source on your file system to Google Drive on which files with the same name in the same folder exist. This exists because in the past(during the early stages before clash renaming was in), drive would explicitly refuse to deal with any clashing paths remotely as also stated in the manifesto under known issues https://github.com/odeke-em/drive#known-issues, however this strict behavior made it unuseable for many users so that solution was a compromise.

Please use the newer drive clashes commands for listing and fixing up clashes.

drive-clashes

$ drive clashes --list clash-test
These paths clash
X /clash-test/t3 0Bwu8laYc9RTPQzlPTm5zaDd4bDQ
X /clash-test/t3 13SvxP_di0SfYDBThPk9pbruY5we7272Uy3NGMNPwW7Q
X /clash-test/t2 0Bwu8laYc9RTPblZkb181TXQ1Rk0
X /clash-test/t2 1w151wYvKpfc-V9yhZIfmBmV6KyvzoHXeVtiGGuFsuqY
X /clash-test/t1 0Bwu8laYc9RTPdmRSWk51S0d6emc
X /clash-test/t1 16CX6di4qz9kTi3bAyu2375yMhDMBZEPsoVCPsiQljBU

$ drive clashes --fix clash-test
Some clashes found, we can fix them by following renames:
/clash-test/t1 16CX6di4qz9kTi3bAyu2375yMhDMBZEPsoVCPsiQljBU -> t1_0
/clash-test/t3 13SvxP_di0SfYDBThPk9pbruY5we7272Uy3NGMNPwW7Q -> t3_0
/clash-test/t2 1w151wYvKpfc-V9yhZIfmBmV6KyvzoHXeVtiGGuFsuqY -> t2_0
Proceed with the changes [Y/N] ? 

Also you can perform some operations by fileId e.g pulling

$ drive pull --id 0fM9rt0Yc9RTPaDdsNzg1dXVjM0E 0fM9rt0Yc9RTPaTVGc1pzODN1NjQ 0fM9rt0Yc9RTPV1NaNFp5WlV3dlU

I use fileIds when I have to inspect specific files.
Most of the

[tgkprog]

Just wanted to say I see this issue too on ubuntu. Version 0.3.7. Another uses pushes changes (from windows). When I pull with --fix-clashes it still complains.

drive_linux pull --fix-clashes <aPath>