Page 1 of 2 12 LastLast
Results 1 to 10 of 42

Hybrid View

Previous Post Previous Post   Next Post Next Post
  1. #1
    Join Date
    Jan 2010
    Posts
    1

    Default Find duplicate files with different names

    Hello all,

    We have bc3 here at my job. We have recently received a large number (90,000) of contracts in pdf. In viewing the contracts we realized that many are duplicated over and over again with different file names. So what I need to do is compare the folder's contents against itself, based on the document's contents NOT the file name.

    Is this something bc3 can handle?

    Thanks!
    Frank

  2. #2
    Join Date
    Oct 2007
    Location
    Madison, WI
    Posts
    4,759

    Default

    Sorry, BC3 doesn't provide duplicate file searching.
    Chris K Scooter Software

  3. #3
    Join Date
    Nov 2004
    Posts
    88

    Default

    I was going to ask a similar question.

    I'm in the process of cleaning up a lot of old files, and the ones that are in my email client (Kmail on Ubuntu/Linux) are in different folders, and have different filenames. KMail creates one file for every email message, so it is quite impossible to track down the dupliate email messages.

    However, I came across a nice little program called 'fdupes", it runs recursively through any path, and stores MD5 for each file, then it must sort to MD5, and shows the (possible) duplicates. I have manually checked a few files that 'fdupes' has picked up, and they are in fact duplicates, and they have very different filenames.

    Now, it would be great if there was some way to feed the results of 'fdupes' into BC3 ??

    I know BC3 folder compare works on exact filename match, but the BC3 filename compare, where there are 2 filenames, ..hmm, can that be run in batch mode, and a list of files fed into it somehow ??

    Peter

  4. #4
    Join Date
    Nov 2004
    Posts
    88

    Default

    Frank, if fdupes would help, and you run Windooze, see if 'fdupes' will run under CYGWIN - http://www.cygwin.com/

    But even if it can pick up all the duplicate PDF's, I'd still want to parse the results through BC3 somehow, just to be 100% sure the file/s were a real duplicate.

    HTH

    Peter

  5. #5
    Join Date
    Oct 2007
    Location
    Bremen, Germany
    Posts
    583

    Default

    One way to do some kind of duplicate file search ist to activate the CRC column in folder-view. You can then sort by CRC values and manually look for duplicates. Duplicates can be compared via context-menu "Compare to... (F7)"
    When the files are spread over different folders you can use the "Flatten Folders" feature.


    Bye
    Christoph

  6. #6
    Join Date
    Nov 2004
    Posts
    88

    Default

    Thanks for the tips; some features there that I didn't know exist. I selected a fairly large path, and then sorted by CRC, and there were quite a few where there where duplicate CRC's, for example see the attachment.

    Although these 2 files were in the same path, there were others that were in different paths, and had different filenames, but the same CRC.

    Now, the big question. Can I be 100% certain that where the same CRC is shown for 2 different filenames, that I can then safely delete one of them (assuming that is the objective, to get rid of duplicate files, where the filename and/or path are different).

    I guess rather than look for duplicate CRC's manually, I could save the folder compare report as plain text, and then parse the file through a script, and look for the CRC (where it is the same as previous line).

    Christoph, your solution would help Frank, to find those duplicate PDF's, where the filename is different.

    Thanks,

    Peter
    Last edited by peterr; 06-Jan-2010 at 05:34 AM.

  7. #7
    Join Date
    Nov 2004
    Posts
    88

    Default

    Quote Originally Posted by peterr View Post
    I guess rather than look for duplicate CRC's manually, I could save the folder compare report as plain text, and then parse the file through a script, and look for the CRC (where it is the same as previous line).
    The path isn't shown as a column, in the Folder Compare Report.

  8. #8
    Join Date
    Oct 2007
    Posts
    437

    Default

    The Folder Compare Report will be fixed to include the path column when appropriate in a future release (probably 3.2).
    Erik Scooter Software

  9. #9
    Join Date
    Oct 2007
    Location
    Bremen, Germany
    Posts
    583

    Default

    Now, the big question. Can I be 100% certain that where the same CRC is shown for 2 different filenames, that I can then safely delete one of them (assuming that is the objective, to get rid of duplicate files, where the filename and/or path are different).
    Using CRC alone brings roughly 99,99999999% sureness that the files are the same. But when you consider the filesize and date+time like in your screenshot then you can be pretty sure.

    Years ago some users suggested to include MD5 calculation besides CRC. But this feature request seems to be a too small entry on the famous internal wishlist of Scootersoftware.


    Bye
    Christoph

  10. #10
    Join Date
    Oct 2007
    Location
    Pennsylvania
    Posts
    1,772

    Default

    I agree. Filesize plus CRC is sufficient to ensure duplicity.
    BC v4.0.7 build 19761
    ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •