Please enable JavaScript to view this site.

WordSmith Tools Help

 

WSImage_572_copy_24

The problem

Sometimes one finds that text files contain duplicate sections, either because the corpus has become corrupted through being copied numerous times onto different file-stores or because they were not edited effectively, e.g. a newspaper has several different editions in the same file. The result can sometimes be that you get a number of repeated concordance lines.

 

Solution

If you choose Edit |Remove Duplicates, Concord goes through your concordance lines and if it finds any two where the  concordance lines match each other (regardless of the filename, date etc.) it will mark one of these for deletion. To establish a match it examines 500 characters centred on the search-word in each line. Every single character including punctuation would need to be identical for a match.

 

Check before you zap...

At the end it will sort all the lines so you can see which ones match each other before you decide finally to zap the ones you really don't want.

 

See also: the Corpus Checker utility's find duplicate contents function which finds near duplicates.

  

Keyboard Navigation

F7 for caret browsing
Hold ALT and press letter

This Info: ALT+q
Nav Header: ALT+n
Page Header: ALT+h
Topic Header: ALT+t
Topic Body: ALT+b
Exit Menu/Up: ESC