Show/Hide Toolbars

WordSmith Tools Manual

Navigation: Concord > editing concordances

Remove duplicates ()

Scroll Prev Top Next More


The problem

Sometimes one finds that text files contain duplicate sections, either because the corpus has become corrupted through being copied numerous times onto different file-stores or because they were not edited effectively, e.g. a newspaper has several different editions in the same file. The result can sometimes be that you get a number of repeated concordance lines.



If you choose Edit |Remove Duplicates, Concord goes through your concordance lines and if it finds any two where the  concordance lines match each other (regardless of the filename, date etc.) it will mark one of these for deletion. To establish a match it examines 500 characters centred on the search-word in each line. Every single character including punctuation would need to be identical for a match.


Check before you zap...

At the end it will sort all the lines so you can see which ones match each other before you decide finally to zap the ones you really don't want.