The tool works in any language. It checks consistency by using known ultra high frequency words and seeking them in your corpus.

 

1. Choose a set of words which ought to be present in virtually any text in the language you're working with.

2. Choose your corpus head folder and check the "include sub-folders" box if your corpus spreads over that folder and sub-folders.

3. Decide on the percentage of the high-consistency words you wish to require. Here I chose 50%.

4. Press Start.

 

Of the thousands of texts in the corpus concerned with water we have identified three which look 'inconsistent', not 'ordinary' texts.

 

Click on any to see what the program found. Here the first text seems to be just a list of possible Christmas gifts. Of the top 12 words 8 were missing. Text 2 also concerned gift ideas and text 3 lists birthdays of well-known people.

 

High consistency words

For English, THE,AND,TO,IN,A,OF,FOR,THAT,ON,IT,IS and WITH are all reliably found in over 95% of corpus texts. For another language, make a word list and check out the Text % column to identify suitable items.