Mar-19-2019, 11:58 PM
(Mar-19-2019, 11:43 PM)Truman Wrote: [ -> ]Wouldn't it be more 'fair' that each word has an equal chance of being selected?
What's fair got to do with it? He wants to create a list of words with the same distribution as that for the original text. Not all the words are used the same number of times in the original text. And you'll note that he's doing it by pairs of words. So you start with 'I', and then pick a word based on it's probability of occurring after 'I' in the original text. Say you pick 'can'. Next you pick the third word based on it's probability of occurring after 'can' in the original text. And so on. This gives you a random sentence that matches the conditional probabilities of the random text. It will probably almost but not really make sense. If you chose the words with the same probability, without looking at what words come after other words, it would be complete gibberish.