scScript Pattern Counting

A good exercise is using Pats to count words in a document. A word can be (over) simplified as one or more contiguous alphanumeric letters (plus underscore for us programmers). These are then separated by a non-word consisting of one or more contiguous non-letter characters. Obviously, the buffer may start with a non-word (or not), so it is best to set the PNotWord Min to 0 instead of 1. PLetter = Pat.Char(Pat.#Alpha, "_"); PWord = Pat.Rep(1, 0,Pat.#Self + Pat.#Max, PLetter); PNotWord = Pat.Rep(0, 0, Pat.#Self + Pat.#Max, Pat.Char(Pat.#Neg, PLetter));

PWord is defined as one or more contiguous PLetter characters, and PNotWord (the stuff between words) is defined as zero or more contiguous non-PLetter characters. Note the use of #Neg in the definition of PNotWord. Both Reps are #Self and #Max, they just need to eat all that matches without worrying about the next Pat.

Pat.Char cannot accept PFunc callbacks. These Pats look at one character at a time and decide whether it is in their inclusion set. Not only would the semantics of a PFunc callback not be clear, the performance hit of invoking it many thousands of time, even on small data files, would be prohibitive.

Pat.Char only maintain inclusion sets of characters, not exclusion sets! When defining a Char Pat, #Neg chars (from a Str or another Char Pat) should only come after the #Pos ones. In a mathematical analogy, it only stores a positive sum; so positive numbers must be added in before the negative ones! Fortunately, scScript is smart enough to automatically include #All characters if the very first Str/Pat is #Neg. Char can accept an #Exact flag, but is #Smart by default, so À in data will match against A if À itself is not included in the Char.

(While it is theoretically possible to also maintain separate exclusion sets in Pat.Char, it would double the workload and present semantic issues of priority in case of conflict!)

A very efficient way to count words is to ignore non-words and just let Pat.Match scan the buffer and do the counting, it will simply skip over any word gaps: N = Pat.Match(BN, PWord, Pat.#Skip);

This function will scan forward to match PWord, then skip the matched word and scan until it hits again, this repeats until the buffer end. The returned Match count will in fact be the word count! But it is critical to use #Skip, as #Slide would count each word multiple times (once for each letter!).

The distinction between #Skip and #Slide emphasizes the importance of procedural understanding when pattern matching, another reason why obfuscating or concealing the process is counter productive. Programmers (especially C programmers) need to understand the operational paradigm, what things mean and how they operate; hiding this behind a syntactic formulation (or opaque object!) just leads to problems. One can also use clever defaults and aids to catch common problems, but they will complicate the user paradigm and confuse even more when there is the inevitable failure.

It is also possible to have a self-scanning PCount pattern: PCount = Pat.Rep(0, 0, Pat.#Max, PNotWord, PWord, Pat.#S_Cnt, "WCnt");

Rep accepts a core Pat or Str to repeat. But it also accepts more than one and will implicitly sequence them, just like Seq Pats. Here, this works as if Pat.Rep was given Pat.Seq(PNotWord, PWord)!

Each PCount repetition first eats up the non-word letters, then matches a full word. The goal is to have as many reps as possible, so it is #Max. The default for Rep is #Cons, but there are no subsequent Pats, so nothing to be considerate of!

The actual match for PCount, like all #Anchor matches, is quite efficient: Pat.Match(BN, PCount, Pat.#Anchor)(PDict);

But PDict is a required out arg in Pat.Match because the desired word count will be stashed there under "WCnt". Once the function returns, the script can simply print out PDict.WCnt or assign it to a variable.

The file Pat2.sc in SampleSC lists a few variations of this match to experiment with. But note that PWord and PNotWord are exact opposites only for ASCII characters. Things get more complicated for higher Unicode (UTF-8) characters, especially since the default #Smart conversion is in effect here. For example Ā is obviously not in #Alpha (as it is not LowASCII), so the #Smart regime will look at its root (A) and match it successfully for PWord. But PNotWord is explicitly the set of all chars, except #Alpha, so will positively match Ā itself. The system will therefore not bother to extract the root and simply declare Ā as being in PNotWord. Pat2.sc lists a very simple way to get the exact (desired) symmetry, so Unicode characters do not end up being in PWord and PNotWord at the same time:
PNotWord = Pat.Rep(0, 0, Pat.#Self + Pat.#Max + Pat.#Neg, PLetter, Pat.#Pos, Pat.Go(1));
(A positive Go(1) is required to advance, Rep would just be an infinite loop without it!!)

A good way to test such Pats during development is to have it write the matching words to *Output*. This can be achieved easily with a callback SFunc: Function WriteWord()() {Write(Pat.GetStr(), " "); Return 1;}

Finally, PWord must be modified to invoke WriteWord after each successful match: PLetter = Pat.Char(Pat.#Alpha, "_"); PWord = Pat.Rep(1, 0, Pat.#Self + Pat.#Max, PLetter, Pat.#S_Func, WriteWord); PNotWord = Pat.Rep(0, 0, Pat.#Self + Pat.#Max + Pat.#Neg, PLetter, Pat.#Pos, Pat.Go(1)); PCount = Pat.Rep(0, 0, Pat.#Max, PNotWord, PWord, Pat.#S_Cnt, "WCnt"); Pat.Match(BN, PCount, Pat.#Anchor, 0, 0, 1186)(PDict);

It is a very good idea to limit the range of a test match, especially one that prints a lot of info, when dealing with large files (Sample_BigC.txt in SampleSC is a huge file). And if word-by-word verification is desired, WriteWord can be made to call Ed.QueryMC!

While writing word after word to *Output* will significantly slow things down, it also provides an opportunity to abort (Ctrl-G) out of the process. The exact timing will vary, but this tends to occur during the SFunc callback (WriteWord) of PWord. If aborted, PCount will never succeed (and finalize), so WCnt will remain 0. If the count is critical, then WriteWord itself must record a running tally.