Menu

scScript Pattern Callbacks

The previous section illustrated three of the four common techniques for getting match results:

  1. Use match count returned by Pat.Match itself.
  2. Read PDict after Stash Directives store info/results.
  3. Process results with SFunc callbacks invoked by Pats.
  4. Process results with RFunc callbacks invoked by Pat.Match call.

RFunc and SFunc serve a similar purpose, both are callbacks used by Pat.Match to inform the scScript and process the match. But SFunc is often called redundantly for #Cons matches that are later invalidated, while RFunc is only invoked for (finalized) successful top-level matches.

SFunc, like other stash directives in individual Pats, are invoked when the particular Pat succeeds. But the Pat may have a #Cons flag and therefore match only provisionally, at the whim of subsequent Pats. So if a subsequent Pat fails, the Match mechanism will backtrack, consider the previous success (that called the SFunc) invalid, and look at another match for the Pat. This cannot be avoided because SFunc and other stash directives for a #Cons Pat cannot be delayed until subsequent ones are resolved, as those later Pats may in fact rely on preceding stash directives for their template data (#Dyn data) or PFunc. But the process of searching, going down blind alleys, and reversing course for a #Cons Pat may be too much for the calling scScript (and its callbacks). A simple script expects only final information and cannot handle the to and fro that comes from #Cons pattern matching.

scScript provides an RFunc callback to avoid this problem. When an RFunc is present, Pat.Match will maintain an internal RStack data structure to record what was matched and in what order. This RStack will carry the burden of searching and backtracking, always storing the final path in the successful match, as invalidated values are popped off. So after the whole match is done, Pat.Match calls RFunc iteratively on each element of the RStack, giving the calling script the illusion of a straightforward (prescient!!) path without any backtracking.

There are a few operational points when matching complex multi-level Pats:

An excellent variation of the previous examples, that will (eventually) illustrate the dangers of SFunc calls, is counting comment words in C source files (and now you know why the sample data is C code and not a chapter of War and Peace). There are two comment types in C, // initiated line comments as well as /* and */ delimited block comments. Naturally, comment delimiters should be ignored if in a string, just as string delimiters are ignored in a comment.

A logical way to start such an endeavor is by writing a Pat outline first:
PStr = Pat.Span('"', '"'); PLCom = Pat.Span("//", "\n"); PBCom = Pat.Span("/*", "*/"); PCCnt = Pat.Alt(PStr, PLCom, PBCom);

The repeated application of PCCnt (in #Skip mode) will find all PLCom and PBCom without looking inside C strings! But there are two problems:

  1. C sources may also include char specs for quotes ('"' or '\"').
  2. Strings may include escaped quotes!

Problem (1) can be easily solved by manually including char specs (2 versions) as the first Alt args, so as to catch them first before they can gum up the works:
PCCnt = Pat.Alt(|>'"'<|, |>'\"'<|, PStr, PLCom, PBCom);

The long string format is used here to avoid an extra escape layer required in short string formats. Otherwise the backslash itself (and possibly the double quote) in the second string would need an escape. It would have to be written as '\\"' or worse "\\\"", making it very hard to read. (Shades of RE!)

The solution for problem (2) starts with a more flexible Pat to eat chars between quotes. Then it becomes (seemingly) easy to match a C Str despite included escaped quotes:
PAnyC = Pat.Rep(0, 0, Pat.Any()); PStr = Pat.Seq('"', PAnyC, Pat.#Neg, |>\"<|, Pat.#Pos, '"');

The idea is to start with one quote, skip any \", but stop at the closing quote. Sadly, the clumsy formulation above will fail, even if there is only a single escaped quote! PAnyC will expand to eat just the backslash "\", leaving the quote immediately following it to prematurely end the string.

A working, but convoluted, approach would be:
PStr = Pat.Seq('"', PAnyC, Pat.Rep(0, 0, |>\"<|, PAnyC), '"');

This breaks the target string into a number of segments, each separated with an escaped double quote. So the inner Rep will match any number of segments following the initial one, such as:
"----------", "----\"----", "----\"----\"----" ...

The segmenting approach works, but an elegant and slightly faster solution uses Alt to specifically eat escaped quotes (\") inside the C String: (given in SampleSC/Pat3.sc)
PInStr = Pat.Rep(0, 0 Pat.Alt(|>\"<|, Pat.Any())); PStr = Pat.Seq(|>"<|, PInStr, |>"<|); PLCom = Pat.Span("//", "\n"); PBCom = Pat.Span("/*", "*/"); PPCnt = Pat.Alt(|>'"'<|, |>'\"'>|, PStr, PLCom, PBCom);

Note: Pat.Match(BN, PPCnt, Pat.#Skip) will return a large number, it counts all hits on PPCnt!

This works well and leaves PLCom and PBCom to capture comments for word counting. Word counts are easy, they were done before:
PLetter = Pat.Char(Pat.#Alpha, "_"); PWord = Pat.Rep(1, 0, Pat.#Self + Pat.#Max, PLetter, Pat.#S_Func, WriteWord); PNotWord = Pat.Rep(0, 0, Pat.#Self + Pat.#Max + Pat.#Neg, PLetter, Pat.#Pos, Pat.Go(1)); PCount = Pat.Rep(0, 0, Pat.#Max, PNotWord, PWord, Pat.#S_Cnt, "WCnt");

It seems easy to transform PLCom/PBCom for counting:
PLCom = Pat.Seq("//", PCount, "\n"); PBCom = Pat.Seq("/*", PCount, "*/");

But this approach will fail for subtle but important reasons:


  1. PCount has to be #Cons as it has to bridge from the opening delimiter to the closing one in PLCom/PBCom. So PCount first tries 0 words, then 1, then 2... until finally reaching where the calling Seq would match the closing delimiter. But each time PCount tries a Rep, it will invoke its SFunc thinking it has succeeded, only for the calling Seq to fail and dash its hopes.

  2. PNotWord also has a problem. It is #Self and #Max, so what stops it from eating the closing delimiters? A comment line can end with a punctuation (or extra space) char; in that case PNotWord will read it, the subsequent Newline, and the beginning comment characters of the next line! It is also quite common in C to have line comments with no words at all, just a lot of stars, dashes, or other ornamentation.

One solution is to restore PLCom/PBCom back to Span again, give them both an SFunc (not a problem as both Spans will be #Self), and launch a new Match from inside the SFunc:
PInStr = Pat.Rep(0, 0, Pat.Alt(|>\"<|, Pat.Any())); PStr = Pat.Seq(|>"<|, PInStr, |>"<|); PLCom = Pat.Span("//", "\n", Pat.#S_Func, CountWords); PBCom = Pat.Span("/*", "*/", Pat.#S_Func, CountWords); PPCnt = Pat.Alt(|>'"'<|, |>'\"'<|, PStr, PLCom, PBCom); Function CountWords()() { Var Range = Pat.GetRange(); Var PosS = Range & 0x00FFFFFFFF; Var PosE = Range >> 32; WrdCnt += Pat.Match(BN, PWord, Pat.#Skip, PosS, PosS, PosE); Return 1; }

Warning: It is imperative to include "Return 1;" at the end of every callbacks. Otherwise, the match will fail and the results will be hard to comprehend and debug! Details of Pat.Match behavior are generally quite cryptic, but can be illuminated by turning on debugging flags in scScript sources.

The above relies on Match being re-entrant and working from inside (one of its own) callbacks. But this is very much in the spirit of C, just break the problem down and let brute force have its day without much further thought from the script writer.

Another, perhaps more elegant, solution is to introduce a PNLet, where PCount happily loops on non-letters (individually) or PWord Pats:
PInStr = Pat.Rep(0, 0, Pat.Alt(|>\"<|, Pat.Any())); PStr = Pat.Seq(|>"<|, PInStr, |>"<|); PLetter = Pat.Char(Pat.#Alpha, "_"); PNLet = Pat.Seq(Pat.#Neg, PLetter, Pat.#Pos, Pat.Go(1)); PWord = Pat.Rep(1, 0, Pat.#Self + Pat.#Max, PLetter, Pat.#S_Func, GotAWord); PCount = Pat.Rep(0, 0, Pat.Alt(PNLet, PWord)); PLCom = Pat.Seq("//", PCount, "\n"); PBCom = Pat.Seq("/*", PCount, "*/"); PPCnt = Pat.Alt(|>'"'<|, |>'\"'<|, PStr, PLCom, PBCom);

Each call to GotAWord would simply increment the word count! This is dead simple as PWord is #Self and matches with finality. (Incidentally, a call to Pat.GetCnt() from inside GotAWord would return the char count in the given word. So one could tally exactly what percentage of the source file has been allocated to comments!)

The #Neg logic flag, used here and previously, seems quite simple, but introduces some surprising complexity, especially when applying to multiple args. The simple Pat:
P1 = Pat.Seq("A", "B", "C", Pat.#Neg, "D");

will happily match "ABC", but not "ABCD". In that sense P1 does not care if "ABC" is followed by another character, as long as it is not "D".

But things get more complicated with:
P2 = Pat.Seq("A", "B", "C", Pat.#Neg, "D", "E", "F");

Once again it will happily match "ABC". But rather than specifically excluding "ABCDEF" as one might naively expect, P2 really excludes "ABCD" (and by extension "ABCDEF"), "ABCE", and "ABCF". So while the Seq forms a conjunction of its #Pos templates, it forms a disjunction of its #Neg ones! Perhaps surprising, this is in fact quite logical and practical, allowing:
P3 = Pat.Seq("A", "B", "C", Pat.#Neg, "D", "E", "F", #Pat.Pos, Pat.Any(), "X", "Y", "Z");

to match any string of the form "ABC*XYZ" as long as * is not "D", "E", or "F". If one really does want "ABC" but not "ABCDEF" (the naive initial expectation of P2), it can be written as:
P4 = Pat.Seq("A", "B", "C", Pat.#Neg, Pat.Seq("D", "E", "F"));

so the inner Pat.Seq creates a conjunction of "DEF", which is then explicitly negated en masse for the outer Seq. Of course, this trivial case is best written as:
P4a = Pat.Seq("ABC", Pat.#Neg, "DEF");

#Neg flags are seldom used with Alt and Span because exclusionary templates are often too willing to match on their own. But this excess promiscuity is greatly mitigated in #Cons matches where sub-patterns effectively become part of a larger ensemble that must match.

Exclusion, and therefore #Neg flags, do play a large role in Char Pats. Each such Pat is really just a set of characters, and it is often convenient to define a larger set first, only to then exclude a few exceptions. While Char should logically require a #Pos set before losing characters to #Neg, it is smart enough to assume a base of #All characters if its definition starts with an exclusion (but this too can lead you astray if that was not your intention, especially when dealing with #Smart root considerations for UTF-8 chars).

#Smart and #Exact flags were late additions to the pattern matcher. These became necessary as scEmacs was expanded to understand Latin Unicode chars. So scScript now uses an internal table that represents accented chars, their root ASCII letters, and whether they are upper or lower case. As a result, the system can match entree against entrée or indeed against ĔņŤŖēĖ. The default #Smart flag means root (base ASCII) letters in a Pat can match their accented brethren (subject to case #Sen or #Insen flags) as well as their plain unadorned versions. But #Exact forces base (unaccented) letters to only match identical (again subject to case flags) base letters. Whether #Smart or #Exact, accented letters in the Pat can only match identically accented ones in data; so entrée will always match entrée and never entree. (If explicitly excluding a Unicode char with a Char Pat, make it #Neg and #Exact, otherwise its root may match anyway!)