Version 1.16 of VitRegex is released
Having been away for a break, I came back with fresh eyes and found a bunch more improvements and optimisations.
- where there were invalid character-class ranges like [z-a] or [7-4] or
[\x61-\x41] the code silently matched nothing as the bitmap setting loop
would not execute and those characters were silently excluded. I decided
that this could lead to hard-to-diagnose bugs for users so the compiler
has been changed to throw an error and the compilation fails hence forcing
the user (programmer) to correct the regex. The message is of the form:
[VitRegex.compileRegex] Invalid character class range:
start ('zz') > end ('xx') in: <regex>
where zz and xx are the decimal values of the characters.
- in extended mode comments are skipped (ie. between # and EOL). The code
for this has been tightened up to consume LF after CR in CRLF typical
Windows style EOL character pair.
- also in extended mode we now skip unescaped whitespace (space, tab, CR,
LF) inside character classes
- previously nested named groups with the same name were not reported as a
compile error. We now check for duplicate named groups outside of
alternation and give a compile error when found. Note that PCRE allows
duplicate named groups in alternation branches ((?<n>...) | (?<n>...))
which seems reasonable so we have adopted the same approach here.
- new helper method mergeTokenFirstChars(pTarget) which ORs current token's
first-char bits into passed pTarget (32-byte bitmap). This is called from
four places in analyseFirstCharSet and serves to remove code duplication
and guarantee a consistent approach across all paths. Note: all paths use
tmpBitmap as the intermediate accumulator and the caller ORs tmpLongs into
mergedLongs. For the 'G' group cases in the top-level alternation path,
each inner branch is OR'd into tmpBitmap via the helper (since the helper
itself uses OR, multiple inner branches accumulate correctly), then the
outer merge handles tmpBitmap -> mergedBitmap as normal.
- improvements to required tokens search logic where a prefix is found
in order to more often avoid going into tryMatch due to earlier rejection.
- if start position for split() is beyond the end of the data (including
empty data) simply return one blank line
- performance improvements in split() where splitting on a pure literal but
not starting at position 1. We can use st.findChars directly in a loop
rather than falling back to using vr.findPos (which has more overheads,
including initialiseMatchGroups, the recursion depth reset, all the
optimisation checks, and the tryMatch call itself). Also when doing a
noCase split, we can still use st.findChars but on the lowered text copy.
- enhancement to analyseLiteralPrefix so that phase 2 now handles both bare
literals (as before) *and* mandatory non-alternating groups containing
only simple literals. So (A) (B) can now yield prefix "A B" not just "A "
- enhancements in findPos when searching for required tokens. We store
the position of the first one found so we do not have to search again
for all the required tokens if a subsequent search starts prior to
this first required token match in the haystack.
- added further comments in the code to aid future maintenance
- in the debug/tracing version there are traces for each call of tryMatch.
There are 53 of these:
5 from findPos and 48 recursive calls from tryMatch itself.
These 53 calls have had some descriptive comments added to aid tracing
the code.
- change mergeTokenFirstChars to use 8 long operations rather than 32 byte
operations for 'C' character classes.
- new optimisation in analyseRequiredToken as I realised that nextAltIndex
on the opening ( token is already set by linkPatterns (Pass 3) to the
position of the first | directly inside the group. If it is > 0 there is
alternation whereas if it is 0 there is none. This enables us to avoid
calls to scanForAlternation for every mandatory group it encounters.
scanForAlternation does a linear scan of the group contents so it is
good to avoid this by short-circuiting it in the common case where a
group does not contain alternation.