\n// This token stream ignores the value of {@link Token//getChannel}. If your\n// parser requires the token stream filter tokens to only those on a particular\n// channel, such as {@link Token//DEFAULT_CHANNEL} or\n// {@link Token//HIDDEN_CHANNEL}, use a filtering token stream such a\n// {@link CommonTokenStream}.
This field is set to -1 when the stream is first constructed or when\n\t// {@link //setTokenSource} is called, indicating that the first token has\n\t// not yet been fetched from the token source. For additional information,\n\t// see the documentation of {@link IntStream} for a description of\n\t// Initializing Methods.
For example, {@link CommonTokenStream} overrides this method to ensure\n// that\n// the seek target is always an on-channel token.
\n // The default value is {@code false} to avoid the performance and memory\n // overhead of copying text for every token unless explicitly requested.
\n// This token factory does not explicitly copy token text when constructing\n// tokens.
\n// This token stream provides access to all tokens by index or when calling\n// methods like {@link //getText}. The channel filtering is only used for code\n// accessing tokens via the lookahead methods {@link //LA}, {@link //LT}, and\n// {@link //LB}.
\n// By default, tokens are placed on the default channel\n// ({@link Token//DEFAULT_CHANNEL}), but may be reassigned by using the\n// {@code ->channel(HIDDEN)} lexer command, or by using an embedded action to\n// call {@link Lexer//setChannel}.\n//
\n// Note: lexer rules which use the {@code ->skip} lexer command or call\n// {@link Lexer//skip} do not produce tokens at all, so input text matched by\n// such a rule will not be available as part of the token stream, regardless of\n// channel.
If {@code ctx} is {@code null} and the end of the rule containing\n// {@code s} is reached, {@link Token//EPSILON} is added to the result set.\n// If {@code ctx} is not {@code null} and the end of the outermost rule is\n// reached, {@link Token//EOF} is added to the result set.
If {@code ctx} is {@code null} and {@code stopState} or the end of the\n// rule containing {@code s} is reached, {@link Token//EPSILON} is added to\n// the result set. If {@code ctx} is not {@code null} and {@code addEOF} is\n// {@code true} and {@code stopState} or the end of the outermost rule is\n// reached, {@link Token//EOF} is added to the result set.
If the symbol type does not match,\n// {@link ANTLRErrorStrategy//recoverInline} is called on the current error\n// strategy to attempt recovery. If {@link //getBuildParseTree} is\n// {@code true} and the token index of the symbol returned by\n// {@link ANTLRErrorStrategy//recoverInline} is -1, the symbol is added to\n// the parse tree by calling {@link ParserRuleContext//addErrorNode}.
To support output-preserving grammar transformations (including but not\n// limited to left-recursion removal, automated left-factoring, and\n// optimized code generation), calls to listener methods during the parse\n// may differ substantially from calls made by\n// {@link ParseTreeWalker//DEFAULT} used after the parse is complete. In\n// particular, rule entry and exit events may occur in a different order\n// during the parse than after the parser. In addition, calls to certain\n// rule entry methods may be omitted.
With the following specific exceptions, calls to listener events are\n// deterministic, i.e. for identical input the calls to listener\n// methods will be the same.
If {@code listener} is {@code null} or has not been added as a parse\n// listener, this method does nothing.
\n// ParseTree t = parser.expr();\n// ParseTreePattern p = parser.compileParseTreePattern(\"<ID>+0\",\n// MyParser.RULE_expr);\n// ParseTreeMatch m = p.match(t);\n// String id = m.get(\"ID\");\n//
E.g., given the following input with {@code A} being the current\n// lookahead symbol, this function moves the cursor to {@code B} and returns\n// {@code A}.
\n// A B\n// ^\n//
\n// return getExpectedTokens().contains(symbol);\n//
Stack tops equal, parents merge is same; return left graph.\n//
Same stack top, parents differ; merge parents giving array node, then\n// remainders of those graphs. A new root node is created to point to the\n// merged parents.\n//
Different stack tops pointing to same parent. Make array node for the\n// root where both element in the root point to the same (original)\n// parent.\n//
Different stack tops pointing to different parents. Make array node for\n// the root where each element points to the corresponding original\n// parent.\n//
These local-context merge operations are used when {@code rootIsWildcard}\n// is true.
{@link //EMPTY} is superset of any graph; return {@link //EMPTY}.\n//
{@link //EMPTY} and anything is {@code //EMPTY}, so merged parent is\n// {@code //EMPTY}; return left graph.\n//
Special case of last merge if local context.\n//
These full-context merge operations are used when {@code rootIsWildcard}\n// is false.
Must keep all contexts; {@link //EMPTY} in array is a special value (and\n// null parent).\n//
Different tops, different parents.\n//
Shared top, same parents.\n//
Shared top, different parents.\n//
Shared top, all shared parents.\n//
Equal tops, merge parents and reduce top to\n// {@link SingletonPredictionContext}.\n//
Used for XPath and tree pattern compilation.
\n// Since tokens on hidden channels (e.g. whitespace or comments) are not\n// added to the parse trees, they will not appear in the output of this\n// method.\n// /\nRuleContext.prototype.getText = function() {\n\tif (this.getChildCount() === 0) {\n\t\treturn \"\";\n\t} else {\n\t\treturn this.children.map(function(child) {\n\t\t\treturn child.getText();\n\t\t}).join(\"\");\n\t}\n};\n\n// For rule associated with this parse tree internal node, return\n// the outer alternative number used to match the input. Default\n// implementation does not compute nor store this alt num. Create\n// a subclass of ParserRuleContext with backing field and set\n// option contextSuperClass.\n// to set it.\nRuleContext.prototype.getAltNumber = function() { return INVALID_ALT_NUMBER; }\n\n// Set the outer alternative number for this context node. Default\n// implementation does nothing to avoid backing field overhead for\n// trees that don't need it. Create\n// a subclass of ParserRuleContext with backing field and set\n// option contextSuperClass.\nRuleContext.prototype.setAltNumber = function(altNumber) { }\n\nRuleContext.prototype.getChild = function(i) {\n\treturn null;\n};\n\nRuleContext.prototype.getChildCount = function() {\n\treturn 0;\n};\n\nRuleContext.prototype.accept = function(visitor) {\n\treturn visitor.visitChildren(this);\n};\n\n//need to manage circular dependencies, so export now\nexports.RuleContext = RuleContext;\nvar Trees = require('./tree/Trees').Trees;\n\n\n// Print out a whole tree, not just a node, in LISP format\n// (root child1 .. childN). Print just a node if this is a leaf.\n//\n\nRuleContext.prototype.toStringTree = function(ruleNames, recog) {\n\treturn Trees.toStringTree(this, ruleNames, recog);\n};\n\nRuleContext.prototype.toString = function(ruleNames, stop) {\n\truleNames = ruleNames || null;\n\tstop = stop || null;\n\tvar p = this;\n\tvar s = \"[\";\n\twhile (p !== null && p !== stop) {\n\t\tif (ruleNames === null) {\n\t\t\tif (!p.isEmpty()) {\n\t\t\t\ts += p.invokingState;\n\t\t\t}\n\t\t} else {\n\t\t\tvar ri = p.ruleIndex;\n\t\t\tvar ruleName = (ri >= 0 && ri < ruleNames.length) ? ruleNames[ri]\n\t\t\t\t\t: \"\" + ri;\n\t\t\ts += ruleName;\n\t\t}\n\t\tif (p.parentCtx !== null && (ruleNames !== null || !p.parentCtx.isEmpty())) {\n\t\t\ts += \" \";\n\t\t}\n\t\tp = p.parentCtx;\n\t}\n\ts += \"]\";\n\treturn s;\n};\n\n","/* Copyright (c) 2012-2017 The ANTLR Project. All rights reserved.\n * Use of this file is governed by the BSD 3-clause license that\n * can be found in the LICENSE.txt file in the project root.\n */\n//\n\n// A token has properties: text, type, line, character position in the line\n// (so we can ignore tabs), token channel, index, and source from which\n// we obtained this token.\n\nfunction Token() {\n\tthis.source = null;\n\tthis.type = null; // token type of the token\n\tthis.channel = null; // The parser ignores everything not on DEFAULT_CHANNEL\n\tthis.start = null; // optional; return -1 if not implemented.\n\tthis.stop = null; // optional; return -1 if not implemented.\n\tthis.tokenIndex = null; // from 0..n-1 of the token object in the input stream\n\tthis.line = null; // line=1..n of the 1st character\n\tthis.column = null; // beginning of the line at which it occurs, 0..n-1\n\tthis._text = null; // text of the token.\n\treturn this;\n}\n\nToken.INVALID_TYPE = 0;\n\n// During lookahead operations, this \"token\" signifies we hit rule end ATN state\n// and did not follow it despite needing to.\nToken.EPSILON = -2;\n\nToken.MIN_USER_TOKEN_TYPE = 1;\n\nToken.EOF = -1;\n\n// All tokens go to the parser (unless skip() is called in that rule)\n// on a particular \"channel\". The parser tunes to a particular channel\n// so that whitespace etc... can go to the parser on a \"hidden\" channel.\n\nToken.DEFAULT_CHANNEL = 0;\n\n// Anything on different channel than DEFAULT_CHANNEL is not parsed\n// by parser.\n\nToken.HIDDEN_CHANNEL = 1;\n\n// Explicitly set the text for this token. If {code text} is not\n// {@code null}, then {@link //getText} will return this value rather than\n// extracting the text from the input.\n//\n// @param text The explicit text of the token, or {@code null} if the text\n// should be obtained from the input along with the start and stop indexes\n// of the token.\n\nObject.defineProperty(Token.prototype, \"text\", {\n\tget : function() {\n\t\treturn this._text;\n\t},\n\tset : function(text) {\n\t\tthis._text = text;\n\t}\n});\n\nToken.prototype.getTokenSource = function() {\n\treturn this.source[0];\n};\n\nToken.prototype.getInputStream = function() {\n\treturn this.source[1];\n};\n\nfunction CommonToken(source, type, channel, start, stop) {\n\tToken.call(this);\n\tthis.source = source !== undefined ? source : CommonToken.EMPTY_SOURCE;\n\tthis.type = type !== undefined ? type : null;\n\tthis.channel = channel !== undefined ? channel : Token.DEFAULT_CHANNEL;\n\tthis.start = start !== undefined ? start : -1;\n\tthis.stop = stop !== undefined ? stop : -1;\n\tthis.tokenIndex = -1;\n\tif (this.source[0] !== null) {\n\t\tthis.line = source[0].line;\n\t\tthis.column = source[0].column;\n\t} else {\n\t\tthis.column = -1;\n\t}\n\treturn this;\n}\n\nCommonToken.prototype = Object.create(Token.prototype);\nCommonToken.prototype.constructor = CommonToken;\n\n// An empty {@link Pair} which is used as the default value of\n// {@link //source} for tokens that do not have a source.\nCommonToken.EMPTY_SOURCE = [ null, null ];\n\n// Constructs a new {@link CommonToken} as a copy of another {@link Token}.\n//\n//
\n// If {@code oldToken} is also a {@link CommonToken} instance, the newly\n// constructed token will share a reference to the {@link //text} field and\n// the {@link Pair} stored in {@link //source}. Otherwise, {@link //text} will\n// be assigned the result of calling {@link //getText}, and {@link //source}\n// will be constructed from the result of {@link Token//getTokenSource} and\n// {@link Token//getInputStream}.
If {@code context} is {@code null}, it is treated as\n// {@link ParserRuleContext//EMPTY}.
This method updates {@link //dipsIntoOuterContext} and\n// {@link //hasSemanticContext} when necessary.
This cache makes a huge difference in memory and a little bit in speed.\n // For the Java grammar on java.*, it dropped the memory requirements\n // at the end from 25M to 16M. We don't store any of the full context\n // graphs in the DFA because they are limited to local context only,\n // but apparently there's a lot of repetition there as well. We optimize\n // the config contexts before storing the config set in the DFA states\n // by literally rebuilding them with cached subgraphs only.
I tried a cache for use during closure operations, that was\n // whacked after each adaptivePredict(). It cost a little bit\n // more time I think and doesn't save on the overall footprint\n // so it's not worth the complexity.
We track these variables separately for the DFA and ATN simulation\n// because the DFA simulation often has to fail over to the ATN\n// simulation. If the ATN simulation fails, we need the DFA to fall\n// back to its previously accepted state, if any. If the ATN succeeds,\n// then the ATN does the accept and the DFA simulator that invoked it\n// can simply return the predicted token type.
If {@code speculative} is {@code true}, this method was called before\n// {@link //consume} for the matched character. This method should call\n// {@link //consume} before evaluating the predicate to ensure position\n// sensitive values, including {@link Lexer//getText}, {@link Lexer//getLine},\n// and {@link Lexer//getcolumn}, properly reflect the current\n// lexer state. This method should restore {@code input} and the simulator\n// to the original state before returning (i.e. undo the actions made by the\n// call to {@link //consume}.
The {@code skip} command does not have any parameters, so this action is\n// implemented as a singleton instance exposed by {@link //INSTANCE}.
This action is implemented by calling {@link Lexer//pushMode} with the\n// value provided by {@link //getMode}.
The {@code popMode} command does not have any parameters, so this action is\n// implemented as a singleton instance exposed by {@link //INSTANCE}.
This action is implemented by calling {@link Lexer//popMode}.
The {@code more} command does not have any parameters, so this action is\n// implemented as a singleton instance exposed by {@link //INSTANCE}.
This action is implemented by calling {@link Lexer//mode} with the\n// value provided by {@link //getMode}.
This class may represent embedded actions created with the {...}\n// syntax in ANTLR 4, as well as actions created for lexer commands where the\n// command argument could not be evaluated when the grammar was compiled.
{...}
Custom actions are implemented by calling {@link Lexer//action} with the\n// appropriate rule and action indexes.
This action is implemented by calling {@link Lexer//setChannel} with the\n// value provided by {@link //getChannel}.
This action is not serialized as part of the ATN, and is only required for\n// position-dependent lexer actions which appear at a location other than the\n// end of a rule. For more information about DFA optimizations employed for\n// lexer actions, see {@link LexerActionExecutor//append} and\n// {@link LexerActionExecutor//fixOffsetBeforeMatch}.
Note: This class is only required for lexer actions for which\n// {@link LexerAction//isPositionDependent} returns {@code true}.
This method calls {@link //execute} on the result of {@link //getAction}\n// using the provided {@code lexer}.
The executor tracks position information for position-dependent lexer actions\n// efficiently, ensuring that actions appearing only at the end of the rule do\n// not cause bloating of the {@link DFA} created for the lexer.
Normally, when the executor encounters lexer actions where\n// {@link LexerAction//isPositionDependent} returns {@code true}, it calls\n// {@link IntStream//seek} on the input {@link CharStream} to set the input\n// position to the end of the current token. This behavior provides\n// for efficient DFA representation of lexer actions which appear at the end\n// of a lexer rule, even when the lexer rule matches a variable number of\n// characters.
Prior to traversing a match transition in the ATN, the current offset\n// from the token start index is assigned to all position-dependent lexer\n// actions which have not already been assigned a fixed offset. By storing\n// the offsets relative to the token start index, the DFA representation of\n// lexer actions which appear in the middle of tokens remains efficient due\n// to sharing among tokens of the same length, regardless of their absolute\n// position in the input stream.
If the current executor already has offsets assigned to all\n// position-dependent lexer actions, the method returns {@code this}.
This method calls {@link IntStream//seek} to set the position of the\n// {@code input} {@link CharStream} prior to calling\n// {@link LexerAction//execute} on a position-dependent action. Before the\n// method returns, the input position will be restored to the same position\n// it was in when the method was invoked.
\n// The basic complexity of the adaptive strategy makes it harder to understand.\n// We begin with ATN simulation to build paths in a DFA. Subsequent prediction\n// requests go through the DFA first. If they reach a state without an edge for\n// the current symbol, the algorithm fails over to the ATN simulation to\n// complete the DFA path for the current input (until it finds a conflict state\n// or uniquely predicting state).
\n// All of that is done without using the outer context because we want to create\n// a DFA that is not dependent upon the rule invocation stack when we do a\n// prediction. One DFA works in all contexts. We avoid using context not\n// necessarily because it's slower, although it can be, but because of the DFA\n// caching problem. The closure routine only considers the rule invocation stack\n// created during prediction beginning in the decision rule. For example, if\n// prediction occurs without invoking another rule's ATN, there are no context\n// stacks in the configurations. When lack of context leads to a conflict, we\n// don't know if it's an ambiguity or a weakness in the strong LL(*) parsing\n// strategy (versus full LL(*)).
\n// When SLL yields a configuration set with conflict, we rewind the input and\n// retry the ATN simulation, this time using full outer context without adding\n// to the DFA. Configuration context stacks will be the full invocation stacks\n// from the start rule. If we get a conflict using full context, then we can\n// definitively say we have a true ambiguity for that input sequence. If we\n// don't get a conflict, it implies that the decision is sensitive to the outer\n// context. (It is not context-sensitive in the sense of context-sensitive\n// grammars.)
\n// The next time we reach this DFA state with an SLL conflict, through DFA\n// simulation, we will again retry the ATN simulation using full context mode.\n// This is slow because we can't save the results and have to \"interpret\" the\n// ATN each time we get that input.
\n// CACHING FULL CONTEXT PREDICTIONS
\n// We could cache results from full context to predicted alternative easily and\n// that saves a lot of time but doesn't work in presence of predicates. The set\n// of visible predicates from the ATN start state changes depending on the\n// context, because closure can fall off the end of a rule. I tried to cache\n// tuples (stack context, semantic context, predicted alt) but it was slower\n// than interpreting and much more complicated. Also required a huge amount of\n// memory. The goal is not to create the world's fastest parser anyway. I'd like\n// to keep this algorithm simple. By launching multiple threads, we can improve\n// the speed of parsing across a large number of files.
\n// There is no strict ordering between the amount of input used by SLL vs LL,\n// which makes it really hard to build a cache for full context. Let's say that\n// we have input A B C that leads to an SLL conflict with full context X. That\n// implies that using X we might only use A B but we could also use A B C D to\n// resolve conflict. Input A B C D could predict alternative 1 in one position\n// in the input and A B C E could predict alternative 2 in another position in\n// input. The conflicting SLL configurations could still be non-unique in the\n// full context prediction, which would lead us to requiring more input than the\n// original A B C.\tTo make a\tprediction cache work, we have to track\tthe exact\n// input\tused during the previous prediction. That amounts to a cache that maps\n// X to a specific DFA for that context.
\n// Something should be done for left-recursive expression predictions. They are\n// likely LL(1) + pred eval. Easier to do the whole SLL unless error and retry\n// with full LL thing Sam does.
\n// AVOIDING FULL CONTEXT PREDICTION
\n// We avoid doing full context retry when the outer context is empty, we did not\n// dip into the outer context by falling off the end of the decision state rule,\n// or when we force SLL mode.
\n// As an example of the not dip into outer context case, consider as super\n// constructor calls versus function calls. One grammar might look like\n// this:
\n// ctorBody\n// : '{' superCall? stat* '}'\n// ;\n//
\n// Or, you might see something like
\n// stat\n// : superCall ';'\n// | expression ';'\n// | ...\n// ;\n//
\n// In both cases I believe that no closure operations will dip into the outer\n// context. In the first case ctorBody in the worst case will stop at the '}'.\n// In the 2nd case it should stop at the ';'. Both cases should stay within the\n// entry rule and not dip into the outer context.
\n// PREDICATES
\n// Predicates are always evaluated if present in either SLL or LL both. SLL and\n// LL simulation deals with predicates differently. SLL collects predicates as\n// it performs closure operations like ANTLR v3 did. It delays predicate\n// evaluation until it reaches and accept state. This allows us to cache the SLL\n// ATN simulation whereas, if we had evaluated predicates on-the-fly during\n// closure, the DFA state configuration sets would be different and we couldn't\n// build up a suitable DFA.
\n// When building a DFA accept state during ATN simulation, we evaluate any\n// predicates and return the sole semantically valid alternative. If there is\n// more than 1 alternative, we report an ambiguity. If there are 0 alternatives,\n// we throw an exception. Alternatives without predicates act like they have\n// true predicates. The simple way to think about it is to strip away all\n// alternatives with false predicates and choose the minimum alternative that\n// remains.
\n// When we start in the DFA and reach an accept state that's predicated, we test\n// those and return the minimum semantically viable alternative. If no\n// alternatives are viable, we throw an exception.
\n// During full LL ATN simulation, closure always evaluates predicates and\n// on-the-fly. This is crucial to reducing the configuration set size during\n// closure. It hits a landmine when parsing with the Java grammar, for example,\n// without this on-the-fly evaluation.
\n// SHARING DFA
\n// All instances of the same parser share the same decision DFAs through a\n// static field. Each instance gets its own ATN simulator but they share the\n// same {@link //decisionToDFA} field. They also share a\n// {@link PredictionContextCache} object that makes sure that all\n// {@link PredictionContext} objects are shared among the DFA states. This makes\n// a big size difference.
\n// THREAD SAFETY
\n// The {@link ParserATNSimulator} locks on the {@link //decisionToDFA} field when\n// it adds a new DFA object to that array. {@link //addDFAEdge}\n// locks on the DFA for the current decision when setting the\n// {@link DFAState//edges} field. {@link //addDFAState} locks on\n// the DFA for the current decision when looking up a DFA state to see if it\n// already exists. We must make sure that all requests to add DFA states that\n// are equivalent result in the same shared DFA object. This is because lots of\n// threads will be trying to update the DFA at once. The\n// {@link //addDFAState} method also locks inside the DFA lock\n// but this time on the shared context cache when it rebuilds the\n// configurations' {@link PredictionContext} objects using cached\n// subgraphs/nodes. No other locking occurs, even during DFA simulation. This is\n// safe as long as we can guarantee that all threads referencing\n// {@code s.edge[t]} get the same physical target {@link DFAState}, or\n// {@code null}. Once into the DFA, the DFA simulation does not reference the\n// {@link DFA//states} map. It follows the {@link DFAState//edges} field to new\n// targets. The DFA simulator will either find {@link DFAState//edges} to be\n// {@code null}, to be non-{@code null} and {@code dfa.edges[t]} null, or\n// {@code dfa.edges[t]} to be non-null. The\n// {@link //addDFAEdge} method could be racing to set the field\n// but in either case the DFA simulator works; if {@code null}, and requests ATN\n// simulation. It could also race trying to get {@code dfa.edges[t]}, but either\n// way it will work because it's not doing a test and set operation.
\n// Starting with SLL then failing to combined SLL/LL (Two-Stage\n// Parsing)
\n// Sam pointed out that if SLL does not give a syntax error, then there is no\n// point in doing full LL, which is slower. We only have to try LL if we get a\n// syntax error. For maximum speed, Sam starts the parser set to pure SLL\n// mode with the {@link BailErrorStrategy}:
\n// parser.{@link Parser//getInterpreter() getInterpreter()}.{@link //setPredictionMode setPredictionMode}{@code (}{@link PredictionMode//SLL}{@code )};\n// parser.{@link Parser//setErrorHandler setErrorHandler}(new {@link BailErrorStrategy}());\n//
\n// If it does not get a syntax error, then we're done. If it does get a syntax\n// error, we need to retry with the combined SLL/LL strategy.
\n// The reason this works is as follows. If there are no SLL conflicts, then the\n// grammar is SLL (at least for that input set). If there is an SLL conflict,\n// the full LL analysis must yield a set of viable alternatives which is a\n// subset of the alternatives reported by SLL. If the LL set is a singleton,\n// then the grammar is LL but not SLL. If the LL set is the same size as the SLL\n// set, the decision is SLL. If the LL set has size > 1, then that decision\n// is truly ambiguous on the current input. If the LL set is smaller, then the\n// SLL conflict resolution might choose an alternative that the full LL would\n// rule out as a possibility based upon better context information. If that's\n// the case, then the SLL parse will definitely get an error because the full LL\n// analysis says it's not viable. If SLL conflict resolution chooses an\n// alternative within the LL set, them both SLL and LL would choose the same\n// alternative because they both choose the minimum of multiple conflicting\n// alternatives.
\n// Let's say we have a set of SLL conflicting alternatives {@code {1, 2, 3}} and\n// a smaller LL set called s. If s is {@code {2, 3}}, then SLL\n// parsing will get an error because SLL will pursue alternative 1. If\n// s is {@code {1, 2}} or {@code {1, 3}} then both SLL and LL will\n// choose the same alternative because alternative one is the minimum of either\n// set. If s is {@code {2}} or {@code {3}} then SLL will get a syntax\n// error. If s is {@code {1}} then SLL will succeed.
\n// Of course, if the input is invalid, then we will get an error for sure in\n// both SLL and LL parsing. Erroneous input will therefore require 2 passes over\n// the input.
\n// The prediction context must be considered by this filter to address\n// situations like the following.\n//
\n// \n// grammar TA;\n// prog: statement* EOF;\n// statement: letterA | statement letterA 'b' ;\n// letterA: 'a';\n// \n//
\n// grammar TA;\n// prog: statement* EOF;\n// statement: letterA | statement letterA 'b' ;\n// letterA: 'a';\n//
\n// If the above grammar, the ATN state immediately before the token\n// reference {@code 'a'} in {@code letterA} is reachable from the left edge\n// of both the primary and closure blocks of the left-recursive rule\n// {@code statement}. The prediction context associated with each of these\n// configurations distinguishes between them, and prevents the alternative\n// which stepped out to {@code prog} (and then back in to {@code statement}\n// from being eliminated by the filter.\n//
\n// In some scenarios, the algorithm described above could predict an\n// alternative which will result in a {@link FailedPredicateException} in\n// the parser. Specifically, this could occur if the only configuration\n// capable of successfully parsing to the end of the decision rule is\n// blocked by a semantic predicate. By choosing this alternative within\n// {@link //adaptivePredict} instead of throwing a\n// {@link NoViableAltException}, the resulting\n// {@link FailedPredicateException} in the parser will identify the specific\n// predicate which is preventing the parser from successfully parsing the\n// decision rule, which helps developers identify and correct logic errors\n// in semantic predicates.\n//
If {@code D} is {@link //ERROR}, this method returns {@link //ERROR} and\n// does not change the DFA.
\n// When using this prediction mode, the parser will either return a correct\n// parse tree (i.e. the same parse tree that would be returned with the\n// {@link //LL} prediction mode), or it will report a syntax error. If a\n// syntax error is encountered when using the {@link //SLL} prediction mode,\n// it may be due to either an actual syntax error in the input or indicate\n// that the particular combination of grammar and input requires the more\n// powerful {@link //LL} prediction abilities to complete successfully.
\n// This prediction mode does not provide any guarantees for prediction\n// behavior for syntactically-incorrect inputs.
\n// When using this prediction mode, the parser will make correct decisions\n// for all syntactically-correct grammar and input combinations. However, in\n// cases where the grammar is truly ambiguous this prediction mode might not\n// report a precise answer for exactly which alternatives are\n// ambiguous.
\n// This prediction mode may be used for diagnosing ambiguities during\n// grammar development. Due to the performance overhead of calculating sets\n// of ambiguous alternatives, this prediction mode should be avoided when\n// the exact results are not necessary.
\n// This method computes the SLL prediction termination condition for both of\n// the following cases.
COMBINED SLL+LL PARSING
When LL-fallback is enabled upon SLL conflict, correct predictions are\n// ensured regardless of how the termination condition is computed by this\n// method. Due to the substantially higher cost of LL prediction, the\n// prediction should only fall back to LL when the additional lookahead\n// cannot lead to a unique SLL prediction.
Assuming combined SLL+LL parsing, an SLL configuration set with only\n// conflicting subsets should fall back to full LL, even if the\n// configuration sets don't resolve to the same alternative (e.g.\n// {@code {1,2}} and {@code {3,4}}. If there is at least one non-conflicting\n// configuration, SLL could continue with the hopes that more lookahead will\n// resolve via one of those non-conflicting configurations.
Here's the prediction termination rule them: SLL (for SLL+LL parsing)\n// stops when it sees only conflicting configuration subsets. In contrast,\n// full LL keeps going when there is uncertainty.
HEURISTIC
As a heuristic, we stop prediction when we see any conflicting subset\n// unless we see a state that only has one alternative associated with it.\n// The single-alt-state thing lets prediction continue upon rules like\n// (otherwise, it would admit defeat too soon):
{@code [12|1|[], 6|2|[], 12|2|[]]. s : (ID | ID ID?) ';' ;}
When the ATN simulation reaches the state before {@code ';'}, it has a\n// DFA state that looks like: {@code [12|1|[], 6|2|[], 12|2|[]]}. Naturally\n// {@code 12|1|[]} and {@code 12|2|[]} conflict, but we cannot stop\n// processing this node because alternative to has another way to continue,\n// via {@code [6|2|[]]}.
It also let's us continue for this rule:
{@code [1|1|[], 1|2|[], 8|3|[]] a : A | A | A B ;}
After matching input A, we reach the stop state for rule A, state 1.\n// State 8 is the state right before B. Clearly alternatives 1 and 2\n// conflict and no amount of further lookahead will separate the two.\n// However, alternative 3 will be able to continue and so we do not stop\n// working on this state. In the previous example, we're concerned with\n// states associated with the conflicting alternatives. Here alt 3 is not\n// associated with the conflicting configs, but since we can continue\n// looking for input reasonably, don't declare the state done.
PURE SLL PARSING
To handle pure SLL parsing, all we have to do is make sure that we\n// combine stack contexts for configurations that differ only by semantic\n// predicate. From there, we can do the usual SLL termination heuristic.
PREDICATES IN SLL+LL PARSING
SLL decisions don't evaluate predicates until after they reach DFA stop\n// states because they need to create the DFA cache that works in all\n// semantic situations. In contrast, full LL evaluates predicates collected\n// during start state computation so it can ignore predicates thereafter.\n// This means that SLL termination detection can totally ignore semantic\n// predicates.
Implementation-wise, {@link ATNConfigSet} combines stack contexts but not\n// semantic predicate contexts so we might see two configurations like the\n// following.
{@code (s, 1, x, {}), (s, 1, x', {p})}
Before testing these configurations against others, we have to merge\n// {@code x} and {@code x'} (without modifying the existing configurations).\n// For example, we test {@code (x+x')==x''} when looking for conflicts in\n// the following configurations.
{@code (s, 1, x, {}), (s, 1, x', {p}), (s, 2, x'', {})}
If the configuration set has predicates (as indicated by\n// {@link ATNConfigSet//hasSemanticContext}), this algorithm makes a copy of\n// the configurations to strip out all of the predicates so that a standard\n// {@link ATNConfigSet} will merge everything ignoring predicates.
The basic idea is to split the set of configurations {@code C}, into\n// conflicting subsets {@code (s, _, ctx, _)} and singleton subsets with\n// non-conflicting configurations. Two configurations conflict if they have\n// identical {@link ATNConfig//state} and {@link ATNConfig//context} values\n// but different {@link ATNConfig//alt} value, e.g. {@code (s, i, ctx, _)}\n// and {@code (s, j, ctx, _)} for {@code i!=j}.
Reduce these configuration subsets to the set of possible alternatives.\n// You can compute the alternative subsets in one pass as follows:
{@code A_s,ctx = {i | (s, i, ctx, _)}} for each configuration in\n// {@code C} holding {@code s} and {@code ctx} fixed.
Or in pseudo-code, for each configuration {@code c} in {@code C}:
\n// map[c] U= c.{@link ATNConfig//alt alt} // map hash/equals uses s and x, not\n// alt and not pred\n//
The values in {@code map} are the set of {@code A_s,ctx} sets.
If {@code |A_s,ctx|=1} then there is no conflict associated with\n// {@code s} and {@code ctx}.
Reduce the subsets to singletons by choosing a minimum of each subset. If\n// the union of these alternative subsets is a singleton, then no amount of\n// more lookahead will help us. We will always pick that alternative. If,\n// however, there is more than one alternative, then we are uncertain which\n// alternative to predict and must continue looking for resolution. We may\n// or may not discover an ambiguity in the future, even if there are no\n// conflicting subsets this round.
The biggest sin is to terminate early because it means we've made a\n// decision but were uncertain as to the eventual outcome. We haven't used\n// enough lookahead. On the other hand, announcing a conflict too late is no\n// big deal; you will still have the conflict. It's just inefficient. It\n// might even look until the end of file.
No special consideration for semantic predicates is required because\n// predicates are evaluated on-the-fly for full LL prediction, ensuring that\n// no configuration contains a semantic context during the termination\n// check.
CONFLICTING CONFIGS
Two configurations {@code (s, i, x)} and {@code (s, j, x')}, conflict\n// when {@code i!=j} but {@code x=x'}. Because we merge all\n// {@code (s, i, _)} configurations together, that means that there are at\n// most {@code n} configurations associated with state {@code s} for\n// {@code n} possible alternatives in the decision. The merged stacks\n// complicate the comparison of configuration contexts {@code x} and\n// {@code x'}. Sam checks to see if one is a subset of the other by calling\n// merge and checking to see if the merged result is either {@code x} or\n// {@code x'}. If the {@code x} associated with lowest alternative {@code i}\n// is the superset, then {@code i} is the only possible prediction since the\n// others resolve to {@code min(i)} as well. However, if {@code x} is\n// associated with {@code j>i} then at least one stack configuration for\n// {@code j} is not in conflict with alternative {@code i}. The algorithm\n// should keep going, looking for more lookahead due to the uncertainty.
For simplicity, I'm doing a equality check between {@code x} and\n// {@code x'} that lets the algorithm continue to consume lookahead longer\n// than necessary. The reason I like the equality is of course the\n// simplicity but also because that is the test you need to detect the\n// alternatives that are actually in conflict.
CONTINUE/STOP RULE
Continue if union of resolved alternative sets from non-conflicting and\n// conflicting alternative subsets has more than one alternative. We are\n// uncertain about which alternative to predict.
The complete set of alternatives, {@code [i for (_,i,_)]}, tells us which\n// alternatives are still in the running for the amount of input we've\n// consumed at this point. The conflicting sets let us to strip away\n// configurations that won't lead to more states because we resolve\n// conflicts to the configuration with a minimum alternate for the\n// conflicting set.
CASES
EXACT AMBIGUITY DETECTION
If all states report the same conflicting set of alternatives, then we\n// know we have the exact ambiguity set.
|A_i|>1 and\n// A_i = A_j for all i, j.
|A_i|>1
A_i = A_j
In other words, we continue examining lookahead until all {@code A_i}\n// have more than one alternative and all {@code A_i} are the same. If\n// {@code A={{1,2}, {1,3}}}, then regular LL prediction would terminate\n// because the resolved set is {@code {1}}. To determine what the real\n// ambiguity is, we have to know whether the ambiguity is between one and\n// two or one and three so we keep going. We can only stop prediction when\n// we need exact ambiguity detection when the sets look like\n// {@code A={{1,2}}} or {@code {{1,2},{1,2}}}, etc...
\n// map[c.{@link ATNConfig//state state}] U= c.{@link ATNConfig//alt alt}\n//
For context dependent predicates, we must pass in a local context so that\n// references such as $arg evaluate properly as _localctx.arg. We only\n// capture context dependent predicates in the context in which we begin\n// prediction, so we passed in the outer context here in case of context\n// dependent predicate evaluation.
\n// The evaluation of predicates by this context is short-circuiting, but\n// unordered.
This is a one way link. It emanates from a state (usually via a list of\n// transitions) and has a target state.
Since we never have to change the ATN transitions once we construct it,\n// we can fix these transitions as specific classes. The DFA transitions\n// on the other hand need to update the labels as it adds transitions to\n// the states. We'll use the term Edge for the DFA to distinguish them from\n// ATN transitions.
I use a set of ATNConfig objects not simple states. An ATNConfig\n// is both a state (ala normal conversion) and a RuleContext describing\n// the chain of rules (if any) followed to arrive at that state.
A DFA state may have multiple references to a particular state,\n// but with different ATN contexts (with same or different alts)\n// meaning that state was reached via a different set of rule invocations.
We only use these for non-{@link //requiresFullContext} but\n\t// conflicting states. That\n\t// means we know from the context (it's $ or we don't dip into outer\n\t// context) that it's an ambiguity not a conflict.
This list is computed by {@link\n\t// ParserATNSimulator//predicateDFAState}.
Because the number of alternatives and number of ATN configurations are\n// finite, there is a finite number of DFA states that can be processed.\n// This is necessary to show that the algorithm terminates.
Cannot test the DFA state numbers here because in\n// {@link ParserATNSimulator//addDFAState} we need to know if any other state\n// exists that has this exact set of ATN configurations. The\n// {@link //stateNumber} is irrelevant.
\n// This implementation prints messages to {@link System//err} containing the\n// values of {@code line}, {@code charPositionInLine}, and {@code msg} using\n// the following format.
\n// line line:charPositionInLine msg\n//
The default implementation simply calls {@link //endErrorCondition} to\n// ensure that the handler is not in error recovery mode.
The default implementation simply calls {@link //endErrorCondition}.
The default implementation returns immediately if the handler is already\n// in error recovery mode. Otherwise, it calls {@link //beginErrorCondition}\n// and dispatches the reporting task based on the runtime type of {@code e}\n// according to the following table.
The default implementation resynchronizes the parser by consuming tokens\n// until we find one in the resynchronization set--loosely the set of tokens\n// that can follow the current rule.
Implements Jim Idle's magic sync mechanism in closures and optional\n// subrules. E.g.,
\n// a : sync ( stuff sync )* ;\n// sync : {consume to what can follow sync} ;\n//
If the sub rule is optional ({@code (...)?}, {@code (...)*}, or block\n// with an empty alternative), then the expected set includes what follows\n// the subrule.
During loop iteration, it consumes until it sees a token that can start a\n// sub rule or what follows loop. Yes, that is pretty aggressive. We opt to\n// stay in the loop as long as possible.
ORIGINS
Previous versions of ANTLR did a poor job of their recovery within loops.\n// A single mismatch token or missing token would force the parser to bail\n// out of the entire rules surrounding the loop. So, for rule
\n// classDef : 'class' ID '{' member* '}'\n//
This functionality cost a little bit of effort because the parser has to\n// compare token set at the start of the loop and at each iteration. If for\n// some reason speed is suffering for you, you can turn off this\n// functionality by simply overriding this method as a blank { }.
This method is called when {@link //singleTokenDeletion} identifies\n// single-token deletion as a viable recovery strategy for a mismatched\n// input error.
The default implementation simply returns if the handler is already in\n// error recovery mode. Otherwise, it calls {@link //beginErrorCondition} to\n// enter error recovery mode, followed by calling\n// {@link Parser//notifyErrorListeners}.
This method is called when {@link //singleTokenInsertion} identifies\n// single-token insertion as a viable recovery strategy for a mismatched\n// input error.
The default implementation attempts to recover from the mismatched input\n// by using single token insertion and deletion as described below. If the\n// recovery attempt fails, this method throws an\n// {@link InputMismatchException}.
EXTRA TOKEN (single token deletion)
{@code LA(1)} is not what we are looking for. If {@code LA(2)} has the\n// right token, however, then assume {@code LA(1)} is some extra spurious\n// token and delete it. Then consume and return the next token (which was\n// the {@code LA(2)} token) as the successful result of the match operation.
This recovery strategy is implemented by {@link\n// //singleTokenDeletion}.
MISSING TOKEN (single token insertion)
If current token (at {@code LA(1)}) is consistent with what could come\n// after the expected {@code LA(1)} token, then assume the token is missing\n// and use the parser's {@link TokenFactory} to create it on the fly. The\n// \"insertion\" is performed by returning the created token as the successful\n// result of the match operation.
This recovery strategy is implemented by {@link\n// //singleTokenInsertion}.
EXAMPLE
For example, Input {@code i=(3;} is clearly missing the {@code ')'}. When\n// the parser returns from the nested call to {@code expr}, it will have\n// call chain:
\n// stat → expr → atom\n//
\n// => ID '=' '(' INT ')' ('+' atom)* ';'\n// ^\n//
This method determines whether or not single-token insertion is viable by\n// checking if the {@code LA(1)} input symbol could be successfully matched\n// if it were instead the {@code LA(2)} symbol. If this method returns\n// {@code true}, the caller is responsible for creating and inserting a\n// token with the correct type to produce this behavior.
If the single-token deletion is successful, this method calls\n// {@link //reportUnwantedToken} to report the error, followed by\n// {@link Parser//consume} to actually \"delete\" the extraneous token. Then,\n// before returning {@link //reportMatch} is called to signal a successful\n// match.
\n// This error strategy is useful in the following scenarios.
\n// {@code myparser.setErrorHandler(new BailErrorStrategy());}
If the state number is not known, this method returns -1.
If the set of expected tokens is not known and could not be computed,\n// this method returns {@code null}.