Recognition of Tokens Lexical-Analyzer Generator Lex Lexical Analysis (2) Sukree Sinthupinyo 1 1 Department of Computer Engineering Chulalongkorn University 14 July 2012 Lexical Analysis (2)
Recognition of Tokens Lexical-Analyzer Generator Lex Outline Recognition of Tokens 1 Transition Diagrams Completion of the Running Example Architecture of a Transition-Diagram-Based Lexical Analyzer Recognizing a keyword Lexical-Analyzer Generator Lex 2 Use of Lex Structure of Lex Lex Demonstration Lexical Analysis (2)
Transition Diagrams Recognition of Tokens Completion of the Running Example Lexical-Analyzer Generator Lex Architecture of a Transition-Diagram-Based Lexical Analyzer Recognizing a keyword Example We will use this grammar as our first example Lexical Analysis (2)
Transition Diagrams Recognition of Tokens Completion of the Running Example Lexical-Analyzer Generator Lex Architecture of a Transition-Diagram-Based Lexical Analyzer Recognizing a keyword Example Lexical Analysis (2)
Transition Diagrams Recognition of Tokens Completion of the Running Example Lexical-Analyzer Generator Lex Architecture of a Transition-Diagram-Based Lexical Analyzer Recognizing a keyword Tokens, Patterns, and Attribute Values Lexical Analysis (2)
Transition Diagrams Recognition of Tokens Completion of the Running Example Lexical-Analyzer Generator Lex Architecture of a Transition-Diagram-Based Lexical Analyzer Recognizing a keyword Outline Recognition of Tokens 1 Transition Diagrams Completion of the Running Example Architecture of a Transition-Diagram-Based Lexical Analyzer Recognizing a keyword Lexical-Analyzer Generator Lex 2 Use of Lex Structure of Lex Lex Demonstration Lexical Analysis (2)
Transition Diagrams Recognition of Tokens Completion of the Running Example Lexical-Analyzer Generator Lex Architecture of a Transition-Diagram-Based Lexical Analyzer Recognizing a keyword Transition Diagrams We’ll convert the regular-expression to transition diagrams Nodes represent states . Edges represent methods of how to change the state. If we find an edge labeled a connecting from current state to another state and the next symbol is a , we will move to such state. Lexical Analysis (2)
Transition Diagrams Recognition of Tokens Completion of the Running Example Lexical-Analyzer Generator Lex Architecture of a Transition-Diagram-Based Lexical Analyzer Recognizing a keyword Transition Diagrams ( cont. ) Conventions about transition diagrams Accepting or final state indicates that a lexeme has been found. It’s represented by a double circle. If it is necessary to retract the forward pointer one position, then we shall place a * near that accepting state. There is one start state , or initial state . Lexical Analysis (2)
Transition Diagrams Recognition of Tokens Completion of the Running Example Lexical-Analyzer Generator Lex Architecture of a Transition-Diagram-Based Lexical Analyzer Recognizing a keyword Transition Diagram for relop Lexical Analysis (2)
Transition Diagrams Recognition of Tokens Completion of the Running Example Lexical-Analyzer Generator Lex Architecture of a Transition-Diagram-Based Lexical Analyzer Recognizing a keyword Recognition of Reserved Words and Identifiers How to recognize keywords, such as if , then and, else . Install the reserved words in the symbol table initially. Any new identifier will be an id . Create separate transition diagrams for each keyword. Lexical Analysis (2)
Transition Diagrams Recognition of Tokens Completion of the Running Example Lexical-Analyzer Generator Lex Architecture of a Transition-Diagram-Based Lexical Analyzer Recognizing a keyword Outline Recognition of Tokens 1 Transition Diagrams Completion of the Running Example Architecture of a Transition-Diagram-Based Lexical Analyzer Recognizing a keyword Lexical-Analyzer Generator Lex 2 Use of Lex Structure of Lex Lex Demonstration Lexical Analysis (2)
Transition Diagrams Recognition of Tokens Completion of the Running Example Lexical-Analyzer Generator Lex Architecture of a Transition-Diagram-Based Lexical Analyzer Recognizing a keyword Completion of the Running Example A transition diagram for unsigned numbers. A transition diagram for whitespace. Lexical Analysis (2)
Transition Diagrams Recognition of Tokens Completion of the Running Example Lexical-Analyzer Generator Lex Architecture of a Transition-Diagram-Based Lexical Analyzer Recognizing a keyword Outline Recognition of Tokens 1 Transition Diagrams Completion of the Running Example Architecture of a Transition-Diagram-Based Lexical Analyzer Recognizing a keyword Lexical-Analyzer Generator Lex 2 Use of Lex Structure of Lex Lex Demonstration Lexical Analysis (2)
Transition Diagrams Recognition of Tokens Completion of the Running Example Lexical-Analyzer Generator Lex Architecture of a Transition-Diagram-Based Lexical Analyzer Recognizing a keyword Architecture of a Transition-Diagram-Based Lexical Analyzer Example Code for relop Lexical Analysis (2)
Transition Diagrams Recognition of Tokens Completion of the Running Example Lexical-Analyzer Generator Lex Architecture of a Transition-Diagram-Based Lexical Analyzer Recognizing a keyword Architecture of a Transition-Diagram-Based Lexical Analyzer (cont.) Lexical Analysis (2)
Transition Diagrams Recognition of Tokens Completion of the Running Example Lexical-Analyzer Generator Lex Architecture of a Transition-Diagram-Based Lexical Analyzer Recognizing a keyword Architecture of a Transition-Diagram-Based Lexical Analyzer (cont.) Lexical Analysis (2)
Transition Diagrams Recognition of Tokens Completion of the Running Example Lexical-Analyzer Generator Lex Architecture of a Transition-Diagram-Based Lexical Analyzer Recognizing a keyword Architecture of a Transition-Diagram-Based Lexical Analyzer (cont.) Try sequentially. The function fail() will reset the pointer forward and starts the next transition diagram. Run in parallel. Feed the next input to all of them and allowing each one to make whatever transition it required. Combine all the transition diagrams into one. We combine states 0, 9, 12, and 22 into one start state. Lexical Analysis (2)
Transition Diagrams Recognition of Tokens Completion of the Running Example Lexical-Analyzer Generator Lex Architecture of a Transition-Diagram-Based Lexical Analyzer Recognizing a keyword Outline Recognition of Tokens 1 Transition Diagrams Completion of the Running Example Architecture of a Transition-Diagram-Based Lexical Analyzer Recognizing a keyword Lexical-Analyzer Generator Lex 2 Use of Lex Structure of Lex Lex Demonstration Lexical Analysis (2)
Transition Diagrams Recognition of Tokens Completion of the Running Example Lexical-Analyzer Generator Lex Architecture of a Transition-Diagram-Based Lexical Analyzer Recognizing a keyword KMP Algorithm Knuth, Morris, and Pratt algorithm Recognize a single keyword b 1 b 2 . . . b n in a text string. For example, the diagram for the keyword ababaa is We must find the failure function , f ( s ) computed as follow. Lexical Analysis (2)
Transition Diagrams Recognition of Tokens Completion of the Running Example Lexical-Analyzer Generator Lex Architecture of a Transition-Diagram-Based Lexical Analyzer Recognizing a keyword Failure Function ababaa Lexical Analysis (2)
Transition Diagrams Recognition of Tokens Completion of the Running Example Lexical-Analyzer Generator Lex Architecture of a Transition-Diagram-Based Lexical Analyzer Recognizing a keyword Failure Function cont. To find whether a keyword b 1 b 2 . . . b n is a substring of a string a 1 a 2 . . . a m , we use the following algorithm. Lexical Analysis (2)
Use of Lex Recognition of Tokens Structure of Lex Lexical-Analyzer Generator Lex Lex Demonstration Outline Recognition of Tokens 1 Transition Diagrams Completion of the Running Example Architecture of a Transition-Diagram-Based Lexical Analyzer Recognizing a keyword Lexical-Analyzer Generator Lex 2 Use of Lex Structure of Lex Lex Demonstration Lexical Analysis (2)
Use of Lex Recognition of Tokens Structure of Lex Lexical-Analyzer Generator Lex Lex Demonstration Use of Lex Process of Lex Lexical Analysis (2)
Use of Lex Recognition of Tokens Structure of Lex Lexical-Analyzer Generator Lex Lex Demonstration Outline Recognition of Tokens 1 Transition Diagrams Completion of the Running Example Architecture of a Transition-Diagram-Based Lexical Analyzer Recognizing a keyword Lexical-Analyzer Generator Lex 2 Use of Lex Structure of Lex Lex Demonstration Lexical Analysis (2)
Use of Lex Recognition of Tokens Structure of Lex Lexical-Analyzer Generator Lex Lex Demonstration Form of Lex declarations %% translation rules %% auxiliary functions Lexical Analysis (2)
Use of Lex Recognition of Tokens Structure of Lex Lexical-Analyzer Generator Lex Lex Demonstration Declarations Lexical Analysis (2)
Use of Lex Recognition of Tokens Structure of Lex Lexical-Analyzer Generator Lex Lex Demonstration Translations Rules Lexical Analysis (2)
Use of Lex Recognition of Tokens Structure of Lex Lexical-Analyzer Generator Lex Lex Demonstration Auxiliary Functions Lexical Analysis (2)
Recommend
More recommend
