path: root/man1p/lex.1p

.\" Copyright (c) 2001-2003 The Open Group, All Rights Reserved 
.TH "LEX" P 2003 "IEEE/The Open Group" "POSIX Programmer's Manual"
.\" lex 
.SH NAME
lex \- generate programs for lexical tasks (\fBDEVELOPMENT\fP)
.SH SYNOPSIS
.LP
\fBlex\fP \fB[\fP\fB-t\fP\fB][\fP\fB-n|-v\fP\fB][\fP\fIfile\fP \fB...\fP\fB]\fP\fB\fP
.SH DESCRIPTION
.LP
The \fIlex\fP utility shall generate C programs to be used in lexical
processing of character input, and that can be used as an
interface to \fIyacc\fP. The C programs shall be generated from \fIlex\fP
source code and
conform to the ISO\ C standard. Usually, the \fIlex\fP utility shall
write the program it generates to the file
\fBlex.yy.c\fP; the state of this file is unspecified if \fIlex\fP
exits with a non-zero exit status. See the EXTENDED
DESCRIPTION section for a complete description of the \fIlex\fP input
language.
.SH OPTIONS
.LP
The \fIlex\fP utility shall conform to the Base Definitions volume
of IEEE\ Std\ 1003.1-2001, Section 12.2, Utility Syntax Guidelines.
.LP
The following options shall be supported:
.TP 7
\fB-n\fP
Suppress the summary of statistics usually written with the \fB-v\fP
option. If no table sizes are specified in the \fIlex\fP
source code and the \fB-v\fP option is not specified, then \fB-n\fP
is implied.
.TP 7
\fB-t\fP
Write the resulting program to standard output instead of \fBlex.yy.c\fP.
.TP 7
\fB-v\fP
Write a summary of \fIlex\fP statistics to the standard output. (See
the discussion of \fIlex\fP table sizes in Definitions in lex .) If
the \fB-t\fP option is specified and \fB-n\fP is not specified, this
report shall
be written to standard error. If table sizes are specified in the
\fIlex\fP source code, and if the \fB-n\fP option is not
specified, the \fB-v\fP option may be enabled.
.sp
.SH OPERANDS
.LP
The following operand shall be supported:
.TP 7
\fIfile\fP
A pathname of an input file. If more than one such \fIfile\fP is specified,
all files shall be concatenated to produce a
single \fIlex\fP program. If no \fIfile\fP operands are specified,
or if a \fIfile\fP operand is \fB'-'\fP , the standard
input shall be used.
.sp
.SH STDIN
.LP
The standard input shall be used if no \fIfile\fP operands are specified,
or if a \fIfile\fP operand is \fB'-'\fP . See
INPUT FILES.
.SH INPUT FILES
.LP
The input files shall be text files containing \fIlex\fP source code,
as described in the EXTENDED DESCRIPTION section.
.SH ENVIRONMENT VARIABLES
.LP
The following environment variables shall affect the execution of
\fIlex\fP:
.TP 7
\fILANG\fP
Provide a default value for the internationalization variables that
are unset or null. (See the Base Definitions volume of
IEEE\ Std\ 1003.1-2001, Section 8.2, Internationalization Variables
for
the precedence of internationalization variables used to determine
the values of locale categories.)
.TP 7
\fILC_ALL\fP
If set to a non-empty string value, override the values of all the
other internationalization variables.
.TP 7
\fILC_COLLATE\fP
.sp
Determine the locale for the behavior of ranges, equivalence classes,
and multi-character collating elements within regular
expressions. If this variable is not set to the POSIX locale, the
results are unspecified.
.TP 7
\fILC_CTYPE\fP
Determine the locale for the interpretation of sequences of bytes
of text data as characters (for example, single-byte as
opposed to multi-byte characters in arguments and input files), and
the behavior of character classes within regular expressions.
If this variable is not set to the POSIX locale, the results are unspecified.
.TP 7
\fILC_MESSAGES\fP
Determine the locale that should be used to affect the format and
contents of diagnostic messages written to standard
error.
.TP 7
\fINLSPATH\fP
Determine the location of message catalogs for the processing of \fILC_MESSAGES
\&.\fP 
.sp
.SH ASYNCHRONOUS EVENTS
.LP
Default.
.SH STDOUT
.LP
If the \fB-t\fP option is specified, the text file of C source code
output of \fIlex\fP shall be written to standard
output.
.LP
If the \fB-t\fP option is not specified:
.IP " *" 3
Implementation-defined informational, error, and warning messages
concerning the contents of \fIlex\fP source code input shall
be written to either the standard output or standard error.
.LP
.IP " *" 3
If the \fB-v\fP option is specified and the \fB-n\fP option is not
specified, \fIlex\fP statistics shall also be written to
either the standard output or standard error, in an implementation-defined
format. These statistics may also be generated if table
sizes are specified with a \fB'%'\fP operator in the \fIDefinitions\fP
section, as long as the \fB-n\fP option is not
specified.
.LP
.SH STDERR
.LP
If the \fB-t\fP option is specified, implementation-defined informational,
error, and warning messages concerning the contents
of \fIlex\fP source code input shall be written to the standard error.
.LP
If the \fB-t\fP option is not specified:
.IP " 1." 4
Implementation-defined informational, error, and warning messages
concerning the contents of \fIlex\fP source code input shall
be written to either the standard output or standard error.
.LP
.IP " 2." 4
If the \fB-v\fP option is specified and the \fB-n\fP option is not
specified, \fIlex\fP statistics shall also be written to
either the standard output or standard error, in an implementation-defined
format. These statistics may also be generated if table
sizes are specified with a \fB'%'\fP operator in the \fIDefinitions\fP
section, as long as the \fB-n\fP option is not
specified.
.LP
.SH OUTPUT FILES
.LP
A text file containing C source code shall be written to \fBlex.yy.c\fP,
or to the standard output if the \fB-t\fP option is
present.
.SH EXTENDED DESCRIPTION
.LP
Each input file shall contain \fIlex\fP source code, which is a table
of regular expressions with corresponding actions in the
form of C program fragments.
.LP
When \fBlex.yy.c\fP is compiled and linked with the \fIlex\fP library
(using the \fB-l\ l\fP operand with \fIc99\fP), the resulting program
shall read character input from the standard input and shall
partition it into strings that match the given expressions.
.LP
When an expression is matched, these actions shall occur:
.IP " *" 3
The input string that was matched shall be left in \fIyytext\fP as
a null-terminated string; \fIyytext\fP shall either be an
external character array or a pointer to a character string. As explained
in Definitions in lex ,
the type can be explicitly selected using the \fB%array\fP or \fB%pointer\fP
declarations, but the default is
implementation-defined.
.LP
.IP " *" 3
The external \fBint\fP \fIyyleng\fP shall be set to the length of
the matching string.
.LP
.IP " *" 3
The expression's corresponding program fragment, or action, shall
be executed.
.LP
.LP
During pattern matching, \fIlex\fP shall search the set of patterns
for the single longest possible match. Among rules that
match the same number of characters, the rule given first shall be
chosen.
.LP
The general format of \fIlex\fP source shall be:
.sp
.RS
.nf

\fIDefinitions\fP
\fB%%\fP
\fIRules\fP
\fB%%\fP
\fIUser\fPSubroutines
.fi
.RE
.LP
The first \fB"%%"\fP is required to mark the beginning of the rules
(regular expressions and actions); the second
\fB"%%"\fP is required only if user subroutines follow.
.LP
Any line in the \fIDefinitions\fP section beginning with a <blank>
shall be assumed to be a C program fragment and shall
be copied to the external definition area of the \fBlex.yy.c\fP file.
Similarly, anything in the \fIDefinitions\fP section
included between delimiter lines containing only \fB"%{"\fP and \fB"%}"\fP
shall also be copied unchanged to the external
definition area of the \fBlex.yy.c\fP file.
.LP
Any such input (beginning with a <blank> or within \fB"%{"\fP and
\fB"%}"\fP delimiter lines) appearing at the
beginning of the \fIRules\fP section before any rules are specified
shall be written to \fBlex.yy.c\fP after the declarations of
variables for the \fIyylex\fP() function and before the first line
of code in \fIyylex\fP(). Thus, user variables local to
\fIyylex\fP() can be declared here, as well as application code to
execute upon entry to \fIyylex\fP().
.LP
The action taken by \fIlex\fP when encountering any input beginning
with a <blank> or within \fB"%{"\fP and
\fB"%}"\fP delimiter lines appearing in the \fIRules\fP section but
coming after one or more rules is undefined. The presence
of such input may result in an erroneous definition of the \fIyylex\fP()
function.
.SS Definitions in lex
.LP
\fIDefinitions\fP appear before the first \fB"%%"\fP delimiter. Any
line in this section not contained between \fB"%{"\fP
and \fB"%}"\fP lines and not beginning with a <blank> shall be assumed
to define a \fIlex\fP substitution string. The
format of these lines shall be:
.sp
.RS
.nf

\fIname substitute\fP
.fi
.RE
.LP
If a \fIname\fP does not meet the requirements for identifiers in
the ISO\ C standard, the result is undefined. The string
\fIsubstitute\fP shall replace the string { \fIname\fP} when it is
used in a rule. The \fIname\fP string shall be recognized in
this context only when the braces are provided and when it does not
appear within a bracket expression or within double-quotes.
.LP
In the \fIDefinitions\fP section, any line beginning with a \fB'%'\fP
(percent sign) character and followed by an
alphanumeric word beginning with either \fB's'\fP or \fB'S'\fP shall
define a set of start conditions. Any line beginning
with a \fB'%'\fP followed by a word beginning with either \fB'x'\fP
or \fB'X'\fP shall define a set of exclusive start
conditions. When the generated scanner is in a \fB%s\fP state, patterns
with no state specified shall be also active; in a
\fB%x\fP state, such patterns shall not be active. The rest of the
line, after the first word, shall be considered to be one or
more <blank>-separated names of start conditions. Start condition
names shall be constructed in the same way as definition
names. Start conditions can be used to restrict the matching of regular
expressions to one or more states as described in Regular Expressions
in lex .
.LP
Implementations shall accept either of the following two mutually-exclusive
declarations in the \fIDefinitions\fP section:
.TP 7
\fB%array\fP
Declare the type of \fIyytext\fP to be a null-terminated character
array.
.TP 7
\fB%pointer\fP
Declare the type of \fIyytext\fP to be a pointer to a null-terminated
character string.
.sp
.LP
The default type of \fIyytext\fP is implementation-defined. If an
application refers to \fIyytext\fP outside of the scanner
source file (that is, via an \fBextern\fP), the application shall
include the appropriate \fB%array\fP or \fB%pointer\fP
declaration in the scanner source file.
.LP
Implementations shall accept declarations in the \fIDefinitions\fP
section for setting certain internal table sizes. The
declarations are shown in the following table.
.sp
.ce 1
\fBTable: Table Size Declarations in \fIlex\fP\fP
.TS C
center; l2 l2 l.
\fBDeclaration\fP	\fBDescription\fP	\fBMinimum Value\fP
%\fBp\fP \fIn\fP	Number of positions	2500
%\fBn\fP \fIn\fP	Number of states	500
%\fBa\fP \fIn\fP	Number of transitions	2000
%\fBe\fP \fIn\fP	Number of parse tree nodes	1000
%\fBk\fP \fIn\fP	Number of packed character classes	1000
%\fBo\fP \fIn\fP	Size of the output array	3000
.TE
.LP
In the table, \fIn\fP represents a positive decimal integer, preceded
by one or more <blank>s. The exact meaning of these
table size numbers is implementation-defined. The implementation shall
document how these numbers affect the \fIlex\fP utility and
how they are related to any output that may be generated by the implementation
should limitations be encountered during the
execution of \fIlex\fP. It shall be possible to determine from this
output which of the table size values needs to be modified to
permit \fIlex\fP to successfully generate tables for the input language.
The values in the column Minimum Value represent the
lowest values conforming implementations shall provide.
.SS Rules in lex
.LP
The rules in \fIlex\fP source files are a table in which the left
column contains regular expressions and the right column
contains actions (C program fragments) to be executed when the expressions
are recognized.
.sp
.RS
.nf

\fIERE action
ERE action\fP\fB...
\fP
.fi
.RE
.LP
The extended regular expression (ERE) portion of a row shall be separated
from \fIaction\fP by one or more <blank>s. A
regular expression containing <blank>s shall be recognized under one
of the following conditions:
.IP " *" 3
The entire expression appears within double-quotes.
.LP
.IP " *" 3
The <blank>s appear within double-quotes or square brackets.
.LP
.IP " *" 3
Each <blank> is preceded by a backslash character.
.LP
.SS User Subroutines in lex
.LP
Anything in the user subroutines section shall be copied to \fBlex.yy.c\fP
following \fIyylex\fP().
.SS Regular Expressions in lex
.LP
The \fIlex\fP utility shall support the set of extended regular expressions
(see the Base Definitions volume of
IEEE\ Std\ 1003.1-2001, Section 9.4, Extended Regular Expressions),
with the following additions and exceptions to the syntax:
.TP 7
\fB"..."\fP
Any string enclosed in double-quotes shall represent the characters
within the double-quotes as themselves, except that
backslash escapes (which appear in the following table) shall be recognized.
Any backslash-escape sequence shall be terminated by
the closing quote. For example, \fB"\\01"\fP \fB"1"\fP represents
a single string: the octal value 1 followed by the character
\fB'1'\fP .
.TP 7
<\fIstate\fP>\fIr\fP,\ <\fIstate1,state2,\fP...>\fIr\fP
.sp
The regular expression \fIr\fP shall be matched only when the program
is in one of the start conditions indicated by \fIstate\fP,
\fIstate1\fP, and so on; see Actions in lex . (As an exception to
the typographical conventions of
the rest of this volume of IEEE\ Std\ 1003.1-2001, in this case <\fIstate\fP>
does not represent a metavariable, but
the literal angle-bracket characters surrounding a symbol.) The start
condition shall be recognized as such only at the beginning
of a regular expression.
.TP 7
\fIr\fP/\fIx\fP
The regular expression \fIr\fP shall be matched only if it is followed
by an occurrence of regular expression \fIx\fP (
\fIx\fP is the instance of trailing context, further defined below).
The token returned in \fIyytext\fP shall only match
\fIr\fP. If the trailing portion of \fIr\fP matches the beginning
of \fIx\fP, the result is unspecified. The \fIr\fP expression
cannot include further trailing context or the \fB'$'\fP (match-end-of-line)
operator; \fIx\fP cannot include the \fB'^'\fP
(match-beginning-of-line) operator, nor trailing context, nor the
\fB'$'\fP operator. That is, only one occurrence of trailing
context is allowed in a \fIlex\fP regular expression, and the \fB'^'\fP
operator only can be used at the beginning of such an
expression.
.TP 7
{\fIname\fP}
When \fIname\fP is one of the substitution symbols from the \fIDefinitions\fP
section, the string, including the enclosing
braces, shall be replaced by the \fIsubstitute\fP value. The \fIsubstitute\fP
value shall be treated in the extended regular
expression as if it were enclosed in parentheses. No substitution
shall occur if { \fIname\fP} occurs within a bracket expression
or within double-quotes.
.sp
.LP
Within an ERE, a backslash character shall be considered to begin
an escape sequence as specified in the table in the Base
Definitions volume of IEEE\ Std\ 1003.1-2001, Chapter 5, File Format
Notation (
\fB'\\\\'\fP , \fB'\\a'\fP , \fB'\\b'\fP , \fB'\\f'\fP , \fB'\\n'\fP
, \fB'\\r'\fP , \fB'\\t'\fP , \fB'\\v'\fP ). In
addition, the escape sequences in the following table shall be recognized.
.LP
A literal <newline> cannot occur within an ERE; the escape sequence
\fB'\\n'\fP can be used to represent a
<newline>. A <newline> shall not be matched by a period operator.
.br
.sp
.ce 1
\fBTable: Escape Sequences in \fIlex\fP\fP
.TS C
center; l1 lw(30)1 lw(30).
\fBEscape\fP	T{
.na
\fB\ \fP
.ad
T}	T{
.na
\fB\ \fP
.ad
T}
\fBSequence\fP	T{
.na
\fBDescription\fP
.ad
T}	T{
.na
\fBMeaning\fP
.ad
T}
\\\fIdigits\fP	T{
.na
A backslash character followed by the longest sequence of one, two, or three octal-digit characters (01234567). If all of the digits are 0 (that is, representation of the NUL character), the behavior is undefined.
.ad
T}	T{
.na
The character whose encoding is represented by the one, two, or three-digit octal integer. If the size of a byte on the system is greater than nine bits, the valid escape sequence used to represent a byte is implementation-defined. Multi-byte characters require multiple, concatenated escape sequences of this type, including the leading \fB'\\'\fP for each byte.
.ad
T}
\\x\fIdigits\fP	T{
.na
A backslash character followed by the longest sequence of hexadecimal-digit characters (01234567abcdefABCDEF). If all of the digits are 0 (that is, representation of the NUL character), the behavior is undefined.
.ad
T}	T{
.na
The character whose encoding is represented by the hexadecimal integer.
.ad
T}
\\c	T{
.na
A backslash character followed by any character not described in this table or in the table in the Base Definitions volume of IEEE\ Std\ 1003.1-2001, Chapter 5, File Format Notation ( \fB'\\\\'\fP , \fB'\\a'\fP , \fB'\\b'\fP , \fB'\\f'\fP , \fB'\\n'\fP , \fB'\\r'\fP , \fB'\\t'\fP , \fB'\\v'\fP ).
.ad
T}	T{
.na
The character \fB'c'\fP , unchanged.
.ad
T}
.TE
.TP 7
\fBNote:\fP
If a \fB'\\x'\fP sequence needs to be immediately followed by a hexadecimal
digit character, a sequence such as
\fB"\\x1"\fP \fB"1"\fP can be used, which represents a character containing
the value 1, followed by the character
\fB'1'\fP .
.sp
.LP
The order of precedence given to extended regular expressions for
\fIlex\fP differs from that specified in the Base Definitions
volume of IEEE\ Std\ 1003.1-2001, Section 9.4, Extended Regular
Expressions. The order of precedence for \fIlex\fP shall be as shown
in the following table, from high to low. 
.TP 7
\fBNote:\fP
The escaped characters entry is not meant to imply that these are
operators, but they are included in the table to show their
relationships to the true operators. The start condition, trailing
context, and anchoring notations have been omitted from the
table because of the placement restrictions described in this section;
they can only appear at the beginning or ending of an
ERE.
.sp
.sp
.sp
.ce 1
\fBTable: ERE Precedence in \fIlex\fP\fP
.TS C
center; l2 l.
\fBExtended Regular Expression\fP	\fBPrecedence\fP
collation-related bracket symbols	[= =] [: :] [. .]
escaped characters	\\<\fIspecial character\fP>
bracket expression	[ ]
quoting	"..."
grouping	( )
definition	{\fIname\fP}
single-character RE duplication	* + ?
concatenation	\ 
interval expression	{m,n}
alternation	|
.TE
.LP
The ERE anchoring operators \fB'^'\fP and \fB'$'\fP do not appear
in the table. With \fIlex\fP regular expressions, these
operators are restricted in their use: the \fB'^'\fP operator can
only be used at the beginning of an entire regular expression,
and the \fB'$'\fP operator only at the end. The operators apply to
the entire regular expression. Thus, for example, the pattern
\fB"(^abc)|(def$)"\fP is undefined; it can instead be written as two
separate rules, one with the regular expression
\fB"^abc"\fP and one with \fB"def$"\fP , which share a common action
via the special \fB'|'\fP action (see below). If the
pattern were written \fB"^abc|def$"\fP , it would match either \fB"abc"\fP
or \fB"def"\fP on a line by itself.
.LP
Unlike the general ERE rules, embedded anchoring is not allowed by
most historical \fIlex\fP implementations. An example of
embedded anchoring would be for patterns such as \fB"(^|\ )foo(\ |$)"\fP
to match \fB"foo"\fP when it exists as a
complete word. This functionality can be obtained using existing \fIlex\fP
features:
.sp
.RS
.nf

\fB^foo/[ \\n]      |
" foo"/[ \\n]    /* Found foo as a separate word. */
\fP
.fi
.RE
.LP
Note also that \fB'$'\fP is a form of trailing context (it is equivalent
to \fB"/\\n"\fP ) and as such cannot be used with
regular expressions containing another instance of the operator (see
the preceding discussion of trailing context).
.LP
The additional regular expressions trailing-context operator \fB'/'\fP
can be used as an ordinary character if presented
within double-quotes, \fB"/"\fP ; preceded by a backslash, \fB"\\/"\fP
; or within a bracket expression, \fB"[/]"\fP . The
start-condition \fB'<'\fP and \fB'>'\fP operators shall be special
only in a start condition at the beginning of a
regular expression; elsewhere in the regular expression they shall
be treated as ordinary characters.
.SS Actions in lex
.LP
The action to be taken when an ERE is matched can be a C program fragment
or the special actions described below; the program
fragment can contain one or more C statements, and can also include
special actions. The empty C statement \fB';'\fP shall be a
valid action; any string in the \fBlex.yy.c\fP input that matches
the pattern portion of such a rule is effectively ignored or
skipped. However, the absence of an action shall not be valid, and
the action \fIlex\fP takes in such a condition is
undefined.
.LP
The specification for an action, including C statements and special
actions, can extend across several lines if enclosed in
braces:
.sp
.RS
.nf

\fIERE\fP \fB<\fP\fIone or more blanks\fP\fB> {\fP \fIprogram statement
                           program statement\fP \fB}
\fP
.fi
.RE
.LP
The default action when a string in the input to a \fBlex.yy.c\fP
program is not matched by any expression shall be to copy the
string to the output. Because the default behavior of a program generated
by \fIlex\fP is to read the input and copy it to the
output, a minimal \fIlex\fP source program that has just \fB"%%"\fP
shall generate a C program that simply copies the input to
the output unchanged.
.LP
Four special actions shall be available:
.sp
.RS
.nf

\fB|   ECHO;   REJECT;   BEGIN
\fP
.fi
.RE
.TP 7
\fB|\fP
The action \fB'|'\fP means that the action for the next rule is the
action for this rule. Unlike the other three actions,
\fB'|'\fP cannot be enclosed in braces or be semicolon-terminated;
the application shall ensure that it is specified alone, with
no other actions.
.TP 7
\fBECHO;\fP
Write the contents of the string \fIyytext\fP on the output.
.TP 7
\fBREJECT;\fP
Usually only a single expression is matched by a given string in the
input. \fBREJECT\fP means "continue to the next
expression that matches the current input", and shall cause whatever
rule was the second choice after the current rule to be
executed for the same input. Thus, multiple rules can be matched and
executed for one input string or overlapping input strings.
For example, given the regular expressions \fB"xyz"\fP and \fB"xy"\fP
and the input \fB"xyz"\fP , usually only the regular
expression \fB"xyz"\fP would match. The next attempted match would
start after \fBz.\fP If the last action in the
\fB"xyz"\fP rule is \fBREJECT\fP, both this rule and the \fB"xy"\fP
rule would be executed. The \fBREJECT\fP action may be
implemented in such a fashion that flow of control does not continue
after it, as if it were equivalent to a \fBgoto\fP to another
part of \fIyylex\fP(). The use of \fBREJECT\fP may result in somewhat
larger and slower scanners.
.TP 7
\fBBEGIN\fP
The action: 
.sp
.RS
.nf

\fBBEGIN\fP \fInewstate\fP\fB;
\fP
.fi
.RE
.LP
switches the state (start condition) to \fInewstate\fP. If the string
\fInewstate\fP has not been declared previously as a
start condition in the \fIDefinitions\fP section, the results are
unspecified. The initial state is indicated by the digit
\fB'0'\fP or the token \fBINITIAL\fP.
.sp
.LP
The functions or macros described below are accessible to user code
included in the \fIlex\fP input. It is unspecified whether
they appear in the C code output of \fIlex\fP, or are accessible only
through the \fB-l\ l\fP operand to \fIc99\fP (the \fIlex\fP library).
.TP 7
\fBint\ \fP \fIyylex\fP(\fBvoid\fP)
.sp
Performs lexical analysis on the input; this is the primary function
generated by the \fIlex\fP utility. The function shall return
zero when the end of input is reached; otherwise, it shall return
non-zero values (tokens) determined by the actions that are
selected.
.TP 7
\fBint\ \fP \fIyymore\fP(\fBvoid\fP)
.sp
When called, indicates that when the next input string is recognized,
it is to be appended to the current value of \fIyytext\fP
rather than replacing it; the value in \fIyyleng\fP shall be adjusted
accordingly.
.TP 7
\fBint\ \fP \fIyyless\fP(\fBint\ \fP \fIn\fP)
.sp
Retains \fIn\fP initial characters in \fIyytext\fP, NUL-terminated,
and treats the remaining characters as if they had not been
read; the value in \fIyyleng\fP shall be adjusted accordingly.
.TP 7
\fBint\ \fP \fIinput\fP(\fBvoid\fP)
.sp
Returns the next character from the input, or zero on end-of-file.
It shall obtain input from the stream pointer \fIyyin\fP,
although possibly via an intermediate buffer. Thus, once scanning
has begun, the effect of altering the value of \fIyyin\fP is
undefined. The character read shall be removed from the input stream
of the scanner without any processing by the scanner.
.TP 7
\fBint\ \fP \fIunput\fP(\fBint\ \fP \fIc\fP)
.sp
Returns the character \fB'c'\fP to the input; \fIyytext\fP and \fIyyleng\fP
are undefined until the next expression is
matched. The result of using \fIunput\fP() for more characters than
have been input is unspecified.
.sp
.LP
The following functions shall appear only in the \fIlex\fP library
accessible through the \fB-l\ l\fP operand; they can
therefore be redefined by a conforming application:
.TP 7
\fBint\ \fP \fIyywrap\fP(\fBvoid\fP)
.sp
Called by \fIyylex\fP() at end-of-file; the default \fIyywrap\fP()
shall always return 1. If the application requires
\fIyylex\fP() to continue processing with another source of input,
then the application can include a function \fIyywrap\fP(),
which associates another file with the external variable \fBFILE *\fP
\fIyyin\fP and shall return a value of zero.
.TP 7
\fBint\ \fP \fImain\fP(\fBint\ \fP \fIargc\fP, \fBchar *\fP\fIargv\fP[])
.sp
Calls \fIyylex\fP() to perform lexical analysis, then exits. The user
code can contain \fImain\fP() to perform
application-specific operations, calling \fIyylex\fP() as applicable.
.sp
.LP
Except for \fIinput\fP(), \fIunput\fP(), and \fImain\fP(), all external
and static names generated by \fIlex\fP shall begin
with the prefix \fByy\fP or \fBYY\fP.
.SH EXIT STATUS
.LP
The following exit values shall be returned:
.TP 7
\ 0
Successful completion.
.TP 7
>0
An error occurred.
.sp
.SH CONSEQUENCES OF ERRORS
.LP
Default.
.LP
\fIThe following sections are informative.\fP
.SH APPLICATION USAGE
.LP
Conforming applications are warned that in the \fIRules\fP section,
an ERE without an action is not acceptable, but need not be
detected as erroneous by \fIlex\fP. This may result in compilation
or runtime errors.
.LP
The purpose of \fIinput\fP() is to take characters off the input stream
and discard them as far as the lexical analysis is
concerned. A common use is to discard the body of a comment once the
beginning of a comment is recognized.
.LP
The \fIlex\fP utility is not fully internationalized in its treatment
of regular expressions in the \fIlex\fP source code or
generated lexical analyzer. It would seem desirable to have the lexical
analyzer interpret the regular expressions given in the
\fIlex\fP source according to the environment specified when the lexical
analyzer is executed, but this is not possible with the
current \fIlex\fP technology. Furthermore, the very nature of the
lexical analyzers produced by \fIlex\fP must be closely tied to
the lexical requirements of the input language being described, which
is frequently locale-specific anyway. (For example, writing
an analyzer that is used for French text is not automatically useful
for processing other languages.)
.SH EXAMPLES
.LP
The following is an example of a \fIlex\fP program that implements
a rudimentary scanner for a Pascal-like syntax:
.sp
.RS
.nf

\fB%{
/* Need this for the call to atof() below. */
#include <math.h>
/* Need this for printf(), fopen(), and stdin below. */
#include <stdio.h>
%}
.sp

DIGIT    [0-9]
ID       [a-z][a-z0-9]*
.sp

%%
.sp

{DIGIT}+ {
    printf("An integer: %s (%d)\\n", yytext,
        atoi(yytext));
    }
.sp

{DIGIT}+"."{DIGIT}*        {
    printf("A float: %s (%g)\\n", yytext,
        atof(yytext));
    }
.sp

if|then|begin|end|procedure|function        {
    printf("A keyword: %s\\n", yytext);
    }
.sp

{ID}    printf("An identifier: %s\\n", yytext);
.sp

"+"|"-"|"*"|"/"        printf("An operator: %s\\n", yytext);
.sp

"{"[^}\\n]*"}"    /* Eat up one-line comments. */
.sp

[ \\t\\n]+        /* Eat up white space. */
.sp

\&.  printf("Unrecognized character: %s\\n", yytext);
.sp

%%
.sp

int main(int argc, char *argv[])
{
    ++argv, --argc;  /* Skip over program name. */
    if (argc > 0)
        yyin = fopen(argv[0], "r");
    else
        yyin = stdin;
.sp

    yylex();
}
\fP
.fi
.RE
.SH RATIONALE
.LP
Even though the \fB-c\fP option and references to the C language are
retained in this description, \fIlex\fP may be
generalized to other languages, as was done at one time for EFL, the
Extended FORTRAN Language. Since the \fIlex\fP input
specification is essentially language-independent, versions of this
utility could be written to produce Ada, Modula-2, or Pascal
code, and there are known historical implementations that do so.
.LP
The current description of \fIlex\fP bypasses the issue of dealing
with internationalized EREs in the \fIlex\fP source code or
generated lexical analyzer. If it follows the model used by \fIawk\fP
(the source code is
assumed to be presented in the POSIX locale, but input and output
are in the locale specified by the environment variables), then
the tables in the lexical analyzer produced by \fIlex\fP would interpret
EREs specified in the \fIlex\fP source in terms of the
environment variables specified when \fIlex\fP was executed. The desired
effect would be to have the lexical analyzer interpret
the EREs given in the \fIlex\fP source according to the environment
specified when the lexical analyzer is executed, but this is
not possible with the current \fIlex\fP technology.
.LP
The description of octal and hexadecimal-digit escape sequences agrees
with the ISO\ C standard use of escape sequences. See
the RATIONALE for \fIed\fP for a discussion of bytes larger than 9
bits being represented by octal values.
Hexadecimal values can represent larger bytes and multi-byte characters
directly, using as many digits as required.
.LP
There is no detailed output format specification. The observed behavior
of \fIlex\fP under four different historical
implementations was that none of these implementations consistently
reported the line numbers for error and warning messages.
Furthermore, there was a desire that \fIlex\fP be allowed to output
additional diagnostic messages. Leaving message formats
unspecified avoids these formatting questions and problems with internationalization.
.LP
Although the \fB%x\fP specifier for \fIexclusive\fP start conditions
is not historical practice, it is believed to be a
minor change to historical implementations and greatly enhances the
usability of \fIlex\fP programs since it permits an
application to obtain the expected functionality with fewer statements.
.LP
The \fB%array\fP and \fB%pointer\fP declarations were added as a compromise
between historical systems. The System V-based
\fIlex\fP copies the matched text to a \fIyytext\fP array. The \fIflex\fP
program, supported in BSD and GNU systems, uses a
pointer. In the latter case, significant performance improvements
are available for some scanners. Most historical programs should
require no change in porting from one system to another because the
string being referenced is null-terminated in both cases. (The
method used by \fIflex\fP in its case is to null-terminate the token
in place by remembering the character that used to come right
after the token and replacing it before continuing on to the next
scan.) Multi-file programs with external references to
\fIyytext\fP outside the scanner source file should continue to operate
on their historical systems, but would require one of the
new declarations to be considered strictly portable.
.LP
The description of EREs avoids unnecessary duplication of ERE details
because their meanings within a \fIlex\fP ERE are the
same as that for the ERE in this volume of IEEE\ Std\ 1003.1-2001.
.LP
The reason for the undefined condition associated with text beginning
with a <blank> or within \fB"%{"\fP and
\fB"%}"\fP delimiter lines appearing in the \fIRules\fP section is
historical practice. Both the BSD and System V \fIlex\fP
copy the indented (or enclosed) input in the \fIRules\fP section (except
at the beginning) to unreachable areas of the
\fIyylex\fP() function (the code is written directly after a \fIbreak\fP
statement). In some cases, the System V \fIlex\fP generates an error
message or a syntax error, depending on the form of indented
input.
.LP
The intention in breaking the list of functions into those that may
appear in \fBlex.yy.c\fP \fIversus\fP those that only
appear in \fBlibl.a\fP is that only those functions in \fBlibl.a\fP
can be reliably redefined by a conforming application.
.LP
The descriptions of standard output and standard error are somewhat
complicated because historical \fIlex\fP implementations
chose to issue diagnostic messages to standard output (unless \fB-t\fP
was given). IEEE\ Std\ 1003.1-2001 allows this
behavior, but leaves an opening for the more expected behavior of
using standard error for diagnostics. Also, the System V behavior
of writing the statistics when any table sizes are given is allowed,
while BSD-derived systems can avoid it. The programmer can
always precisely obtain the desired results by using either the \fB-t\fP
or \fB-n\fP options.
.LP
The OPERANDS section does not mention the use of \fB-\fP as a synonym
for standard input; not all historical implementations
support such usage for any of the \fIfile\fP operands.
.LP
A description of the \fItranslation table\fP was deleted from early
proposals because of its relatively low usage in historical
applications.
.LP
The change to the definition of the \fIinput\fP() function that allows
buffering of input presents the opportunity for major
performance gains in some applications.
.LP
The following examples clarify the differences between \fIlex\fP regular
expressions and regular expressions appearing
elsewhere in this volume of IEEE\ Std\ 1003.1-2001. For regular expressions
of the form \fB"r/x"\fP , the string
matching \fIr\fP is always returned; confusion may arise when the
beginning of \fIx\fP matches the trailing portion of \fIr\fP.
For example, given the regular expression \fB"a*b/cc"\fP and the input
\fB"aaabcc"\fP , \fIyytext\fP would contain the
string \fB"aaab"\fP on this match. But given the regular expression
\fB"x*/xy"\fP and the input \fB"xxxy"\fP , the token
\fBxxx\fP, not \fBxx\fP, is returned by some implementations because
\fBxxx\fP matches \fB"x*"\fP .
.LP
In the rule \fB"ab*/bc"\fP , the \fB"b*"\fP at the end of \fIr\fP
extends \fIr\fP's match into the beginning of the
trailing context, so the result is unspecified. If this rule were
\fB"ab/bc"\fP , however, the rule matches the text
\fB"ab"\fP when it is followed by the text \fB"bc"\fP . In this latter
case, the matching of \fIr\fP cannot extend into the
beginning of \fIx\fP, so the result is specified.
.SH FUTURE DIRECTIONS
.LP
None.
.SH SEE ALSO
.LP
\fIc99\fP , \fIed\fP , \fIyacc\fP
.SH COPYRIGHT
Portions of this text are reprinted and reproduced in electronic form
from IEEE Std 1003.1, 2003 Edition, Standard for Information Technology
-- Portable Operating System Interface (POSIX), The Open Group Base
Specifications Issue 6, Copyright (C) 2001-2003 by the Institute of
Electrical and Electronics Engineers, Inc and The Open Group. In the
event of any discrepancy between this version and the original IEEE and
The Open Group Standard, the original IEEE and The Open Group Standard
is the referee document. The original Standard can be obtained online at
http://www.opengroup.org/unix/online.html .