\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage{xcolor,tokcycle,verbatimbox,tabstackengine,lipsum,lmodern}
\colorlet{bred}{red!90!blue}
\usepackage{caption}
\captionsetup{font={sf, footnotesize},labelfont={sf},textfont={footnotesize}}
\fboxsep=1.5pt\relax
\makeatletter
\let\svlt<
\let\svgt>
\catcode`>=\active %
\catcode`<=\active %
\def\openesc#1{\if\svlt#1\itshape\rmfamily\let\xsvgt\@empty\else
  \svlt\itshape\rmfamily#1\let\xsvgt\svgt\fi}
\def\closeesc{\upshape\ttfamily\xsvgt}
\def\vbdelim{\small\catcode`<=\active\catcode`>=\active 
\def<{\openesc}
\def>{\closeesc}}
\catcode`>=12 %
\catcode`<=12 %
\newcommand\TokCycle{\textsf{tokcycle}}
\newcommand\Character{\textit{Character}}
\newcommand\Group{\textit{Group}}
\newcommand\Macro{\textit{Macro}}
\newcommand\Space{\textit{Space}}
\newcommand\CGMS{\textit{C-G-M-S}}
\newcommand\CGMnS{\textit{C}, \textit{G}, \textit{M}, and \textit{S}}
\newcommand\macname[1]{\texttt{\char92#1}} 
\newcommand\thevbox{\medskip\theverbbox\par\medskip\noindent}
\newenvironment{specfig}
{\par\bigskip\vbox\bgroup\small\sffamily\centering}
{\egroup\medskip\par}
\newtoks\gtoks
\newcommand\svgtoks[1]{\global\gtoks\expandafter{\the#1}}
\frenchspacing

\begin{document}

%%% AGAIN

%%%

\begin{myverbbox}{\argone}#1\end{myverbbox}
\begin{myverbbox}[\vbdelim\normalsize]{\vbnewcytoks}
\newcytoks{<new token register>}
\end{myverbbox}

{\centering{\LARGE Package \TokCycle{} \Large (v\tcver)\par}
Steven B Segletes
{\footnotesize $<$SSegletes@verizon.net$>$}
\\\small{contributor: Christian Tellechea}\footnote{
{I am extremely grateful to Christian $<$unbonpetit@netc.fr$>$}
  for his assistance in the development of this package.
The \macname{addcytoks} macro was provided by him.
He gave constant reminders on what the parser should be able
  to achieve, thus motivating me to spend the extra time striving for
  a generality of application that would not come naturally to me.
I value highly his collegiality and hold his expertise in the highest 
  regard.
}%
\\\today\par}

\begin{quote}
The \TokCycle{} package helps one to build tools to process 
  tokens from an input stream. 
If a macro to process an arbitrary single 
  (non-macro, non-space) token can be built, then 
  \TokCycle{} can provide a wrapper for cycling through an 
  input stream (including macros, spaces, and groups) 
  on a token-by-token basis, using the 
  provided macro on each successive character.

\TokCycle{} characterizes each successive token
  in the input stream as a \Character, a \Group, a \Macro, or 
  a \Space.
Each of these 
  token categories are processed with a unique directive,
  to bring about the desired effect of the token cycle.
\textit{If}-condition flags are provided to identify active,
  implicit, and catcode-6 tokens as they are digested.
The package provides a number of options
  for handling groups.
\end{quote}
\enlargethispage{1.4\baselineskip}

\tableofcontents

\makeatother
\section{The \TokCycle{} macros and environments}\label{s:intro}

The purpose of the \TokCycle{} package is to provide a tool
  to assist the user in developing token-processing macros and
  environments.
The types of processing are limited only by the creativity of the user,
  but examples might include letter-case-operations, letter spacing, 
  dimensional manipulation,
  simple-ciphering, \{group\} manipulation, macro removal, etc.
In one sense, it can be thought of as a streaming editor that operates
  on \LaTeX{} input streams.

The package can be loaded into both plain \TeX{}, by way of the 
  invocation \verb|\input tokcycle.tex|
  as well as \LaTeX, via \verb|\usepackage{tokcycle}|.
It provides a total of 6 macros/pseudo environments, based on
  three criteria:
\begin{itemize}
\item Two pseudo-environments with the phrase ``\texttt{tokencycle}''
  in the name, and four macros containing the phrase ``\texttt{tokcycle}''.
The pseudo-\hspace{0pt}environments operate within a group and typeset
  their result upon completion.
The macros operate within the document's current scope, but do not typeset the
  result automatically.
In the case of both macros and pseudo-environments, the transformed result
  is available for later use, being stored in the package
  token list named \macname{cytoks}.

\item Two macros and one pseudo-environment containing the 
  phrase ``\texttt{xpress}''.
Without the phrase, the macro/environment requires four
  \textit{processing directives} to be \textit{explicitly} specified,
  followed by the input stream.
With the phrase present, only the input stream is to be provided.  
In the \texttt{xpress} case, the \textit{processing directives} are 
  to have been separately specified via external macro and/or are taken 
  from the most recent \TokCycle{} macro invocation (failing that, 
  are taken from the package initialization).

\item Two macros containing the phrase ``\texttt{expanded}''.
When present, the input stream of the token cycle is subject to the
  new \TeX{} primitive, \macname{expanded}, prior to processing by
  the \TokCycle{} macro.
Expansion of specific macros in the input stream can be inhibited 
  in the input stream with the use of \macname{noexpand}.
Note that there are no \texttt{expanded} environments in \TokCycle,
  as \TokCycle{} environments do not pre-tokenize their
  input stream.

\end{itemize}

The basic approach of all \TokCycle{} macros/environments is to
  grab each successive token (or group) from the input stream, 
  decide what category it is, and use the currently active
  processing directives to make any desired transformation of the 
  token (or group).
Generally, with rare exception, the processed tokens should be stored
  in the token register \macname{cytoks}, using the tools provided
  by the package.
The cycle continues until the input stream is terminated.

As tokens/groups are read from the input stream, they are
  categorized according to four type classifications, and are
  subjected to the user-specified processing directive associated 
  with that category of token/group.
The \TokCycle{} categories by which the input stream
  is dissected include \Character{}, 
  \Group{}, \Macro{}, and \Space{}.

Catcode-0 tokens are directed to the \Macro{} directive for processing.
Catcode-10 tokens are directed to the \Space{} directive.
When an explicit catcode-1 token\footnote{
  Throughout this document the terms \textit{catcode-} and \textit{cat-}
  are used interchangeably.
}
  is encountered in the \TokCycle{} input
  stream, the contents of the entire group (sans the grouping) are directed
  to the \Group{} directive for further processing, which may in turn,
  redirect the individual tokens to the other categories.
The handling options of implicit cat-1 and 2 tokens are described later in
  this document (section~\ref{s:impgrp}).
Valid tokens that are neither catcode 0, 1, 2, nor 10, except where noted,
  are directed to the \Character{} directive for processing.

The syntax of the non-\texttt{xpress} macros/environments is\medskip

\begin{verbbox}[\vbdelim]
\tokcycle <<or>> \expandedtokcycle
  {<Character processing directive>}
  {<Group-content processing directive>}
  {<Macro processing directive>}
  {<Space processing directive>}
  {<token input stream>}
\end{verbbox}
\theverbbox\medskip\par\noindent
or, alternately, for the pseudo-environment,

\begin{verbbox}[\vbdelim]
\tokencycle
  {<Character processing directive>}
  {<Group-content processing directive>}
  {<Macro processing directive>}
  {<Space processing directive>}<token input stream>\endtokencycle
\end{verbbox}
\thevbox
For the \texttt{xpress} macros, the syntax is

\begin{verbbox}[\vbdelim]
\tokcyclexpress <<or>> \expandedtokcyclexpress
 {<token input stream>}
\end{verbbox}
\thevbox
or, alternately, for the \texttt{xpress}-pseudo-environment,

\begin{verbbox}[\vbdelim]
\tokencyclexpress<token input stream>\endtokencyclexpress
\end{verbbox}
\thevbox
\begin{sloppypar}

In addition to the above macros/environments, the means is
  provided to define new \TokCycle{} environments:

\begin{verbbox}[\vbdelim]
\tokcycleenvironment\environment_name
  {<Character processing directive>}
  {<Group-content processing directive>}
  {<Macro processing directive>}
  {<Space processing directive>}
\end{verbbox}
\thevbox
This will then permit simplified invocations of the form

\begin{verbbox}[\vbdelim]
\environment_name<token input stream>\endenvironment_name
\end{verbbox}
\thevbox
More recently, an even more versatile \textit{extended} version has 
  been made available:

\begin{verbbox}[\vbdelim]
\xtokcycleenvironment\environment_name
  {<Character processing directive>}
  {<Group-content processing directive>}
  {<Macro processing directive>}
  {<Space processing directive>}
  {<Set-up code>}
  {<Close-out code>}
\end{verbbox}
\thevbox
This will also permit simplified invocations of the form

\begin{verbbox}[\vbdelim]
\environment_name<token input stream>\endenvironment_name
\end{verbbox}
\thevbox
However, with this extended environment definition, additional
  set-up and close-out code will be executed prior to and following
  the token cycle, within the scope of the environment.
The close-out code, while executed after the token cycle, occurs
  before the \macname{cytoks} token list is actually typeset; therefore
  the close-out code can both execute macros and/or add tokens
  to the imminently-typeset token list.
Any definitions or counters that change as a result of executing
  the \CGMS{} directives are, in fact, \textit{past tense} when the 
  close-out code is executed.

Finally, the set-up and close-out code, as well as the \CGMS{} directives
  can invoke \macname{tcafterenv}, whose argument is executed upon the 
  exit from the scope of \textit{this} defined environment.\footnote{
The \macname{tcafterenv} macro is recognized \textit{only} in the
  context of an extended \macname{xtokcycleenvironment} definition.
It is not available for direct use in the package's \macname{tokencycle} 
  pseudo-environment, nor in \macname{tokcycleenvironment} definitions.
}
As with all \TokCycle{} environments, those created by way
  of \macname{tokcycleenvironment} and \macname{xtokcycleenvironment},
  while operating in the scope of a group, will preserve the contents
  of \macname{cytoks} when exiting their scope.

\subsection{Externally specified directives and directive resets}

For use in \texttt{xpress} mode, the directives for the \CGMS{} 
  categories may be externally pre-specified, respectively,
  via the four macros \macname{Characterdirective}, 
  \macname{Groupdirective}, \macname{Macrodirective}, 
  and \macname{Spacedirective}, each taking an argument containing
  the particulars of the directive specification.

Each of these directives may be individually reset to the package
  default with the following argument-free invocations: 
  \macname{resetCharacterdirective}, \macname{resetGroupdirective}, 
  \macname{resetMacrodirective}, or \macname{resetSpacedirective}.
In addition, \macname{resettokcycle} is also provided, which
  not only resets all four directives collectively, but it also
  resets, to the default configuration, the manner in which explicit 
  and implicit group tokens are processed.
Finally, it resets the \macname{aftertokcycle} macro to empty.
\end{sloppypar}

The default directives at package outset and upon reset are

\begin{verbbox}[\vbdelim]
\Characterdirective{\addcytoks{#1}}
\Groupdirective{\processtoks{#1}}
\Macrodirective{\addcytoks{#1}}
\Spacedirective{\addcytoks{#1}}
\aftertokcycle{}
\stripgroupingfalse
\stripimplicitgroupingcase{0}
\end{verbbox}
\thevbox
The interpretation of these directives will be explained in the
  remainder of this document.
Let it suffice for now to say that the default directive settings
  pass through the input stream to the output stream, without
  change.\footnote{
  Except, possibly, in the case of catcode-6 tokens, which will be 
  later addressed in sections~\ref{s:cat6} and \ref{s:whennot}.
}



\section{Commands in the \TokCycle{} directives}\label{s:coms}

The command-line token cycling tools provided in the package are 
  listed in section~\ref{s:intro}.  
For each of those commands and/or pseudo-environments, the 
  user must (explicitly or implicitly) detail a set of directives to 
  specify the manner in which the \Character{}, \Group{}, \Macro{}, 
  and \Space{} tokens found in the input stream are to be processed.
The \CGMS{} processing
  directives consist of normal \TeX/\LaTeX{} commands and user-defined
  macros to accomplish the desired effect.
There are, however, several macros provided by the package to assist
  the user in this endeavor.

The recommended way to apply this package is to collect the
  \TokCycle-transformed results of the input stream in a token register 
  provided by the package, named \macname{cytoks}.
Its contents can then be typeset via \verb|\the\cytoks|.
The macro for appending things to \macname{cytoks}, to be used in
  the package directives, is \macname{addcytoks}.

\subsection{Adding tokens to the output stream: \macname{addcytoks}}

The macro provided to append tokens to the \macname{cytoks} token register is
  named \macname{addcytoks[]\{\}}.
Its mandatory argument consists of tokens denoting \textit{what} you 
  would like to append to the \macname{cytoks} register, while the 
  optional argument denotes \textit{how} you would like them appended (valid 
  options include positive integers \verb|[<|\textit{n}\verb|>]| and 
  the letter \texttt{[x]}).

When the optional argument is omitted, the specified tokens are appended
  \textit{literally} to the register (without expansion).
An integer option, call it \textit{n}, takes the  
  the mandatory argument, and expands it $n$-times, and appends the
  result to \macname{cytoks}.
The \texttt{[x]} option employs the \macname{expanded} primitive to maximally
  expand the argument before appending it to \macname{cytoks}.

The \texttt{[x]} option will prove useful when the \Character{} or other
  directives involve a transformation that is fully expandable.
Its use will allow the expanded
  result of the transformation to be placed in the token list, rather
  than the unexpanded transformation instructions.

\subsubsection{\argone}

In the context of the \CGMnS{} processing directives, one may refer to
  \argone{} (for example, in the argument to \macname{addcytoks} or
  \macname{processtoks}).
\TeX{} users know that the first parameter to a \TeX{} macro is denoted
  as \argone.
The specification of all \TokCycle{} processing directives is structured 
  in such a way that ``\argone{}'' may be directly employed to indicate 
  the current token (or group) under consideration in the input stream.

\subsubsection{Transforming the Input Stream}

Within the \TokCycle{} \CGMS{} directives, the command
  \verb|\addcytoks{#1}| is used to echo the token being processed to
  the output stream that is being collected in the \macname{cytoks}
  token list.
However, one may do more than merely echo the input---one may transform
  it.
If the user creates a macro, let's call it \macname{xform}, to take
  a single character of input and convert it in some way to something
  else, then this macro can be used inside the \Character{} directive
  as \verb|\addcytoks{\xform{#1}}|.
In this case, if an `\texttt{e}' were the token under consideration
  by the \Character{} directive, `\verb|\xform{e}|' would be added 
  to the \macname{cytoks} token list.
Since \TokCycle{} operates upon each successive token in the input 
  stream, this \macname{xform} macro that operates on a single token 
  can be used to completely transform the \TokCycle{} input stream.

If the \macname{xform} macro is expandable, reaching its termination in
  a known number, $n$, of expansions, \macname{addcytoks} can take
  advantage of that with its optional argument, performing the 
  said number of expansions of \macname{xform} \textit{before}
  adding the result to \macname{cytoks}.
If the expandability of \macname{xform} is known, but the number $n$ is 
  not, the optional argument \verb|[x]| to \macname{addcytoks} may
  be used to fully expand the argument (by way of the 
  \macname{expanded} primitive).
In that case, the resulting \macname{cytoks} result will show 
  no evidence of the \macname{xform} macro, but only the results
  of its transformation.

The accompanying examples document, \texttt{tokcycle-examples.pdf},
  is full of examples showing specific \TokCycle-induced transformations.

\subsection{\Group{} directive: \macname{ifstripgrouping}, and \macname{processtoks}}

The \Group{} directive is unique, in that it is the only directive whose
  argument (\argone) may consist of more than a single token.
There are two issues to consider when handling the tokens comprising a group:
  do I retain or remove the grouping (cat-1,2 tokens)? 
Do I process the group's token content
  individually through the token cycle, or collectively as a unit?

Grouping in the output stream is determined by the externally set 
  condition \macname{ifstripgrouping}.
The package default is \macname{stripgroupingfalse}, such that any 
  explicit grouping that appears in the input stream will be echoed 
  in the output stream.
The alternative, \macname{stripgroupingtrue}, is dangerous 
  in the sense that it will strip the cat-1,2 grouping from the 
  group's tokens, thereby 
  affecting or even breaking code that utilizes such grouping for macro arguments.
Apply \macname{stripgroupingtrue} with care.

The issue of treating the tokens comprising the content of a 
  group individually or collectively
  is determined by the choice of macro inside the \Group{} directive.
Within the \Group{} directive, the argument \argone{} contains the 
  collective tokens of the group (absent the outer cat-1,2 grouping).
The default directive \macname{processtoks\{\argone\}} will recommit the 
  group's tokens individually to be processed through the token cycle.
In contrast, the command \macname{addcytoks\{\argone\}} in the \Group{}
  directive would directly add the collective group
  of tokens to the \macname{cytoks} register, without being processed
  individually by \TokCycle.

\subsubsection{The \macname{groupedcytoks} macro for more sophisticated
  group processing}
\label{s:grpcy}

\begin{sloppypar}
To this point, there have been presented two options for handling grouped
  material in the token cycle: process each of the grouped tokens (via
  \macname{processtoks}) or echo the grouped tokens (via 
  \macname{addcytoks}).  Obviously, one may also discard the grouped
  tokens altogether by employing neither of the above choices in the
  context of the \Group{} directive.
\end{sloppypar}

However, there are times where you might wish to perform a macro task
  (for example, one that might add additional tokens to \macname{cytoks}) 
  immediately \textit{outside} of the group, before or after.
If you were to add the task macro before or after the \macname{processtoks}
  invocation, it will still be performed, by default, \textit{inside} 
  the \macname{cytoks} group.

The macro \macname{groupedcytoks} allows one to manually specify the
  grouping duties of the \Group{} directive.
Thus, the following two group-processing configurations are 
  functionally identical: the default

\begin{verbbox}[\vbdelim]
  \stripgroupingfalse
  \Groupdirective{\processtoks{#1}}
\end{verbbox}
\thevbox
and 

\begin{verbbox}[\vbdelim]
  \stripgroupingtrue
  \Groupdirective{\groupedcytoks{\processtoks{#1}}}
\end{verbbox}
\thevbox
Therefore, unless the goal is to introduce additional nesting levels
  into the input stream, the first rule of using \macname{groupedcytoks} 
  is to \macname{stripgroupingtrue} before entering the token cycle.
The argument of \macname{groupedcytoks} specifies the tasks that are to 
  occur inside a \macname{cytoks} grouping.
So, in the following example

\begin{verbbox}[\vbdelim]
  \stripgroupingtrue
  \Groupdirective{\taskA
    \groupedcytoks{\taskB\processtoks{#1}\taskC}%
    \taskD}
\end{verbbox}
\thevbox
the tasks A and D will occur \textit{outside} of the \macname{cytoks}
  group.
Tasks B and C occur inside the group, immediately before and after processing
  the tokens of the group, respectively.
Thus, the use of \macname{groupedcytoks} in this way permits \TokCycle{}
  taskings to occur outside of the grouping applied to the output.

If \macname{stripgroupingtrue} had been omitted while still using
  \macname{groupedcytoks}, tasks A and D would have been in the 
  explicit group, and tasks B and C, along with the grouped tokens
  \argone{}, would have been nested within an additional group.

\subsubsection{Implicit grouping tokens: \macname{stripimplicitgroupingcase}}
\label{s:impgrp}

Implicit grouping tokens (e.g., \macname{bgroup} \& \macname{egroup})
  can be handled in one of \textit{three} separate ways.
Therefore, rather than using an \textit{if}-condition, the external
  declaration \macname{stripimplicitgroupingcase\{\}} is provided, which takes
  one of 3 integers as its argument (0, 1, or $-$1).
The package-default case of ``0'' indicates that the implicit-group tokens
  will not be stripped, but rather echoed directly into the
  output stream.
The case of ``1'' indicates that the implicit-group tokens
  will be stripped and not placed into the output stream
  (as with explicit grouping, this is a dangerous case and should be
  specified with care).

Finally, the special case of $-$1 indicates that the implicit-group
  tokens should instead be passed to the \Character{} directive for
  further handling (note that the \macname{implicittoktrue} condition will
  be set\footnote{as well as the internal condition \macname{tc@implicitgrptrue}}).
Such a special treatment has limited application---for example,
  when the intent is to detokenize these tokens.

\subsection{Truncating the input stream}

The basic process of \TokCycle{} is one in which an input stream (or argument)
  of tokens is examined and processed based on directives
  set up by the user.
Typically, the input stream is processed token-by-token to exhaustion, which 
  occurs when a defined terminating token is reached.
In section~\ref{s:escape}, we will see how a sequence of tokens can be exempted
  from any processing that would otherwise occur in the directive, 
  and be passed instead directly to the output stream.

However, in this section, we will examine how \TokCycle{} can be directed
  to \textit{dynamically} truncate, that is, discard the remaining 
  input stream based on the tokens found therein.
Truncation commands may be issued within the \Character, \Macro,
  and \Space{} directives.  
More will be said about the \Group{} directive in section~\ref{s:truncingrp}.

Truncation can be made to apply either to the remainder of the current 
  \TokCycle-nesting level (corresponding to an explicit catcode-1,2 
  group in the input stream)\footnote{
Note that implicit groups ({\ttfamily\macname{bgroup}...\macname{egroup}})
  do not create a new \TokCycle{} nesting level, nor do 
  {\ttfamily\macname{begingroup}...\macname{endgroup}}.  
Only explicit catcode-1,2 groups, in the nature of {\ttfamily\{...\}},
  create a nested \TokCycle{} level.
The current \TokCycle{} nesting level may be obtained from the \TeX{} count 
  \macname{tcdepth}, which starts at zero upon \TokCycle{} entry.
This count is incremented and decremented as groups are entered and
  exited in the input stream.
It may be examined within the \CGMS{} directives in order to
  guide decisions.
}
  or for the remainder of the total \TokCycle{} 
  input stream.

\subsubsection{\macname{truncategroup} and \macname{truncategroupiftokis}}
\label{s:truncgrp}

As to truncating the input stream for the remainder of the \TokCycle{}
  nesting level, the command is simply \macname{truncategroup}.
So, for example, the directive 

\begin{verbbox}[\vbdelim]
  \Spacedirective{\truncategroup}
\end{verbbox}
\thevbox
would direct \TokCycle{}, if it ever finds a (catcode-10) space token 
  in the input stream, to discard the remainder of the tokens within 
  the group in which that space was initially found.

More often, however, one would desire to issue the truncation
  \textit{conditionally}.
One could use one of the conditional commands of section~\ref{s:misc},
  for example, 

\begin{verbbox}[\vbdelim]
\tctestifcon{<condition>}{\truncategroup}{<code if condition not met>}
\end{verbbox}
\thevbox
Alternatively, if the \textit{condition} is the occurrence of a particular
  single token in the input stream, one may use an abbreviated syntax:

\begin{verbbox}[\vbdelim]
\truncategroupiftokis{<tok>}{<code if condition not met>}
\end{verbbox}
\thevbox
Thus, an example to echo macro tokens to the output stream, unless
  a \macname{relax} token is found, in which case terminate the group,
  would be

\begin{verbbox}[\vbdelim]
\Macrodirective{\truncategroupiftokis{\relax}{\addcytoks{#1}}}
\end{verbbox}
\thevbox
Obviously, checks for particular character tokens would be placed in the
  \Character{} directive, rather than the \Macro{} directive.

\subsubsection{\macname{truncatecycle} and \macname{truncatecycleiftokis}}

There are two commands that are in every way analogous to the 
  group-\hspace{0pt}terminating macros of section~\ref{s:truncgrp}.
These commands are \macname{truncatecycle} and 
  \macname{truncatecycleiftokis}.
But in this case, all tokens are discarded to the end of the input
  stream, not just the current group (i.e., \TokCycle-nesting level).
One point to note, however, is that open group levels will be
  closed by the \macname{truncatecycle} command, so that the tokens,
  both those being executed as well as those retained in 
  \macname{cytoks}, will be group-balanced.

\subsubsection{Truncating from within the \Group{} Directive}
\label{s:truncingrp}

The truncate commands described in prior sections may \textit{not}
  be executed from the \Group{} directive, as has already been
  mentioned.
What does termination even mean in the \macname{Groupdirective}
  context?
The \Group{} directive is executed whenever an explicit catcode-1,2
  group is encountered in the input stream.
The argument, \argone, to that directive will be the complete
  contents of the group (sans the catcode-1,2 delimiters).

So, to truncate the group from the outset, not doing anything with
  \argone{} will accomplish the desired result.
For example, to truncate all group-level-3 tokens, you could say,
  using the conditional commands of section~\ref{s:misc},

\begin{verbbox}[\vbdelim]
\Groupdirective{\tctestifnum{\tcdepth=3}{}{\processtoks{#1}}}
\end{verbbox}
\thevbox
There is no need to use an explicit \macname{truncategroup} macro
  to accomplish it.

On the other hand, to truncate the complete token cycle if, for example,
  group-level-3 tokens are encountered, you have to set some sort of
  flag that is immediately picked up by one of the other directives.
Here is an example of how that can be done:

\begin{verbbox}[\vbdelim]
\Groupdirective{\tctestifnum{\tcdepth=3}{\processtoks{\truncatenow}}%
                                        {\processtoks{#1}}}
\Macrodirective{\truncatecycleiftokis{\truncatenow}{\addcytoks{#1}}}
\end{verbbox}
\thevbox
Note that, as long as \macname{stripgroupingfalse} is active, the above
  examples will result in empty, rather than absent, level-3 groups,
  since the group open occurs before the truncation is processed.
If the presence of even an empty group is to be avoided in such a case,
  one may employ the \macname{groupedcytoks} techniques described in
  section~\ref{s:grpcy}.

\subsection{Escaping content from \TokCycle{} processing\label{s:escape}}

There are times you may wish to prevent tokens in the \TokCycle{} 
  input stream from being operated on by \TokCycle.
Rather, you just want
  the content passed through unchanged to the output;  that is,
with the intent to have multi-token content bypass the 
  \TokCycle{} directives altogether.

\begin{sloppypar}
The method developed by the package is to enclose the escaped
  content in the input stream between a set of \TokCycle{} escape 
  characters, initially set to a vertical rule character found 
  on the keyboard: ``$|\mkern1mu$''.
The main proviso is that the escaped content cannot straddle a 
  group boundary (the complete group may, however, be escaped).
The escape character can be changed with
\begin{verbbox}[\vbdelim]
\settcEscapechar{<escape-token>}\end{verbbox}
\theverbbox
$\!\!\!$.
\end{sloppypar}

\subsection{Flagged tokens}

Certain token types are trapped and flagged via true/false 
  \textit{if}-conditions.
These \textit{if}-conditions can be examined within the appropriate 
  directive (generally the \Character{} directive), to direct the
  course of action within the directive.

\subsubsection{Active characters}

\paragraph{\macname{ifactivetok}:}

Active (cat-13) tokens that occur in the input stream result in the
  flag \macname{ifactivetok} being set \macname{activetoktrue}.
Note that the expansion of the token's active \macname{def}
  occurs \textit{after} \TokCycle{} processing.
With active \macname{let}'s, there is no text substitution; however,
  the assignment is already active at the time of \TokCycle{} processing.
%
The only exception to this rule is with pre-expanded input,
  \macname{expandedtokcycle[xpress]}.
If an active token's substitution is governed by a \macname{def}, the 
  text substitution will have occurred before reaching the token cycle.

\paragraph{\macname{ifactivetokunexpandable}:} 

This flag is similar to \macname{ifactivetok}, in that a token must
  be active for this to be set true.
However, in addition, it is only
  true if the active token is \macname{let} to a character or a 
  primitive, neither of which can be expanded.
Active characters assigned via \macname{def} or else \macname{let} to a
  macro will \textit{not} qualify as \macname{activetokunexpandabletrue}.

\paragraph{\macname{ifactivechar}:}

This flag, rather than testing the token, tests the character code of
  the token, to see if it is set active.  
Generally, the token and its character code will be syncronized
  in their \textit{activeness}.
However, if a token is tokenized when active, but the corresponding character 
  code is made non-active in the meantime, prior to the token reaching
  \TokCycle{} processing, this flag will be set \macname{activecharfalse}.
A similar discrepency will arise if a token is not active when tokenized,
  but the character code is made active in the interim, prior to
  \TokCycle{} processing.

\subsubsection{Implicit tokens: \macname{ifimplicittok}}

Implicit tokens, those assigned via \macname{let} to characters\footnote{%
Some clarification may be needed.
Control sequences and active characters that are \macname{let} 
  to something other than a cat-0 control sequence will be flagged 
  as implicit.
If implicit, a token will be processed 
  through the \Character{} directive (exceptions noted).
On the other hand, if a control sequence or active character is \macname{let} 
  to a cat-0 control sequence, it will be directed to the \Macro{} 
  directive for processing, without the implicit flag.
}%
  , are flagged with
  a \texttt{true} setting for \macname{ifimplicittok}.
Generally, implicit tokens will be directed to the \Character{}
  directive for processing.
There are, however, two exceptions:
i)~implicit grouping tokens (e.g., \macname{bgroup} and \macname{egroup})
  will not appear in any directive unless the
  \macname{stripimplicitgroupingcase} has been set to $-$1; and
ii)~implicit space tokens (e.g., \macname{@sptoken}) will be processed 
  by the \Space{} directive.

\subsubsection{Active-Implicit Tokens, Including Spaces}

One may occasionally run across a token that is both active and implicit.
For example, in the following code, \texttt{Q} is made both active and implicit:

\begin{verbbox}[\vbdelim]
\catcode`Q=\active
\let Qx
\end{verbbox}
\thevbox
In general, both \macname{ifactivetok} and \macname{ifimplicittok}
  tests can be performed together to determine such cases.

This is true even in the case of active-implicit catcode-10 spaces, which
  are always processed through the \Space{} directive.
However, because of the process \TokCycle{} uses in digesting space
  tokens, the actual token passed to the \Space{} directive (so-called 
  \verb|#1|), when an active-implicit space is encountered in the input 
  stream, is a generic implicit space token named \macname{tc@sptoken}.
However, the catcode-12 version of the active character that 
  produced it will be, for that moment, retained in a definition
  named \macname{theactivespace}.
This can be useful if detokenization is required of the spaces.
Such an example is described in the \texttt{tokcycle-examples} 
  adjunct document.


\subsubsection{Parameter (cat-6) tokens (e.g., \texttt{\#}): \macname{ifcatSIX}}
\label{s:cat6}

Typically, category-code 6 tokens (like \texttt{\#}) are used to 
  designate a parameter (e.g., \argone--\texttt{\#9}).
Since they are unlikely to be used in that capacity inside a 
  tokcycle input stream, the package behavior is to convert them
  into something cat-12 and set the \textit{if}-condition \macname{catSIXtrue}.
In this manner, \macname{ifcatSIX} can be used inside the
  \Character{} directive to trap and convert cat-6 tokens into something of
  the user's choosing.

\begin{sloppypar}
As to the default nature of this conversion (if no special actions are
  taken), explicit cat-6 characters are converted
  into the identical character with category code of 12.  
On the other hand, implicit cat-6 macros (e.g., \verb|\let\myhash#|) 
  are converted into a fixed-name macro, \macname{implicitsixtok}, 
  whose cat-12 substitution text is a \macname{string} of the original
  implicit-macro name.
\end{sloppypar}

\subsubsection{Parameters (\argone--\texttt{\#9}): 
\macname{whennotprocessingparameter}}
\label{s:whennot}

While, generally, one would not intend to use parameters 
  in the \TokCycle{} input stream, the package provides, not a flag,
  but a macro to allow it.
The macro is to be used \textit{within} the \Character{} directive
  with the syntax:\smallskip

\begin{verbbox}[\vbdelim]
\whennotprocessingparameter#1{<non-parameter-processing-directive>}\end{verbbox}
\theverbbox\smallskip

\noindent Here, the \argone{} doesn't refer to \argone{} as
  it might appear in the input stream, but to the sole parameter of
  the \Character{} directive, referring to the current token being
  processed.
With this syntax, when the token under consideration is a 
  parameter (e.g., \argone--\texttt{\#9}), that parameter is 
  added to the \macname{cytoks} register.  
If the token under consideration is not a parameter, the code
  in the final argument is executed.

\subsection{Misc: general \textit{if}-condition tools}
\label{s:misc}

\TeX{} comes equipped with a variety of \macname{if...} condition
  primitives.
When dealing with macros for which the order of expansion is important,
  the \macname{else} and \macname{fi} can sometimes get in the way
  of proper expansion and execution.
These four restructured \textit{if} macros came in handy writing the 
  package, and may be of use in creating your directives,
  preventing \macname{else} and \macname{fi} from getting in the 
  way:\medskip

\begin{verbbox}[\vbdelim]
\tctestifcon{<TeX-if-condition>}{<true-code>}{<false-code>}
\tctestifx{<\ifx-comparison-toks>}{<true-code>}{<false-code>}
\tctestifnum{<\ifnum-condition>}{<true-code>}{<false-code>}
\tctestifcatnx<tok1><tok2>{<true-code>}{<false-code>}
\end{verbbox}
\theverbbox
  
\medskip\noindent
Note that unlike those macros requiring three arguments, the lone
  exception is \macname{tctestifcatnx}, which requires four, comparing the
  catcodes of the two input toks \textbf{without expansion} (\texttt{nx}
  denotes \macname{noexpand}).

\subsection{\TokCycle{} Completion: \macname{aftertokcycle}
  and \macname{tcendgroup}}

The \TokCycle{} \textit{macros}, upon completion, do nothing.
Unlike \macname{tokencycle} \textit{environments}, they don't even 
  output \macname{the}\macname{cytoks} token register.
A command has been provided, \macname{aftertokcycle}, which takes 
  an argument to denote the actions to be executed following
  completion of \textit{all} subsequent \TokCycle{} \textit{macro}
  invocations within the scope of the current group.
It might be specified as simply as \verb|\aftertokcycle{\the\cytoks}|,
  so as to have the output stream automatically typeset by 
  future invocations of the \macname{tokcycle} macro.

The meaning of \macname{aftertokcycle} can be reset with
  \macname{aftertokcycle\{\}}, but is also reset as part of
  \macname{resettokcycle}.
Unlike macros, the \TokCycle{} environments are unaffected
  by \macname{aftertokcycle}, as they actually set it locally 
  (within their grouped scope) to accomplish their own purposes.

Speaking of the \TokCycle{} environments, there is a command employed by
  the environments \macname{tokencycle} and \macname{tokencyclexpress}
  that is likewise available to the user: \macname{tcendgroup}.
This is a version of \macname{endgroup} that additionally transmits the 
  current value of the \macname{cytoks} token list \textit{outside of 
  the group}.
So if a user wishes to confine certain \TokCycle{} activity inside a 
  group, but wishes to have access to the \macname{cytoks} result afterward,
  he may simply wrap the activity in \verb|\begingroup...\tcendgroup|.

\subsection{Accommodating catcode-1,2 changes: \macname{settcGrouping}%
  \label{s:grouping}}

In order to avoid making the \TokCycle{} parser overly complex, requiring
  multiple passes of the input stream, the package defaults to using
  catcode-1,2 braces \texttt{\{~\}} to bring about grouping in the
  output stream, regardless of what the actual cat-1,2 tokens are in
  the input stream.
As long as their sole purpose in the token 
  cycle is for grouping and scoping, this arrangement will produce
  the expected output.

However, if the actual character-code of these tokens is important
  to the result (e.g., when detokenized), there is one other option. 
The package allows the external specification of which cat-1,2
  tokens should be used in the \TokCycle{} output stream.
The syntax is \verb|\settcGrouping{{#1}}|, to use the standard braces
  for this purpose (default).  
If angle-brackets \verb|< >| were to be the \textit{new} 
  grouping tokens, then, after their catcodes were changed, and
  \macname{bgroup} and \macname{egroup} were reassigned, one would
  invoke \verb|\settcGrouping<<#1>>|.
These will then be the grouping tokens in the \TokCycle{} output
  stream until set to something else.

\section{Usage Examples}

See the adjunct file, \texttt{tokcycle-examples.pdf}, for an 
  array of \TokCycle{} examples.

\section{Summary of known package limitations}

The goal of this package is not to build the perfect token-stream
  parser.
It is, rather, to provide the means for users to build useful 
  token-processing tools for their \TeX/\LaTeX{} documents.

What follows are the known limitations of the package.
  which arise, in part, from the single-pass parsing algorithm
  embedded in the package.
Surely, there are more cases associated with arcane catcode-changing
  syntax that are not accounted for; 
I encourage you to bring them to my attention.
If I can't fix them, I can at least disclaim and declaim them below:

\begin{itemize}

\item One must inform the package (via \macname{settcGrouping}) of 
  changes to the cat-1,2 tokens \textit{if}
  there is a need to detokenize the output with the specified
  bracing group; however, 
  grouping will still be handled properly (i.e., cat-1,2 tokens will
  be detected), even if the package is not notified.
See section~\ref{s:grouping}.

\item Should one need to keep track of the \textit{names} of implicit 
  cat-6 tokens (e.g., \verb|\svhash| following \verb|\let\svhash#|), 
  care must be exercised.
When encountered in the input stream, a cat-12 \macname{string} of 
  the implicit-cat-6 token \textit{name} is stored in the \macname{implicitsixtok} 
  macro.
It is the \macname{implicitsixtok} token which is passed to the 
  \Character{} directive as \verb|#1|, in lieu of the actual cat-6 token.
If there is at most a single implicit-cat-6 named token in the input stream,
  then \macname{implicitsixtok} will reflect its name.
However, if there is more than one, \macname{implicitsixtok} will be redefined
  on the fly with each occurrence.

Therefore, should one intend to add more than one such token \textit{name}
  to the token list by way of \macname{addcytoks}, \textit{it must be expanded 
  first} (e.g., \macname{addcytoks[1]\{\argone\}}), in order to capture the 
  \textit{name} of that particular cat-6 implicit token, rather than the name 
  of the last one in the input stream.
There is no similar problem for explicit cat-6 tokens.
In any event, all cat-6 tokens are trapped and flag as \texttt{true}
  the \macname{ifcatSIX} condition, which is how the user should detect
  their presence.

\item A similar warning is to be made for active-implicit spaces as was
  made for implicit-cat-6 tokens.  
In the case of active-implicit spaces, the name of the active character is 
  momentarily stored in the \macname{theactivespace} macro.
If one wishes to output the active \textit{name} to the \macname{cytoks} token list,
  one must make sure that \macname{theactivespace} is expanded once before
  being added to the token list.

\end{itemize}

\section*{Acknowledgments}

In addition to Christian Tellechea, a contributor to this 
  package,
the author would like to thank Dr.\@ David Carlisle for his 
  assistance in understanding some of the nuances of token
  registers.
Likewise, his explanation about how a space token is defined 
  in \TeX{} (see \verb|https://tex.stackexchange.com|\allowbreak
  \verb|/questions/64197/|\allowbreak\verb|pgfparser-module-and-blank-spaces/64200#64200|) 
  proved to be useful here.
The \texttt{tex.stackexchange} site provides a wonderful
  opportunity to interact with the leading developers and
  practitioners of \TeX{} and \LaTeX{}.

\vspace{1in}
\section*{Source Code}

%\vspace{-0.1in}
\Large\texttt{tokcycle.sty}
\verbfilenobox[\footnotesize]{tokcycle.sty}%
%\vspace{-0.1in}

\noindent\texttt{tokcycle.tex}
\verbfilenobox[\footnotesize]{tokcycle.tex}


\end{document}

