Syntax Manual :: Chapter 4: Extending Syntax

Syntax— Lexical Analysis for Syntax Highlighting | Syntax Version: 1.0.0 Manual Last Updated: 2005-06-18 20:25 UTC

| Previous (3. Syntax Highlighting) | Up # 4. Extending Syntax ## 4.1. Introduction Because of Syntax’s modular design, it is pretty straightforward to create your own syntax modules. The hardest part is doing the actual tokenizing of your chosen syntax. You can use the existing syntax modules to guide your own implementation if you wish, but note that each module will have to solve unique problems, because of the uniqueness of each different syntax. ## 4.2. Interface Your new syntax implementation should extend `Syntax::Tokenizer`—this sets up a rich domain-specific language for scanning and tokenizing. Then, all you need to implement is the `#step` method, which should take no parameters. Each invocation of `#step` should extract at least one token, but may extract as many as you need it to. (Fewer is generally better, though.) Additionally, you may also implement `#setup`, to perform any initialization that should occur when tokenizing begins. Similarly, `#teardown` may be implemented to do any cleanup that is needed. ## 4.3. Scanning API Within a tokenizer, you have access to a rich set of methods for scanning the text. These methods correspond to the methods of the StringScanner class (i.e., `scan`, `scan_until`, `bol?`, etc.). Additionally, subgroups of recent regexps (used in `scan`, etc.) can be obtained via `subgroup`, which takes as a parameter the group you want to query. Tokenizing proceeds as follows: Identify a token (using `#peek`, `#scan`, etc.). Start a new token group (using `#start_group`, passing the symbol for the group and optionally any text you want to seed the group with). Append text to the current group either with additional calls to `#start_group` using the same group, or with `#append` (which just takes the text to append to the current group) Instead of `#start_group`, you can also use `#start_region`, which begins a new region for the given group, and `#end_region`, which closes the region. Here is an example of a very, very simple tokenizer, that simple extracts words and numbers from the text: Simple tokenizer [ruby]
1 2 3 4 5 6 7 8 9 10 11 12 13 |
    require 'syntax'

    class SimpleTokenizer < Syntax::Tokenizer
      def step
        if digits = scan(/\d+/)
          start_group :digits, digits
        elsif words = scan(/\w+/)
          start_group :words, words
        else
          start_group :normal, scan(/./)
        end
      end
    end
## 4.4. Registering Your New Syntax Once you’ve written your new syntax module, you need to register it with the Syntax library so that it can be found and used by the framework. To do this, just add it to the `Syntax::SYNTAX` hash: Registering a new syntax [ruby]
1 2 3 4 5 6 7 |
    require 'syntax'

    class SimpleTokenizer < Syntax::Tokenizer
      ...
    end

    Syntax::SYNTAX['simple'] = SimpleTokenizer
That’s it! Once you’ve done that, you can now use your syntax just by requiring the file that defines it, and then using the standard Syntax framework methods: Using your new syntax [ruby]
1 2 3 4 5 |
    require 'simple-tokenizer'
    require 'syntax/convertor/html'

    convertor = Syntax::Convertors::HTML.for_syntax "simple"
    puts convertor.convert( "hello 15 worlds!" )
Previous (3. Syntax Highlighting) | Up