Building a Tiny Note Taking App Grammer

Introduction

Growing up, Pidgin English was everywhere. It is essentially a blend of broken English, slang, and words borrowed from native languages. The look on people's faces when they hear me speaking Pidgin English over the phone always cracks me up. It's usually a mixture of surprise and confusion—confusion because they can't quite make sense of what I'm saying, and surprise because they can still recognize a few English words.

Like every language, Pidgin English has rules. They may not be formally written down, but native speakers instinctively know when something "sounds right" or "sounds wrong." That got me thinking: if every language follows some set of rules, why do we need to formally define a grammar when designing a programming language?

Motivation

While reading a book on programming language design, I reached the chapter on context-free grammars and realized I didn't truly understand them. I could follow the examples, but I couldn't explain why the grammar came before the lexer or how a parser used those grammar rules to construct a syntax tree.

Rather than pushing ahead, I decided to take a detour. Instead of building a full programming language, I designed a tiny markup language for a note-taking application. It was small enough to experiment with, yet rich enough to explore grammars, tokenization, parsing, and rendering from first principles.

Designing the Language

I wanted the language to be small enough that I could understand every part of it, yet expressive enough to demonstrate the core concepts involved in language design. I began by defining a Document, which serves as the root node of every document. From there, I introduced the concept of block elements, consisting of headings, paragraphs, and lists. Finally, I defined a set of inline elements that can appear within blocks: bold, italic, underline, links, and plain text.

The distinction between block and inline elements is important. Block elements define the overall structure of a document, while inline elements describe how content within a block should be presented. For example, a heading or paragraph occupies its own section of the document, whereas bold or italic formatting only affects a portion of the text inside those blocks.

Defining the Grammar

With the language constructs defined, the next step is to formally describe how they fit together. This is the job of a grammar. A grammar is simply a set of production rules that describe the valid structure of a language.

Each production has the form A -> B, which can be read as "A consists of B" or "A can be expanded into B." The symbol + means "one or more" occurrences of the preceding item.

I begin by describing what a document can contain. Since a document is the root of the language, it consists of one or more block elements.

Document -> Block+

Next, I define what a Block is. A block can be one of three things: a Paragraph, a List, or a Heading. The pipe (|) symbol simply means "or." You can therefore read the following production as "A block is either a paragraph, a list, or a heading."

Block -> Paragraph | List | Heading

A Paragraph begins with a paragraph marker, contains one or more inline elements, and ends with another paragraph marker. The same marker is used to indicate both the beginning and end of a paragraph; it is the parser's responsibility to determine whether a marker is opening or closing the paragraph.

Paragraph -> PARAGRAPH_MARKER Inline+ PARAGRAPH_MARKER

A List begins with a list marker, contains one or more list items, and ends with another list marker. Since every list must contain at least one item, the grammar uses the + operator to indicate one or more list items.

List -> LIST_MARKER ListItem+ LIST_MARKER

Each ListItem begins with a list item marker and may contain one or more inline elements. A list item may also contain another list, allowing lists to be nested to an arbitrary depth.

ListItem -> LIST_ITEM_MARKER (Inline | List)+ LIST_ITEM_MARKER

An Inline element represents content that can appear inside block elements. In this language, an inline may be bold text, italic text, underlined text, plain text, or a hyperlink.

Inline -> Bold | Italic | Underline | Text | Link

Bold, italic, and underline all follow the same pattern. Each begins with its respective marker, contains one or more inline elements, and ends with the same marker. This allows formatting to be nested, making expressions such as bold text containing italic text perfectly valid.

Bold -> BOLD_MARKER Inline+ BOLD_MARKER
Italic -> ITALIC_MARKER Inline+ ITALIC_MARKER
Underline -> UNDERLINE_MARKER Inline+ UNDERLINE_MARKER

A Link is slightly different because it stores both a destination and the text that will be displayed to the user. I therefore introduced the concept of a link target, followed by a delimiter, before the inline content.

Link -> LINK_MARKER LINK_TARGET Delimiter Inline+ LINK_MARKER

Finally, a Heading begins with a heading marker followed by one or more inline elements. The heading marker also stores the heading level, allowing the language to distinguish between first, second, and third level headings.

Heading -> HEADING Inline+

Although the grammar appears simple, it serves as the blueprint for everything that follows. The tokenizer uses it to identify the language's terminals, while the parser follows these production rules to construct the document tree. One of the biggest lessons I learned during this project was that defining the grammar first made both the tokenizer and parser significantly easier to design.

Building the Tokenizer

With the grammar complete, I now have a clear list of terminals that make up my language. The responsibility of the tokenizer is simply to scan through the source text and emit these terminals as tokens. It does not understand the structure of the language or whether a bold marker is opening or closing a bold section. Its only responsibility is to recognize valid terminals and produce a stream of tokens that the parser can later consume.

For example, given the text ### Hello **World**, the tokenizer recognizes a heading marker, plain text, a bold marker, more text, and finally another bold marker. The parser will later determine that the second bold marker closes the first.

Writing the Parser

Unlike the tokenizer, the parser understands the structure of the language. It consumes the token stream produced by the tokenizer and builds a hierarchical representation of the document. One realization that greatly simplified my implementation was that every production rule in my grammar naturally became a parsing function. For example, the production Heading -> HEADING Inline+ translated almost directly into a parseHeading() function.

It is also the parser that gives meaning to the token stream. Since the tokenizer emits the same BOLD_MARKER token for both the opening and closing marker, the parser is responsible for determining whether a marker begins or ends a bold section.

Constructing the Document Model (AST)

The parser produces what is commonly referred to as an Abstract Syntax Tree, or AST. Although the name may sound intimidating, it is simply a tree that represents the structure of the document. Each node in the tree corresponds to one construct in the language such as a heading, paragraph, bold section, or text node.

One advantage of using an AST is that it becomes the single source of truth for the document. Rather than manipulating HTML directly, the application modifies the tree and allows different renderers to transform it into other representations. In my implementation, every node is also assigned a unique identifier so it can be efficiently located later using an index.

Rendering HTML

Once the document tree has been constructed, rendering HTML becomes a relatively straightforward task. The renderer simply walks the tree recursively and generates the corresponding HTML for each node. A heading node becomes an HTML heading element, a bold node becomes a <b> element, and plain text becomes a text node.

I chose to implement the renderer using the Visitor pattern. This keeps the rendering logic separate from the parser and allows me to easily add other renderers in the future, such as Markdown, PDF, or even a custom editor view.

End-to-End Example

To demonstrate the complete pipeline, consider the following input:

### Hello **World**

The tokenizer first converts the character stream into the following tokens:

HEADING(level=3)
TEXT("Hello ")
BOLD_MARKER
TEXT("World")
BOLD_MARKER

The parser then consumes these tokens to construct the following document tree:

Document
└── Heading(level=3)
    ├── Text("Hello ")
    └── Bold
        └── Text("World")

Finally, the HTML renderer walks the tree and produces the following HTML:

<h3>Hello <b>World</b></h3>

Although this example is intentionally simple, it demonstrates the complete lifecycle of a document as it moves from plain text to a structured tree and finally into rendered HTML.

Lessons Learned

Before starting this project, grammars felt almost like a dark art. By designing a small language from scratch, I realized they are simply a formal way of describing the valid structure of a language. Once the grammar was defined, the responsibilities of the tokenizer and parser became much clearer. The tokenizer recognizes terminals, while the parser understands structure and context.

Perhaps the biggest takeaway was realizing that building a compiler is less about writing clever code and more about carefully separating responsibilities. Each component performs one well-defined task, making the overall system surprisingly easy to reason about.

Next Steps

Building the language, tokenizer, parser, and renderer was only the first milestone. My original goal was to build a rich text editor that could eventually support collaborative editing across multiple devices. The next challenge is allowing users to edit the document while treating the document tree as the single source of truth.

Future articles will explore document models, selections, editing commands, incremental rendering, and real-time synchronization. Ironically, what began as a detour to better understand grammars has evolved into one of the most enjoyable projects I have worked on.