Skip to content

zimscraperlib.rewriting.js_ast

The little bit of JavaScript parsing the JS rewriter needs.

js.py wraps a script in a block so wombat can shadow window, document and friends. A block is a scope, so every top-level const, let and class in the script becomes block-scoped too, and stops being visible to any other script on the page. wabac.js solves this by parsing the script and hoisting those names back out; this module is the parsing half of that, kept behind one function so the choice of parser is one import to change.

Only the top level matters. Nothing nested can leak a global, so this never walks into a function body, and it answers three questions:

  • which const, let, var and class names the script declares at the top level, and of what kind
  • where each declaration starts, so a let keyword can be removed
  • whether the script calls document.write() at the top level

Why tree-sitter and not a pure-Python parser: the scripts this runs on are whatever the live web served. esprima (the obvious pure-Python choice) is ES2017 and refuses optional chaining, class fields and for await, all of which are ordinary in shipped code today; tree-sitter parses them, and is error-tolerant besides, so a script it cannot fully understand still yields the declarations it could read rather than an exception.

Classes:

Functions:

  • node_text –

    The source a node covers. A missing node reads as no text, so callers

  • parse_top_level –

    Read a script's top-level declarations, or None when it cannot be read.

Declaration dataclass 

Declaration(name: str, kind: str, start: int)

One name a script declares at its top level.

Attributes:

kind instance-attribute 

kind: str

name instance-attribute 

name: str

start instance-attribute 

start: int

TopLevel dataclass 

TopLevel(
    declarations: list[Declaration],
    has_document_write: bool,
)

Attributes:

declarations instance-attribute 

declarations: list[Declaration]

has_document_write instance-attribute 

has_document_write: bool

node_text 

node_text(node: Node | None, source: bytes) -> str

The source a node covers. A missing node reads as no text, so callers can ask for an optional field without a guard at every site.

Source code in src/zimscraperlib/rewriting/js_ast.py 
53
54
55
56
57
58
def node_text(node: Node | None, source: bytes) -> str:
    """The source a node covers. A missing node reads as no text, so callers
    can ask for an optional field without a guard at every site."""
    if node is None:
        return ""
    return source[node.start_byte : node.end_byte].decode("utf-8", errors="replace")

parse_top_level 

parse_top_level(text: str) -> TopLevel | None

Read a script's top-level declarations, or None when it cannot be read.

None means "no opinion", and the caller leaves the script alone — which is what happened to every script before this existed. wabac.js wraps its whole parseGlobals in a try/catch for the same reason, and so does this: nothing here may throw into a scrape.

Source code in src/zimscraperlib/rewriting/js_ast.py 
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
def parse_top_level(text: str) -> TopLevel | None:
    """Read a script's top-level declarations, or None when it cannot be read.

    None means "no opinion", and the caller leaves the script alone — which is
    what happened to every script before this existed. wabac.js wraps its whole
    parseGlobals in a try/catch for the same reason, and so does this: nothing
    here may throw into a scrape."""
    try:
        source = text.encode("utf-8")
        root = _PARSER.parse(source).root_node
        declarations: list[Declaration] = []
        has_document_write = False
        for node in root.named_children:
            if node.type == "lexical_declaration":
                # `const` or `let` — `using` has its own node type.
                kind = node_text(node.children[0], source)
                for name in _identifiers(node, source):
                    declarations.append(Declaration(name, kind, node.start_byte))
            elif node.type == "variable_declaration":
                for name in _identifiers(node, source):
                    declarations.append(Declaration(name, "var", node.start_byte))
            elif node.type == "class_declaration":
                name_node = node.child_by_field_name("name")
                declarations.append(
                    Declaration(node_text(name_node, source), "class", node.start_byte)
                )
            elif not has_document_write and _is_document_write(node, source):
                has_document_write = True
        return TopLevel(
            declarations=declarations, has_document_write=has_document_write
        )
    except Exception:
        return None