BeautifulSoup4 vs AngleSharp: Parsing 50,000 HTML Documents

Overview

BeautifulSoup4 is the default HTML parsing library for Python. It powers web scrapers, data pipelines, content extractors, and testing utilities across millions of projects. When you need to parse an HTML page, find all links, extract table data, or pull headings — BS4 with html.parser is almost always the first thing you reach for.

html.parser is Python's built-in HTML parser, written entirely in Python. Every token — every tag open, attribute, text node, tag close — is processed by a Python method call. BS4 then wraps the resulting parse tree in a layer of Python objects: Tag, NavigableString, ResultSet. Querying with find_all("a") walks a Python list of Python objects.

AngleSharp is a pure C# HTML5-conformant parser. It tokenises HTML using a compiled state machine, builds a typed DOM tree in managed memory, and exposes it through standard CSS selector querying. Everything is JIT-compiled and operates on contiguous memory.

This benchmark runs both on 50,000 HTML documents — the same pipeline: parse → collect all <a href> links → count all <td> cells → read the <h1> text.

Benchmark Setup

• Corpus: 50,000 synthetic HTML product-listing pages (mix of nav links, a data table with 5–15 rows, a linked-product list)
• Pipeline: parse → find_all("a", href=True) → find_all("td") → find("h1").get_text()
• Python: beautifulsoup4 4.12, built-in html.parser
• .NET: AngleSharp 1.x, HtmlParser.ParseDocument(), QuerySelectorAll()
• Validation: href count and td count must match exactly between runtimes

Results

At 5,000 documents .NET is 6× faster. The advantage compounds: at 50,000 documents the gap reaches 17.4× — both the parser and the query engine benefit from JIT warm-up while Python's per-call overhead grows linearly.

Why the Gap Exists

The html.parser tokeniser. Python's built-in html.parser processes each HTML token by calling a Python method: handle_starttag, handle_endtag, handle_data. For a typical page with 80–120 tags, that is 160–240 Python method dispatches per document. At 50,000 documents that is up to 12 million Python method calls just for tokenisation.

BeautifulSoup's object layer. BS4 wraps every token from the parser into a Python object. Every tag becomes a Tag instance with a __dict__, parent/child references, a sibling linked list, and an attribute dict. A page with 80 tags creates 80 Python heap objects, each requiring allocation, initialisation, and eventual GC. For 50,000 pages, that is 4 million objects created and discarded.

find_all traversal. soup.find_all("a", href=True) walks the parse tree through Python generator chains — __iter__ on every Tag, type checks in Python, attribute dict lookups for each node. The result is a Python list built one Python object at a time.

AngleSharp's tokeniser. AngleSharp implements the HTML5 tokenisation spec as a compiled C# state machine. Each character transition is a table lookup or a switch branch in JIT-compiled native code. No Python method calls — no dispatch overhead at all.

AngleSharp's DOM. The DOM tree nodes are typed C# objects on the managed heap. QuerySelectorAll("td") compiles the CSS selector once and evaluates it through a DFS over strongly-typed node references — the JIT can inline property accesses and eliminate virtual dispatch for sealed types. No boxing, no Python attribute dicts.

Key Code

# BeautifulSoup4 — html.parser dispatches a Python method per token
from bs4 import BeautifulSoup

def parse_one(html: str):
    soup  = BeautifulSoup(html, "html.parser")          # 160-240 Python method calls
    hrefs = [a["href"] for a in soup.find_all("a", href=True)]  # Python generator walk
    tds   = len(soup.find_all("td"))                    # second full tree traversal
    h1    = soup.find("h1").get_text(strip=True)        # Python string allocation
    return len(hrefs), tds, h1

// AngleSharp — compiled state machine, CSS selector query, zero Python overhead
var parser = new HtmlParser();   // reused across all documents

using var doc = parser.ParseDocument(html);     // HTML5 tokeniser → typed DOM
var hrefs = doc.QuerySelectorAll("a[href]");    // compiled CSS selector
var tds   = doc.QuerySelectorAll("td").Length;
var h1    = doc.QuerySelector("h1")?.TextContent?.Trim();

The AngleSharp HtmlParser instance is created once and reused across all 50,000 documents. Its internal state machine and compiled CSS selectors persist in memory. BS4 reconstructs its internal tree structures fresh for every call to BeautifulSoup(html, ...).

The Compounding Effect

The speedup grows with document count — not because there is any warm-up for the Python side, but because AngleSharp's JIT improves over the first few hundred documents and then plateaus. The JIT compiles the ParseDocument hot path, inlines the CSS selector evaluation, and eliminates virtual dispatch on the sealed DOM node types.

Python has no equivalent mechanism. Every call to BeautifulSoup(html, "html.parser") dispatches the same Python bytecodes at the same speed. There is no JIT, no profile-guided optimisation, no inline caching across calls.

Real-World Impact

A content pipeline crawling 500,000 pages per night: Python finishes in eleven minutes, .NET finishes in thirty-five seconds. At 5 million pages the difference is measured in hours.

Why Not lxml?

lxml wraps libxml2, a C library. Replacing html.parser with lxml in BS4 (BeautifulSoup(html, "lxml")) speeds up the tokenisation step significantly. But it does not change BS4's own overhead: the Python object layer, the find_all traversal, and the get_text allocation remain. In practice, BS4 with lxml is roughly 2–3× faster than with html.parser — still 5–8× behind AngleSharp.

If you need maximum Python HTML parsing speed you would use lxml's own API (lxml.etree) and XPath directly. That removes BS4's overhead entirely. But then you are also giving up BS4's ergonomics — the reason most engineers reached for it in the first place.