Rewrite of my HTML minifier

I have a custom HTML minifier that I use with my website. Recently, with the growing number of pages on my website, running the minifier on all the pages became a bit slow.

I didn’t want to bother with re-writing the minifier in another language like C++, but I also didn’t want to interrupt my edit-build-preview cycle by waiting for a slow minifier to finish. So I profiled the minifier script with Python’s built-in cProfile module, and found some hotspots.

The first thing that caught my attention was the function that minified the CSS using scss, the SASS compiler. This function was mainly called with two kinds of input.

1. The main CSS of the website.
2. Individual style attributes of HTML tags.

Both of those are repeated a lot within the same page, and across pages. So I cached the results in RAM using the functools.cache decorator. This was essentially a one-line change, and it ended up giving a decent speedup.

The other optimization was a lot more involved. In order to iterate the HTML elements and output a minified version, I was parsing each HTML file using BeautifulSoup with the html5lib parser. This turned out to be very slow, and accounted for most of the time spent in the minifier.

I tried switching the parser from html5lib to lxml, but that didn’t help too much. Even with the faster parser, I was constructing a DOM tree in memory and doing a ton of unnecessary work due to the use of BeautifulSoup.

I ripped all of that out, and re-wrote most of the minifier using the built-in html.parser module. This module allows you to parse HTML documents in a low-level way. Basically, you feed the parser a stream of bytes, and it runs callback functions whenever a tag starts or ends.

Aside from the current tag, nothing else is stored in memory, and no DOM tree is created. The callbacks provide enough information to do the minification correctly.

This change gave a huge speedup. The minifier used to run on multiple processes and take around two minutes on all cores. Now it runs on a single thread and completes in two seconds (for around 450 pages).

I think if I add so many pages to my website that the minifier becomes slow again, I’ll just enable multiprocessing and get a speedup that way. But for now, it doesn’t look like I will need to optimize this minifier any further.