Possible low hanging optimization fruit

[foxfirefey]

So, I've been doing some heavy web page optimization at my day job and on a whim I ran an analyzer on some DW pages and found what I think could be a very easy-to-make-happen optimization: we're not gzipping the JS/CSS static files when we serve them.

Examples:

* The new entry page: 221.2KiB (70% reduction) -- quite the savings when the entire bundle is 349.9 KB
* My reading page: 149.3KiB (68% reduction) -- when the total page is 541.5 K, so decent

Would this be as easy to set up as I think it would, or are there other reasons not to do it?

Comments

[mark]

It would be nice to do, but it would be non-trivial since those files are served by Perlbal which doesn't have the ability to gzip for us. It wouldn't be hard to add, but someone would have to. (At least, I think it doesn't. Pretty sure nobody ever implemented that.)

[foxfirefey]

Oh, good to know, thanks!

[exor674]

If we add it, would it be worth it to add gzipping to the build-static.pl script, that way we don't have to gzip the static files on the fly?

[pauamma]

What are you doing in my brain? :-)

[mark]

I expect that would be hard to do because of CONCAT_RES. We'd have to know what concatenations we're using (there are several, depends on the pages) and that'd be a hard list to maintain. We'd inevitably forget one.

Doing the compression on the fly is fine since we cache the results in varnish. We wouldn't be compressing them very often, just when they change or when they fall out of the cache.

Since doing it at build time doesn't save us much, I'd be disinclined to take the effort of doing that.

[alierak]

... but I think Varnish can?

[mark]

I am pretty sure it can't, but I'd be excited to learn otherwise!

[alierak]

Commence excitement!

[foxfirefey]

Eeee!

[mark]

varnishd (varnish-3.0.2 revision cbf1284)

EXCITEMENT. Added to the config:

    # gzip outgoing HTML/CSS/JS.
    if (req.url ~ "\.(html|css|js)(\?v=\d+)?$" &&
            beresp.http.content-type ~ "text") {
        set beresp.do_gzip = true;
    }

So far, it looks good. I see static content is responding to gzip requests, and varnishd's CPU is not crazy higher. It might be up a bit, but it still gets lost in the noise of the Perlbals.

I'm going to be offline for the next ~24 hours. If something is wrong, you can revert the above section from lb02's /etc/varnish/default.vcl and run /root/bin/reload-varnish-config on that machine. (I'm quite sure you know all that, but meh.)

Edit: This doesn't really compress most of our web output. Just the static files. It'd be nice to figure out if we can compress our HTML output, although pattern-matching all of that might be interesting.

We should probably make it a compression-opt-out instead of choosing what to compress. That way we just have to match the media URLs and also PNG, GIF, JPG, etc.

[alierak]

Yay, fun! Looks like js was content-type "application/x-javascript" so I changed that regexp to "text|javascript".

[alierak]

So, apparently we already do compress most of the dynamic html on the site.

root@sb-web03:~# cat /etc/apache2/mods-enabled/deflate.conf 
<IfModule mod_deflate.c>
          AddOutputFilterByType DEFLATE text/html text/plain text/xml
</IfModule>

But, it gets weirder. Some of the webservers have this kind of stuff instead:

root@sb-web01:~# cat /etc/apache2/mods-enabled/deflate.conf
&;lt;IfModule mod_deflate.c>
          # these are known to be safe with MSIE 6
          AddOutputFilterByType DEFLATE text/html text/plain text/xml

          # everything else may cause problems with MSIE 6
          AddOutputFilterByType DEFLATE text/css
          AddOutputFilterByType DEFLATE application/x-javascript application/javascript application/ecmascript
          AddOutputFilterByType DEFLATE application/rss+xml
</IfModule>

Thanks to Varnish, we've presumably been caching some gzipped stuff all along, but not actively preferring it.