Dreamwidth Development

I have been researching this, and while I've found a lot of articles on how to set up faceted searching using existing search engines (mostly Solr, though a few with Sphinx), and a bunch of articles on frontend design for faceted search.... there is not a lot on optimizing data structure or queries that I can find. And I know basically nothing about code optimization so - tossing this out here!

( Data structures )

( Query Structure )

Tragically the already existing options are 'use Solr', 'use something written on a Java server backend', or 'use a client-side Javascript library (which won't work with JS off and will probably not work on a 1300+ collection, but we didn't stress-test it to see)'

(Apologies to anyone seeing this twice, I posted it in dw_dev_training but Mark and D suggested here was probably a better fit!)

So, this is the gigantic project that's been kicking around my head the last few months, because I love our themes but right now, the organization is nooooot there. And I love organization. The full bugzilla listing has all the debates and links and what not, but I'm going to add the summary of UI implementation I ran by Denise below:

( quoted from Bugzilla )

Because the scope of this runs through Perl, MySQL, BML, and Javascript, today I sat down and wrote out a vague work-flow for how looking at themes by category in /customize should run, and how adding themes as an admin should run, so I have an idea of what I need to do.

( very rough, mind you )

I'd really appreciate feedback on if I missed any important stuff, logistics of the UI implementation, and how exactly we should store this information in the databases. Or even on the list of categories I have so far, though there will probably be a bigger, more official RFC when things get far enough.

...mostly because as anyone who gets all the bugzilla and/or style notifs knows, I did a lot of that this week *g*

First, preview images - I'd been doing them with Ubuntu's screenshot utility and photoshop, which is 1) really slow and 2) doesn't really adequately strip and compress images. So! Enter Screen Capture for Google Chrome, which I've been using with the 'full page' setting, and set to skip pop-ups and save straight to a temp image directory, where I can then rename the screenshots. After that, geekosaur and exor674 (with a little sideline ImageMagick input from tamouse_ helped me write this handy little shell script, to process all the pngs in the directory you run them on - it resizes them, crops them, strips color profiles, and indexes them.

#!/bin/bash for f in *.png; do convert -resize 150x114^ -gravity north -extent 150x114 -sharpen 25 +dither -type Palette "$f" png8:"out.png" && pngnq -n 256 -s 1 -f out.png && pngcrush -c 3 -bit_depth 8 -rem alla -brute out-nq8.png "$f" >/dev/null && rm out.png out-nq8.png done

I've been using it on my own machine, as I run Ubuntu, but I believe all the appropriate packages are now on the Dreamhack server, too, for server-side use.

Second, rb posted a handy little script for sorting properties for color themes and generating appropriate headers. I updated it to match our current formatting for files, add sorting for a few more properties, and output more verbose instructions on what to do next at the end.

#!/usr/bin/perl # # my ( $theme_human, $author_name, $layout_human, $is_nonfree ) = @ARGV; sub diehelp { warn "Syntax: newtheme ThemeName AuthorName LayoutName IsNonfree"; exit; } if ( !$theme_human ) { warn "No theme name provided."; diehelp; } if ( !$layout_human ) { warn "No layout name provided."; diehelp; } ( $layout_name = lc ( $layout_human ) ) =~ s/\s//g; ( $theme_name = lc ( $theme_human ) ) =~ s/\s//g; my ( @dropped, @css, @set, $in_css ); while ( <STDIN> ) { my $line = $_; if ( $in_css ) { if ( $line =~ m/"""; }/ ) { $in_css = 0; next; } push( @css, $line ); } elsif ( $line =~ m/function Page::print_theme_stylesheet/ ) { $in_css = 1; } elsif ( /^\.$/ ) { break; } else { next unless $line; if ( $line =~ m/^layerinfo / ) { push( @dropped, $line ); next; } push( @set, $line ); } } # process @set lines my ( @unknown, @presentation, @page, @entries, @modules, @fonts, @images ); foreach ( @set ) { if ( /userlite_interaction_links = / || /_management_links = / || /module.*order/ || /module.*show/ ) { push( @dropped, $_ ); } elsif ( /font/ ) { push( @fonts, $_ ); } elsif ( /module/ ) { push( @modules, $_ ); } elsif ( /entry/ ) { push( @entries, $_ ); } elsif ( /comment/ ) { push( @entries, $_ ); } elsif ( /links/ || /layout/ ) { push( @presentation, $_ ); } elsif ( /color/ ) { push( @page, $_ ); } elsif ( /image/ ) { push( @images, $_ ); } else { push( @unknown, $_ ); } } my $filename = "$ENV{LJHOME}/temp/$layout_name-$theme_name.s2"; open TMP_FILE, "> $filename"; print TMP_FILE <<"EOT"; #NEWLAYER: $layout_name/$theme_name layerinfo type = "theme"; layerinfo name = "$theme_human"; layerinfo redist_uniq = "$layout_name/$theme_name"; layerinfo author_name = "$author_name"; set layout_authors = [ { "name" => "$author_name", "type" => "user" } ]; EOT # print @set lines sub print_section { my ( $name, @lines ) = @_; if (@lines) { print TMP_FILE<<"EOT"; ##=============================== ## $name ##=============================== EOT foreach ( @lines ) { print TMP_FILE $_; } } } print_section( "Presentation", @presentation ); print_section( "Page", @page ); print_section( "Entry", @entries ); print_section( "Module", @module ); print_section( "Fonts", @fonts ); print_section( "Images", @images ); if ( @css ) { print TMP_FILE "\n"; print TMP_FILE 'function Page::print_theme_stylesheet() { """' . "\n"; foreach ( @css ) { print TMP_FILE " " . $_; } print TMP_FILE '"""; }' . "\n"; } print_section( "Unknown - DELETE THIS SECTION after reclassifying lines", @unknown ); print_section( "Dropped - DELETE THIS SECTION after verifying none of it is needed", @dropped ); print TMP_FILE "\n"; close TMP_FILE; print "Parsed theme now saved in file: $filename\n"; print "Be sure to check this for hardcoded font sizes and color codes that can be shortened.\n"; print "This new text needs to be put into the existing file named:\n"; print "$ENV{LJHOME}/bin/upgrading/s2layers/$layout_name/theme.s2\n\n"; if ( @images ) { print "This layout appears to have image(s). Change their url to $layout_name/$theme_name(_imagename, if multiple), rename the image to $theme_name(_imagename), and put in:\n"; print "$ENV{LJHOME}/htdocs/stc/$layout_name/$theme_name.png\n\n"; } print "Theme also needs a preview screenshot. Resize to 150x114px and put in:\n"; print "$ENV{LJHOME}/htdocs/img/customize/previews/$layout_name/$theme_name.png\n\n"; print "(for additional reference on cleaning themes, see http://wiki.dwscoalition.org/notes/Newbie_Guide_for_People_Patching_Styles#Adding_a_New_Color_Theme )";

This is written to be used server-side right now, with a command-line input like './newtheme.pl "Human-Readable Theme Name" user "Layout This Theme Is For" yes < inputfile' I've been running it client-side by changing line 74, 'my $filename = "$ENV{LJHOME}/temp/$layout_name-$theme_name.s2";' to 'my $filename = "temp_themes/$layout_name-$theme_name.s2";' (otherwise it tries to write to root, and That's Bad. And unnecessary)

Third, for the tiny handful of people doing S2 work on gedit (shhh, I like it, okay), I made some mods to the CSS syntax highlighting file (pastebin because the file is big). I tried to get it to make a custom S2 language, but it was being fussy about parsing it, and a lot of the .s2 files are largely CSS anyway. So this adds highlighting for S2 variables and S2 keywords! It goes in a file called CSS.lang in home/.local/usr/share/gtksourceview-2.0/language-specs, which will probably not exist if you haven't made custom languages. If you're on newer versions that use GTK3, that 2.0 should be a 3.0. You may have to manually select the highlighting, for some reason it only registers the .s2 extension some of the time.