Dreamwidth Development

Data and query structure for faceted styles search

Mon, 17 Jun 2013 05:31:40 GMT

Posted by: momijizukamori

I have been researching this, and while I've found a lot of articles on how to set up faceted searching using existing search engines (mostly Solr, though a few with Sphinx), and a bunch of articles on frontend design for faceted search.... there is not a lot on optimizing data structure or queries that I can find. And I know basically nothing about code optimization so - tossing this out here!

( Data structures )

( Query Structure )

Tragically the already existing options are 'use Solr', 'use something written on a Java server backend', or 'use a client-side Javascript library (which won't work with JS off and will probably not work on a 1300+ collection, but we didn't stress-test it to see)'

comments

Styles have been moved!

Sat, 15 Jun 2013 09:39:39 GMT

Posted by: ninetydegrees

Here are the new locations:

bin/upgrading/s2layers/... => styles/...
bin/upgrading/s2layers.dat => styles/s2layers.dat

ext/dw-nonfree/bin/upgrading/s2layers/... => ext/dw-nonfree/styles
ext/dw-nonfree/bin/upgrading/s2layers.dat => ext/dw-nonfree/styles/s2layers.dat

If you're working on a styles bug, you may want to save your files, update your code and work on a new branch (or do what's necessary to update your current branch and reapply your changes).

comments

Bug 3459 - new styles categorization system (brainstorming + RFC)

Tue, 14 May 2013 20:07:34 GMT

Posted by: momijizukamori

(Apologies to anyone seeing this twice, I posted it in dw_dev_training but Mark and D suggested here was probably a better fit!)

So, this is the gigantic project that's been kicking around my head the last few months, because I love our themes but right now, the organization is nooooot there. And I love organization. The full bugzilla listing has all the debates and links and what not, but I'm going to add the summary of UI implementation I ran by Denise below:

( quoted from Bugzilla )

Because the scope of this runs through Perl, MySQL, BML, and Javascript, today I sat down and wrote out a vague work-flow for how looking at themes by category in /customize should run, and how adding themes as an admin should run, so I have an idea of what I need to do.

( very rough, mind you )

I'd really appreciate feedback on if I missed any important stuff, logistics of the UI implementation, and how exactly we should store this information in the databases. Or even on the list of categories I have so far, though there will probably be a bigger, more official RFC when things get far enough.

comments

The Great Big Styles Move

Fri, 26 Apr 2013 14:03:12 GMT

Posted by: ninetydegrees

Dear fellow devs,

bug 4726 will move all folders in /bin/upgrading/s2layers/ as well as /bin/upgrading/s2layers.dat to /bin/styles. I have this mostly patched so, if you're currently working on any of these files, please set your bug as blocking mine so I can keep my patch up-to-date and know when it's safe to push it.

Thank you!

P.S. baggyeyes, I've already done this for your bug as I know you're working on it. :)

comments

Reminder: when making changes to Core2 check all styles please

Sun, 30 Sep 2012 14:28:31 GMT

Posted by: ninetydegrees

If you make any change to Core2 --frontend but also backend, please check whether this also affects any of our official styles and open a bug for them if you don't want to update them as part of your own bug. It's the only way to make sure everything stays up-to-date and functional, and only requires a simple 'grep -ril'. :)

comments

Opinions sought: Too much info on tag usage available to styles?

Mon, 10 Sep 2012 16:35:40 GMT

Posted by: swaldman

I've been working on Bug 1723, and while doing that I've noticed something that I don't think is right.

The following information on tag usage is exposed to the style system (via the TagDetail class):

use_count: The number of times that the tag in question is used in the journal (at present it's the total number of times, but bug 1723 will most likely change it to being an approximation of the number of times that it's used in posts that the current user can see. This isn't important now :-))
visibility: The most-relaxed security level that the tag is ever used in.
security_counts: How many times it is used at each security level, including security levels that the current user does not have access to.

(1) is fine, and is used by all styles (there is a privacy leak at present, but bug 1723 is meant to fix that)
(2) seems like info that styles don't really need to know, but it can't hurt
(3) concerns me, because it exposes information that the current user shouldn't have.

For example, let's say that user A has not been granted access by user B. User B posts a lot with the tag "hatemyboss". Most of these posts are protected or private, but one of the posts is public. Because of the public post, user A is able to see the "hatemyboss" tag. If user A were to go away and write his own style that used the info in (3) above, he would be able to find out how often user B has used "hatemyboss" in protected, private and filtered posts.

I'm told that none of DW's official styles use the info in (3). This doesn't mean that a custom style doesn't use it somewhere, on DW or another site using the code.

I think that there are three options:
(a) Leave things as they are
(b) Stop exposing the security_counts variable to the styles system
(c) Keep the variable, but force all the values to zero

I prefer (c), because it deals with the privacy issue without totally breaking any custom styles that use this information. We could add a note in core2.s2 explaining what is going on, and maybe remove the variable totally in a year's time or something.

What are other peoples' opinions?

If anybody wants to have a look for themselves, search for:

"class TagDetail" in core2.s2
"$t->{security_counts}" in S2.pm

comments

Some tools for S2 patching

Thu, 19 Apr 2012 06:05:37 GMT

Posted by: momijizukamori

...mostly because as anyone who gets all the bugzilla and/or style notifs knows, I did a lot of that this week *g*

First, preview images - I'd been doing them with Ubuntu's screenshot utility and photoshop, which is 1) really slow and 2) doesn't really adequately strip and compress images. So! Enter Screen Capture for Google Chrome, which I've been using with the 'full page' setting, and set to skip pop-ups and save straight to a temp image directory, where I can then rename the screenshots. After that, geekosaur and exor674 (with a little sideline ImageMagick input from tamouse_ helped me write this handy little shell script, to process all the pngs in the directory you run them on - it resizes them, crops them, strips color profiles, and indexes them.

I've been using it on my own machine, as I run Ubuntu, but I believe all the appropriate packages are now on the Dreamhack server, too, for server-side use.

Second, rb posted a handy little script for sorting properties for color themes and generating appropriate headers. I updated it to match our current formatting for files, add sorting for a few more properties, and output more verbose instructions on what to do next at the end.

#!/usr/bin/perl
#
#

my ( $theme_human, $author_name, $layout_human, $is_nonfree ) = @ARGV;

sub diehelp {
    warn "Syntax: newtheme ThemeName AuthorName LayoutName IsNonfree";
    exit;
}

if ( !$theme_human ) {
    warn "No theme name provided.";
    diehelp;
}
if ( !$layout_human ) {
    warn "No layout name provided.";
    diehelp;
}

( $layout_name = lc ( $layout_human ) ) =~ s/\s//g;
( $theme_name = lc ( $theme_human ) ) =~ s/\s//g;

my ( @dropped, @css, @set, $in_css );
while ( <STDIN> ) {
    my $line = $_;

if ( $in_css ) {
        if ( $line =~ m/"""; }/ ) {
            $in_css = 0;
            next;
        }
        push( @css, $line );
    } elsif ( $line =~ m/function Page::print_theme_stylesheet/ ) {
        $in_css = 1;
    } elsif ( /^\.$/ ) {
        break;
    } else {
        next unless $line;
        if ( $line =~ m/^layerinfo / ) {
            push( @dropped, $line );
            next;
        }

push( @set, $line );
    }
}

# process @set lines
my ( @unknown, @presentation, @page, @entries, @modules, @fonts, @images );
foreach ( @set ) {
    if ( /userlite_interaction_links = / || /_management_links = / || /module.*order/ || /module.*show/ ) {
        push( @dropped, $_ );
    } elsif ( /font/ ) {
        push( @fonts, $_ );
    } elsif ( /module/ ) {
        push( @modules, $_ );
    } elsif ( /entry/ ) {
        push( @entries, $_ );
    } elsif ( /comment/ ) {
        push( @entries, $_ );
    } elsif ( /links/ || /layout/ ) {
        push( @presentation, $_ );
    } elsif ( /color/ ) {
        push( @page, $_ );
    } elsif ( /image/ ) {
        push( @images, $_ );
    } else {
        push( @unknown, $_ );
    }
}

my $filename = "$ENV{LJHOME}/temp/$layout_name-$theme_name.s2";
open TMP_FILE, "> $filename";

print TMP_FILE <<"EOT";
#NEWLAYER: $layout_name/$theme_name
layerinfo type = "theme";
layerinfo name = "$theme_human";
layerinfo redist_uniq = "$layout_name/$theme_name";
layerinfo author_name = "$author_name";

set layout_authors = [ { "name" => "$author_name", "type" => "user" } ];
EOT

# print @set lines
sub print_section {
    my ( $name, @lines ) = @_;

if (@lines) {
        print TMP_FILE<<"EOT";

##===============================
## $name
##===============================

EOT
        foreach ( @lines ) {
            print TMP_FILE $_;
        }
    }
}

print_section( "Presentation", @presentation );
print_section( "Page", @page );
print_section( "Entry", @entries );
print_section( "Module", @module );
print_section( "Fonts", @fonts );
print_section( "Images", @images );
if ( @css ) {
    print TMP_FILE "\n";
    print TMP_FILE 'function Page::print_theme_stylesheet() { """' . "\n";
    foreach ( @css ) {
        print TMP_FILE "    " . $_;
    }
    print TMP_FILE '"""; }' . "\n";
}
print_section( "Unknown - DELETE THIS SECTION after reclassifying lines", @unknown );
print_section( "Dropped - DELETE THIS SECTION after verifying none of it is needed", @dropped );
print TMP_FILE "\n";
close TMP_FILE;

print "Parsed theme now saved in file: $filename\n";
print "Be sure to check this for hardcoded font sizes and color codes that can be shortened.\n";
print "This new text needs to be put into the existing file named:\n";
print "$ENV{LJHOME}/bin/upgrading/s2layers/$layout_name/theme.s2\n\n";
if ( @images ) {
    print "This layout appears to have image(s). Change their url to $layout_name/$theme_name(_imagename, if multiple), rename the image to $theme_name(_imagename), and put in:\n";
    print "$ENV{LJHOME}/htdocs/stc/$layout_name/$theme_name.png\n\n";
    }
print "Theme also needs a preview screenshot. Resize to 150x114px and put in:\n";
print "$ENV{LJHOME}/htdocs/img/customize/previews/$layout_name/$theme_name.png\n\n";
print "(for additional reference on cleaning themes, see http://wiki.dwscoalition.org/notes/Newbie_Guide_for_People_Patching_Styles#Adding_a_New_Color_Theme )";

This is written to be used server-side right now, with a command-line input like './newtheme.pl "Human-Readable Theme Name" user "Layout This Theme Is For" yes < inputfile' I've been running it client-side by changing line 74, 'my $filename = "$ENV{LJHOME}/temp/$layout_name-$theme_name.s2";' to 'my $filename = "temp_themes/$layout_name-$theme_name.s2";' (otherwise it tries to write to root, and That's Bad. And unnecessary)

Third, for the tiny handful of people doing S2 work on gedit (shhh, I like it, okay), I made some mods to the CSS syntax highlighting file (pastebin because the file is big). I tried to get it to make a custom S2 language, but it was being fussy about parsing it, and a lot of the .s2 files are largely CSS anyway. So this adds highlighting for S2 variables and S2 keywords! It goes in a file called CSS.lang in home/.local/usr/share/gtksourceview-2.0/language-specs, which will probably not exist if you haven't made custom languages. If you're on newer versions that use GTK3, that 2.0 should be a 3.0. You may have to manually select the highlighting, for some reason it only registers the .s2 extension some of the time.

comments

Info gathering; S2 links that do not preserve style=

Thu, 19 Jan 2012 09:57:36 GMT

Posted by: exor674

I have noticed certain in-journal links such as next and previous do not preserve the style= setting and really should.

Can people please look at links and compile a list of places that do not preserve this option so I can open a bug.

Thanks.

comments

S2 Patch Assist Script

Sat, 07 Jan 2012 10:27:12 GMT

Posted by: jeshyr

Right before my brain got refried by this stupid disease I was working on a script to turn the theme layers copied from DW itself into the right format to paste into the s2 code in the sourcecode. I was doing it as I learned how so it's only been tested on about 5 different situations at the moment and I'm quite sure it doesn't do everything so check all output please!!

And yes, my brain got fried (again) by this stupid disability so I'm on indefinite leave from developing for the mean time - and VERY disappointed about it.

At the moment you want to pipe the theme layer through STDIN to the script, so something like

$ ./newtheme.pl NewThemeName Rising "Tabula Rasa" yes < ~/file-with-theme-layer.txt

it will tell you what file it writes the output to, it will be something in $LJHOME/temp/

Script

#!/usr/bin/perl
#
#

my ( $theme_human, $author_name, $layout_human, $is_nonfree ) = @ARGV;

sub diehelp {
    warn "Syntax: newtheme ThemeName AuthorName LayoutName IsNonfree";
    exit;
}

if ( !$theme_human ) {
    warn "No theme name provided.";
    diehelp;
}
if ( !$layout_human ) {
    warn "No layout name provided.";
    diehelp;
}

( $layout_name = lc ( $layout_human ) ) =~ s/\s//g;
( $theme_name = lc ( $theme_human ) ) =~ s/\s//g;

my ( @dropped, @css, @set, $in_css );
while ( <STDIN> ) {
    my $line = $_;

push( @set, $line );
    }
}

# process @set lines
my ( @unknown, @presentation, @page, @entries, @modules, @fonts );
foreach ( @set ) {
    if ( /userlite_interaction_links = / || /_management_links = / || /module.*order/ || /module.*show/ ) {
        push( @dropped, $_ ); 
    } elsif ( /font/ ) {
        push( @fonts, $_ );
    } elsif ( /module/ ) {
        push( @modules, $_ );
    } elsif ( /entry/ ) {
        push( @entries, $_ );
    } elsif ( /comment/ ) {
        push( @entries, $_ );
    } elsif ( /links/ ) {
        push( @presentation, $_ );
    } elsif ( /color/ ) {
        push( @page, $_ );
    } else {
        push( @unknown );
    }
}

my $filename = "$ENV{LJHOME}/temp/$layout_name-$theme_name.s2";
open TMP_FILE, "> $filename";

print TMP_FILE <<"EOT";
#NEWLAYER: $layout_name/$theme_name
layerinfo "type" = "theme";
layerinfo "name" = "$theme_human";
layerinfo "redist_uniq" = "$layout_name/$theme_name";
layerinfo "author_name" = "$author_name";

set layout_authors = [ { "name" => "$author_name", "type" => "user" } ];
EOT

# print @set lines
sub print_section {
    my ( $name, @lines ) = @_;

if (@lines) {
        print TMP_FILE<<"EOT";

##===============================
## $name
##===============================
EOT
        foreach ( @lines ) {
            print TMP_FILE $_;
        }
    }
}

print_section( "Presentation", @presentation );
print_section( "Page", @page );
print_section( "Entry", @entries );
print_section( "Module", @module );
print_section( "Fonts", @fonts );
if ( @css ) {
    print TMP_FILE "\n";
    print TMP_FILE 'function Page::print_theme_stylesheet() { """' . "\n";
    foreach ( @css ) {
        print TMP_FILE "    " . $_;
    }
    print TMP_FILE '"""; }' . "\n";
}
print_section( "Unknown - DELETE THIS SECTION after reclassifying lines", @unknown );
print_section( "Dropped - DELETE THIS SECTION after verifying none of it is needed", @dropped );
print TMP_FILE "\n";
close TMP_FILE;

print "Parsed theme now saved in file: $filename\n";
print "This new text needs to be put into the existing file named:\n";
print "$ENV{LJHOME}/bin/upgrading/s2layers/$theme_name/theme.s2\n\n";
print "Theme also needs a preview screenshot. Resize to 150x114px and put in:\n";
print "$ENV{LJHOME}/htdocs/img/customize/previews/$theme_name/$layout_name.png\n\n";

Feedback, cases that break it, improvements, etc. would all be great.

r

Edit: Now with super bonus functioning HTML tags!

comments