karzilla: a green fist above the word SMASH! (Default)
[staff profile] karzilla
We're about to deploy a new backend interface for file storage, called BlobStore, which [staff profile] mark wrote over the past few months with the intention of standardizing how file storage is handled in our code and making it work with any number of possible underlying technologies. It currently supports MogileFS and local disk, and we plan to add support for S3 in the future.

At this point, MogileFS is considered legacy technology. If your site is set up to use MogileFS, that configuration will continue to work under BlobStore for now. However, no new code that requires MogileFS will be accepted.

What you need to know if you are writing code: the new methods are implemented in cgi-bin/DW/BlobStore.pm and are pretty straightforward. For the most part they serve as drop-in replacements for the MogileFS file methods.

What you need to know if you are running a server: if you try to do anything related to uploading images, including userpics, you will get a fatal error unless you have defined either %LJ::BLOBSTORE or %LJ::MOGILEFS_CONFIG. So if you were already using MogileFS, you're fine, but if not, you will need to set up local disk storage in one of your local config files. The stock etc/config-private.pl in dw-free will have an example %LJ::BLOBSTORE that you can uncomment and use.

What you need to know if your existing userpics disappear: If you were running a server without MogileFS, all of your system's userpics were stored in a database table, and use of that table is no longer supported. I'm working on a new version of the migrate-userpics.pl script that can be used to move the images into your BlobStore once you've got that configured. (Update: this is available in bin/upgrading/migrate-userpics.pl.)

Obviously this will all need to be documented on the wiki somewhere, but I've got my hands full right now making sure everything is nailed down to push these changes into production in a few days. Let me know if there's anything I didn't cover here that needs to be addressed.
kareila: Rosie the Riveter "We Can Do It!" with a DW swirl (dw)
[personal profile] kareila
I am excited to announce that I have just published to the wiki a new Template Toolkit / Foundation Conversion Guide! Covering such thrilling and educational topics as:

  • Which page components belong in a controller vs. a template?

  • What do I need to do to convert a page to Foundation?

  • How is a BML file like a plate of spaghetti?


As usual, let me know here if you have any questions or comments!
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
If your dev environment is publicly reachable on the internet, it will get spam! I've written a quick guide to spam prevention.
kareila: "PERL!" (perl)
[personal profile] kareila
I updated http://wiki.dreamwidth.net/wiki/index.php/Dev_Testing with information about setting up the new test config files and initializing the test database. If you give it a go and run into trouble, leave a comment here. ♥
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
I created the Explanations wiki page after seeing [staff profile] mark waving the code machete tonight: he took out a lot of old and unused stuff, which is great, but a few (very out of date) old server-admin documentation files that were casualties had some explanations of things that I don't think are documented anywhere else.

That does not mean the entire server-admin documentation should survive -- out of date documentation is often worse than no documentation! -- but between that and the talk I gave at LCA, "When Your Codebase Is Nearly Old Enough To Vote", it occurred to me that we should probably document a few of those things that live in our heads that we find ourselves explaining to new contributors -- things like why an entry URL uses the anum and not the jitemid, why you can't just change a text string in a patch and have it update the text on the live site, why you occasionally hear oldtimers calling the Recent Entries page 'lastn', stuff like that.

So, if you've been around for a while and can think of a few things that wouldn't necessarily make sense right off the top of your head -- or if you're new and ran into one or more of those things at some point and had to have someone explain them to you -- please add them to that page! As I said in my talk: institutional memory is a great thing to have, but a lousy thing to have to rely on.
fu: Close-up of Fu, bringing a scoop of water to her mouth (Default)
[personal profile] fu

With [staff profile] denise's help (the bulk of this was from her really!), we've made major changes to the dev-facing wiki documentation for clarity.

Among other things:

  • merged multiple/redundant pages and sections

  • improved linking

  • (hopefully) reduced the complexity of paths through the wiki for someone just getting started

The biggest change is to Dev Getting Started, which is now greatly expanded, with a much clearer flow, and more focus on someone totally new to DW/development. The resources for someone more experienced have been moved to Dev Quick Start .

The contents of Version Control have been merged with Newbie Guide: How To in Git and the latter is the canonical page for git info -- though now I'm tempted to go rename it to Version Control because it's shorter. Git How To? ;)

Git instructions in some pages have been updated to be much simpler with a pointer to the appropriate section in the git commands in case that's needed.

And the Directory Structure has been expanded to cover more subdirectories.

Beginner Dev Checklist needs some more effort to pull it apart: plan is to integrate it into other pages as appropriate and then get rid of it (since it's not sufficiently different from Dev Getting Started to warrant its own page)

Would appreciate if you poked around through the various pages and let me know if there's anything still left unclear, or if you're aware of similar pages that can be merged into these existing ones!

kaberett: A sleeping koalasheep (Avatar: the Last Airbender), with the dreamwidth logo above. (dreamkoalasheep)
[personal profile] kaberett
I've synthesised discussions about and practical use of GHI into a single wiki page. Please shout if you want anything added or changed!

I've also gone through and replaced as many references to Bugzilla with references to GHI as (1) makes sense and (2) I am immediately able. There's about 20 pages still on my hit-list, not all of which I have the knowledge to deal with - if you feel like pitching in, please take one and let me know when it's done (or, if you don't have wiki access, let me know what the appropriate edits would be).

List of pages outstanding )
rax: (Silver whaaaaaaaaaaaaaaaaaaaaaaaat)
[personal profile] rax

I'm trying to crawl and parse comments on a community for a fandom event (http://hs-worldcup.dreamwidth.org , if you're curious). I've run into a bunch of issues and lack of API documentation, and talked to DW Support a couple of times, and feel like I am further away from successfully doing anything than when I started. Before I say anything else, here is What I Am Really Trying To Do:
  • take an individual community post (example: http://hs-worldcup.dreamwidth.org/3493.html#comments) 
  • download all of the comments with some sort of threading information --- the data I need in particular is comment subject, comment author, comment content, whether or not it's a reply and if so to what
  • parse out that data and do transformations to it and add it to a database (which is not super relevant to this question I don't think but I can go into more detail if necessary)
I looked into the API for getting comments which led me in a roundabout way to www.livejournal.com/developer/exporting.bml . I'm probably missing something obvious here, but I don't actually see how this tells me how to make an API call? It gives me a GET request, but not what to send the GET request to? Also, support told me the only action DW supports here is "Get all comments for a community," not "Get all comments for a page," and I should probably just crawl the pages. Is that what other folks have done when doing this?

If that is what I should do, how do I get around the adult content warning? Is there a flag I can pass with the URL or something? Do I need to do something more complicated than just using curl to grab the pages? Is there something I can pass to say "just give me one piece of HTML with all 5000 comments on it it will be easier for both of us probably?"

Thank you for any suggestions or advice you might have.

fu: Close-up of Fu, bringing a scoop of water to her mouth (Default)
[personal profile] fu

SCSS directory structure

All our SCSS files are in htdocs/scss/; compiled versions are automatically placed in htdocs/stc/css. We care about the former when editing, and the latter when including them in the webpage itself.

One of the things that SCSS is really good for is making it easy to organize CSS. I'd like us to move away from CSS for individual pages, and instead concentrate on having components that are used on multiple pages.

  • htdocs/scss/foundation - stylesheets from the Foundation library. The main files to touch here are foundation.scss, which specifies the components we'll be using on all pages, and _variables.scss, which sets default behavior / appearance variables that are common to all site skins.

    _variable.scss should not contain any colors, because we have to take into account dark-on-light vs light-on-dark skins. Sizes / measurements are perfectly acceptable here!

  • htdocs/scss/components - stylesheets for individual components. These are included only on the pages that need them, but contain all the styling that's required within. Everything here should have a corresponding example of the HTML structure over on /dev/style-guide

    This folder also contains Foundation components that aren't used on enough pages to warrant being included on all of them (htdocs/scss/foundation/foundation.scss), but that we still need on individual pages. htdocs/scss/components/progress-bars is an example.

    For components where you want access to variables, useful for things like line-height, include the following code at the top:

    $include-html-classes: false;
    @import "foundation/variables", "foundation/components/global";

    That won't work for colors though! Things that specify colors will have to go into the site skins.

    To use a component from a .tt file:

    [% dw.need_res( { group => "foundation" }, "stc/css/components/component-name.css" ); %]
  • htdocs/scss/mixins - mixins are fragments of code that can be used by multiple elements. Currently we have one mixin which hides elements visually but keeps them visible to screenreader. This replaces the... three or four different ways we were doing it before

    To use a mixin, from an SCSS file:

    @import "mixins/screenreader-friendly";
    .element-name {
        @extend %screenreader-friendly;
  • htdocs/scss/pages - stylesheets for individual pages. Directory should mirror the URL structure of your pages as much as possible. But really we should try not to have anything here...

  • htdocs/scss/skins - stylesheets for site skins. Includes the actual skins and shared files, e.g., skiplinks, alert colors

JS directory structure

There's a lot more of the old JS hanging around in htdocs/js, but new files should follow the same basic structure as above:

  • htdocs/js/foundation - scripts from the Foundation library

  • htdocs/js/jquery - scripts from the jQuery / jQuery UI library

  • htdocs/js/components - individual components, suitable for inclusion on multiple pages

  • htdocs/js/pages - onload for individual pages; no examples yet. Preferably just the bare minimum of

    $(document).load( function() {

    } );

fu: Close-up of Fu, bringing a scoop of water to her mouth (Default)
[personal profile] fu

So I'm here to present to you all our new style guide. That's on my public dev server, not yet on dreamwidth.org, so don't worry if it's a bit slow.

That page contains the components that we'll be using across the site. The goal is to have every component documented there and to have as little per-page styling as possible. That does two things: make appearance / interactions consistent across the site (good for users); make it easy to refer to design decisions when writing pages (good for developers, especially those of us who aren't really frontend people)

We've used the Foundation framework as our basis for our redesign. That gives us a clean, responsive design that is really easy to make work on large and small screens -- we've had a huge problem with our site headers and certain pages on phones / tablets; it's about time to fix that.

I recommend going over the Foundation documentation btw. They have excellent documentation. It's a good way to get started.

All our Foundation work uses SCSS. SCSS is a CSS preprocessor and it works like CSS but with additional features. You can have variables, if statements, mixins (fragments of code that are used by more than one thing).

It requires an additional compilation step to get all this goodness, but that's possible with one command. Try the following:

  • pull in the latest changes on develop

  • Go to /dev/style-guide on your 'hack. It'll be completely unstyled

  • Run this command:

    compass compile

    A couple lines will scroll by, looking like this:

    create htdocs/stc/css/foundation.scss
    create htdocs/stc/css/normalize.scss
    create htdocs/stc/css/skins/celerity.scss
    create htdocs/stc/css/skins/gradation/gradation-horizontal.scss
    create htdocs/stc/css/skins/gradation/gradation-vertical.scss
    create htdocs/stc/css/skins/lynx.scss

    That's good: that means your SCSS files have been turned into CSS, which you can now use.

The SCSS files themselves are in htdocs/scss. These are the files you'll be touching. After they've been compiled, the generated files are placed in htdocs/stc/css. These are the files that you'll be including on the page.

So if you have a file:


You include it by doing this:

    [% dw.need_res( "stc/css/components/foo.css" ) %]

One last thing, the compass compile command is good, but when you're developing, the last thing you want is to have to constantly switch from the file you're editing to your terminal window to run a command. Luckily, someone's thought of that. What you can do instead is this:

     compass watch

That watches for any changes in the SCSS files, and compiles them into .css files automatically. Leave that running in a separate window and it'll just do its thing. Just make sure you stop it after you're done developing for a session (just like you do with your Apache servers), because it does use up some resources!

Note: it's magical but not completely so. If you're working on something in dw-nonfree, you'll have to run compass watch separately in that directory:

    cd $LJHOME/ext/dw-nonfree
    compass watch

But if you're not touching anything in dw-nonfree, then you only need to run it the once.

Please jump in, poke around! Play with things. I'm happy to answer any questions :)

I plan on making several entries on the ongoing conversion. Soon to come: error handling for forms, directory structure for CSS / JS, easier way to format messages for email, etc.

kareila: IT prepares you for a life of fighting with PCs nonstop. (sysadmin)
[personal profile] kareila

All of the information that was previously there assumed you had root on a Debian box. I've updated the page to document the steps needed to get it working for dreamhacks as well. (Until the next time things break, anyway.)



That is to say, if you try using my instructions and run into problems, let me know!
fu: Close-up of Fu, bringing a scoop of water to her mouth (Default)
[personal profile] fu
I started with a desire to document how to submit changes to a release branch, and somehow ended up trying to figure out how we could organize the wiki to make it easy for us to figure out what we have and what we need.

Please read and comment on my entry in dw-wiki. I'm locking comments here so we can centralize all discussion in one place.
foxfirefey: A picture of GIR. (gir)
[personal profile] foxfirefey
So, as far as I can tell from my researching over the past few days, the only real way to get a pull request to review it is to:

* Manually add the submitter's dw-* repo as a remote in the repo on your hack (or Github I suppose, but there is no GUI advantage here).
* Manually pull the branch in question

Is there a better way to do this? Am I missing something? This seems really, well, annoying, with manually crafting URLs and whatnot, and not very user friendly. (I'm trying to make some documents on reviewing pull requests for people who are not [personal profile] fu, since we need to try and spread out that work a little.)
foxfirefey: Dreamwidth: social content with dimension. (dreamwidth)
[personal profile] foxfirefey

As of several weeks ago, all commits to the Dreamwidth codebases have gone to our new repositories on Github:

There are a few relevant wiki documents that have been fully revised to account for this change:

  • Moving your Dreamwidth installation to use Github -- these instructions will tell you how to move your current Dreamhack/dev environment over to a Github based installation
  • Dev Maintenance -- this document tells you how to keep your Github Dreamwidth based installation (and your Dreamwidth forks) updated with the code from the Dreamwidth repositories
  • Draft: Github development process -- This is the document in the least refined state, so keep in mind that it is in stronger need of suggestions and revisions. It goes through the very basics of Git workflow. This and Dev Maintenance might eventually end up merged into one document.

We are working on getting the rest of the wiki development documentation updated (see the dw_wiki post). Feel free to comment to this post with your questions/concerns about the move!

foxfirefey: Dreamwidth: social content with dimension. (dreamwidth)
[personal profile] foxfirefey
As part of our move to Github we have a lot of development documentation to update.

If you want to be part of the vanguard, take a gander at the Moving your Dreamwidth installation to use Github instructions. People following them are encouraged to log into the #dreamwidth-dev channel on IRC for real time assistance.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
[personal profile] crschmidt pointed out in his bug walkthrough that the process of running the automated test suite is not very well documented! And I agree; I know I don't run the tests when I'm patching things since I don't really know much about it, and I don't contribute tests along with my code because I actually don't know how to write them or what they should do. I am probably not the only one! (The only thing I know about the test suite is that you shouldn't run it in production because it does weird and wonky things to the database.)

I do know that our test suite coverage is not extensive at all, and we've talked before about moving to a more test-driven devlopment mindset. If anybody wanted to write up some documentation on the Wiki about both how to run the tests (and what you should look for) and how to write new tests (and when you should), that would be awesome and I would love you forever. (Well, I mean, I love you all forever already. But I'd love you more forever.)
azurelunatic: DW: my eloquence cannot be captured in 140 chars (twitter)
[personal profile] azurelunatic
Because it was a question (thanks, [twitter.com profile] 3sperhand!), I've started a wiki page for documenting the cookies we set, and what's in them, and why.


Anyone with more expertise in this than me, please feel free to jump in and help document.

On use of the wiki )
skakri: (Default)
[personal profile] skakri
Hello, I'd like to know if there is any documentation regarding content importing from LJ-based site (AFAIK, old version, circa "07) and cross-post setup steps? How should I define a new external site?

I'm currently using jbackup.pl + LJ::Simple mash-up, but that won't work in the long run, as even if entry backdate is set, users still receive messages of a new posts, if they're subscribed.
szabgab: (Default)
[personal profile] szabgab

Seeing how complex the deployment of applications developed in various corporations I am doing a little research to see how open source application manage the complexity. For that I was looking for complex (in terms of deployment and development environment) applications to see how they manage the task and how they test their system. Dreamwidth seems like a good case so I jumped on the IRC channel where [personal profile] pauamma gave me a quick list of the entities. Then I was volunteered to raise the issue here to get further clarifications.

So here is the list I got:

  1. perlbal on the load balancer(s)

  2. webservers running Apache+modperl+DW webapp code

  3. database cluster for global data

  4. several (currently 3) database clusters for per-user data. Each cluster holds the same data for different users

  5. Gearman servers, to offload some web-synchronous compute-intensive tasks

  6. TheSchwartz server(s) for asynchronous or non-web tasks.

  7. MogileFS servers for blobby data not stored in a SQL database for various reasons.

  8. Postfix

I found the Production Notes to be also a good source for information but I'd appreciate to get further details if the list is not complete. What would be very interesting to me is to understand what are the responsibilities of the various entities in the system. What kind of data is kept in the various databases. Which entities can be duplicated if load requires it and how the jobs can be divided?
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
I'm really interested in learning more about what kind of performance things we need to keep in mind while coding, and how to design for a well-behaved high-availability web app. I know the very basics, like using memcache whenever possible and using direct assignments instead of shifts, but not much else!

So, is anyone interested in writing tutorials on things to keep in mind when coding stuff for performance factors? As we grow, this is going to start to be stuff we should keep in mind, and I think it's a great opportunity for us all to share knowledge and learn together. No matter how much you know, even if you think it's not much, it's probably more than someone else (ie, me) knows...


dw_dev: The word "develop" using the Swirly D logo.  (Default)
Dreamwidth Open Source Development

March 2017

5678 91011
1920212223 2425


RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Mar. 26th, 2017 05:21 am
Powered by Dreamwidth Studios