The death of cvsreport.pl

[mark]

All the talk about version control systems is great, but when fu and I sat down tonight to talk about the feedback and figure out where to go next, we realized that there's one thing that really complicates the whole situation horrendously: cvsreport.pl and overlaid repositories.

Therefore, it should die. This should happen before we go down the garden path to version control nirvana, so we're going to tackle this. It started tonight and will continue to happen until we get it taken care of.

The plan right now:

* Any modules that are on CPAN should be installed via CPAN or your local package manager. We should not be distributing files locally from random repositories. Dependency management is a burden we should offload where possible. Since we already have bin/checkconfig.pl which people should be running, we can use that to ensure people have the packages needed installed.

* External programs like Perlbal, MogileFS, ddlockd, etc should be managed separately. Since most of them aren't even necessary, we can just offload this to the person who wants to install a production quality system. (And honestly, with the advent of fpm this isn't a huge burden, building packages is so incredibly easy now.)

* Small utility files that were in various repositories (the s2 stuff, miscperl, etc) we will just fold in to dw-free. It adds very little burden to us and it makes things more sane.

* ssldocs should just mirror htdocs. We can't see any harm in allowing all URLs to be valid SSL URLs. This is a necessary step before we roll out an "ssl-everywhere" option for people, so we should just bite the bullet and do it.

This does not solve the problem of dw-nonfree yet. We have some ideas, but those depend on if we end up going with git or hg or whatever. For now we're going to put that off to the end and solves the above dependency issues first and then we'll go from there.

Comments, questions?

Comments

[fu]

Looking forward to this, trying to explain clearly the difference between $LJHOME and $LJHOME/cvs/$repos when someone is just starting out has been tough to say the least, and has confused many.


Here's the specifics for the repositories:
* switch to one-folder $LJHOME (dw-free)
dw-free/

* need to decide what to do. Fold into $LJHOME/local, $LJHOME/private?
dw-nonfree/
dw-private/


* folded into dw-free:
CSS::Cleaner
miscperl

* to fold into dw-free
LJ::UserSearch
bml
s2
js
vcv


* moved to system:
Sys::Syscall
Danga::Socket

* to move to system:
memcached
gearman

* install via CPAN?
TheSchwartz http://code.sixapart.com/svn/TheSchwartz
TheSchwartz-Worker-SendEmail http://code.sixapart.com/svn/TheSchwartz-Worker-SendEmail
hubbub phttp://pubsubhubbub.googlecode.com/svn/trunk




Note: will need to run the command to clean out new files after you next run cvsreport.pl, so:

# get changes
bin/cvsreport.pl --update

# sync, yes twice
bin/cvsreport.pl --sync --cvs
bin/cvsreport.pl --sync --cvs

# remove new files
for i in `./bin/cvsreport.pl -n -1`; do rm -v $i; done


And this right here is a good indication of *why* this change is needed.

[exor674]

Honestly, it'd add a little more code in places ( routing, widgets, stats modules -- but I bet we could refactor that to be ONE PLACE that did all the magic)
and I'm not sure how we'd handle static served files ( htdocs )

but we could easily have a folder like "ext" or something, where we'd have it .dvcsignored so nothing under there would matter to dw-free ), and all of those cgi-bin, blah folders would be considered?

So we could have ext/dw-nonfree/cgi-bin/DW/Controller/DreamwidthSpecific.pm or ext/dw-nonfree/htdocs/img/a.jpg

[afuna]

Hmmm, and just have all the submodules under that? I'm assuming that we'd be able to use the submodules implementation of either hg or git, whichever we end up doing.

htdocs is the sticking point, because we'd have to look it up in multiple places.

[exor674]

Not submodules, because that introduces commit tracking. Hence why we'd have the thing .gitignored or .hgignored. And because we'd not want the pointer to dw-nonfree and dw-private in -free

[exor674]

And actually, with htdocs.

Everything goes through ::trans { for Apache }, and we do the bml rewrite there, I don't think it'd be hard to punt it to a different file.

Not sure about Perlbal or whatever we are serving static files out of nowadays.

[exor674]

p.s. to get all the unused repos out of the way ( in cvs )
mkdir xx
mv bml/ CSS-Cleaner/ Danga-Socket/ Data-ObjectDriver/ djabberd/ dmtp/ miscperl/ mogilefs/ openid/ perlbal/ s2 Sys-Syscall/ Test-FakeApache vcv xx

( you could rm -rv those, but I was being safe, may have missed some, but I have some local folders / repos in the way )

[afuna]

<3 thanks!

[exor674]

( for eventual removal of vcv )

How about local overrides to files? ( config files mainly -- etc/* )
Developers with custom tools? ( yeah, I know, put them outside the tree )
I know this isn't catered for, but also things like how I have the DW logo flipped on my development environment to give a hint that I am actually NOT looking for my changes on live?

[fu]

Those are some of the things we'd like to figure out in the comments of this entry, exactly!

What I have in mind is for things like configs, instead of having etc/config.pl, etc/config-local.pl, etc/config-private.pl, we have:

$LJHOME/etc/config.pl
$LJHOME/local/etc/config.pl
$LJHOME/private(?)/etc/config.pl

We'd need to figure out if that's feasible first of course...

The DW logo case is actually catered for, in a way, because we'd also need to tweak the logos for the dw-nonfree case (my suggestion: custom theme, parent of the non-flipped theme)

[denise]

(Not quite the same, but I have my 'hack set to Tropo Purple, which is the skin I don't use anywhere else)

[fu]

I just figured out what's been bothering me about cvsreport.pl all this time. Usually /build folders are found within the repo folder (where they're ignored), or are a separate folder entirely. But in our case, $LJHOME *is* our /build folder, and it also contains the repos, which is where things begin to get all pear-shaped.

[exor674]

Also, this "everything being available in SSL" is going to require some changes in routing, when we get to it. Open a bug and I'll get it done?

[sophie]

Agreed that everything in this post is a good idea!

Some caveats I'm thinking of - since we're not bundling modules, it will not be immediately obvious when an external module needs updating. We should strongly encourage bin/checkconfig.pl to be run after every update - something that currently isn't even in the updating procedure on the wiki. (I'm glad to see that we don't need --no=ljconfig any more - when did that get fixed?)

If we're going to have ssldocs mirror htdocs, then we shouldn't actually have ssldocs as a separate directory. I'm going to go look at some LJ history to see if I can find any reason why they didn't do it this way initially.

Definitely agreed on folding in s2, miscperl, etc. Pretty sure nobody is going to be using any of those in other projects; in fact, I can't think of a single website other than LJ/DW and their clones that use them. Would obviously not say the same for the big stuff like MogileFS/Perlbal; at this point, these shouldn't even be distributed with our code, as you say.

I would very, very much like if we could have a testing period with a repo or two where these changes were implemented, separate from the main repos (and most likely without any actual updates going on it). That would give me the opportunity to tweak the Dreamhack scripts to work correctly with the changes in an isolated fashion, without affecting anybody else - I could make a copy of my account creation script that would take things from the branch. (It would probably also allow you to get to grips with hosting on github, if we decided that's where we're moving to!)

[vlion]

Plus Ten Thousand on the idea of having a transition/test period before the bulk of devs get it.

[denise]

soooo excited about this!

[exor674]

1. I've actually hacked up a fix for the htdocs problem: http://www.dre.hack.dreamwidth.net/
I've got dw-free checked out to $LJ::HOME there.
( be aware if you view a BML page that's in dw-free, you'll probably get the dw-free sitescheme and not tropo-red... I'll... dig into that )

2. as an experiment I've been doing my work in a clone of the dw-free hg repository -- sooo I can link to this:

https://bitbucket.org/anall/dw-free-work/changesets/tip/branch(%22feature/NoOverlays%22)