mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark

Hi all!

As of today, I've made a few fairly broad changes to the way our repositories are set up. You will need to likely make some changes to your development workflow. But! They should be simple.

First, the changes:

  1. The dw-free repository has been renamed to dreamwidth -- because,

  2. The dw-nonfree repository is no more, it has been merged into the dw-free under the path ext/dw-nonfree/.

  3. Finally, the master branch has been renamed to main.

These two changes mean that no more do we have to track and coordinate changes across two repositories, they can be done atomically in one place. Since we don't really support or test running Dreamwidth on its own (without the branding) this will also allow us to simplify development by ultimately collapsing the code, too.

To update your checkout of Dreamwidth, you will need to do two separate things. First, go to Github and to your existing dw-free repository, go to the settings and:

  1. Rename the repository to dreamwidth

  2. Go to branches and rename the master branch to main

Once that's done, you need to update your local checkout like this:

 cd $LJHOME

 # WARNING: Only do this if you have no code here that you care about!
 rm -rf ext/dw-nonfree

 # Update the repository root (for your "origin" remote, **set your username here**)
 git remote set-url origin git@github.com:YOUR_GITHUB_USERNAME/dreamwidth.git

 # Update the repository root (for your "upstream" remote, please change as needed)
 git remote set-url upstream git@github.com:dreamwidth/dreamwidth.git

 # Fetch only to get the new main branch
 git fetch

 # Switch over to it
 git checkout main

 # Get rid of local master
 git branch -d master

 # Redirect HEAD
 git symbolic-ref refs/remotes/origin/HEAD refs/remotes/origin/main


That's it; you should be good to go now. I'll work on updating the Dreamhack scripts shortly, but wanted to get these instructions out. Please let me know if you have any questions/comments/issues.

kareila: "Mom, I'm hungry." "Hush, I'm coding. You ate yesterday." (coding)
[personal profile] kareila
Hi all! During Saturday's dev chat, one of the discussions that came up was the current status of our work on improving the process of using Github Issues as our bug tracker.

Here is where we are right now:

It is now possible for anyone to use inline text tags to categorize new issues. [personal profile] fu has put up a rough draft of the instructions for this process, along with a list of the labels that are currently recognized. We are also planning to allow issues to be "claimed" in a similar manner, although the format for doing that hasn't been finalized yet.

Another part of Github Issues that we've been experimenting with is the use of milestones. If you look at the milestones on dw-free you can see, for example, the status of open bugs on certain major projects like Foundation and mobile styles. I think this could be really useful, especially in light of recent comments saying that we haven't been doing a good job of communicating the progress of our major development projects within the development community.

There are also other milestones currently displayed on that page such as "unclaimed", "curated", and "in progress" - a lively discussion ensued as to whether these would be better used as tags instead of milestones, and I'd like to continue that discussion below. The main issue with milestones on Github in general is that a bug cannot be assigned to more than one milestone. From a workflow perspective, we would expect to see a progression of untriaged -> unassigned -> claimed/in progress -> pull request -> closed, and the fact that an issue cannot exist in more than one of those states at a time makes it a good candidate for the use of milestones - but could conflict with using milestones for the purpose of tracking large projects.

There was also a great deal of confusion as to what "curated" was supposed to mean in this context. [staff profile] mark said what he desired was a list of three or so "top priority" unclaimed bugs, so that he could easily decide what to spend his limited time on. Again, this may be a better candidate for a tag than a milestone, and maybe with a label that is easier to interpret.

The last issue mentioned is that we still haven't come up with tags for indicating which area of the code is involved with a bug (e.g. styles, notifications, etc.), which is something we relied on pretty heavily with our previous Bugzilla setup in order to find new things to work on.

Please continue the discussion in comments!
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark

We did our first release from git tonight. It went well, I will chalk that up to all of the hard work from folks like [personal profile] fu and [personal profile] exor674 (and others!) to make sure that everything was in place. [personal profile] fu put together a nice "things we have to do" in [site community profile] dw_staff, so that was very helpful.

Anyway, I wanted to document some things that are important to know for the period between releases. This probably won't apply to most of you, but just in case...

So, we are following the document about a successful git branching model. This is what git flow is based on and it's a really efficient model, so we're using it for doing our features and in this case, our release.

The release process was, in essence:

  • Create a release-1.11.0 branch
  • Make final tweaks in this branch to get things ready for release
  • Push this branch live
  • Merge this branch back in to develop and master
  • Add a release tag with the version number on master

This leaves our repository in this state:

  • develop is still open, you can keep committing things to it (no code freeze, yay!)
  • master has stable tags, so you can always roll back to tags here (easy production rolls!)
  • release-1.11.0 is open for doing fixes on the live release (easy fixing production, yay!)

The thing that is important to know, if you are going to be committing fixes that you intend to go live, is that you need to make your changes on the release-x.xx.x branch -- not on develop. Any code committed to develop will, because of its nature, have to wait until the next code push -- weeks away.

If you have stuff you need to get live sooner (fixes to things broken in the last code push or stability/important fixes), you need to adjust your workflow slightly. It's pretty easy though, and goes something like this:

git checkout -b release-1.11.0 dreamwidth/release-1.11.0

Now make your fixes, as per normal, and submit a pull request. As long as your current branch is the release branch, then you will be able to submit a pull request on the appropriate branch, and we can merge it in and push it out to the site quite easily.

fu: Close-up of Fu, bringing a scoop of water to her mouth (Default)
[personal profile] fu

After a bunch of discussion, we've decided to move from Mercurial to Git. Mercurial and Git are different version control systems. If you're not sure what that is, they basically manage code and the history of how the code has changed.

One big reason for using Git is Github. Github is a service that hosts code, and provides web-based tools to manage commits and reviews. This will make it a lot easier to work collaboratively, and simplify the process of submitting patches. The changeover may be a bit of a pain, but we will work to make it as easy as possible!

We've also decided to simplify our development environment. Instead of having the $LJHOME directory contain a bunch of repositories that need to be synced and copied around -- no one has ever understood this setup without a lot of explaining and re-explaining -- we will just have one set of directories. This means we can get rid of cvsreport.pl, which is the script that does the syncing between the $LJHOME directory and the repositories within it.

Nothing will change yet, but I thought (and [staff profile] denise agreed) it might be a good idea to post a very basic overview of how the process will be changing.

I'm still banging on the workflow; right now I'm just working with a couple of the more experienced devs so we can get a general idea of what needs to be done. We'll invite more people for a test run once to iron out the more fiddly bits once we have a list of steps to try.I expect we'll have to adjust again after people have tried it out for real.

Don't worry! Nothing is going to happen without a lot more in the way of information and tutorials and explanations, but here is a very high-level overview.

diagrams and details within )
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark
All the talk about version control systems is great, but when [personal profile] fu and I sat down tonight to talk about the feedback and figure out where to go next, we realized that there's one thing that really complicates the whole situation horrendously: cvsreport.pl and overlaid repositories.

Therefore, it should die. This should happen before we go down the garden path to version control nirvana, so we're going to tackle this. It started tonight and will continue to happen until we get it taken care of.

The plan right now:

* Any modules that are on CPAN should be installed via CPAN or your local package manager. We should not be distributing files locally from random repositories. Dependency management is a burden we should offload where possible. Since we already have bin/checkconfig.pl which people should be running, we can use that to ensure people have the packages needed installed.

* External programs like Perlbal, MogileFS, ddlockd, etc should be managed separately. Since most of them aren't even necessary, we can just offload this to the person who wants to install a production quality system. (And honestly, with the advent of fpm this isn't a huge burden, building packages is so incredibly easy now.)

* Small utility files that were in various repositories (the s2 stuff, miscperl, etc) we will just fold in to dw-free. It adds very little burden to us and it makes things more sane.

* ssldocs should just mirror htdocs. We can't see any harm in allowing all URLs to be valid SSL URLs. This is a necessary step before we roll out an "ssl-everywhere" option for people, so we should just bite the bullet and do it.

This does not solve the problem of dw-nonfree yet. We have some ideas, but those depend on if we end up going with git or hg or whatever. For now we're going to put that off to the end and solves the above dependency issues first and then we'll go from there.

Comments, questions?
kareila: (Default)
[personal profile] kareila
This recent post brought to my attention the fact that it really ought to be simpler for sites using DW code in production to easily keep up with stable milestones of DW development, without having to look through all of our repositories for the most recent milestone tags. Besides, the milestone tags are tied to code pushes on dreamwidth.org and not necessarily an indication of stability!

I propose the following mechanism:

1. Add a "stable" tag to all of our shared repositories. We already do code freezes where we only put in changes to fix bugs that are uncovered immediately after a code push. When the code freeze is lifted, move the "stable" tag to the most recent milestone and leave it there until after the next push/freeze cycle. For example, the most recent cycle would have moved the stable tag from 0.21.2 to 1.0.1 once it became clear there would be no 1.0.2 needed.

2. Add a new site config parameter (something like DW_STABLE) that indicates whether a site is only accepting our "stable" changes.

3. Change the behavior of -update in bin/vcv to only update as far as the "stable" tag if DW_STABLE is set.
kareila: (Default)
[personal profile] kareila
I will shortly be pushing changesets that close the staging and production branches in the dw-free repository. The branch system hasn't been used effectively and causes problems whenever cloning a new copy of the repository, so I'm reverting to using the default branch for all development.

There shouldn't be any user facing changes from this, except for one change to bootstrap.pl. However, if you run into any difficulties please let me know immediately. Thanks!

ETA: my testing shows that you may have to manually do hg update -C tip after pulling in the dw-free changes, since the active branch has been switched. After that everything should work normally, although the branches will not appear to be closed until you update Mercurial to a version that supports branch closings.
kareila: Taking refuge from falling debris under a computer desk. (computercrash)
[personal profile] kareila
I've been reading the O'Reilly Mercurial guide, and one item I've run into is the 'hg rename' command for renaming files. We have NOT been using this. Instead, we've been removing the old file and adding the new file.

This is bad for a couple of reasons. One, recent changes to the old file can get dropped on the floor (this has happened at least once). Two, there's a discontinuity in the revision history. You can still access the revision history of the older file under the original file name, but it would be nicer for the entire revision history to remain easily accessible.

Looking Forward



There are a couple of options you can set up in your .hgrc to help with this:

[diff]
git = true


This will tell hg to use "git-style" diffs, which show that a file has been renamed, not add/removed. There are other differences, such as permission changes, that are ignored by the default diff as well.

[defaults]
addremove = --similarity 100


This tells the "hg addremove" command (which should be executed before every commit, probably - how often do committers forget to add new files to the repo?) to detect when the same file has been removed and readded under a different name, and treat it as a rename instead. Presumably values less than 100 allow fuzzier matching.

Looking Backward



I couldn't figure out a way to correct the revision history in our repository without some hideous branch merging scheme for each changeset containing a failed rename, so for now I'll just make a list of the files that should have been renamed and weren't.

Read more... )

To be clear, I am not placing blame here, but I would like for us all, and most especially my fellow committers, to be aware of the issue and attempt to do the right thing in the future. :)
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
Mark and I talked this over a bit at LCA, and we've implemented a new Bugzilla tag: "from-suggestions". This will let you see improvements our users suggested, if you're looking for a bug to work on and don't know what you want to do next. (I'll be adding the keyword to bugs in batches over time so I don't totally overwhelm with bugmail.)

Right now, our implementation rate runs around 25% (it'll drop a little when next I migrate stuff; I've been putting it off). One of my several-months-out goals is to get that rate to around or above 50%, so we can legitimately say that we've implemented over half of the things that've come through the suggestions process and been approved by the community and by us.

I dare you all. :)
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
So, this week I checked in Bugzilla, and we've got 62 bugs that are assigned, with no patch, and with no activity on the bug since 9/1/09! [staff profile] mark and I would really like to get that number down a bit -- stuff people are working on in the background and stuff that's being blocked by other stuff is fine to keep assigned, but some of the bugs are fairly low-complexity things that people have been asking for, or that are bugfixes as opposed to minor features or display/UI enhancements.

Now, every week is Unassigning Amnesty week around here -- anyone who watches all bugmail can tell you that I do a "I want this! No I don't!" at least once a month -- but this week we're declaring an Unassigning Amnesty in force.

If you've got a bug that's assigned to you, and you don't think you're going to be able to get to it anytime soon, you can toss it back into the fishpond with zero shame. Just change the status from ASSIGNED to NEW, click the 'edit' next to the Assignee line, and tick the 'Reset Assignee to default' tickbox. Then save the bug. Voila, it's off your hook and out of your hands! (How many bad clichés can I fit in here?)

Later on in the week, so that people have a chance to decide which ones they still want, I'll be emailing people who have bugs that haven't been updated in a while and that look abandoned and checking in with them to see if there's anything we can help with/anything they're stuck on, or whether they just don't have the time in their lives right now to do DW dev work. (Which is totally not a shame. Really. Seriously.) Those I don't hear back from on the contacts, I'll be unassigning bugs from them to release them back into the general assignment pool.

If you have a bug that qualifies as "assigned, no patch, no activity since 9/1" that you still want to do, and you don't want to be contacted about it, just drop a quick "This bug is blocked on bug XXX" or "I'm still sketching out the design for this" or "I'm going to post to [site community profile] dw_dev about this one", or, really, anything to let us know that it hasn't slipped through the cracks.

(And thanks, guys, for being so awesome! *g*)
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark
Okay, by request of [personal profile] afuna I am now tagging releases. I am going to adopt the version scheme suggested by [personal profile] damned_colonial:

X.Y.Z

X = major version (closed beta, open beta, release, quarterly milestones, etc)
Y = code push (individual code pushes within a major release)
Z = patches (minor things, probably bug fixes, not part of an announced push)

In this case, last night marked release 0.1.0 of Dreamwidth. A few patches later and things seem stable, so as of a few minutes ago I just tagged us up to 0.1.1.

Tagging is done on all repositories, but since some of our repos move very slowly I do not intend to be tagging them with every single version. The main repository that 99% of our code goes into is of course the dw-free repository. This means that, more than likely, you want to look at that for the latest version of the Dreamwidth code.

This seems more reasonable to me than having to maintain the same list of tags in all of the repositories. Thoughts?
afuna: Cat under a blanket. Text: "Cats are just little people with Fur and Fangs" (Default)
[personal profile] afuna
I've been thinking a bit about organization, and being able to find out when a particular changeset went live. There are a couple of existing resources which we can look up and use to step backwards, or guess:

Code status, which shows the latest commits with live and non-live; [site community profile] dw_maintenance which announces the code push; [site community profile] dw_dev, with the code tours to get a rough idea of what's changed; [community profile] changelog_digest, which tags entries with "code push", so that it is possible to look up exactly when past code pushes happened.

One more thing I'd like to do is to tag code push commits directly in the repository. That would make it easier to look up when code pushes happened, and which changesets were applied to a particular code push.

That would also make it possible to update to a specific tag, and be able to quickly find when a particular feature/bug was introduced and went out -- whether it had always been that way and no one noticed or whether it was something that just went live in the last code push. We could use the tags to jump between code pushes, or to a commit just before a code push to track down bugs or confirm old behavior.

Each major/announced code push would have its own tag (date of push?). Post major-codepush we tend to have a series of minor codepushes to fix new bugs or tweak existing behavior. These could be grouped together into another tag a couple of days after the major code push (which we could also use to signal that we're finally done with the minor post-codepush pushes, and we can review/commit bigger patches now).

Thoughts, etc?
pauamma: Cartooney crab wearing hot pink and acid green facemask holding drink with straw (Default)
[personal profile] pauamma
Looking for people interested in implement more stats collection and display stuff. Anyone interested, see the links below, and or poke me or Afuna.

http://wiki.dwscoalition.org/notes/Stats_Design
http://bugs.dwscoalition.org/show_bug.cgi?id=124
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark
Edit, read first: After a lot of looking and talking to [personal profile] damned_colonial, only Launchpad is under consideration now. But the rest of the entry is preserved for interested parties.

Now the question is, how big of a change is this for us, and does it provide enough positive benefit considering the interruption? I think it does, what do you think?

...

After OSCON, I'm considering hosting the Dreamwidth code on Launchpad or GitHub, which are both open source project hosting environments that give you a lot of really cool collaborative features and help to lower the barriers for people to get working on things. Plus, it makes it really easy to federate the code like we want people to be able to federate the site.

I'd love people to go take a look and comment with their thoughts:

Launchpad has a site tour: https://launchpad.net/+tour/index

GitHub has some 'what we offer' halfway down the front page: http://github.com/

Yes, both of these would require that we learn either Bazaar or Git (as opposed to the Mercurial we use right now). And we'd have to figure out how to setup [site community profile] changelog to work with it. But those are minor hurdles, if the advantages of being in a hosted environment like that are really worth it.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
Have now:

* enabled target milestones in Bugzilla
* milestoned NEARLY EVERYTHING with when we should get it done by

Unmilestoned issues are either "I don't care, get it done whenever" (for minor things) or "prioritized past Q409" (for bigger things). Will work out best way to differentiate that sooner or later.

For information on how this all works, please see Roadmap on-wiki.

Basically, barring the special-case functional milestones of "B - Open Beta" and "L - Site Launch", "target milestone" means "we want to have this done by the end of this time period". From the Wiki:

Our two current milestones are functional: open beta and "site launch", which is when we will consider ourselves out of beta testing. After those milestones have passed, we'll use quarterly milestones: Q309, Q409, Q110, Q210, etc.

For the functional milestones, Target Milestone: Open Beta and Target Milestone: Site Launch means that the item should be finished before that goal.

For the date-based milestones, Target Milestone: QXXX means that the item should be finished by the final day of that quarter: "Q309" means that the item should be finished on or before September 30, 2009.

Blocking flags (blocking-open-beta+, blocking-launch+) means that the items are considered essential to that functional milestone.


It is entirely possible, and in fact expected, that "L - Site Launch" will extend into Q309 -- that's the overlap period between transition from functional milestones to date-based milestones. During that time, we'll be working on both sets of milestones.
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark
We're planning on launching open beta in mid-April.  That's about five weeks away at this point.  As such, I'm going to be going into a focused development mode starting in a few days.  What does this mean for you?  Well, it means you should keep the following things in mind:
 
* Everybody will be asked to take on bugs that are marked as 'blocking-open-beta' and prioritize those.
* I will only be doing reviews of patches that are for bugs in the open-beta list.  (Generally speaking.)
 
Does this mean nothing else will be done or worked on?  Not at all!  You are encouraged to work on whatever you find interesting.  I'd far rather have you working on something you want to work on that doesn't happen to be on the list than work on nothing at all.  :)
 
Also, Pau and Afuna will be reviewing everything they're comfortable with still.  It's just me that's hunkering down.

Any questions?  Fire away!
 

Profile

dw_dev: The word "develop" using the Swirly D logo.  (Default)
Dreamwidth Open Source Development

June 2025

S M T W T F S
1234567
89101112 1314
15161718192021
22232425262728
2930     

Syndicate

RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Jun. 14th, 2025 06:58 pm
Powered by Dreamwidth Studios