Dreamwidth Development

GitHub repository rename and updates

Wed, 19 Oct 2022 08:15:19 GMT

Posted by: mark

Hi all!

As of today, I've made a few fairly broad changes to the way our repositories are set up. You will need to likely make some changes to your development workflow. But! They should be simple.

First, the changes:

The dw-free repository has been renamed to dreamwidth -- because,
The dw-nonfree repository is no more, it has been merged into the dw-free under the path ext/dw-nonfree/.
Finally, the master branch has been renamed to main.

These two changes mean that no more do we have to track and coordinate changes across two repositories, they can be done atomically in one place. Since we don't really support or test running Dreamwidth on its own (without the branding) this will also allow us to simplify development by ultimately collapsing the code, too.

To update your checkout of Dreamwidth, you will need to do two separate things. First, go to Github and to your existing dw-free repository, go to the settings and:

Rename the repository to dreamwidth
Go to branches and rename the master branch to main

Once that's done, you need to update your local checkout like this:

 cd $LJHOME

 # WARNING: Only do this if you have no code here that you care about!
 rm -rf ext/dw-nonfree

 # Update the repository root (for your "origin" remote, **set your username here**)
 git remote set-url origin git@github.com:YOUR_GITHUB_USERNAME/dreamwidth.git

 # Update the repository root (for your "upstream" remote, please change as needed)
 git remote set-url upstream git@github.com:dreamwidth/dreamwidth.git

 # Fetch only to get the new main branch
 git fetch

 # Switch over to it
 git checkout main

 # Get rid of local master
 git branch -d master

 # Redirect HEAD
 git symbolic-ref refs/remotes/origin/HEAD refs/remotes/origin/main

That's it; you should be good to go now. I'll work on updating the Dreamhack scripts shortly, but wanted to get these instructions out. Please let me know if you have any questions/comments/issues.

comments

GitHub Services going away

Mon, 18 Jun 2018 04:48:22 GMT

Posted by: pauamma

(aka, the thing that lets us do changelog posts and a few other things, like - I believe - the autoclaiming thing)

https://developer.github.com/changes/2018-04-25-github-services-deprecation/

What should replace it for us? Or should there be a bug for it?

h/t: the warning at the bottom of recent changelog entries.

comments

"How to fix a mess in git"

Mon, 08 Sep 2014 08:11:25 GMT

Posted by: swaldman

Nice flowchart here.

It's not new, but I haven't seen it here before :-)

comments

Claiming a bug on Github, another way

Tue, 05 Aug 2014 14:05:06 GMT

Posted by: fu

Forgot to say -- if you forget to claim a bug formally, but you submit a pull request that refers to that bug using the "#123" format in a commit, then that will also automatically assign the bug to you.

comments

Github Issues: now on the wiki!

Fri, 01 Aug 2014 13:39:15 GMT

Posted by: kaberett

I've synthesised discussions about and practical use of GHI into a single wiki page. Please shout if you want anything added or changed!

I've also gone through and replaced as many references to Bugzilla with references to GHI as (1) makes sense and (2) I am immediately able. There's about 20 pages still on my hit-list, not all of which I have the knowledge to deal with - if you feel like pitching in, please take one and let me know when it's done (or, if you don't have wiki access, let me know what the appropriate edits would be).

( List of pages outstanding )

comments

Claiming a bug on Github

Wed, 30 Jul 2014 08:45:57 GMT

Posted by: fu

We discovered that you have to be a committer on a repository in order to claim a bug for yourself on Github. If you're not one, you have to ask someone to assign the bug to you.

This didn't really seem feasible, so we've fixed it. You can now claim an issue by leaving a comment with the words "claim", "claimed", or "claiming".

Case doesn't matter -- you can use capital letters if you want. You can also have other words be part of the comment, so you don't need to memorize a specific format. "I'm claiming this" will work just as well as "Claimed!".

Claiming will only work if the current issue is not yet claimed -- this will avoid the problem of accidentally grabbing the bug from someone else during a long discussion about claiming something else in a different context.

You *do* need to be part of the Dreamwidth contributors team on Github before assigning issues will work so we've gone and added everyone who's been active in the past year (if you got an email from Github welcoming you the Dreamwidth contributors team that's what that's about!). If you're a new contributor, you'll be automatically added when you give your CLA to denise.

For existing contributors, do double-check if you're on any of the Dreamwidth teams. If it shows you a list of teams you're on, you're good! If we missed you somehow and tells you that you're not on any teams, let us know and we'll add you ASAP.

comments

Retiring master

Sun, 27 Jul 2014 17:52:17 GMT

Posted by: fu

We have two main branches in the Dreamwidth repositories: master and develop.

master was intended to be used but... never really got any love.

So here's what I propose: remove all the instructions in our documentation which mention master, and just have 'develop`.

Since develop is where all work goes, nothing in the dev workflow will officially change. But occasionally we have accidents where master gets merged into developinstead of the other way around, or someone finds themselves on master when they should be on develop instead -- and removing any mention of master from our instructions should prevent that from ever happening again.

As for what will happen to master... we never actually used it, and at this point ingrained habits for develop + all setups old and new assume develop, so I suggest we continue not using master the way we currently do.

comments

Categorization of Issues on Github

Mon, 07 Jul 2014 17:54:16 GMT

Posted by: kareila

Hi all! During Saturday's dev chat, one of the discussions that came up was the current status of our work on improving the process of using Github Issues as our bug tracker.

Here is where we are right now:

It is now possible for anyone to use inline text tags to categorize new issues. fu has put up a rough draft of the instructions for this process, along with a list of the labels that are currently recognized. We are also planning to allow issues to be "claimed" in a similar manner, although the format for doing that hasn't been finalized yet.

Another part of Github Issues that we've been experimenting with is the use of milestones. If you look at the milestones on dw-free you can see, for example, the status of open bugs on certain major projects like Foundation and mobile styles. I think this could be really useful, especially in light of recent comments saying that we haven't been doing a good job of communicating the progress of our major development projects within the development community.

There are also other milestones currently displayed on that page such as "unclaimed", "curated", and "in progress" - a lively discussion ensued as to whether these would be better used as tags instead of milestones, and I'd like to continue that discussion below. The main issue with milestones on Github in general is that a bug cannot be assigned to more than one milestone. From a workflow perspective, we would expect to see a progression of untriaged -> unassigned -> claimed/in progress -> pull request -> closed, and the fact that an issue cannot exist in more than one of those states at a time makes it a good candidate for the use of milestones - but could conflict with using milestones for the purpose of tracking large projects.

There was also a great deal of confusion as to what "curated" was supposed to mean in this context. mark said what he desired was a list of three or so "top priority" unclaimed bugs, so that he could easily decide what to spend his limited time on. Again, this may be a better candidate for a tag than a milestone, and maybe with a label that is easier to interpret.

The last issue mentioned is that we still haven't come up with tags for indicating which area of the code is involved with a bug (e.g. styles, notifications, etc.), which is something we relied on pretty heavily with our previous Bugzilla setup in order to find new things to work on.

Please continue the discussion in comments!

comments

addenda to yesterday's entry

Wed, 02 Apr 2014 15:41:06 GMT

Posted by: denise

1) People who do not have write access to the repo can't change labels or milestones, so ignore all the bits about changing labels and milestones in yesterday's entry. Somebody with commit access will have to add the labels and change the milestones when an issue is submitted and when somebody decides to start working on an issue.

2) We did discover that it's possible to assign issues to people once we add them to a read-only team membership of the repo. I've added people who have open pull requests and assigned the issues they're working on to them.

comments

Bug tracking moving to Github Issues, effective now

Tue, 01 Apr 2014 15:33:31 GMT

Posted by: denise

This entry will be stickied in dw_dev for at least a few weeks; please point people at it if they haven't seen it!

Apologies for the delay on writing up guidance on this, folks; we were wrangling out the last of the details. (Or rather, we were wrangling enough of the details for me to make this post; I'm sure there are multiple remaining details unwrangled, which we'll figure out as we go.)

Starting now, and thanks to Bugzilla's sad demise, we will be starting a trial period of using Github Issues to keep track of our bugs. It's possible that it won't work out for us, but right now the downsides are outweighed by the very strong advantage of having everything in one place and the heavy integration.

We will not be importing the entire Bugzilla database, for two reasons: one, putting together something to port over all the data is effort that would be best spent elsewhere, and two, it's been a long time since we've been through the list to cull things that are no longer features that are no longer needed and bugs that are no longer manifesting. We decided that starting fresh with a blank slate and only entering things as they come up will be a good way of making sure the list is only full of relevant things.

This post will be about two things: how to log bugs and how to find bugs to work on. There will be a future post on how a particular item (issue) will move through the workflow with you; we have a few last little things to clear up there first, but I wanted to get this posted as soon as possible so people could start work again.

Of note:

* Please only use this process for code-related bugs. We're not yet 100% certain what the process for docs bugs will be. For now, if you spot a FAQ that needs to be changed or needs to be written, report it with a comment on this entry in dw-docs.

* DO NOT use this process for security bugs. If you think you've found a security bug and want to report it, email webmaster@dreamwidth.org (private support category) and we'll take it from there.

* Part of this transition involves declaring total bug amnesty. If you had a bug assigned to you in Bugzilla, and decide that you don't want it anymore (or if you've forgotten about it completely), you're off the hook. (Not that you couldn't have been off the hook at any time anyway, but I know how hard it is to admit defeat sometimes.)

* On the other hand, if you had a bug assigned to you in Bugzilla, and you're still working on it or were almost ready to submit a pull request, we still want it! Open a new issue in Github for it and carry on.

* Just because we're not making a concerted effort to migrate every open bug from the old Bugzilla database doesn't mean that we don't want bugs to be re-reported. If you logged a bug in Bugzilla (or remember a bug from Bugzilla), and the bug is still happening and still annoying you, please open an issue for it. I will be trying to go through the last year or so of still-open bugs from Bugzilla to find "bugs that are still bugs" and re-create them, but I can't guarantee how long it will take me and I'd rather the bugs get logged sooner than I can commit for-sure to doing it, especially so that we have a nice collection of stuff for people to pick from if they just want to pick something up and hack.

That having been said:

( Logging bugs )

( Finding bugs to work on )

There you have it! It will probably take a little bit of getting used to (I know it took me a bit to figure it all out) but -- having gone through the process a few times in the course of figuring things out -- it really is very straightforward once you start doing it. The biggest gotcha, I think, is going to be remembering to set the milestone for all the unclaimed bugs. (That's part of the reason why we're considering using labels for that functionality instead of milestones; there are benefits and detriments to both. Fu and I will decide in a week or two once we see how a small scale test works out.)

Please take this chance to log all the bugs you're still working on, and all the bugs you can think of as still affecting you, over the next few days! Once that's been done, I'll start going through the various "upkeep" tasks (new themes, new embed sources, etc) and add those, then start working through the dw_suggestions posts tagged "bugzilla: migrated" to see which ones should be moved over to GHI.

(ninetydegrees, I know you have spreadsheets for themes that were in Bugzilla and not yet patched; if you want to add those, that would be awesome, but if you don't have the time/energy, I will get to them when I fill in the various "is: upkeep" things.) (Also, I will explain the "is: upkeep" and how that differs from other "is: foo" tags in a few paragraphs!)

Again, I'm sorry for my delay in writing up the guidance for What We're Doing With This -- things took a little more discussion. Thank you all for being so willing to roll with things and try out new workflows.

If you run into questions as you work, just holler.

( Appendix: What the labels mean: a work in progress )

comments

Your branch is ahead of 'upstream/develop' by 540 commits.

Sat, 07 Sep 2013 17:04:42 GMT

Posted by: swaldman

~/dw (develop)$ git status
# On branch develop
# Your branch is ahead of 'upstream/develop' by 540 commits.

Did we ever figure out what causes this? I have a theory, but I don't really understand...

(Theory: the dwu script does "git pull --ff-only upstream develop" straight from the remote repo into the local develop branch. This doesn't update the local copy of "remotes/upstream/develop" that was created by a "git fetch" sometime in the past. This is what the current branch is being compared to. I have no idea whether this makes sense, and if so how to deal with it...)

comments

Release branch 1.17 created

Wed, 17 Jul 2013 19:16:36 GMT

Posted by: mark

I've created release-1.17 branches in both repos in preparation for tomorrow's code push. If you have anything you really want to land in tomorrow's code push, please let me or fu know and we'll see if we can get it in before then.

This frees the develop branch for continuing development. Anything that lands there will be slated for release in the NEXT code push (which will hopefully be next week, on account of next week being OSCON, and our typical enjoyment of Conference Driven Development).

comments

Your own personal DW bug tracker on your GitHub fork

Thu, 06 Jun 2013 23:31:24 GMT

Posted by: foxfirefey

So, sometimes Bugzilla can be slow to reference/search, or sometimes you want to break up a bigger bug into a set of tinier bugs without cluttering up Bugzilla or making a bunch of comments to your bug in Bugzilla that you then have to manually scan for information.

Your personal GitHub fork can have its own simple bug tracker! Log in, go to your fork of dw-free or dw-nonfree, and find the settings in the button bar. Go check the box next to issues. Voila! You now have a very simple, fast issue tracker you can use in a way that is most convenient to you. The link to the issues interface will be a button in the toolbar. Here's mine so far:

https://github.com/foxfirefey/dw-free/issues

You can note that I'm putting the Bugzilla bug number in the header of each issue, just so I know what bug each issue belongs to. I have also made myself a milestone for the next code push (tomorrow!), so I can filter down to which issues I want to have finished by then.

You can also check the wiki box to have your own personal wiki notebook for taking your research notes that aren't relevant to anyone else by checking the wiki box on that settings page.

I hope this is useful for you! I'm thinking it will be very useful for me.

comments

quick tip when reviewing pull requests

Tue, 12 Mar 2013 06:46:07 GMT

Posted by: fu

There are two ways to leave per-line comments when reviewing a pull request:

opening up each commit in a separate tab and going through them one-by-one (commits are listed in both the Discussion and the Commits tabs)
going to the Files Changed tab, and seeing the entire final result

I used to open up each commit individually, but:

if the person is forced to rebase (either to get rid of extra commits, or to clean up their history), the discussion is lost
under the discussions tab, there's no context, just the comment, which makes it possible to know that there's activity going on, but very difficult to keep track of exact context
it's possible that something you reviewed in one commit was fixed later on in the commit series, invalidating your comment
in the email notification, the "view it on github" link shows you the commit, not the associated pull request (and there's no easy way to find which pull request a commit is on, given just the commit)

I'd been ignoring the Files Changed tab for some reason, but I've now found that they actually don't have any the disadvantages of the per-commit view:

if the person is forced to rebase, old discussion is preserved in the pull request. If the rebase altered the diff, you'll get an "comment made on outdated diff"; if the diff is the same after a rebase, it shows up like it did before
under the discussions tab, it shows the context of the diff you're commenting on, along with your comment, which makes it very easy to keep track of the exact reason a comment was made (especially on short comments hehe)
you see the final version of the code that's being submitted
in the email notification, "view it on github" links to the pull request where all the discussion is taking place

All this is to say that if you're poking around and want to comment on something you see in someone else's pull request (and we do encourage code review in this manner!), use the Files Changed tab. It's much more useful.

Some examples:

actual pull request where you can see how easy it is to tell that a review comment has been handled (and what still needs work, or what doesn't need to be changed after all)
test pull request where I committed a change, commented using both the commit/files changed view, rebased+force pushed the rebase (which blew away the comment on the commit, but retained the comment on files changed), and then made the diff outdated -- which collapsed the discussion but still kept it visible

comments

Git tip: moving where a branch is based

Sun, 10 Feb 2013 06:18:26 GMT

Posted by: foxfirefey

I just had to figure out how to make a branch I was developing be based on the "develop" branch instead of the "master" branch. The command to do this ended up being, as far as I can tell:

git rebase --onto develop master Bug3426/editotherstyles

The advice that finally made sense came from this guide and was explained thusly:

git rebase --onto topicC topicA topicB

Where topicC is the newbase, topicA is the oldbase, and topicB is the reference for what HEAD of topicC will become.

comments

Case conflict: Sinisteria.jpeg

Sat, 29 Dec 2012 19:10:40 GMT

Posted by: fu

I had some weird behavior on my computer (running Mac OSX), because we had an image file named Sinisteria.jpeg, and another named sinisteria.jpeg, so I committed a fix.

Depending on your OS, you may need to run an extra step next time you upgrade. You'll need this for Mac OSX for sure, I can't remember how Windows and Linux deal with case in filenames.

Pull in updates as usual. This will delete Sinisteria.jpeg (uppercase)
$ git pull Updating ecc1d71..7cb6ab7 Fast-forward htdocs/img/styles/paperme/Sinisteria.jpeg | Bin 786 -> 0 bytes 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 htdocs/img/styles/paperme/Sinisteria.jpeg

Then check if you accidentally deleted sinisteria.jpeg (lowercase)
$ git status # On branch develop # Changes not staged for commit: # (use "git add/rm ..." to update what will be committed) # (use "git checkout -- ..." to discard changes in working directory) # # deleted: htdocs/img/styles/paperme/sinisteria.jpeg #

If you didn't see the "deleted: ..." bit, you're fine. No need to do anything more.

If you do see the "deleted:..." bit, then do this:
git checkout -- htdocs/img/styles/paperme/sinisteria.jpeg

Now check again to make sure that worked:
$ git status # On branch develop nothing to commit (working directory clean)

Done!

comments

Best way to review pull requests

Fri, 09 Nov 2012 23:13:37 GMT

Posted by: foxfirefey

So, as far as I can tell from my researching over the past few days, the only real way to get a pull request to review it is to:

* Manually add the submitter's dw-* repo as a remote in the repo on your hack (or Github I suppose, but there is no GUI advantage here).
* Manually pull the branch in question

Is there a better way to do this? Am I missing something? This seems really, well, annoying, with manually crafting URLs and whatnot, and not very user friendly. (I'm trying to make some documents on reviewing pull requests for people who are not fu, since we need to try and spread out that work a little.)

comments

Github might be experiencing intermittant downtime

Fri, 19 Oct 2012 22:14:38 GMT

Posted by: foxfirefey

They've been being hit by DDOS attacks--so if you're trying to do something with our Github and it isn't responding, know that this might be the cause!

comments

Something that might be fun to play with our Git repository with

Wed, 10 Oct 2012 19:16:49 GMT

Posted by: foxfirefey

For the folks who like digging into things like this: codeq (warning: Javascript generated blog page, ew)

comments

O'Reilly offering a sale on Git books

Sat, 06 Oct 2012 04:27:22 GMT

Posted by: denise

We still want to buy you books, and since this week O'Reilly is having a sale on resources about Git (our new version control system), we'll even buy you more than one of them if you spot stuff that you think will help :)

Browse the list of eligible titles (or the wider O'Reilly catalog) and if you'd like a copy of the book or the videos, let me know: we'll buy you that plus another title. (Maybe a book on Perl, or Javascript, or jQuery, or UI design, or or or...) I'm trying to figure out a way with O'Reilly that will make it easier for us to pay for ebooks that are then delivered to your O'Reilly account, but in the meantime, we can either do it where you buy it through your O'Reilly account so you can have future access to it and we'll write you a reimbursement check, or I buy it through my O'Reilly account with the DW card and send the file to you. Or you can get the hard copy, of course, if you don't want an ebook version; it's up to you.

Either way: let us buy you stuff! People have been really hesitant to take us up on this, but I promise you, we really want to, and if you meet the criteria (5 patches in last 12 months, at least one in the past two months), we mean you even if you think you haven't done 'enough'.

comments

Reminder: We've moved over to Git

Thu, 27 Sep 2012 08:54:43 GMT

Posted by: fu

As of this code push, we're using the Git repository for our code on production. If you haven't moved your own repositories over to Git yet, you should soon!

foxfirefey has a handy list of resources for the Git transition.

We just had our 101st pull request to the dw-free repository over on Github! Two of those were test pulls, but the others are all legit ;-)

The dw-nonfree repository, on the other hand, has only had 14 pull requests. Since we don't really want much here other than branding stuff, this is as it should be.

comments

Identifying yourself when committing in Git

Mon, 24 Sep 2012 15:55:15 GMT

Posted by: fu

Quick note:

We'd like you to use your Dreamwidth username / email when committing. This makes it a lot easier for us to track who did what in the future. It also helps us match the author of the commit with who you are, and lets us confirm that we've received your CLA.

Edit your config (~/.gitconfig) and add this section:

[user]
        name = USERNAME
        email = USERNAME@dreamwidth.org

Then add your USERNAME@dreamwidth.org email to your Github account, so that the commits on Github will link to your identity on Github.

Please don't forget to tweak this setting on both your local computer and your Dreamhack if you do commits on both. Thank you!

If you're working on other projects, you can also just edit your author identity in just the Dreamwidth repositories (Thanks shadowspar)

comments

Important: Update to "dwu"

Sun, 23 Sep 2012 21:16:36 GMT

Posted by: mark

If you use the "dwu" script from the wiki, you will need to update your local copy. I've made some changes because the last version of the script would merge changes from both develop and master, leading to broken environments and GitHub repositories.

http://wiki.dwscoalition.org/wiki/index.php/Dev_Maintenance#dwu_-_Updating_the_repos

Please use the updated script.

If you have gotten yourself into a broken state, let me know and I can walk you through fixing it. There is some force pushing involved, sadly.

comments

Release notes

Sun, 23 Sep 2012 04:59:18 GMT

Posted by: mark

We did our first release from git tonight. It went well, I will chalk that up to all of the hard work from folks like fu and exor674 (and others!) to make sure that everything was in place. fu put together a nice "things we have to do" in dw_staff, so that was very helpful.

Anyway, I wanted to document some things that are important to know for the period between releases. This probably won't apply to most of you, but just in case...

So, we are following the document about a successful git branching model. This is what git flow is based on and it's a really efficient model, so we're using it for doing our features and in this case, our release.

The release process was, in essence:

Create a release-1.11.0 branch
Make final tweaks in this branch to get things ready for release
Push this branch live
Merge this branch back in to develop and master
Add a release tag with the version number on master

This leaves our repository in this state:

develop is still open, you can keep committing things to it (no code freeze, yay!)
master has stable tags, so you can always roll back to tags here (easy production rolls!)
release-1.11.0 is open for doing fixes on the live release (easy fixing production, yay!)

The thing that is important to know, if you are going to be committing fixes that you intend to go live, is that you need to make your changes on the release-x.xx.x branch -- not on develop. Any code committed to develop will, because of its nature, have to wait until the next code push -- weeks away.

If you have stuff you need to get live sooner (fixes to things broken in the last code push or stability/important fixes), you need to adjust your workflow slightly. It's pretty easy though, and goes something like this:

git checkout -b release-1.11.0 dreamwidth/release-1.11.0

Now make your fixes, as per normal, and submit a pull request. As long as your current branch is the release branch, then you will be able to submit a pull request on the appropriate branch, and we can merge it in and push it out to the site quite easily.

comments

Overdue Announcement: Dreamwidth, Github and you

Sun, 26 Aug 2012 07:26:19 GMT

Posted by: foxfirefey

As of several weeks ago, all commits to the Dreamwidth codebases have gone to our new repositories on Github:

There are a few relevant wiki documents that have been fully revised to account for this change:

Moving your Dreamwidth installation to use Github -- these instructions will tell you how to move your current Dreamhack/dev environment over to a Github based installation
Dev Maintenance -- this document tells you how to keep your Github Dreamwidth based installation (and your Dreamwidth forks) updated with the code from the Dreamwidth repositories
Draft: Github development process -- This is the document in the least refined state, so keep in mind that it is in stronger need of suggestions and revisions. It goes through the very basics of Git workflow. This and Dev Maintenance might eventually end up merged into one document.

We are working on getting the rest of the wiki development documentation updated (see the dw_wiki post). Feel free to comment to this post with your questions/concerns about the move!

comments