Bugzilla etiquette, responsibilities, and business priorities

[mark]

Recently we had a bit of confusion in Bugzilla about exactly who does what, who's responsible for what, and when it might be appropriate to circumvent someone else's priority in order to get something done more quickly. (Read: step in on someone else's bug to fix something.)

The situation: let's say that denise writes a patch. Then afuna comes along and commits that patch. A week later, I push it live, woo! It turns out, oh noes!, there was a bug in that code that went live. Who is responsible for fixing this?

The short and correct answer is: once code has gone live on the site and contains a bug, it is everybody's responsibility to ensure a timely fix is made.

When it comes down to it, this is a balancing act between Dreamwidth the community project and Dreamwidth the business. We spend most of our time optimizing for the former because I believe that's the right thing to do. However, from time to time, we have to make concessions in the name of the latter. (Example, I'd much rather be working on some features right now, but I need to do the payment system. Also, The review queue needs some serious work, but alas, I have to deprioritize it for a little while!)

This balancing act comes to the front again when we talk about the difference between code that is still theoretical (in Bugzilla as patches) and code that is live and available for users to interact with. As soon as code is live, I expect anybody who can step in to fix problems to do so, and I expect people who submit patches to understand that once it goes live, they lose a lot of that ownership that they are given while working on it to begin with.

I don't want people being afraid to fix bugs because of code/bug ownership or interpersonal issues -- we're all on the same team here. We're all working together to make sure this site continues to be as amazing as it has been, to keep a wonderful home for us and our friends, etc etc. This is something that is going to take people working together to accomplish, and 'this is my bug, go away' can be counter-productive to those goals in some cases.

Now, also consider that it does depend somewhat on the severity of the bug in question. If it's a minor issue (colors/font wrong, text out of place, ugly, etc) then you should ask the original author first before you touch the bug and start fixing it. It's just polite. In this example, Denise put all her time into the original feature or fix, more than likely she feels bad that it was broken and would like the opportunity to fix it. If Afuna swoops in and fixes it, that hinders Denise's ability to learn and it doesn't let her fully resolve the issue. (If in doubt on priority, ask Denise or me -- we'll be happy to tell you.)

But, and I want to stress this again, the decision on priority is going to be one that is fuzzy. Denise may think the bug is not worth an immediate fix from someone else, but Afuna might think it is. If Afuna comes in and fixes it, then Denise might be annoyed, but Afuna did nothing wrong. If the priority was wrongly determined, feel free to point that out and discuss it, but let's avoid any anger and frustration: everybody here did the right thing and was working to the betterment of the site.

Does anybody have any thoughts on this subject? I'm happy to discuss it and try to find something that elegantly balances the priorities and needs of the project with the goals and desires of our development community.

Comments

[exor674]

Any policy on what to do if one accidentally fixes a bug assigned to someone else while doing something unrelated? ( either in the case of "same cause, different bugs" and "This problem was blocking my bug, didn't look to see if it already existed cause I was in a coding groove and it was a quick fix"? )

[mark]

"Same cause, different bugs" -- well, if one of them is assigned to you and you fix it, then you should resolve the other one as quickly as possible to let them know that you took care of it. Or, if the other person is already well on their way and is going to be able to solve it, you could let them do it too. I'm thinking particularly in cases where the other person is a newer developer, I'd rather see them fix the bug than someone who won't learn anything by it.

For the blocking issue, you should just talk to the other person. Ask them how far they've gotten, and if they have an ETA (but don't be rude about it, be gentle). If they haven't started, or have barely started, you could ask to take it over.

If you just see something and fix it, though, that's tricky. I don't want people to be frustrated by working on something and having other people trounce their work. I guess it will have to come down to communication and judgment -- talk to whomever has the bug assigned. Of course, if you don't know about a bug for the problem, then you can't really be blamed for fixing it.

Some of the weird edge cases we'll just have to handle when they come up and trust people to realize we're all working to make Dreamwidth a better place.

[alierak]

I'd also suggest that it's everybody's responsibility to ensure that patches are tested / reviewed before being committed or pushed live. Of course it's first of all the bug owner's job to make a decent patch, but once it's in the review queue anyone can try it out and give useful feedback, whether or not you feel authoritative enough to approve or reject it.

[mark]

Yeah, I agree.

This is something I've been thinking more about lately, actually. Using Mercurial "properly" and splitting things into a Dev repository and a Production repository. Then we can have things committed to Dev pretty early on, even while probably buggy, and have people go test them and submit more fixes.

Oh, maybe Dev, Staging, and Production. Once we're ready to start tightening the screws on the build, we ship it to Staging. Anything that needs fixing is fixed in Stage, development can continue in Dev. Once it's stable, it moves to Production.

[denise]

If possible, I'd also like to have the ability for individual people to opt into Staging on the live site somehow, so people can volunteer into it. There are a lot of people who'd like to play with the latest & greatest in exchange for tolerating a few bugs, and there are sometimes bugs that don't manifest until you're dealing with a larger set of use cases.

[kareila]

This. I cannot tell you how frustrated I have been trying to test things without a realistic amount of account/post/etc. data to work with. Being able to test things against the live site would make the review process much more effective, as well as more accessible to a wider range of volunteers.

[yvi]

Seconding this.

[mark]

I usually use the importer and just import my DW account onto my dev server. That gives me plenty of data to play with, typically.

If I need several accounts to play with, I just have all my test accounts import... test, test2, test3 all end up little xb95s! Aw so cute, aren't they?

...

Anyway, maybe worth a shot if you haven't tried this before.

[kareila]

Whoa, that never occurred to me. Hmm.

Still doesn't help with lots of accounts, though... like the birthday module problem I was working on several months ago.

[denise]

I wonder if we could get people to volunteer their (public) data for a test data suite that we could have as part of Dreamhacks. I know I wouldn't mind throwing my public posts into the pot.

[yvi]

*facepalm* why didn't I think of this when doing the tag merge? why????????

[jadelennox]

*facepalm*

Yeah, I'm not very bright. Genious timesaver.

[denise]

You aren't the only one. *laugh*

[pauamma]

That works well if you have a dreamhack, but I believe Denise meant betatesting for standard site users, as is possible on LJ.

[pauamma]

That would require dedicating one or more servers to push staging to, aka beta servers. IIRC, Mark nixed the idea when I suggested it a few months ago, because it would significantly lower reliability as it would leave only 2 web servers for normal, non-testing usage. (Mark, is still applicable?)

[mark]

Yes, the same constraints are active.

[pauamma]

There's one consideration you didn't mention and which I think is relevant: how close are we to a code push? If there's one looming (ie, in less than the average/expected bugfix turnaround time on a broken uncommitted patch), it's reasonable to let a committer Just Fix It IMO, because the alternative (waiting for a fix from the patch submitter) is risking havihng to rush in a fix either right before the push or after the push, to fix broken stuff that went live. This may be applicable aw well if something small is missing to get a feature we rilly, rilly want to work, and a push is looming. (Even though, in that case, nothing is *broken* per se.)

With all the above (and the rest of the discussion) in mind, I would favor the 3-repo approach, with staging being short-lived (it's copied or synced from dev when Mark decides a codepush is in order, and ditched or idle once the dust settles after the code push), and the following rules for who can fix what when :
- In dev: Original patch author, unless it breaks things all devs need, eg syntax error in a module everyone uses (http://hg.dwscoalition.org/dw-free/rev/fb8df8611779) or bug in bin/upgrading/upgrade-db*.pl (http://hg.dwscoalition.org/dw-free/rev/a49d7c481fa6, http://hg.dwscoalition.org/dw-free/file/763971242f25/bin/upgrading/update-db-general.pl). In those cases, anyone can step in, and if the fixer is a committer), it's left to committer's discretion whether to upload a corrected/corrective patch to the bug, or to commit directly from their dreamhack.
- In staging: Anyone. Preferred patches either work around (eg, disable feature) or back out the buggy patch. (If it's buggy, it should be fixed in dev.) If the fix is obvious, you can do that instead, but it should be reviewed by the original patch author before any commitback to dev, and the original author is free to substitute their own instead.
- In prod: No one, fixes should happen in staging and be pushed to prod (after testing) instead. If the bug isn't testable or the fix nor verifiable in staging, then committers and ops working together.

Note: I'm not sure who have commit rights to staging. Maybe all dev committers, maybe only some, maybe only Mark and ops. I expect Mark and ops only would have commit to production. (Can't be Mark only, or we can't fix stuff that slipped past staging if Mark is practicing his violin or socializing with bears.)

[mark]

I like the split approach, it makes it a lot easier to understand who does what and when. When we move to Dev/Staging/Prod I think we should adopt this.

As to the last bit, yeah, ops folks currently have full commit rights everywhere and I expect that to stand. If there's a bug and they need to fix something in Prod they'll be able to. (And actually, I expect we can give commit to Prod as a staff-commit-bit, so fu would get it, f.ex.)

Another thing, stuff doesn't have to be committed to go live. Sometimes the fastest fix is one done in the /home/dw/current direction on sb-admin01 and pushed out seconds later. In some circumstances, that's certainly the right thing to do.

[pauamma]

stuff doesn't have to be committed to go live.

Point, but these should still get backcommitted to prod and staging ASAP, otherwise no one except you and ops have visibility, which is (IMO) bad.

[mark]

Oh I agree it's terribly bad. But as far as 'Mark is out riding wild bears!!!' (love that example, btw) goes, there are ways to take care of things even if commit privileges aren't sorted correctly.

[ninetydegrees]

I think you also need to factor in the level of experience of the people involved. Had I been you, I might not have even blinked an eye. But I'm not you. I have zero experience here and I don't know what's ok and what isn't because there wasn't any guide anywhere (and I expect future volunteers won't necessarily read this post). To me, it can't hurt to leave a comment in all cases, and take the time to explain if you think it may be needed. Better some communication than no communication at all. And, yeah, it's just polite. :)