dw_dev | Things Real Dreamwidth Programmers Do

Entry tags:

introduction

Things Real Dreamwidth Programmers Do

Lots of new developers - including myself - are very nervous about screwing things up. Lots of old developers have told me not to worry, but I keep worrying. I don't want to seem stupid or break something important!

So I thought we could make a list of some wonderful hilarious things that Dreamwidth developers have done and still do. I've put the names of people who told me I could share these, but you can be sure that pretty much all of them make everybody around grin and admit that they do something very similar too, actually.

Our fearless leaders have commented on the topic when I told them I was making the list:

denise: you haven't arrived until you've made at least one mistake that brings the site to its knees ;)

mark, when told about this list's creation: if we're going to create a list of all the shit I've done over the years it will be a very long list.

Things Real Dreamwidth Programmers Do (Or Have Done)

Everybody: Spent ages searching for the bug, only to realise the file you're editing is not actually the same file you're running.

fu managed to do this one while I was actually writing up this list, and not just on her Dreamhack but on Dreamwidth itself!

exor674 I STILL have to look at the TT docs every damn time I have to do things. and I mean the "how to make pages" not the scary "doing weird and obscure things in the plugins" (note:

exor674 is the project head for TT conversion)

shadowspar: I spend a lot of time with 'perldoc -f <most-any-perl-function>'

azurelunatic: Forget to convert null entries to zeroes before doing mathematical operations that's likely to involve counting or dividing.

mark: the other day I oopsed a database and had to rebuild it. while debugging the slow page load thing, I installed something, that uninstalled MySQL :P

azurelunatic: And for my part, I totally submit suggestions all the time that are already logged in Bugzilla.

mark: when I was at Mozilla, I accidentally clobbered the database that contained the crash logs. i.e., every time Firefox crashes and you let it send in the crash log. this was at the Firefox 3 launch. I blew it away. no backups.

denise: like the time

mark helped me troubleshoot my broken email for like three hours, only for me to discover the reason it wasn't working was because i'd let the domain registration lapse

azurelunatic: Forget to increment the serial number on the DNS file, and spend a half-hour cursing and weeping in class.

So next time you're feeling stupid, please remember that these things are all perfectly normal programmer behaviour:

Asking others for help

Forgetting how you did the same thing yesterday

Asking others for more help

Forgetting to restart Apache

Finding yourself looking up the same bit of Perl syntax for the 37th time

Forgetting to upload the file you edited

Making typos and not noticing

Forgetting to restart Apache again

Asking for help yet again

All of these are things that Mark, Denise, Fu, and all of our senior coders do on a regular basis, therefore we have proof they are totally normal and expected and no cause for shame or guilt.

If you care to, please share other silly things you've done in the comments!

Flat | Top-Level Comments Only

I was writing a 'only let admin people log in' function for a mu*. I managed to write the test perfectly backwards, so that all the admin users (and only the admin users) were locked out of the server, and not noticed until it went live.

This is when I learned to backup binaries. Especially since it took about an hour to actually get the dang thing to recompile. Not to mention the bad shutdown of the server.

Not on DW either: When trying to do essentially the same thing (make something wizard-only on a MOO), I instead made every user account including guest accounts a wizard. (= vs. ==).

Edited (Missing )) 2012-02-24 10:44 (UTC)

I made that mistake once on a MOO I was wizarding on, yeah. Once.

After being the one to clean up everything that resulted, I have never forgotten the difference again.

Thanks for posting this, it's a super important message!

Computers, servers, web sites, and all the assorted technical bits and bobs that make them are hard. Asking for help, needing help, getting help, and offering help are totally normal and expected. (It's actually strange if you never need help.)

And as an aside, Dreamwidth development is not run like most other projects. We value contributors over contributions. This web site is about the people, not the lines of code. I have a personal dislike of the word meritocracy because of all the baggage it has and the way people use it as an excuse to demean and discourage. Not here.

If you keep that thought front and center -- people first! -- the rest of it starts to fall into place. We're a team and we will support each other. That's what's important here.

/me wishes she had all the hearts in the world to give to Mark.

Me too!!

I'll second that (or, spying rb below) third it.

One of these days I'm going to write the huge long rant about how the word 'meritocracy' is actually used in open source to mean 'ogliarchy' and how if people really believed in evaluation of people by merit alone they'd be a lot less hostile to newcomers. And then I will post it, and I will turn off my internet for two weeks :P

But seriously, "we're a team and we support each other" is absolutely it. If we make it unpleasant for people to code with us, why would people want to code with us? And I would rather a dozen newcomers-to-programming who are eager and willing to learn and enthusiastic about the project and the product than one burnt-out, bitter, jaded 'rockstar' who screams at anybody who doesn't do something perfectly the first time, even if the 'rockstar' were chewing through all the incredibly complex projects. :)

"ogliarchy"? Government by the oily? :-)

Sadly, it's been done many times and still hasn't made much of an impact on the ideo(t)logues. :/

As far as I can see, most of open source isn't a meritocracy - it's a reputocracy. Sometimes, reputation is connected to merit, but it's far from a clear correlation.

Sometimes whole *teams* of developers and beta testers and regular users overlook some weird interaction that's only obvious once you're looking at it from the correct angle, and afterwards leaves people smacking their foreheads and going "HOW IN THE NAME OF ------- DID I MISS THAT. HOW." If you're lucky, it's not an enormous security hole. If you're lucky, even if it is, no one will have exploited it before it can be fixed.

The lesson to be learned here is, if you notice something that you think is so blatantly obvious that surely someone else has noticed it, but no one has mentioned it, feel free to bring it up and don't be shy about it. Sometimes you may be the lucky one to examine it from the correct perspective to spot the possible problem or adverse interaction.

Dreamwidth development culture is at least mildly unusual in that reporting bugs tends to get greeted with glee (more things getting fixed means fewer things broken!) rather than woe (more things to fix, dammit!) as some projects do.

OMG YES. Projects that get annoyed at folks who report bugs make me get my rageface on.

The bugs are there, whether we've found them or not. At least once we've found them, we can do something about them. =)

And by all means, don't hesitate to speak up if something seems awry, even if you think someone else might have noticed it. They might not have! In software development as a whole, most bugs that get past QA are failures of imagination, not of effort. It's not that people aren't trying hard enough to find the bugs -- they are -- but that nobody managed to come up with the particular combination of things that triggers the bug.

And coming from the other end, there is absolutely no need to wait for 100% reproducible results when turning in a bug report, especially if it's something weird. More people watching for the results means the circumstances may get figured out sooner.

Yes! The bug tracker is documentation as much as it is a list of things to fix. So many times one thinks "I've seen this wonkiness before..." or "I've heard this weirdness mentioned before..." and if it's in the bug tracker, you already have something to go on. =)

And the bug tracker has better search than many places. (I am still not sure whether the inside of my head counts, because what I do is remember the shape of the thing and am supernaturally good at remembering the keywords to find it again, but I do need the actual search engine to find it.)

This made me think of Bug 1964 which I worked on - when I started the bug was basically a list of "this came up in support and may be related" :)

Backing this up: several of my friends and I jokingly identify as "human fuzzers". A "fuzzer" is network penetration testing terminology for a tool that builds nonsensical data packets to throw at a network service (including the network itself; some of them build invalid network packets at various levels) to look for combinations that cause various kinds of service failure.

In our case, we have a certain tendency to put together uses of various programs that are obvious to us, but which those programs' designers and testers apparently never considered.

The fact of the matter is that generally the people who design something (a tool, a program, a web service) have certain notions about how they will be used, and the more time they put into the design, the more likely they will overlook even apparently similar alternative ways of using them. (Think of it as hyperfocusing. And the more you think about something, the more likely you'll get stuck into certain ruts, usually without realizing it.) As someone who's been on both the using and designing side of this, I can say I'm always happy to hear about it if there's some option or possibility I've missed.

But I have certainly known designers and developers who become very annoyed in that case; sometimes because they're so caught up in their design that alternatives come across to them as being in some sense wrong, other times because they're kicking themselves in the butt for blindness but are bad at keeping their self-annoyance out of their interactions with others. (And I will also admit that at times I have had problems with the latter. I hope I've been getting better at catching myself and if necessary taking some deep breaths.)

I am a babydev and managed to screw something up on my second patch *g* This? YEP. That would be mostly my fault. I could have sworn I'd accounted for the closing div needed for odd numbers of icons, but nope, apparently not.

(We'll see if any bug reports come in for my other patch, which was a style)

Ow, this - when you're thinking "need to do this" while coding and you're *so sure* you did it but in fact you didn't - totally did that yesterday in the work for my PhD. Spent the whole day trying to find a bug in an algorithm, only to find out that it was in fact perfectly correct, I had just accidentally mixed up the top and bottom coordinate while drawing the result. And I clearly remember thinking "no, in this case I get the bottom, need to subtract height first!" when I was writing it, but when I finally looked at it again, there was only glaring whitespace...

It even looks liek you meant to account for it! the proper if statement was there already and everything! ♥

it's okay, you don't get the pony for this push.

fu broke creating accounts for a day or so. *G*

I introduced a potential security hole into dreamwidth which I then found a year later, nobody else having noticed it. And that it is much as I will say about it because I don't want anyone to go looking for the exploit, but seriously. For a YEAR there was a security hole in the dreamwidth code that I put there.

:-(

Aaaand it wasn't just you who missed it - whoever did your code reviews and commits missed it too. So any "blame" is always spread between multiple peoples.

Forgetting to restart Apache again

^- Even from a non-dreamwidth perspective this can never be said enough.

Shut up and reboot works wonders for jails/servers too sometimes. @_@;;

(I broke the DNS for my work place, following instructions for what should of been a routine job [so routine that they didn't update the instructions TO NEVER, EVER RUN THAT SCRIPT! or um, do what they have done now, and deleted the script from the server.] This effectively meant, that I broke the internet. Fortunately it could be restored! ... By someone else. :( )

Oh, *ow*. What did it do, if you recall/can say?

I am not a programmer, nor do I play one on TV, but I just wanted to thank you for posting this and tell you that it cheered me up a lot.

I tend to idealize and idolize, and this is actually quite an encouraging thing to be reminded of here.

Thanks again.

my favorite interview question to ask to an applicant is "talk about something you've done that was a total screwup," because I figure anyone who can't answer that question either have no experience or is lying. :D (also it is useful to see how people learn from what they did wrong.)

My favorite massive mistake that I made was typing "rm -r .*" as root. This was back in the day, on an old Ultrix machine, before systems got smart enough to recognize that it was dangerous for ".*" to (correctly, according to regular expression specification) expand to "../*, ../../*, ../../../*, ...".

For non-UNIX geeks, what this means is that I was trying to delete a few files that began with ".", and ended up deleting the entire operating system.

I think I must've done that once too. I can also recall trying to create a tar file, but accidentally putting it inside of one of the directories that was being tarred up. It kept on going and filled up the disk writing a copy of the tar file into itself.

OHH, I did that too...

I think some variant of that one gets done at least once by everyone who's ever used a Unix/Linux system.

Also, my favorite version of this was discovering the hard way that an earlyish (but it had been around for several years) version of Midnight Commander would happily follow .. when told to remove a directory and everything under it. You, um, tend to assume your file manager is smarter than that....

(A handy trick: most of the time, you want .??*. This will miss names like .a, but the number of possible such names is small enough that manual cleanup is easy.)

(Also, minor pedantry; they're not regular expressions but file globs. Globs are much simpler, which as usual is both blessing and curse. On the other hand, you'd curse a lot more if the shell made you use regexes.)

Probably the thing I remember most clearly was a little screwup with moving files on the first unix box I ever administered. Fixing it turned out to be a big deal that I now remember as "the mating of the dinosaurs". There's a very brief reference to it here:

"Within a week I'd figured out how to boot it single-user and normally, how to get around in SunOS and the boot monitor, how to make it unbootable by moving /sbin to a different partition, how to restore it by lugging the disk over to a friend's place and connecting it to his Sun 3/260 one night."

I think we were concerned about how little space there was in the root partition and wanted to try moving something to another partition. Not exactly familiar with the boot process, and how the OS would be needing to load a few important files like /sbin/init before it could even get to the point where it could mount other partitions... I probably moved /sbin and left a symlink. Geez.

So, the external disk was this 80-pound VME chassis, an SMD disk I think? It showed up as xy-something under SunOS. Anyway, it took some doing to find someone with compatible hardware, muscle the disk across campus, and then devise and execute a recovery plan in a living room full of loud people. Mounting my disk's partitions on temporary mount points on someone else's server, booted from their internal disk, so I could move the files back. Seems simple enough now!

Youch...

It's always hardest to fix things at the start because you're feeling awful and anxious and guilty and have to deal with that as well as the cognitive challenge itself...

Rands In Repose: When the Sky Falls is quite apropos.

The steps he's talking about are the exact procedure I (and everyone else senior in the team) used whilst on call in the unix operations team for a national ISP every time something broke and we had to fix the sky.

He's talking about bigger corporate circumstances than apply here, but the principles are extremely sound, and can be applied to much smaller groups.

The old "edit the config file for deployment, leave the local one untouched and then can't work out why it's not working in your sandbox" is classic.

Databases are the worst because anything is so hard to undo. Back when I was a grad, I kicked off a batch job in production which meant that the table was locked and no updates or inserts would work - for two hours in business hours.

Then just this week I ran an update, cleverly commented out the WHERE clause and updated all 500 rows instead of the 1 I intended. Luckily this was in dev this time.

I honestly don't remember exactly what the problem was, but I was responsible for the bug that caused DW to have to do a second seed account sale when the site went into open beta (there was only supposed to be one). There was something that prevented a large number of users from buying seed accounts (I think it may have been people who were creating brand new accounts, as opposed to those who had accounts already but were upgrading), so we felt that we had to give those people another chance by doing another sale. Oops.

Oh god, that! Yeah, it was that you could buy a seed account for an existing account, but not for an account that hadn't been created yet.

(It worked out better in the long run, though. If we hadn't had that extra money things would've gone a bit worse when PayPal was stupid at us.)

This is actually another good point - that you can NEVER know all the things that might happen (or fail-to-happen) as a result of what you do.

Somebody else might learn something while fixing your bug, and that thing they learned might turn out to be critically useful to them in the future. Or a billion other things I can't think of ...

(Late to the party, but) Your bug was greatly appreciated! That's how I got my seed account! :D

About 6 years ago I rebuilt a wordpress theme from scratch to suit my style and for a few other people, this was back when a) I was learning PHP and b) wordpress themes were hard to come by and a PITA to hack in any way.

Spent ages on it, checked it in multiple browsers, got it all going really well. I loved it, it was brilliant.

Put it live on my site and several people reported weird problems that I could not replicate. Eventually a friend installed it on his blog and went through it line by line.

There were a few JS calls that I'd overlooked. They were calling some Flash files that did some weird font glittery things to anything within an H1, H2 or H3 tag.

I couldn't find it, couldn't replicate it, couldn't see it. Then I remembered something.

At the time I was living in a rented flat that had old wiring, broadband wasn't an option, so I was on a 56K dial up. So I had, deliberately, uninstalled Flash, completely, from my machine.

The entire theme didn't work on any machine that had Flash on it, because I'd forgotten to check it with a standard configuration. Took me ages to fix it, had to learn the basics of JS in order to only call the bits I wanted, not the Flash.

After nearly 20 years doing this programming game professionally, in the last 10 years or so, I've been in the habit of writing an automated test suite right along with whatever code I'm writing.

If that test suite and the new code I've written work first time, I'm immediately freaked out and stare at the code suspiciously a few times before believing I wrote both the code and the test suite correctly. Usually I then frob the code so it's wrong, run the test suite to make sure, then frob it back.

That freakout probably tells you how rare it is to get code right first time. :-)

As far as silly things done goes... This morning's was wondering why work people haven't got my email telling them I'm working from home today when it's sitting there right in my outbox because I haven't turned on the secure tunnel to the work email server...

Whenever we do a code push, and no problems get reported within the first hour or so,

fu and I always wind up staring at each other and thinking, "It's quiet .... too quiet...."

"Be vewwy vewwy qwuiet, we're huntin' incwidwents."

(reply from suspended user)

geekosaur suggests adding, quoting something

batrachian said: "printing the 'things real DW programmers do' list and sticking it to one's desk as brainweasel repellant"!

(I will confess to having come back and looked at this post a few times myself for encouragement!)

Such a cool idea!!

Just the other day I was working on a WordPress patch to update some things to the new version of an internal API.

Made some changes, saved, looks like we're on the right track, finished up all my changes. Saved. Reloaded the browser. Deprecated notices referring to the old version. Saved again, just in case I forgot to save the first time. Reloaded again. Still getting notices.

Went back to the editor, hand-combed through every damned closing bracket and semicolon to find what I forgot. Nothing. Reload. Still notices.

Sighed HARD. Reverted all my work. Redid it.

Realized that the test install I was working on was actually the NEXT TAB OVER.

Oh, God, you have no idea how often I do that. I've actually taken to opening the site/page/etc that I'm working on in a whole separate window, with no additional tabs in it, and moving that window to my laptop's built-in screen rather than the external monitor, to make it really clear what I'm looking at. And my Dreamhack's site scheme is set to a scheme I don't use on Dreamwidth itself, so I don't get confused as to which site I'm looking at, for that very reason.

Flat | Top-Level Comments Only

Things Real Dreamwidth Programmers Do

Things Real Dreamwidth Programmers Do (Or Have Done)

Not on DW, but....

Re: Not on DW, but....

Re: Not on DW, but....

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

Tests passed first time = Freakout

Re: Tests passed first time = Freakout

Re: Tests passed first time = Freakout

no subject

no subject

no subject

no subject