sophie: A cartoon-like representation of a girl standing on a hill, with brown hair, blue eyes, a flowery top, and blue skirt. ☀ (Default)
[personal profile] sophie
Hi all,

As you may know, about 2-3 weeks ago the Dreamhack server died. Since then, [staff profile] mark and I have been working on getting its replacement going, and updating a few things.

It should be ready to go in a few days, and I wanted to make a few notes for when it comes back up:
  • Firstly and most importantly, you'll need to re-apply for a Dreamhack if you want one, and you'll be set up as if you were a new user. Any changes that you pushed to GitHub will be available, but any other data you may have had will be gone - apologies for that.

  • I do still have email addresses for everybody who had an account when the server went down, and I'll send out a one-time email to everybody when the server is up to point them to this post. After that, the only people who will receive emails about Dreamhacks will be those who have applied for one.

  • The address you need to use to log into the server via SSH will be different from the Web address domain. The email you receive when applying for a Dreamhack will state this clearly.

  • The new server will have an increased quota. The earlier quota of 500MiB was enough at first, but since then the space taken by a base install of Dreamwidth has risen to 270MiB. In light of this, I've raised the quota to 750MiB.

  • You won't need to do a one-off compilation of the stylesheets and JavaScript any more unless you make changes to them - the new-user script will automatically do that for you, and your Dreamhack will have working CSS out of the box.

  • Each user will automatically get a test database called "test_dreamhack_<user>", accessible using the same database user and password as the main database. You'll still need to configure it properly yourself for now, but the installer will at least copy the required files to $LJHOME/ext/local/t for you to configure. Later on I'm hoping that it'll be possible to have it configured automatically.

  • The official email address to contact me has changed - you should now use my Dreamwidth email address (sophie at dreamwidth dot org). Automated emails will come from this address, so if you had the previous email whitelisted you may want to whitelist this new one instead.
Thanks for your patience with this - there's a lot that's been going on, but the new server should be ready to go very soon! Watch this space. :)

If you have any questions, please leave a comment! I'll answer any questions you might have.
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark
Hi all,

I hate to be the bearer of bad news but the Dreamhack server has suffered an unexpected, and fatal, hardware failure. The system is offline for now while we set up a new one.

Further unfortunately, at some point during the various acquisitions and migrations that it has gone through over the years, the automated backups stopped working and I never noticed. All data that was on the Dreamhack server has been lost and cannot be recovered. (We've asked the host -- they said there's nothing they can do.)

I'm really sorry to anybody who had work on the server and now doesn't anymore. This is really, definitely not what we want -- even if it is a development environment, we do want it to be reasonably dependable as far as relying on it not to lose things.

This week we're going to work on setting up a new one and we'll get everybody back up and running. Any code you've submitted as PRs or to your fork on Github will be retrievable, of course. We're happy to help with that when things are back up if you need a hand.

Again, I'm sorry for any patches or code you've lost through this.
sophie: A cartoon-like representation of a girl standing on a hill, with brown hair, blue eyes, a flowery top, and blue skirt. ☀ (Default)
[personal profile] sophie
Hello! Your friendly neighbourhood Dreamhack admin here.

I've updated the version of Locale::Country that the Dreamhack machine uses - we were on 3.25, and we're now on 3.42.

I also installed the Log::Log4perl module, as it's now being used by the codebase. This should make update-db.pl work again!

If there are any issues (related to these or otherwise), please comment and let me know!
sophie: A cartoon-like representation of a girl standing on a hill, with brown hair, blue eyes, a flowery top, and blue skirt. ☀ (Default)
[personal profile] sophie
The Dreamhack machine currently seems to be suffering from a performance issue where if you try to start your Apache, it might tell you that Apache failed to start. In reality, the Apache server *has* started, but the 'start-apache' script is timing out before Apache can report that fact.

If the last line of your Apache log file is:

NOTE: Google::Checkout::* Perl modules were not found.

then Apache has most likely started successfully. (This is a normal warning on Dreamhack machines; we haven't used the Google Checkout code for some time and thus the modules are not installed on the Dreamhack machine.) You can verify this by navigating to your Dreamhack's Web address.

We're investigating this at the moment. This might require a reboot of the Dreamhack machine in the near future; we'll post about it here if that's the case.

[edit: We're going to reboot the Dreamhack machine. I'll edit this post again when it's up.]

[edit 2: The Dreamhack machine is back up, and I've rolled out a new version of start-apache that can wait a bit longer if Apache doesn't start up immediately. It prints out a "Waiting for Apache to start" message, and then checks every second (for up to 6 seconds) whether Apache started, so it shouldn't slow things down for anybody.]
kareila: Taking refuge from falling debris under a computer desk. (computercrash)
[personal profile] kareila
The server that hosts the Dreamhacks seems to be having hardware issues and is currently not responding. We have a ticket open with the hosting service and will update this post when the server is back online. Sorry for the inconvenience!

Update: It is back up now. Let us know if it stops responding again!
sophie: A cartoon-like representation of a girl standing on a hill, with brown hair, blue eyes, a flowery top, and blue skirt. ☀ (Default)
[personal profile] sophie
Hi,

Soon - probably within the next month - I'm planning to add code on the Dreamhack server which will automatically shut down Apache instances when the owner logs out (or at some point thereafter). I'm doing this because running Apache instances do take up resources on the Dreamhack server and a lot of people don't stop their Apache instances when they're not working on them, and it does make things a bit harder.

Currently I'm thinking of either shutting down the Apache instance either 7 days after the owner last logged out (unless they're currently logged in, of course) or shutting it down immediately on logout. If anybody has any issues with either of these options, please let me know in the comments - I'm very willing to listen to people on this.

If you have any other comments or questions, please do also comment. I'd be happy to answer any questions on this.
kareila: (Default)
[personal profile] kareila
As mentioned previously, we're about to reboot the Dreamhack server. Hopefully it won't be down too long; I'll update this post when it seems to be back up.

Update: seems to be back now.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
The company that hosts the Dreamhack machine has let us know that they're prepping to migrate the machine to their new platform. We've been warned that due to the large number of files on the machine, the migration is likely to take a long time or fail to complete successfully.

In order to minimize migration-related downtime and maximize the chance of not having failures, we're temporarily zipping up the home directories of anyone who hasn't touched any files in their account since 2014. Nothing will be permanently deleted -- just temporarily turned from many files into one file. (The problem isn't with the amount of data, but the number of files.)

We'll let you know when the migration is happening. EDIT: We're going to aim for Thursday night, probably sometime in the evening, but depends on when [personal profile] alierak has the chance to babysit it.
kaberett: Overlaid Mars & Venus symbols, with Swiss Army knife tools at other positions around the central circle. (Default)
[personal profile] kaberett
[personal profile] cesy has just been tripped up by my pull request #1751: she updated dw-free but not dw-nonfree, and as a consequence Apache wouldn't start. This suggests that you may need to update -nonfree at this point regardless of whether you actually use it; here are the instructions.
sophie: A cartoon-like representation of a girl standing on a hill, with brown hair, blue eyes, a flowery top, and blue skirt. ☀ (Default)
[personal profile] sophie
As a reminder, the Dreamhack machine is going to be taken offline at some time between 1am EDT and 6am EDT on April 28th for up to an hour, as the datacenter will be doing maintenance work!

You can use the links in the above paragraph to check when this is in your local timezone.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
The datacenter where the Dreamhack box is hosted will be taking the machine offline for up to an hour on the morning of April 28, sometime between 1AM and 6AM EST, for datacenter maintenance.

I'll try to remember to post a reminder closer to the date, but if I don't, somebody either poke me or post the reminder!
sophie: A cartoon-like representation of a girl standing on a hill, with brown hair, blue eyes, a flowery top, and blue skirt. ☀ (Default)
[personal profile] sophie
I'm just about to reboot the Dreamhack server. I'll update this post when it's done!

[edit 17:09 GMT: The server has been rebooted.]
sophie: A cartoon-like representation of a girl standing on a hill, with brown hair, blue eyes, a flowery top, and blue skirt. ☀ (Default)
[personal profile] sophie
Hi all,

The Dreamhack machine has had to be rebooted because of performance issues. I'll update this post when everything's back to normal. Anybody who had Dreamhacks running will have had their Dreamhacks turned off.

Again, will update when everything's back up and running!

[edit about 30 minutes afterwards: The Dreamhack machine is back and appears to be in good working order!]
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
If your dev environment is publicly reachable on the internet, it will get spam! I've written a quick guide to spam prevention.
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark

Hi all,

Sorry for the delay, there was a most unfortunate timing as I was in Europe without Internet access. Anyway, the Dreamhack server is back up and running so you should be able to log in and such.

I'll also be making sure that at least [personal profile] alierak has access to the control panel so this kind of thing won't be a single point of failure on me in the future.

karzilla: a green fist above the word SMASH! (Default)
[staff profile] karzilla
The dreamhack machine is not responding to SSH requests. [personal profile] sophie is aware of the issue but says that [staff profile] mark is the only one with the necessary access to administer the machine, and he is currently unavailable. We'll get things fixed up as soon as we can!
sophie: A cartoon-like representation of a girl standing on a hill, with brown hair, blue eyes, a flowery top, and blue skirt. ☀ (Default)
[personal profile] sophie
The new Apache system-wide configuration that I talked about a few posts ago has now been rolled out across all Dreamhacks.

The instructions with the show-apache2-config command detailed in [staff profile] mark's post are now deprecated. I have replaced the "show-apache2-config" command with a command that simply prints out the new config, so you will not lose anything by following the instructions, but they are no longer necessary.

For the two users who had changed their httpd.conf files in a manner other than through Mark's script, I've put their changes into their local httpd.conf files. I'm also adding some more lines to the system-wide httpd.conf in a new pull request since some of the changes would be useful to everybody.

You should now be able to continue 'hacking as normal. If you have any questions or something goes wrong, please leave a comment!
sophie: A cartoon-like representation of a girl standing on a hill, with brown hair, blue eyes, a flowery top, and blue skirt. ☀ (Default)
[personal profile] sophie
I have a bit of an apology to make.

I haven't been doing a lot lately, and I'm keenly aware of that. I could use this space to find a bunch of excuses about why this is the case, but I'm not going to do that. I'm just going to say that I apologise, and that I'm going to try to make things better. Starting now. (Actually, starting yesterday.)

Because I haven't been keeping in touch, I know I haven't been very responsive to questions and comments. I want to change this, too. As such, I want to do a redo of a previous post that I made a few years ago: What do you find difficult on a Dreamhack? I want to try to help people and fix any issues they might be having.

Also, if you have any questions or comments about the Dreamhack service in general, please do leave a comment, or send me a PM! A comment would be preferable, but I'll be glad to respond and/or help if needed regardless.
sophie: A cartoon-like representation of a girl standing on a hill, with brown hair, blue eyes, a flowery top, and blue skirt. ☀ (Default)
[personal profile] sophie
Now that the Dreamhack machine has been upgraded, I'm looking at making things a bit easier to maintain, starting with the Apache configuration files.

[staff profile] mark has already posted instructions to update your configuration so that it will work, but ideally, that shouldn't need to be an issue - it should Just Work. And as it turns out, very few people ever modified their default Apache config files; there are very few cases where it's necessary. In fact, of the three people that have, two of them had only done so for Apache 2.4 compatibility.

It makes sense to me to have this sort of configuration change done on a global basis rather than needing each person to change their Apache config themselves. I do recognise that some people will still need to make local changes, though, so I'm going to be setting it up so that the configuration file as it is now will simply become an Include line to a central configuration file which uses environment variables (which will be in a ~/apache/conf/envvars file) to allow Apache to know where everything is.

This will also involve adding one line to the start-apache and stop-apache scripts pointing to the new envvars file.

I'll make the changes for everybody so that nobody needs to worry about how to do this. Where people have modified the defaults, I'll make sure that those changes are still in the new files. I'll also back up the old Apache configs to a ~/apache/conf/httpd.conf.2.4-backup file in case anything goes wrong.

I plan to do this change in a few days from now - if anybody has any questions or concerns, please let me know! This change does mean that nobody will need to modify their Apache config any more, so when this goes through the instructions in [staff profile] mark's post will no longer be needed, although it will still be necessary to update your code to the latest version and, if you have any branches you're working on currently, pull the latest changes from develop into those branches in order to get the code change for Apache 2.4.

If anybody needs any help, please let me know in the comments!
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark

Hi all,

I've now updated our Hack server to Ubuntu 14.04 and Apache 2.4 to mirror our current plans for production. This update has been a long time coming, and there were some bumps on the way. You will need to update your Hack's Apache configuration in order to start your Hack back up.

The steps for updating your configuration:

  1. Log in to your Hack
  2. Run: /dreamhack/bin/show-apache2-config > apache/conf/httpd.conf
  3. Upgrade your code to the latest (however you do that)
  4. Restart Apache (again, however you do that)

Basically, since we've now upgraded to Apache 2.4, there is a code change you need that was just landed to the develop branch about 24 hours ago. So if you're in the middle of working on some code, you'll need to rebase off of the latest develop to keep working on it.

Profile

dw_dev: The word "develop" using the Swirly D logo.  (Default)
Dreamwidth Open Source Development

March 2017

S M T W T F S
   1234
5678 91011
12131415161718
1920212223 2425
262728293031 

Syndicate

RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Mar. 26th, 2017 05:19 am
Powered by Dreamwidth Studios