sophie: A cartoon-like representation of a girl standing on a hill, with brown hair, blue eyes, a flowery top, and blue skirt. ☀ (Default)
[personal profile] sophie
As mentioned in my last post, I will be taking the Dreamhack server down for maintenance in 30 minutes! I'll edit this post once it starts and again when it's back up.

10:04am BST: I have restricted logins to admins and am beginning the maintenance now. This will involve a reboot and some diagnostics before opening it back up.

10:58am BST: The maintenance is done, and logins are unrestricted again!
sophie: A cartoon-like representation of a girl standing on a hill, with brown hair, blue eyes, a flowery top, and blue skirt. ☀ (Default)
[personal profile] sophie
Hi all,

I'm going to need to schedule some maintenance for the Dreamhack server as I've just discovered that the quota settings haven't taken effect. I'm therefore going to schedule maintenance for Wednesday 2017-04-19, 10am BST (5am EDT, 2am PDT) (about 36 hours from now) for up to 2 hours. (It shouldn't actually take that long, but I'm reserving it just in case.)

This maintenance will involve a reboot of the Dreamhack server. I will also be restricting logins until the maintenance is done, as the nature of the maintenance means that it will need as little data modification done as possible.

I'll post again once the maintenance starts, and edit that post when it ends.
sophie: A cartoon-like representation of a girl standing on a hill, with brown hair, blue eyes, a flowery top, and blue skirt. ☀ (Default)
[personal profile] sophie
Apologies again for the delay on this - Dreamhack applications are now officially open again!

As I explained in my previous post (which you should read if you haven't already - there's a lot of info in there), anybody who wants a Dreamhack should apply through this form, even if you already had one before. I will, as a one-time thing, email everybody who had an account before to let them know that the service is back up and that they'll need to re-apply.

My apologies once again for the delay! If you find any issues with the service, please feel free to let me know.
sophie: A cartoon-like representation of a girl standing on a hill, with brown hair, blue eyes, a flowery top, and blue skirt. ☀ (Default)
[personal profile] sophie
Hi all,

As you may know, about 2-3 weeks ago the Dreamhack server died. Since then, [staff profile] mark and I have been working on getting its replacement going, and updating a few things.

It should be ready to go in a few days, and I wanted to make a few notes for when it comes back up:
  • Firstly and most importantly, you'll need to re-apply for a Dreamhack if you want one, and you'll be set up as if you were a new user. Any changes that you pushed to GitHub will be available, but any other data you may have had will be gone - apologies for that.

  • I do still have email addresses for everybody who had an account when the server went down, and I'll send out a one-time email to everybody when the server is up to point them to this post. After that, the only people who will receive emails about Dreamhacks will be those who have applied for one.

  • The address you need to use to log into the server via SSH will be different from the Web address domain. The email you receive when applying for a Dreamhack will state this clearly.

  • The new server will have an increased quota. The earlier quota of 500MiB was enough at first, but since then the space taken by a base install of Dreamwidth has risen to 270MiB. In light of this, I've raised the quota to 750MiB.

  • You won't need to do a one-off compilation of the stylesheets and JavaScript any more unless you make changes to them - the new-user script will automatically do that for you, and your Dreamhack will have working CSS out of the box.

  • Each user will automatically get a test database called "test_dreamhack_<user>", accessible using the same database user and password as the main database. You'll still need to configure it properly yourself for now, but the installer will at least copy the required files to $LJHOME/ext/local/t for you to configure. Later on I'm hoping that it'll be possible to have it configured automatically.

  • The official email address to contact me has changed - you should now use my Dreamwidth email address (sophie at dreamwidth dot org). Automated emails will come from this address, so if you had the previous email whitelisted you may want to whitelist this new one instead.
Thanks for your patience with this - there's a lot that's been going on, but the new server should be ready to go very soon! Watch this space. :)

If you have any questions, please leave a comment! I'll answer any questions you might have.
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark
Hi all,

I hate to be the bearer of bad news but the Dreamhack server has suffered an unexpected, and fatal, hardware failure. The system is offline for now while we set up a new one.

Further unfortunately, at some point during the various acquisitions and migrations that it has gone through over the years, the automated backups stopped working and I never noticed. All data that was on the Dreamhack server has been lost and cannot be recovered. (We've asked the host -- they said there's nothing they can do.)

I'm really sorry to anybody who had work on the server and now doesn't anymore. This is really, definitely not what we want -- even if it is a development environment, we do want it to be reasonably dependable as far as relying on it not to lose things.

This week we're going to work on setting up a new one and we'll get everybody back up and running. Any code you've submitted as PRs or to your fork on Github will be retrievable, of course. We're happy to help with that when things are back up if you need a hand.

Again, I'm sorry for any patches or code you've lost through this.
sophie: A cartoon-like representation of a girl standing on a hill, with brown hair, blue eyes, a flowery top, and blue skirt. ☀ (Default)
[personal profile] sophie
Hello! Your friendly neighbourhood Dreamhack admin here.

I've updated the version of Locale::Country that the Dreamhack machine uses - we were on 3.25, and we're now on 3.42.

I also installed the Log::Log4perl module, as it's now being used by the codebase. This should make update-db.pl work again!

If there are any issues (related to these or otherwise), please comment and let me know!
sophie: A cartoon-like representation of a girl standing on a hill, with brown hair, blue eyes, a flowery top, and blue skirt. ☀ (Default)
[personal profile] sophie
The Dreamhack machine currently seems to be suffering from a performance issue where if you try to start your Apache, it might tell you that Apache failed to start. In reality, the Apache server *has* started, but the 'start-apache' script is timing out before Apache can report that fact.

If the last line of your Apache log file is:

NOTE: Google::Checkout::* Perl modules were not found.

then Apache has most likely started successfully. (This is a normal warning on Dreamhack machines; we haven't used the Google Checkout code for some time and thus the modules are not installed on the Dreamhack machine.) You can verify this by navigating to your Dreamhack's Web address.

We're investigating this at the moment. This might require a reboot of the Dreamhack machine in the near future; we'll post about it here if that's the case.

[edit: We're going to reboot the Dreamhack machine. I'll edit this post again when it's up.]

[edit 2: The Dreamhack machine is back up, and I've rolled out a new version of start-apache that can wait a bit longer if Apache doesn't start up immediately. It prints out a "Waiting for Apache to start" message, and then checks every second (for up to 6 seconds) whether Apache started, so it shouldn't slow things down for anybody.]
kareila: Taking refuge from falling debris under a computer desk. (computercrash)
[personal profile] kareila
The server that hosts the Dreamhacks seems to be having hardware issues and is currently not responding. We have a ticket open with the hosting service and will update this post when the server is back online. Sorry for the inconvenience!

Update: It is back up now. Let us know if it stops responding again!
sophie: A cartoon-like representation of a girl standing on a hill, with brown hair, blue eyes, a flowery top, and blue skirt. ☀ (Default)
[personal profile] sophie
Hi,

Soon - probably within the next month - I'm planning to add code on the Dreamhack server which will automatically shut down Apache instances when the owner logs out (or at some point thereafter). I'm doing this because running Apache instances do take up resources on the Dreamhack server and a lot of people don't stop their Apache instances when they're not working on them, and it does make things a bit harder.

Currently I'm thinking of either shutting down the Apache instance either 7 days after the owner last logged out (unless they're currently logged in, of course) or shutting it down immediately on logout. If anybody has any issues with either of these options, please let me know in the comments - I'm very willing to listen to people on this.

If you have any other comments or questions, please do also comment. I'd be happy to answer any questions on this.
kareila: (Default)
[personal profile] kareila
As mentioned previously, we're about to reboot the Dreamhack server. Hopefully it won't be down too long; I'll update this post when it seems to be back up.

Update: seems to be back now.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
The company that hosts the Dreamhack machine has let us know that they're prepping to migrate the machine to their new platform. We've been warned that due to the large number of files on the machine, the migration is likely to take a long time or fail to complete successfully.

In order to minimize migration-related downtime and maximize the chance of not having failures, we're temporarily zipping up the home directories of anyone who hasn't touched any files in their account since 2014. Nothing will be permanently deleted -- just temporarily turned from many files into one file. (The problem isn't with the amount of data, but the number of files.)

We'll let you know when the migration is happening. EDIT: We're going to aim for Thursday night, probably sometime in the evening, but depends on when [personal profile] alierak has the chance to babysit it.
kaberett: Overlaid Mars & Venus symbols, with Swiss Army knife tools at other positions around the central circle. (Default)
[personal profile] kaberett
[personal profile] cesy has just been tripped up by my pull request #1751: she updated dw-free but not dw-nonfree, and as a consequence Apache wouldn't start. This suggests that you may need to update -nonfree at this point regardless of whether you actually use it; here are the instructions.
sophie: A cartoon-like representation of a girl standing on a hill, with brown hair, blue eyes, a flowery top, and blue skirt. ☀ (Default)
[personal profile] sophie
As a reminder, the Dreamhack machine is going to be taken offline at some time between 1am EDT and 6am EDT on April 28th for up to an hour, as the datacenter will be doing maintenance work!

You can use the links in the above paragraph to check when this is in your local timezone.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
The datacenter where the Dreamhack box is hosted will be taking the machine offline for up to an hour on the morning of April 28, sometime between 1AM and 6AM EST, for datacenter maintenance.

I'll try to remember to post a reminder closer to the date, but if I don't, somebody either poke me or post the reminder!
sophie: A cartoon-like representation of a girl standing on a hill, with brown hair, blue eyes, a flowery top, and blue skirt. ☀ (Default)
[personal profile] sophie
I'm just about to reboot the Dreamhack server. I'll update this post when it's done!

[edit 17:09 GMT: The server has been rebooted.]
sophie: A cartoon-like representation of a girl standing on a hill, with brown hair, blue eyes, a flowery top, and blue skirt. ☀ (Default)
[personal profile] sophie
Hi all,

The Dreamhack machine has had to be rebooted because of performance issues. I'll update this post when everything's back to normal. Anybody who had Dreamhacks running will have had their Dreamhacks turned off.

Again, will update when everything's back up and running!

[edit about 30 minutes afterwards: The Dreamhack machine is back and appears to be in good working order!]
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
If your dev environment is publicly reachable on the internet, it will get spam! I've written a quick guide to spam prevention.
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark

Hi all,

Sorry for the delay, there was a most unfortunate timing as I was in Europe without Internet access. Anyway, the Dreamhack server is back up and running so you should be able to log in and such.

I'll also be making sure that at least [personal profile] alierak has access to the control panel so this kind of thing won't be a single point of failure on me in the future.

karzilla: a green fist above the word SMASH! (Default)
[staff profile] karzilla
The dreamhack machine is not responding to SSH requests. [personal profile] sophie is aware of the issue but says that [staff profile] mark is the only one with the necessary access to administer the machine, and he is currently unavailable. We'll get things fixed up as soon as we can!
sophie: A cartoon-like representation of a girl standing on a hill, with brown hair, blue eyes, a flowery top, and blue skirt. ☀ (Default)
[personal profile] sophie
The new Apache system-wide configuration that I talked about a few posts ago has now been rolled out across all Dreamhacks.

The instructions with the show-apache2-config command detailed in [staff profile] mark's post are now deprecated. I have replaced the "show-apache2-config" command with a command that simply prints out the new config, so you will not lose anything by following the instructions, but they are no longer necessary.

For the two users who had changed their httpd.conf files in a manner other than through Mark's script, I've put their changes into their local httpd.conf files. I'm also adding some more lines to the system-wide httpd.conf in a new pull request since some of the changes would be useful to everybody.

You should now be able to continue 'hacking as normal. If you have any questions or something goes wrong, please leave a comment!

Profile

dw_dev: The word "develop" using the Swirly D logo.  (Default)
Dreamwidth Open Source Development

July 2017

S M T W T F S
      1
2345678
910111213 1415
16171819202122
23242526272829
3031     

Syndicate

RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Jul. 22nd, 2017 12:50 am
Powered by Dreamwidth Studios