10:04am BST: I have restricted logins to admins and am beginning the maintenance now. This will involve a reboot and some diagnostics before opening it back up.
10:58am BST: The maintenance is done, and logins are unrestricted again!
I'm going to need to schedule some maintenance for the Dreamhack server as I've just discovered that the quota settings haven't taken effect. I'm therefore going to schedule maintenance for Wednesday 2017-04-19, 10am BST (5am EDT, 2am PDT) (about 36 hours from now) for up to 2 hours. (It shouldn't actually take that long, but I'm reserving it just in case.)
This maintenance will involve a reboot of the Dreamhack server. I will also be restricting logins until the maintenance is done, as the nature of the maintenance means that it will need as little data modification done as possible.
I'll post again once the maintenance starts, and edit that post when it ends.
As I explained in my previous post (which you should read if you haven't already - there's a lot of info in there), anybody who wants a Dreamhack should apply through this form, even if you already had one before. I will, as a one-time thing, email everybody who had an account before to let them know that the service is back up and that they'll need to re-apply.
My apologies once again for the delay! If you find any issues with the service, please feel free to let me know.
As you may know, about 2-3 weeks ago the Dreamhack server died. Since then, mark and I have been working on getting its replacement going, and updating a few things.
It should be ready to go in a few days, and I wanted to make a few notes for when it comes back up:
- Firstly and most importantly, you'll need to re-apply for a Dreamhack if you want one, and you'll be set up as if you were a new user. Any changes that you pushed to GitHub will be available, but any other data you may have had will be gone - apologies for that.
- I do still have email addresses for everybody who had an account when the server went down, and I'll send out a one-time email to everybody when the server is up to point them to this post. After that, the only people who will receive emails about Dreamhacks will be those who have applied for one.
- The address you need to use to log into the server via SSH will be different from the Web address domain. The email you receive when applying for a Dreamhack will state this clearly.
- The new server will have an increased quota. The earlier quota of 500MiB was enough at first, but since then the space taken by a base install of Dreamwidth has risen to 270MiB. In light of this, I've raised the quota to 750MiB.
- Each user will automatically get a test database called "test_dreamhack_<user>", accessible using the same database user and password as the main database. You'll still need to configure it properly yourself for now, but the installer will at least copy the required files to $LJHOME/ext/local/t for you to configure. Later on I'm hoping that it'll be possible to have it configured automatically.
- The official email address to contact me has changed - you should now use my Dreamwidth email address (sophie at dreamwidth dot org). Automated emails will come from this address, so if you had the previous email whitelisted you may want to whitelist this new one instead.
If you have any questions, please leave a comment! I'll answer any questions you might have.
I hate to be the bearer of bad news but the Dreamhack server has suffered an unexpected, and fatal, hardware failure. The system is offline for now while we set up a new one.
Further unfortunately, at some point during the various acquisitions and migrations that it has gone through over the years, the automated backups stopped working and I never noticed. All data that was on the Dreamhack server has been lost and cannot be recovered. (We've asked the host -- they said there's nothing they can do.)
I'm really sorry to anybody who had work on the server and now doesn't anymore. This is really, definitely not what we want -- even if it is a development environment, we do want it to be reasonably dependable as far as relying on it not to lose things.
This week we're going to work on setting up a new one and we'll get everybody back up and running. Any code you've submitted as PRs or to your fork on Github will be retrievable, of course. We're happy to help with that when things are back up if you need a hand.
Again, I'm sorry for any patches or code you've lost through this.
I've updated the version of Locale::Country that the Dreamhack machine uses - we were on 3.25, and we're now on 3.42.
I also installed the Log::Log4perl module, as it's now being used by the codebase. This should make update-db.pl work again!
If there are any issues (related to these or otherwise), please comment and let me know!
If the last line of your Apache log file is:
NOTE: Google::Checkout::* Perl modules were not found.
then Apache has most likely started successfully. (This is a normal warning on Dreamhack machines; we haven't used the Google Checkout code for some time and thus the modules are not installed on the Dreamhack machine.) You can verify this by navigating to your Dreamhack's Web address.
We're investigating this at the moment. This might require a reboot of the Dreamhack machine in the near future; we'll post about it here if that's the case.
[edit: We're going to reboot the Dreamhack machine. I'll edit this post again when it's up.]
[edit 2: The Dreamhack machine is back up, and I've rolled out a new version of start-apache that can wait a bit longer if Apache doesn't start up immediately. It prints out a "Waiting for Apache to start" message, and then checks every second (for up to 6 seconds) whether Apache started, so it shouldn't slow things down for anybody.]
Update: It is back up now. Let us know if it stops responding again!
Soon - probably within the next month - I'm planning to add code on the Dreamhack server which will automatically shut down Apache instances when the owner logs out (or at some point thereafter). I'm doing this because running Apache instances do take up resources on the Dreamhack server and a lot of people don't stop their Apache instances when they're not working on them, and it does make things a bit harder.
Currently I'm thinking of either shutting down the Apache instance either 7 days after the owner last logged out (unless they're currently logged in, of course) or shutting it down immediately on logout. If anybody has any issues with either of these options, please let me know in the comments - I'm very willing to listen to people on this.
If you have any other comments or questions, please do also comment. I'd be happy to answer any questions on this.
In order to minimize migration-related downtime and maximize the chance of not having failures, we're temporarily zipping up the home directories of anyone who hasn't touched any files in their account since 2014. Nothing will be permanently deleted -- just temporarily turned from many files into one file. (The problem isn't with the amount of data, but the number of files.)
We'll let you know when the migration is happening. EDIT: We're going to aim for Thursday night, probably sometime in the evening, but depends on when alierak has the chance to babysit it.
I'll try to remember to post a reminder closer to the date, but if I don't, somebody either poke me or post the reminder!
The Dreamhack machine has had to be rebooted because of performance issues. I'll update this post when everything's back to normal. Anybody who had Dreamhacks running will have had their Dreamhacks turned off.
Again, will update when everything's back up and running!
[edit about 30 minutes afterwards: The Dreamhack machine is back and appears to be in good working order!]
Sorry for the delay, there was a most unfortunate timing as I was in Europe without Internet access. Anyway, the Dreamhack server is back up and running so you should be able to log in and such.
I'll also be making sure that at least alierak has access to the control panel so this kind of thing won't be a single point of failure on me in the future.