Category: Geek

The Dell I ordered to replace my old Linux box arrived, and after spending a few hours setting it up, it’s amazingly tiny, whisper-quiet, and works pretty well! It spooked me a bit when it suddenly started making sound, as I didn’t realize it has a speaker built right into the case.

It also doesn’t have room for the 8TB internal storage drive. Whoops! I probably misread the storage configuration information when I ordered it and it comes with two internal 2.5″ bays, and no 3.5″. It was super-easy to get at those bays though, so I took the old SSD and added it to the one the system came with.

I think I can still work with this, though. It’s so small that there’s plenty of space to put the storage drive in an external enclosure instead. Just from testing with the two external drives I have right now, I can get 150MB/s out of the USB3 ports (versus the 15-20 I was getting on the old system), so speed certainly won’t be a problem. It’ll just be another wall-wart to deal with…

And after using Ubuntu for probably close to 10 years, I’m giving Fedora a try instead. Just for something different, and I’d been seeing recommendations for it in a few places now. I’ll still mostly be using it via SSH, so there probably won’t be too much difference in practice.

Well, the rescue finished, and after letting e2fsck have its way with the recovered data, it looks like only a handful of files were affected: a Windows ISO I can redownload from MSDN if needed (it was Windows 8.1, so…not likely), a couple of game soundtrack files, and five ripped DVDs, all of which were fairly recently ripped ones so the discs are actually still on hand and hadn’t been packed away in the closet yet. The game soundtrack was the only one I actually needed to restore from backup.

Still, I’m a bit paranoid about corruption, so I’m also running a big ‘diff’ between the recovered data and the backups, just to make sure there isn’t some silent corruption in them. That’ll only take another few days to run. Then I have a script I can use to test the integrity of various file types. And then I need to clone all this so that this isn’t my only copy of the recovered data lest another drive fail… (Update: It has found at least a few more corrupted files, so far. Paranoia works!)

This is all still on external drives though, so I still need to get a new internal storage drive. But I’ve also been meaning to replace this old Linux box, as it’s just a bunch of old parts cobbled together as a temporary workaround when a motherboard failed in the old system. It’s just been ‘temporary’ for a few years now… I’m not feeling too keen on assembling my own system right now though, so instead I ordered a Dell T3420 SFF system. It’s a couple years old, but I don’t need anything new and fancy for this role, and it was reasonably priced enough. The small case size (7.8L) will be a nice change from the Antec P180 behemoth (54L!) currently sitting on my desk. I wish you could order them without any hard drives at all though, as I’ll be replacing them with my own anyway.

The ‘ddrescue’ is still running (current ETA: 2.5 more days), and there’s some good news in that it’s stopped logging errors and so far there’s only around 60MB worth of missing data out of 4TB. I’ve also received a couple of 8TB external drives so I have some spare room now to start parts of the recovery process, like getting data out of the ‘dar’ backups.

Fortunately, test runs on the backup drive show that the backups are not affected by the bad block I found on that drive. Unfortunately, I also realized that since the backup was in progress when the drive failed, I’m not sure if maybe that backup run picked up and stored some corrupted data as it was running.

So, I have the following data to work with:

• A fully-intact base backup that dates from July 2017.
• The partial, might-possibly-have-some-corruption differential backup from just before the failure.
• The data salvaged from the failing drive.

My plan so far is that once the ‘ddrescue’ is done, I take the bad blocks list and run e2fsck against the salvaged data. This should get me a list of files that were hit by the bad blocks and could not be recovered. For each of those files (hopefully the list isn’t too long!), try and find a copy in the base backup. If it’s not in the base backup, or I really need a newer version than that one, get it from the differential backup and try and eyeball it for corruption. If any of the ripped DVDs gets hit (quite likely, they took up a lot of the space), note which one and get ready to re-rip it.

Too bad this is all going to happen at USB2 speeds though…

So, while my drive recovery grinds away (current ETA: 8.5 days) and I wait for the arrival of new drives, I need to step back and rethink how I do my data storage and backup to begin with.

Previously, I thought I had a fairly decent setup: one big storage drive that did automatic weekly differential backups via ‘dar’ to an external USB drive, rotating and starting a new full backup once a month while still keeping one previous backup set around, and every once in a while I would physically swap the external drive with an offsite one, for disaster recovery.

That sounds alright, and it worked well for a while, but the problem was that I let the backup solution fall behind my main storage needs, and started compromising on things. As the amount of data I had grew, I started letting it run less often so that the backup drive wouldn’t fill up so quickly. Soon there just wasn’t room for both the previous and current backup sets, so I got rid of the previous one. I had to do full backups from scratch more often because there just wasn’t much room left for the differentials. I got lazier about how frequently I did the offsite drive swap. And eventually, I’d upgraded my storage drive to 4TB but the backup drive was still a mere 1.5TB, so I had to start excluding stuff like the ripped DVDs from the backup because they just wouldn’t fit. This all left my backups in a more fragile state than I’d realized, leading to my current hassles.

Once this is all cleaned up, I obviously need something better. Something simpler, less prone to error and laziness and compromise.

Right now, my basic line of thought is:

• Still have one big internal storage drive, an external backup drive, and an offsite replica.
• The backup drives must always be at least as large as the storage drive.
• Instead of differentials, the backup drive is a straight mirror of the storage drive.
• The mirroring is done automatically on a schedule via ‘rsync’, instead of my horribly convoluted wrapper script around ‘dar’.

This should avoid most of the aforementioned problems except perhaps the offsite swapping laziness, which I’m just going to have to do better at. The data is guaranteed to fit, there’s no need to compromise or exclude anything, the syncing can be done automatically safely, and in the event of drive failure, the backup drive can swap straight in as the storage drive. The downside is that plain mirroring will also mirror any mistakes I make if I don’t realize it and catch them before the next sync occurs, but hopefully most mistakes will be heat-of-the-moment ones I can restore immediately, and in the worst case I can still resort to the offsite drive for an older copy.

Well the plain ‘dd’ on my failing drive was a bust, as a day later the transfer rate had dropped to tens of kB/s, and it would have taken forever at that rate.

So, instead, I’m using ‘ddrescue’, which is better suited to this kind of rescue as it can skip ‘slow’ areas of the disk, retry bad blocks multiple times from different directions, and will pick up where it left off if interrupted. I can’t compress the output from it though, since it’s no longer just a stream of bytes, so I had to completely move everything else off of the external drive I’m writing to so I could write to it at the raw partition level instead. It’s making good progress though, with an overall transfer rate of 3 MB/s after a day.

What’s becoming obvious from the log is that disk errors are occurring scattered all over the place, not just in one spot, and the concern here is that this may wind up leaving ‘holes’ in random files. Since I’m primarily trying to recover a collection of ripped DVDs, I’d prefer not to find out that a movie is corrupt when I’m halfway through watching it!

Fortunately, as documented here, it looks like I can use the list of bad blocks found by ‘ddrescue’ and feed it to e2fsck when it comes time to try to repair the rescued image, and it should be able to figure out which files are damaged.

Oh goodie. A couple weeks ago, the backup drive for my gaming PC failed, which isn’t too big a deal; I’d been meaning to upgrade it to a larger one anyway.

Then this morning, I noticed that my Linux box’s HD activity light was stuck on. Turned out that my main storage drive was failing, and it was stuck on accessing the hard drive. I have a separate backup drive for this system, but the failure started while it was mid-backup, so the backup is incomplete. And then, for the final kick in the pants, I noticed that there are also bad blocks on the backup drive, so who knows how that’ll affect trying to restore from it.

Fortunately I’m paranoid and I have a second backup drive that I rotate with this one and store offsite, so that backup drive should still be fully intact. Unfortunately, I am also lazy, and it’s been a while since I swapped drives, so it’s going to be an old backup. This is primarily just a ‘media’ drive and all my important personal stuff is on a different drive and largely duplicated on my laptop, at least. There are also a lot of files on the failing drive that weren’t included in the backup since they took up too much space and could be re-obtained if needed. Those are primarily DVD rips, but it’s going to be super-annoying if I have to dig my DVDs out of the closet and re-rip them.

I’d still like to recover as much as I can, though. So, recovering from this is going to be an exercise in trying to reconcile a) an intact but fairly old backup, b) a recent but incomplete and potentially corrupt backup, and c) what I can scrape off the failing drive before it completely gives out. Fun.

I started copying some files off the failing drive and got a bunch of stuff including my music collection and non-Windows gaming files (mainly a lot of Minecraft servers and worlds, and emulation stuff), but then an error hit and I can no longer access the root directory of that partition, so I can’t even get at any of the intact files anymore. So, now I’m ‘dd’ing and compressing the raw partition of the failing drive onto a spare external drive so I can go at it with more in-depth tools later on. Though at its current rate of 364 kB/s, it’ll only take another, uh, 127 days or so. (Hopefully it’ll get past the damaged part and speed up soon.)

Except I don’t really have anywhere I can un-‘dd’ it to right now, since it’s a rather large drive. Guess it’s time to start buying large hard drives in bulk…

When I go to update my phone’s apps and 18 out of 24 fail to update because I’m still on iOS 7, I guess it might finally be time for a new phone…

Also speaking of dead hard drives, now’s as good a chance as any to sort through some old drives left over from old systems, since I’ve now got a USB adapter (kind of like this one) that lets me hook up drives without having to open up the case.

Some of these are dead or dying drives that I didn’t want to immediately throw out because I wanted a chance to erase them first. My secret crab bisque recipe will never be yours, dumpster-divers! The drive I just replaced in this server actually crapped out completely while I was 3/4 of the way through erasing it, so it’s now relegated entirely to the role of ‘squeaky noise-maker’.

Some of them still work perfectly fine, but I’m not sure what to do with them yet. One is a 1TB drive, which is still a good amount of space, but it won’t fit into any of my current systems, and it’s too small to act as a backup drive to them. But it’s too (physically) big to use as an upgrade in my PS3. Probably the best thing to do would be to stick it in a USB enclosure and use it as a General Purpose Portable Drive ™. But then I’ve also got a 500GB drive left over. And a 320GB drive. And a 160GB…

And some of them are ooooold. In one box I found a dusty WD Caviar drive that’s 13 years old and is louder than every other hard drive I have put together, but amazingly enough it still works. But it only holds 30GB; what could I even use it for? And I’m trying to make my workspace quieter, not louder. But then my natural reluctance to throw out anything that might be useful in some theoretical way kicks in…

Hard drive failures suck. A couple weeks ago, I noticed this server box was starting to make some odd squeaking noises, and I was reasonably sure that there weren’t any mice trapped inside. I checked the SMART stats, and sure enough, bad sectors had started appearing on one of the drives. In the days after that, the number kept steadily rising, and just trying to access some directories started spitting out I/O errors.

The data was safe, as it’s pretty robustly backed up, but it presented me with a bit of a dilemma: do I just replace the drive, or do I consider replacing the whole system, as it’s now around five years old?

In the end, I wound up kind-of upgrading it. I kept the system, but upgraded it to 4GB of memory cannibalized from my recently replaced gaming box, and replaced both of the hard drives in it instead of just the failing one. Instead of two 1TB drives, it now has one 240GB SSD, and one 4TB hard drive. The SSD makes for a nice and snappy boot and home drive, but it obviously can’t hold much, so any large collections of data will go on the 4TB drive, with symlinks all over the place as needed. The CPU’s still adequate for its relatively light needs, so with these upgrades, this system should hopefully last for quite a while longer.

Now how long is it going to take to restore all those files…

I’m still having a bit of trouble with the new system, and primarily with the video card. The first problem is that I have to reboot every day or the drivers start ‘wigging out’, constantly issuing a stream of timeout and recovery errors and making the system unusable. This apparently happens after the system has been running for 36 hours, and there are a lot of complaints about it on NVIDIA’s forum and moderators have posted that it’s under investigation, so hopefully a fix isn’t too far away. At least rebooting is super-fast, though I’d be a lot more annoyed if I was currently doing development on this system and had to close and reopen everything every day.

The second problem was that I started getting spontaneous reboots in the middle of games. After checking the temperatures and seeing them get way too high, I cracked open the case and saw that a bundle of loose cables had drooped down into the video card fans, stopping one of them from spinning. There wasn’t really anything I could easily tie them to, but they were going underneath another cable that’s much stiffer, so it seemed like it would suffice to have them rest on top of it instead. Unfortunately that meant rerouting that other cable, which is the USB3 front panel cable that caused me grief before, so I had to pull the 3.5″ drive bay out again to put the USB3 plug back in.

And then in the course of doing that, I somehow yanked out two of those looser cables from the motherboard. They were just the front panel power LED pins, but if I didn’t do anything about them, then those would droop down into the video card fans. Unfortunately those pins are in a very narrow little region between the power supply and the video card, too narrow to reach with the kind of finger-pinching needed to guide the plugs properly. The one time in my life I actually specifically needed needle-nose pliers, and I discover I don’t have any…

So, I had to move the video card out of the way to give me more room to work in, and that meant moving some other cables that ran past the video card, and removing the ‘cage’ on the back of the case to get at some screws, and… In the end I wound up damn near disassembling and reassembling the whole system, just to put one cable on top of another. There’s probably a lesson here about doing proper cabling up front, but who knows if I’ll remember it by the next time I have to assemble a system…

All of my parts finally arrived, so I spent most of last weekend assembling the new PC and the last week reinstalling stuff and getting it set back up again. It is, of course, much faster than my old one, but the main thing I immediately noticed is just how much quieter it is compared to the old system, even when just sitting there idle. When I first powered it on, I was instantly concerned that I’d screwed something up because I couldn’t hear anything. Maybe I can hear myself think now…

As usual, the main hassle was cabling, especially since I went with a smaller mATX case this time. I always seem to screw up the order of something; I attached the USB3 header early on, but then it turned out it was blocking me from moving the video card into place, so I had to unplug it. But then with the card in place, I could no longer reach down to where the USB3 jack was, so I had to remove the 3.5″ drive bay for a second time, plug it in, put the drive bay back, reattach the drives… The SATA plug on the SSD also worries me a bit; there’s nothing for the clip on the plug to latch on to, so the cable sits rather loosely. Oh well, as long as nothing disturbs it. Ideally I’d like not to have to crack this thing open again for another 3-4 years.

I wound up putting Windows 8 on it anyway. I’m still not all that big a fan of the Metro side of things, but once you’ve got it all set up you can pretty much stay in the Desktop mode 99% of the time, and the 8.1 update due out soon is supposed to fix a lot of 8’s problems. Unfortunately my current version of Acronis True Image didn’t work on Win8, despite only being one version behind the current one, so they basically held my backup data hostage until I bought an upgrade…

With the SSD, Win8’s quick boot support, and the UEFI BIOS together, it only takes about 20 seconds to go from clicking ‘Restart’ to the login screen coming back up again. I’ve gone to reboot, turned to do something on the laptop, and turned back to see the login screen and suddenly become uncertain whether it really did reboot or if I accidentally just logged out instead.

The one problem I’m having right now though is that after the system’s been running for a while, the frame rate becomes erratic in games and it starts freezing for a couple seconds. A lot of people seem to be having troubles like these with the latest NVIDIA drivers, so I’ve reverted to the ones that ASUS themselves provide, so we’ll see if that helps. As always, driver issues are the bane of everyone’s existence.

Now I just have to get around to actually playing some games…

Remember that new router I mentioned below? I only just realized that it’s been running continuously since I first set it up, without a single reboot:

I’ve never had any trouble with its wireless reliability or firewalling or NAT, either. It’s nice to finally have a router that just does its job quietly and reliably, a far cry from the previous Linksys that would freeze up and need to be power cycled every week or two and would occasionally drop the wireless.

Well, Haswell and the new Nvidia GPUs are out, so it’s time to start putting together a new system. It’s still too early for prebuilt ones to be available, and I wasn’t happy with the ones I looked at before since they all involved compromising on something due to limited choices, so I guess I’m rolling my own again. So far, the parts I’m looking at are:

CPU: Intel i5-4670. Of the new batch of Haswell chips, this seems to hit the sweet spot for price and performance. There are slightly faster ones, but they start to get much more expensive very fast after this point.

Motherboard: Asus Z87M-Plus uATX. There seems to be a lot less differentiation among motherboards nowadays, perhaps due to more stuff being moved right onto the CPU die. It’s mainly about size and number of USB and PCIe slots, and I don’t really need very many of them. I’m not going SLI, so I don’t need more than 1 PCIe 16x slot, so this board satisfies everything even though it’s on the cheaper end.

Case: SilverStone PS07B uATX. I could reuse my current Antec P180, but it’s a behemoth, and I’d like to try something a bit more compact. I don’t need four external 5 1/4″ bays! It’s my first time looking at uATX cases, so I’m slightly worried about things fitting and cooling, but as far as I can tell from measurements and other peoples’ reports, it should all work.

Video: Asus GTX 770 DirectCUII. Not much differentiation among video cards either, but this new generation should last me a good while and this card is supposed to be particularly cool and quiet. I’m slightly concerned that 2GB might not be enough if games ported from next-gen consoles need more and more texture memory, but it’s easy enough to replace the card down the road if that happens.

Storage: SanDisk Extreme 480GB SSD and Seagate Barracuda 3TB. It’s about time I got an SSD, and this one’s been recommended for being reliable and reasonably performant. It won’t be big enough by itself, so there’s ye olde regular hard drive in there too.

Power Supply: SeaSonic X650 Gold. Not a particularly glamorous component, but I’m going for a modular one to cut down on the mess of cables inside, especially with a smaller case.

I’m also trying to choose quieter parts this time. I didn’t really focus on it last time, and the fans are fairly noticeable even at idle.