As the old saying goes, you learn something new everyday. But sometimes you learn something and forget something. Yesterday was one of those days. While writing up VoodooPad 4, I thought I’d open up some of my VoodooPad files to check out the new features.
I mostly use VP files to keep track of notes and ideas for fiction writing that I do in my spare time; it’s great for building connections between different concepts and characters. And as I used to maintain multiple computers, I’d gotten into the habit of using my iDisk for file storage so I could access said files no matter where I was. As an added benefit, I figured that would give me multiple synchronized copies of files in different locations to help safeguard against data loss.
I’d also assumed that Apple itself backed up the iDisk files, as most large server installations do. This wasn’t blind assumption either; back in 2005, Apple recovered files that had disappeared from my iDisk:
We isolated an issue with your iDisk, which is now fixed and we recovered the missing items for your iDisk Documents folder. To access the recovered data, please connect to your iDisk and navigate to the Recovered Items folder that is now present in your iDisk Documents folder. Please note there may be duplicate copies of some of the files.
Fast-forward to yesterday. I open up the VoodooPad files on my iDisk only to find they won’t open. And when I check them out in the Finder, I discover that’s because they register as 0KB in size.
I should mention that I usually work directly off those files on my iDisk: it appears in the Finder like any other mountable volume, so why shouldn’t it be treated as such? Now, I had encountered sync errors from time to time while working on those files, but if you’ve used an iDisk for any reasonable amount of time, you know that sync errors come up a fair amount. At one point, when presented with the option to choose between two copies of one of my VP files, one from .Mac (as it was at the time) and one from my Mac, I made sure to keep both, just in case.
That’s why it was all the more surprising to end up with suddenly empty files yesterday. So I headed over to MobileMe Support, figuring I would fire off an email and see if they could help me out. Now, despite the many problems MobileMe users have run into, I’ve only had to contact support a couple times in the many years I’ve been a subscriber. So y next surprise of the day was that MobileMe support email is a thing of the past; your only option now is to chat online with a support rep.
With no other option, I jumped into a chat with MobileMe rep Cody H. After explaining my problem, Cody offered this diagnosis:
Cody H.: Okay, I do not recommend running a database, such as Voodoopad, on your MobileMe iDisk. The iDisk is designed to hold data that is static, which means photos, movies, and other files that users can view and download.
Cody H.: The iDisk is not designed to hold files that users actively manipulate, such as a database. You will find that using a database on a local hard disk is much faster than using the database on the iDisk.
Dan Moren: What’s the difference between that and, say, a text file that you edit on the disk?
Cody H.: Manipulating files while they reside on the iDisk, especially when using iDisk Sync could lead to unusable files if computer tries to sync while you are making changes to that file.
That, as I said to Cody, was news to me. I’d been editing files on the iDisk for years, and while I’m not a filesystem engineer, you could say that I’m reasonably computer savvy. Maybe I should have known better, but the past is the past.
That being surprise number three, we move on to surprise number four—the biggie:
Cody H.: I do apologize that you have lost data on your iDisk due to this issue. Please remember that to protect your privacy, Apple does not maintain any data that MobileMe members have manually deleted from the MobileMe servers. This includes files deleted, whether normally or due to corruption, from the iDisk and messages deleted from MobileMe Mail. Preserving data is the responsibility of each MobileMe member.
Cody H.: Please see the “Limitation of Liability” section of the MobileMe Terms of Service.
Ouch. Not good. Well. I’m not a complete idiot, so I do use Time Machine to backup my laptop, which is my main computer. Cody had earlier intimated that maybe that could help.
Er. Surprise number five.
Dan Moren: In terms of trying to recover iDisk files from a Time Machine backup, how would that be accomplished?
Cody H.: Unfortunately Time Machine may not backup the local iDisk, but any files saved to your computer should be backed up. You can contact AppleCare for further information on how to restore information using Time Machine.
Now, this dislodged another piece of information in my brain from last year when Leopard rolled out. In previous versions of OS X, your iDisk was stored as a disk image (DMG) in your Library. This lead to the problem of the miraculously inflating iDisk. Apple sort of compensated for that issue in Leopard by switching to a “sparsebundle” format that can dynamically shift its size.
The other problem with the iDisk image is backing up with Time Machine. Normally, TM checks to see if a file has been altered, and if it has, backs it up. But if you change one file on your iDisk, it can change the entire disk image, which could lead you to backing up a multi-gigabyte file every time you change a single file.
But the sparsebundle format should address that too: Time Machine can look at the file as 8MB-sized “bands” and back up only those bands that have changed. The problem there is that it renders Time Machine useless if you want to go back and recover an individual file. All you can recover is the most recent entire image of your iDisk. If you want to recover an earlier version of any single file, you’re out of luck unless you’ve engaged in some elaborate trickery.
So despite feeling secure in the knowledge that I had multiple copies of these valuable files, the truth ended up being that I had multiple copies of one corrupt version of the same file. Which, as it turns out, is actually worse than useless. It’s more like “maniacal rage-inducing.”
That, then, is my tragic tale of woe and…tragedy. If you take anything away from it, I hope that it is this: don’t rely on your iDisk for backups of critical data. Make sure that you have a copy on a local disk at all times, lest you end up crushed and bereft, like me. I imagine the chances that you’ll be in this exact same scenario are probably slim: I seem to have an encountered a perfect storm of data loss, certainly at least partially of my own making.
I don’t know if there’re things Apple could do to improve the situation so that other people don’t fall through the cracks in the system as I did. From a technical standpoint, I’d sure like to see them have a more robust system for handling working off files on the iDisk, as well as being a little more upfront about what the limitations of the system are.
From my perspective, this situation has encouraged me to take ownership of my data and do an extensive overhaul of the many, many documents I have scattered around myriad computers and hard drives. My goal is to never run into this problem ever again. Digital data has become an increasingly integral part of our lives, and losing it can be as frustrating as losing physical possessions. Take it from me.
Could I theoretically create a disk image inside iDisk, store everything in that image, and then use SuperDuper for routine back ups of that image to a Drobo or somesuch?
I've just been rendered utterly paranoid by this :)
Never, never, never, work directly off of a network volume. Yes, it may work "most" of the time, but you are simply asking for trouble. Using a network volume on a local LAN is dangerous enough. Working directly off of volume on the Internet is pure foolish. (I've been designing computer networks for almost 20 years, so I can say these things with some degree of confidence).
I might be off here but isn't ones contact/address book a database file and mine gets synced often with no issue? So I guess my question is how is that different from your files? Thanks for posting this I will be re-thinking my work flow a bit.
Thanks for your cautionary tale. I actually have the same setup for sharing writing ideas amongst my different computers, Voodoo Pad is my best bud. I occasionally backup everything on my idisk to a separate hard drive, but I'd hate to lose a month or so worth of written notes because of this. I think I'll look at upgrading to Voodoo Pad 4 and setting up a webdav sever to keep my notes up to date instead.
Thanks for sharing.
OUCH! That's gotta hurt!
My iDisk is empty because of other bad news stories I have read about it. I use only the email functions of MobileMe.
You need "Dropbox". It is a new startup product that stores all your files locally AND up on the cloud. This allows you to have automated backups (a) on Time Machine, (b) up on the Cloud, and (c) on as many other computers that you add to it. Pretty darn impossible to have a disaster like you've experienced. Apple should buy this company!
Why can't you just restore your entire iDisk via TM to the way it was, say 1 month ago? It should be able to restore all of the changed bands back to a specific point in time unless you've been re-backing up the busted version for so long it's fallen off the tail end...
I'm sorry for your data woes!
I'm also sorry to hear about Apple not backing up iDisk. I guess I understand why, but I was really hoping I could use my iDisk to backup all of my data while on the road, particularly when crossing the border (US/Can) given the (new?) ability of US customs to just seize your laptop until they've had a chance to look at its contents over the course of possibly weeks!
I thought if I loaded my data onto iDisk, then even if the worst happened, I could rent a replacement MacBook until I got my original back. I may have to re-think that plan now...
Do you have multiple computers? If so, I would highly recommend switching to Dropbox.That way the copy of your files (including VP files) are stored locally on each machine, but instantly synced each time there's a change. It's PFM. Works like a charm. I've moved all my "in process" files as well as my VP file to my Dropbox.
getdropbox.com
For my money, Drop Box is what iDisk should be... but isn't.
Drop Box is a regular folder on your hard drive, it just gets backed up to Amazon's S3 service.
It *will* be backed up via Time Machine, just as a regular folder would.
And it syncs much faster than iDisk does.
There's also some kind of "revision keeping" done on the website, but I don't know exactly how it works.
Up to 2GB is free, 50GB is $100/yr. I keep just my "active" files on there, so 2GB is plenty of space.
(Anything not active gets put on my Drobo.)
I wish iDisk worked better, but it doesn't, and I don't expect it will any time soon.
https://www.getdropbox.com/
(No connection, just a happy user.)
ps - iDisk is a WebDav, and things like Yojimbo or anything that uses SQLite refuses to go on a WebDAV, if I recall correctly.
Apple should fix Time Machine to backup data on your iDisk exactly like it does on other disks. I specifically configured Time Machine to not backup the iDisk dmg, maybe I should reconsider that.
I'm confused. The iDisk IS a local disk. Yes, it's a sparsbundle file, but it's local. So I'm not clear what the MMe rep was telling you.
If you can't work off of files on the local disk (database or not), then what the heck are you supposed to do with them?
I work off my local iDisk all the time. If sync occsionally tells me there's a conflict, I take the one with the latest date (which unsurprisingly is the one I'm working on).
As for TM, I knew when I moved to iDisk that if I lose a file I'll have to restore the whole disk just to get it. That sucks, but then so does losing a file and having to restore it in the first place.
Try a backup product such as memeo (think it's the correct spelling) which allows you to create backup programs/routines, that will incrementally update any file that has been changed, amended or replaced. It also allows you to keep multiple copies of the files so you have an older backup just in case of corruption. It comes as standard with Western Digital external drives. They also make a remote version of the software to allow remote backup and retreival of data. Will disagree with Neil in regards to working off network shares with live documents or enviroments. Most certainly with multiple user databases, you'll always run off of a dedicated database server, and it will be backed up incrementally to another machine. You'd never keep the database locally as the drive may die or corrupt on that machine, and remote backup is problematic. The same goes for user profiles and home directories. However, if you're running off a home network, keeping it local and backing up is most likely the general option, but you can run a good disk array of some description as a local network and work from that. Even then, you should back it up. With things like mobile me, cloud computing, you're always in the hands of those running it, and quite frankly, should never rely on it for anything important, and never work directly off it. Better to always carry a small hard drive with you for backup and when required, working from. I use mac's and pc, but I wished apple users would go look at the big world around them, and realise there is an abundance of software that will run on macs, that does a much better job than the apple equivalent. Give memeo a try, see what you make of it. I have a feeling that it will restore your faith in backup's, and knowing that even if a new version of a file is corrupted, you still have a version that will work.
@phil, the difference is that MobileMe understands the Address Book database, it is built to sync with it. The Voodoo Pad database, on the other hand, is just a file to MobileMe and Voodoo Pad is not using the standard open/edit in memory/save paradigm.
@Kim, customs agents cannot seize your computer, they can copy the image to read at their leisure (you can refuse, but they can refuse you entry). Keeping your files on your iDisk (with syncing turned off) is a good way to keep your data safe from prying eyes. One would hope that the election of a constitutional law professor would spell the end of this violation of the Bill of Rights.
Everyone, iDisk is safe and backed up. However, when you delete a file, any backups are also removed by policy. There was a time when people complained that Google didn't do this.
Dave-O, you're incorrect regarding the seizure of equipment by customs officers.
They can if they choose, seize and hold any electronic device capable of holding information including laptops, cell phones, pda's and even MP3/MP4 players. They may simply choose to copy the disk/contents over, but are not obliged or required to do this in lieu of returning the physical device. This also applies to phyisical printed documents as well.
You can get this information from the US customs website in a downloadable file on Procedures of Search Authority. It's actually been in place for a number of years, they've just made a point recently of using it more often. Another way of endearing the masses to feel free to travel to the US.
Yes, I totally agree and hope that the newly elected president, will reverse the uneeded scrutiny and prying into anything that is seen fit by customs and federal agencies, without just cause, or due to their boredom needing to be relieved. That would be a glorious change, and many others are sure to follow.