Goodbye WinFS, We Hardly Knew Ye

One of the uber-cool features of Vista was supposed to be WinFS, the filesystem that was actually a database. It's not a new concept - Oracle wanted to create their own OS based on the concept (but since stuck with Linux crawling with Oracle contributions), and IBM has been doing that with the AS/400 since horses were brought to the Americas. Lotus Notes keeps most of it's file info in database tables, formatted in largely unconstrained recordsets. But those implementations were more server-centric, used to accelerate huge tables of data. WinFS was supposed to bring order to the average consumer - and finally catalog the bazillions of MP3 files we all have.

It's a good idea when you think about it... and something that Google Desktop is already capitalizing on. Now that hard drives have reached astronomical sizes it's getting harder and harder to find that Christmas card list you built two years ago. Google Desktop extracts a basic summary out of each file - the meta-data out of the data - and keeps that in a big honkin' index. That way when you search for a file, it can do an extremely fast lookup in the index instead of sequentially looking at each individual file on the drive.

WinFS takes it one step further... all "files" exist purely as records in a database schema. This can be pretty liberating when you stop to think about it - no longer do you care about what directory a file is in. You don't even need directories anymore. You can instead organize files and information any way you want as "views," or collections of records.

For example, say you have a lecture your grandmother gave on neutrino collisions, saved as an MP3. In the old filesystem model you might have a folder for MP3's, a folder for family collections and a folder for general relativity. Which one do you put it in? With WinFS you don't care - you store the document once, then you can have different views called "MP3's" that show you all your MP3 files, another view called "Family Collections" that have family members in the ID3 genre, and another view called "General Relativity" that sorts off the file's topic. The file is in one physical place on disk, but is listed under three separate views.

It's easy to see how this can be utilized with multimedia files and office documents... your music can have a ton of different, virtual "folders" by album, title, artist and genre, but the file is still only in one place. Using the old file system model, you'd have to copy the file over and over again.

There's one problem tho: this method requires a lot of computing beef. You're basically running a massive, 750 GB SQL Server 2003 database on a consumer OS. Not only does this eat memory and CPU cycles like candy, but you also come up with a whole mess of file integrity issues. BeOS tried to do this before, but there's a slew of problems running a filesystem-as-a-database-in-userspace. First off... if you lose power in the middle of a write, how do you recover? Was the data lost in the filesystem, in the database, or when being committed to disk? What about deadlocks? If you have 5000 I/O requests hitting the filesystem at one time, to you lock the database and handle each request sequentially? Do you do some sort of record-level locking? How to you stop deadlock states from a huge queue of I/O, especially with writing out to temporary files or performing huge non-sequential reads?

As far as gaming goes, WinFS would likely be a nightmare. Not only would your resources become sparse, but doing large sequential reads or small file lookups would suddenly become a drain. No longer can you just jump to an .ogg and start streaming it... you have to basically change an open file request to execute an SQL statement which gets the file as a record and then returns the file handle. Your filesystem with quick search time has just become an unnecessary bottleneck.

In the end, you have to start building a journalled filesystem on top of your journalled file system. Maybe create a separate partition that's database-centric, another partition that's node (or allocation table)-centric. Then Occam's razor comes by and stabs you repeatedly, harshing you for building some weird golem of code.

You know what ultimately can get you 90% of the way with 10% of the complexity? Symbolic links, a good journalling file system (like EXT3 or ReiserFS) and a good desktop search tool. Which is where we're at today.

It seems as if Microsoft has finally come to the realization that a filesystem based on SQL Server was just unwieldly. They first receded WinFS' scope by just bolting it on to NTFS, tantamount to the weird file system amalgam. Now it seems they've dropped WinFS altogether, perhaps realizing that the BeOS developers had a point after all.

We should probably all be happy about that. It's taken Linux devs log enough to reverse-engineer NTFS, after all.