And So It Begins - Vector Processing Standards Battle!

I'm not sure why I'm such a fan of this topic... maybe just because I enjoy watching the inevitable march towards entirely new CPU architectures.

Nvidia just released something that's been on Apple's wishlist for a while: OpenCL 1.0. Finally a "standard", royalty-free specification for developing applications to leverage vector processing units currently available on GPUs. While the processors on such high-end video cards aren't geared towards general computing per se, they absolutely blaze through certain workloads - especially those that work through sequential processing pipelines.

Microsoft's competing specification for some reason is available for DirectX 11 only, which makes absolutely no blimmin' sense to me. This basically means that your specification is limited to Vista... which rather defeats the concept behind a "standard." Not only do you get vendor lock-in, you get implementation lock-in. Sweet.

Can you imagine what this might do for Nvidia tho? Picture it now: tons of cheap commodity motherboards laid end-to-end, each with six PCI-E slots filled to the brim with Nvidia cards and running OpenCL apps on a stripped-down Linux distro. Supercomputing clusters for cheap.

Although I imagine the electricity bill might suffer.

Someone to Give Me the Time

It's been really interesting to see the responses from Blitz, Fly Object Space and GigaSpaces concerning state management as well as Newton and Rio concerning service discovery. I'm definitely learning as I go, but the good thing is that it seems like there are many in the community eager to help.

Now I'm working on another issue with enterprise service development - scheduled services. There are some services out there who may want a have an event fire in 1000 milliseconds, or five minutes, or an hour, or somewhere in between. This would appear to be an easy thing to solve at first blush - until you consider volume, quality of service and scalability. It's a steep drop into complexity at that point.

Here's the thing: you could easily just do a scheduled executor in J2SE, but once your VM dies then your pending events die too. You could submit a scheduled job to something like clustered Quartz instances, but then you must have a reliable back-end database to write to (no native replication). You could use something like Moab Cluster Suite, but it seems to live outside the muuuuuuuuch more simple realm of event scheduling.

So let's think outside the box and use some replicated object store that isn't necessarily meant for scheduling. How about we slap a time to live (TTL) on a JMS message, throw it on a queue and wait for it to hit the dead letter queue? That might work at times, but TTLs are really intended for quality of service and not for scheduled events. Unless you have a consumer attached to the former queue constantly polling for messages you're not guaranteed to land in the latter dead letter queue.

How about using Camel's Delayer Enterprise Integration Pattern? Nope - that's just a Thread.sleep on the local VM. Doesn't do you much good once the VM dies. How about a delayed message using JBoss Messaging? I've heard tell that it exists, but I can't find much reference to it in the documentation.

This isn't a new problem - there's even JSR 236 that is intended to address this problem. But it's been hanging around since 2004 with very little activity of note, so I doubt it's going to have much hope of working by Monday.

Until JSR 236 is addressed I'll likely have to just find a way to deal with this on my own. Maybe create a JobStore for Quartz that's backed by a JMS topic? Or just suck it up and build a clustered Quartz instance with a fault-tolerant database?

Gah. Sticky wicket.

Chipped Chrome

VIA recently announced that they have opened their reference documentation for their GPUs and are even now actively working with the openChrome project. For me, however, it's too little too late.

I've finally grown tired of even trying to get accelerated video working on my VIA-based MythTV box. XvMC support is simply non-existent and accelerated anything just doesn't work. With standard 480i broadcast TV I had no problem being CPU-bound for MPEG2 decoding, but it just doesn't fly with 720p and a pcHDTV card. I throw in the towel.

I'm looking at what Shuttle has to offer instead with either an Intel, NVIDIA or AMD platform. It appears that the graphic chipset choices break down into either GeForce 8, Intel GMA X4500HD, Intel GMA 3100 or GeForce 7050PV. The best NVIDIA choice appears to be the 7050PV, as it seems to enjoy known XvMC acceleration on MythTV's feature matrix and some have even reported getting it to work in a dual-head environment. The Intel cards should theoretically offer great performance for the power and enjoy good Linux driver support due to Intel's great contributions to X.Org, however Myth's wiki seems to know painfully little about the Intel GMA's. As far as XvMC support goes, Intel's chipsets don't seem to have a great track record.

So GeForce 7050 seems to be the most sane choice for those who are tired of fisticuffs with xorg.conf. But wait! We have audio to worry about too. If I'm going with a Shuttle GeForce 7050, then it looks like I'm going with a Realtek ALC888DD. Here again, MythTV notes that yet another Intel chipset is a true pain to work with. Still, I noticed that the MythTV hardware database notes there was at least one other Shuttle user with a similar setup that was able to get things to work.

Weirdly enough, both of the GeForce 7050 Shuttle boxes I found were AMD boxes. Go figure...

Empty Spaces

Yup, I'm still trying to work my way around Jini. This time it's JavaSpaces.

Apache River hasn't gone much farther from the last time I looked at it, but I liked the bare-bones reference implementation aspect. GigaSpaces seems a bit thick for my tastes and seems to be tightly coupled with their application server. I thought Blitz JavaSpaces might be a better fit, especially if I could use their fault tolerant edition.

I was able to get Blitz up and running then configured it to do unicast discovery to a pre-existing Jini registrar without a problem. I was having problems getting my client to connect in its security context, so I decided to dig a little deeper. As I did I also kept an eye towards fault-tolerance, but found that branch seemingly suspended. I later found a post from the author indicating he didn't really see a good motivation for moving forward with his fault-tolerance work:

In my spare moments I've been doing a re-implementation but the fact of the matter is that it's not a trivial problem to solve (though I believe I do have a solution). And here's the rub, this work doesn't pay the bills which means that it's going to take a long time to implement because I have to do a day's work first. For those who don't know, most of Blitz has been written during time between periods of employment - not over weekends and evenings as you might expect.
This presents me with a problem - users seem to want this feature but I'm struggling to see doing this as a good thing. Here's some of my reasons:

1. I'd be building a significant feature which will, judging by demand, make a lot of money for those who use it but zilch for me.
   
2. Not only do I earn nothing from this venture but I have to earn a significant amount of cash just to allow me time to develop the feature. Basically, I'd be financing everybody else's money making ventures.
   
3. One of GigaSpaces key value adds is the clustering/replication feature - they are fully commercial and need to earn a crust plus they're one of only a few credible companies that can provide corporate grade support for JavaSpaces. Were I to do this work for Blitz I'd maybe be damaging the market I've been helping to create.
   

Right now, I feel like the price of this piece of work is too high for me personally and for others in the commercial JINI world (and like it or not they are an important element in any future success for JINI). I can see why Blitz users might want this feature - they can avoid paying Gigaspaces a license for starters.

So... it seems like the development of an enterprise ready Blitz isn't in the cards. Casually strolling through Wikipedia's definition of a Tuple space brought up Fly Object Space, a tuple space that is not a JavaSpace implementation. While it doesn't fit into the Jini realm I know and love, it is a more minimalistic implementation of an object space that fits my desire of something smaller and to-the-point. It doesn't appear to support replication or fail-over on the non-commercial level, but I'm checking to see if there are plans to support it on a commercial level.

It's tough. I need an object space that has a minimalistic implementation, has a small footprint and can at least run active/passive for fault tolerance. Maybe I might have to dust off my old Terracotta instance and try out SemiSpace.

EDIT: Be sure and see Nati Shalom's comments following this post.

Suspend to RAM - Actually Works? Really?

Let it be known that today I was actually able to get my Linux laptop, using the proprietary NVIDIA drivers no less, to suspend to RAM. I'll give you a moment to pick yourself up off the floor.

Using a less-than-stock build of OpenSUSE 11.0, KDE 4.1.3, the latest NVIDIA drivers and a Dell Precision M6300 I was able to successfully both SUSPEND TO and RESUME FROM RAM. I crap you not. I even took a picture.

Wow. That's... like... historic.

Dark Power Adjusting Laptop Brightness

One reason why I was really looking forward to KDE4 was the level of abstraction it offered from services and hardware while offering a lot of unification as far as end-user interaction and desktop integration.

One fantastic example of this has become PowerDevil, which was introduced around the KDE 4.1 time but is now standard in KDE 4.2. Its functionality is based upon Solid, KDE4's hardware abstraction layer (which also abstracts audio & bluetooth).

PowerDevil runs as a fully-fledged KDE4 service, meaning it doesn't need to be some awkward "TSR" or persistent applet in your system tray. That also means that it runs much leaner than kpowersave, which largely monitored events and then attempted to send system calls along to the appropriate background resources.

The PowerDevil coders may talk down the control panel UI, but it works rather well. And while it doesn't have a Plasmoid (applet) yet or much in the way of UI, the beauty of KDE4 means it doesn't immediately need to. Since PowerDevil is well integrated into the KDE4 desktop, KRunner displays all the immediate options you need when you type "power profile" into the runner dialog. Very nice.

This kind of desktop integration is exactly what will make KDE4 a success in the long run, and it's great to see projects like PowerDevil emerge that take advantage of what KDE4, Solid and Plasma have to offer.

We Thank Thee, O Great NVIDIA...

All the way back in July I mentioned how slow and unstable KDE 4 is with NVIDIA, and why it's not their fault. Through August/September/October I've been living in a time warp and finally emerged out the other side... and behold! NVIDIA has graced our presence with a new driver release that has several KDE 4 compositing & Plasma fixes!!!

I clapped, I was so happy. Take pity on me.

Everything Breaks at Once

It seems that when one thing breaks, everything breaks. Ceiling starts leaking, TV loses convergence, power supply on computer goes out, server's UPS battery dies, the lithium battery pack for a portable DVD player starts to swell and explode, iPod ends up in the toilet, phone LCD cracks. It's partly because I'm over-tired and accidentally breaking crap, but it also appears that entropy has hit my living room en force.

I re-caulked the upstairs shower, replaced the TV, dug a new power supply out of the closet and got a new iPod. However - I've been busy with other stuff, so I haven't swapped the PSU in the workstation yet and I haven't completely re-installed all the home theater stuff, including MythTV box. And since I've already reduced every other electrical doodad in my house to its bare components, I might as well strip apart the Myth box and update it as well.

I have a pcHDTV card, now sitting dormant in the aforementioned workstation with a fried PSU. I'm thinking of ripping it out of the workstation and placing it into the MythTV Mini-ITX box. There's only room for one PCI card... so I'll have to remove the old Hauppauge cable TV tuner. Yet the Hauppauge WinTV-PVR-150 is also the IrDA receiver for my remote control - so replacing the tuner card would mean losing the DVR's infrared receiver.

I started looking around for alternatives and found a whole slew of Linux-ready solutions. An easy solution that I didn't think of at first was using a generic HID Windows Media Center remote, since it just pretends to be a USB keyboard. Re-map the keycodes and you're pretty much ready to go... assuming you're willing to re-map a universal remote and the Myth keyboard bindings to accommodate the case-sensitivity.

One could also use the receiver from a SnapStream Firefly Mini - they appear to be well supported by MythTV and would be a nice out-of-the-box alternative, given that everything is ready to go. Hopefully the reports of limited range wouldn't be a problem given my spartan living room setup.

Of course, like every self-respecting home theater geek, I already have a programmable universal remote that controls my whole setup. I don't need the additional remote that the previous two solutions bring, so it seems unnecessary. The first thing I would do is throw away the packaged remote, re-program my Harmony 670 and just use the new IR receiver. So it probably makes sense to just buy the receiver if at all possible.

There is a fairly flexible Linux-happy USB infrared receiver/transmitter that is supported by recent versions of lirc and MythTV, but it might be more hardware than I need.

IguanaWorks makes some powerful receivers and transmitters as well, in both serial and USB flavors. Considering serial access is considered the "legit" way for IrDA access, it seems like the cleanest way to go. They are evidently fairly powerful and receptive, but I would still need to get an extension USB cable to rope around to the front of the MythTV box.

Another cottage shop sells a RS232 LIRC receiver built into a DB9 backshell. It's much cheaper and much lower profile and designed to work with lirc natively. The most simple solution to the problem it seems.

I'll probably wander around the local electronics shops looking for a Firefly Mini or MCE remote. If I can't find one this week I might put in an order for a serial port receiver. Never figured I'd use a 9-pin serial port again, that's for sure.

Massive Server Farms Can Equal Massive Failure

A quick trip over to the Amazon Web Service Status Page reveals that massive server resources don't exactly equate to massive uptime numbers. The S3 storage cloud has been down pretty much all day due to "an issue with the communication between several Amazon SQS components." This has affected both the EU and the US, causing some big headaches.

I'm not all that disturbed. I'm just happy that someone makes even bigger impacting gaffes than I do.

Where Are the Java Physics APIs?

If I'm going to try and port Deskblocks to Java then I'll need to find a native Java physics API. If my only choices are native libraries then it makes more sense just to stick to Qt 4 - I'd rather keep this entirely in the "compile once run anywhere domain." Managing native libraries with wrappers functions is just a pain in the butt.

I was surprised to see slim pickins for physics API. For 2D physics I just found Phys2D and for 3D physics I found JBullet, a port of the Bullet physics API. Both seem to be great projects, and both seem to be currently active. Indeed, I'll probably give Phys2D a try and see if I can use it. I guess I just expected an ODE port by now.

KDE 4 Really Fast, Except When It's Really Slow

I've enjoyed openSUSE 11.0 not necessarily because it works flawlessly, but because it's the best working install of KDE 4. KDE 4 has a lot of great potential, but it's not fully realized until you hit KDE 4.1. Since 4.1 is the actual version of KDE 4 meant for widespread usage, I've been downloading the unstable builds from SuSE's build service.

Things have been working fine except for one extremely nagging thing - the initial draw of a konsole window takes 5-10 seconds. It's extremely obnoxious, especially when you need a terminal so often.

I found out this was due to NVIDIA's drivers on newer boards - as described in the KDE techbase and illustrated in the Ubuntu forums and How-To's. Luckily it was an easy fix - running nvidia-settings -a InitialPixmapPlacement=2 -a GlyphCache=1 as root set the appropriate settings in the NVIDIA driver to allow windows to resize at the speed they should. But I don't necessarily blame NVIDIA - the blame deserves to be cast further.

The Linux Hater's Blog brings up a great point about Xorg not being about to allocate offscreen buffers - something that I didn't realize. Xorg lacks a memory manager, so all the stuff you need for full OpenGL support just can't be done. It simply can't be done with Xorg. All the points made in the rant are absolutely right - the memory management infrastructure for pbuffer and framebuffer objects have to be there, otherwise you're hosed.

The core issue that comes from this deficiency is that X11 in and of itself is inherently unable to support OpenGL. Lack of offscreen buffers means that all the great stuff you should be able to do directly in hardware can only be accomplished with a software renderer. Of course, this defeats NVIDIA's entire business model of making the GPU the most important part of your workstation. So they had to massively replace parts of X11; the NVIDIA Linux drivers must, by sheer necessity, replace huge chunks of the XOrg implementation.

After reading the Linux Hater's post a lot of other stuff made sense - why NVIDIA's drivers are so invasive, why you magically don't need to install Xgl to run Compiz Fusion if you are using the proprietary nvidia driver (because it already replaced Xorg for you, thanks), and why KDE's desktop effects had window resizing slowdowns.

NVIDIA didn't break things - they fixed things. They're just trying to live in our broken world.

Java DeskBlocks

Right before I started my latest hacking-for-cash endeavor I was working on DeskBlocks, a physics sandbox rendered directly on the desktop. I was using Qt 4 for development, mostly so I could use ODE and refresh my C++ coding. One problem in development was that things would work perfectly fine in X11 - rounded edges, nice circles bouncing around, render speed was great - then things would work horribly in Windows. Or vice-versa. I could never get things to behave properly cross-platform.

With Java's big update, it's windowing toolkit now allows for translucent and shaped windows on platforms that support it. That means the rendering issues I had with Qt 4 may be solved with Java 6u10.

It makes me start thinking if I should move the project to Java instead. It appears there are Java bindings for ODE... so it just might work.

Real-Life High Dynamic Range Lighting

I've been a sucker for HDR in gaming for a while now. So when I saw mention on Hack a Day about turning your point-and-shoot camera into a full-featured model that allows you to do stop-motion and high dynamic range photography my curiosity was piqued.

The Stuck In Customs HDR Tutorial gives a good reason why HDR photography can be so appealing: our eye adjusts constantly as it is observing its environment, quickly dilating and contracting the pupil to modify the range of light and color hitting the retina. HDR photography does the same thing, re-sampling the image to take in a varying amount of exposure and light at different depths of field.

Hack a Day turned me on to using the CHDK firmware add-on with a Canon SD870 IS. The CHDK add-on software allowed me to do exposure bracketing in continuous shooting mode.

Luckily HDR photography is all the rage lately, so I even had a Grumpy Editor's guide to HDR with Linux. It was great - it introduced me to PFScalibration and Hugin's image alignment, which are both nicely wrapped together in the fantastically easy to use Qtpfsgui toolkit.

Of course I had to turn the flash off and lower the resolution (to allow the continuous mode to write to the SD card faster), but in the end I had a perfect stack of images at varying exposures to import into Qtpfsgui. A tree turns into something more provocative pretty quickly:



I was amazed at the quality of open-source options for photography - Qtpfsgui was great for HDR, and Rawstudio was even more fantastic in dealing with my RAW digital negatives. The SD870 IS doesn't have native RAW file support, but thanks to CHDK and DNG4PS-2 I was able to quickly pull DNG files off of my SD card and start editing them in Rawstudio.

Unfortunately I don't have much if any time to try out new things, but this was a pretty pleasant diversion for the evening.

OpenSUSE 11.0 - Desktop Linux Actually Done Right

I've been pretty unhappy with SuSE Linux as of late. I thought 10.1 was well done, but subsequent releases were of poor build quality and in places just sloppy. There were times it began to show promise, only to fall just short.

I've been living inside of openSUSE 11.0 for the past several days, and tried out RC1 as well. All in all it's a very impressive distribution, and I'm surprised they were able to put this level of polish on KDE 3.5, KDE 4 and Gnome at the same time.

Package management, the Achilles' heel of SuSE up to this point, is scores better. YaST2 actually loads its package management tool quickly, and metadata indexing doesn't take decades like it used to. Compiz is nicely integrated, and KDE 4 is actually quite well done. I'm installing the unstable 4.1 packages right now, and we'll see how that looks as well.

There are some speed issues, but then again I'm using KDE 4 with desktop effects cranked up. However, the fixes to package management alone make SuSE a fantastic winner for the desktop Linux space. Honestly, once KDE 4.1 comes out I wouldn't be surprised at all to see more corporate desktops and developer machines turning to SuSE Linux Enterprise Desktop.

Counting on Trolls Under Your Bridge

It seems no matter if you're in a huge corporate project or a smaller cadre of independent developers, the rules of managing people and code remain the same. And open-source projects tend to manage people much better than their corporate brethren.

In projects with multiple people involved you have to continually worry about people leaving, contributing crappy code or going on a complete tangent. Usually in corporate life people just want the thing to work, and if it randomly happens to hit production and not have complete show-stopping errors, great. But the code could be complete cruft and no one would care.

With an open-source project, however, you must have complete transparency. Brian Fitzpatrick and Ben Collins-Sussman recently gave a presentation on managing people in OSS projects effectively. Code reviews, definite goals and communicating "just enough" were all key. Hrm... how many commercial software developers do the same thing?

It's not without its difficulties, however. Especially with the kernel, contributions can be a big mixed bag. Sifting the good contributions from the pointless ones can be time-consuming and tedious. So projects are split into more minute portions, and newbies are either mentored or given a sandbox to play in.

It seems like there is a role model out there for taking a wide variety of people with an even wider variety of backgrounds and time commitments and having them all contribute to a well-developed end product. It's a shame it isn't emulated more often.

Google Sites = The Reason for a Google Account

The whole reason I wanted to start developing on Google App Engine was because I wanted to start building a repository of code samples, How-To's, documentation, projects, blog posts, all that stuff. Something akin to Confluence. Google already did it for me.

Google just recently launched Google Sites, which basically becomes a content management system for whatever you like. It's exactly what I needed - and I'm planning on moving my docs, code snippets and projects over soon.

Wow... Actual Support! They Know We Exist!

Wanted to buy an album I had just discovered... but didn't want to haul myself to the neighborhood Best Buy. So I wanted to see if the band had an online purchase method. They didn't... but they sold through Amazon. I knew Amazon sold DRM-less MP3's, so I decided to check it out. For whole album downloads you are required to use their lil' download app, so imagine my aghast expression when I saw:

Sweet. Works only on 32-bit Linux (can work on 64 but has problems with library dependencies), but otherwise purchasing was swell. No issues.

Thanks Amazon for realizing Linux users buy music, too!

Spent

A while back I received this e-mail, completely out of the blue:

Wise men say, you only have to resign yourself to what you cannot improve

I'm not sure why I got the e-mail. Or who sent it. No real idea. Could well be spam. But it kinda stuck with me for some odd reason.

I've been working the 80 hour weeks lately, as promised. That means that I've had to give up working on my open-source projects. I feel real pangs of guilt when people ask for bugfixes or when the next release will come out... especially when users as kind as tomasio even volunteer their own time for icon assets. But I've cut sleep down to a few wee hours and just have nothin' left.

I'm hoping I can get everything going at work, get things on a stable foundation, then give myself free time once again. At least, that's the lie I tell myself.

Two Great Tastes - maemo & Qt

Fresh off the wire, it has been announced that Nokia will introduce Qt, my favorite C++ toolkit, to the maemo platform, my favorite portable hardware platform. Two great tastes that go great together.

If this kind of platform expansion and cooperation with Qt developers (such as KDE authors) is what will come of the Nokia acquisition of Trolltech, it may not be as bad as I predicted. Here's to hoping that Nokia sees Qt as a toolkit that will serve mobile, embedded and desktop platforms. Especially with the recent fame of low-cost low-footprint laptops, Qt and maemo have to be getting some additional attention.

If the WiMAX-enabled Nokia 810 becomes more powerful and popular, the addition of Qt and simplified cross-compiling could provide a huge increase of third-party applications hosted on an already open mobile device.

Java 6.9

Someone just directed my attention to the Java 6 update 10 intro on Sun's site. What the living...?

First off, this isn't a minor update. This is taking a backhoe to the foundations of Java, hitting a water line, but digging a basement anyway. Why this wasn't released in 7 I don't know... I guess it's because the update is largely centered around deployment of Java as a platform and not adding any functionality to the underlying API. But damn, it's an overhaul.

First, Java is now chopped neatly into libraries, so you only download what you need. That means Java installations can be one-third of what they were in the previous release. Java can now be downloaded and installed more efficiently as well, thanks to some much easier-to-use JavaScript and HTML-fu.

Konqueror has already done this for a while now, but applets will now execute within a full JVM instead of a half-baked nsplugin. This allows for more robust applets and, from my experience with Konqueror, plugins that are more crash-resistant.

Finally the fairly... blech... look of Java has been completely overhauled with Nibus, long at last. Previously I've had to use javootoo's Look and Feel libraries to make things look remotely presentable. Now Nimbus should be able to fill that gap nicely by adding more modern window decorations and UI components.

These were all desperately needed improvements to have Java make inroads into the desktop space. Let's hope it isn't too late.