Apache, Linux, Image::Magick, and TiVo Home Media Option

Updated 10/28/2003 with screenshots

Having been massively disappointed in the O'Reilly TiVo[1] book due to the misleading copy that neglected to mention you shouldn't bother to buy it if you have a Series 2 TiVo, I decide to make the best of it and use the Perl module on their developer site to get the foolish beast talking to my Linux server.

While I'm a SunOS guy at heart, my Linux box is about 100 times as fast[1.5] as my aging SPARC 20, so I figured that serving images and MP3s would be more practical off of the latter.

So step one was to get and install the Perl module. That was simple enough, merely downloading TiVo.pm and deciding where to plop it down. I decided on /usr/lib/perl5/site_perl because that seemed appropriate - definitely specific to the site.

Next I had to install the TiVoConnect CGI script in the right place and set up Apache to actually pay attention to it. The CGI script needs a location for a filesystem cache (/tmp seems like an obvious choice), and real locations for the /Photos and /Music partitions from which it will serve pictures and music. No big deal.

First, the TiVo docs say in a sideways manner that all requests will begin with the prefix /TiVoConnect, followed by the parameters, for example:

GET /TiVoConnect?Command=QueryContainer&Container=/Music

    # Trying to set up TiVo for Home Media Option.
    #
    Alias /TiVoConnect "/var/www/cgi-bin/TiVoConnect.cgi"

http://customersupport.tivo.com/tivoknowbase/root/public/tv1158.htm?

"TiVo Desktop tries to reach the network on port 8101 first, then tries 8102, then 8200."

Turns out that the comments in the httpd.conf file don't exactly tell the truth:

    # If you want to use server side includes, or CGI outside
    # ScriptAliased directories, uncomment the following lines.
    #
    # To use CGI scripts:
    #
    #AddHandler cgi-script .cgi

	Now from here (the main menu), we go to the Music & Photos option. I know that the Windoze desktop software auto-magically shows up, but after an impatient minute or two, I give up and choose the "Manually add a Server" option ...
	Music on TiVo Online Photos on TiVo Online Manually add a Server

...and set up the IP on the TiVo, but immediately get a "No servers were found at 192.168.1.172" error. I tell it to leave the server there anyways and see what happens - nothing. Now my list is:

192.168.1.172 Music on TiVo Online Photos on TiVo Online Manually add a Server

... not quite what I expected! If I pick my server, I just get the damnable "No such server" message again. Looking at the logs, I can see that the TiVo talked to Apache, but it doesn't seem to reflect a damn thing on the screen, just the IP of the server on the Music & Photos screen.

Well, it so happens that there's a magic beacon that the server is supposed to send out that tells the TiVo the server is out there. But thankfully, after an annoying (but relatively short, about 22-25 seconds) delay, it seems to pick it up anyways, and the music and photos options show up in my TiVo menu:

Music My Personal Photos Music on TiVo Online Photos on TiVo Online Manually add a Server

They're still empty, so I plop a couple JPEGs into the Photos directory, and go on in. But what the hell, nothing is there. To make a long story short, it turns out that while their Perl module will cache the directory objects it creates, it doesn't ever *UN*cache them - they just hang around forever. After some gnashing of teeth, I figured this one out and got the images to show up in my menu - but they wouldn't display.

It was now that I began Adventures with Perl. The TiVo.pm module needs a number of modules to be installed, and will silently fail if they're not. CPAN is your friend, and ImageMagick is a pain in the ass. :-)

Well, first I found and installed ImageMagick, which includes PerlMagick, which is cool, but it wasn't immediately apparent that the usual configure/make/make install deal does NOT install the Perl module. It took a 'find' in the /usr/lib/perl5 hierarchy to prove this to myself. So I figured I'd use CPAN to install it. Hah.

Well, to be fair, the usual:

    % perl -MCPAN -e shell

    cpan shell -- CPAN exploration and modules installation (v1.76)
    ReadLine support enabled

    cpan> 

But in any case, I now had the magic key - a second setup/make/install sequence was necessary to get the Perl module installed, and this second build sequence was located in the PerlMagick subdirectory of the ImageMagick build. With some convincing, I got it to use /usr/lib/perl5 even though ImageMagick itself was installed in /usr/local, and I had Image::Magick installed.

So, finally, I deleted the cache in /tmp again, and this time the images showed up. And the MP3s that I copied over from my friend's band. Good deal.

... Except one little detail - the TiVo's image details page looks like this (I'll bother with a pretty picture later):

+---------+ | | my_picture.jpg | () * | Taken: unknown | * *| Imported: unknown |--\__/\__| Modified: unknown +---------+ (View photo ) (Rotate 90d Clockwise) (Play slide show ) (Don't do anything )

Pretty Picture

Cool! It can rotate! Except ... it can't. Rotate doesn't do a damn thing. Well, forget that for now. Let's move on to sound.

In any case, this meant back to the old printf() debugging. [3] Except that this being CGI, using print statements just wrecks the returned XML. So instead I created a tivo log file and dumped everything to that. Worked fairly well, except that eventually I ended up with print statements all over the place. But it makes it easy to tell what's going on, and I can turn them all off by setting debug to 0, so not that big of a deal. Except that I've gone back in time to 1990.

So anyways, I can see WAV as a type in TiVo.pm ... but it won't load them. More on that later. MP3s are fine, and who knows what the hell M3Us and PLSs are .. but those don't load either. Oh well. Guess it's time to figure out that stupid rotate thing after all.

Well, without the scary details, I did manage to get rotate working, but it took some trickiness with the cache to make it stick and make it not munge the damn images. Initially, I'd hit rotate, and it would rotate the little preview image just fine, but when I went to "View Image", the full-size image was unchanged. There was a bunch of other intermediary fun and games as well. The original Perl code to load an image was like this (I've omitted some junk):

    ##
    ## TiVo::Item::JPG->send( )
    ##
    ##  TiVo::Item send extension supporting image transforms
    ##
    sub send {

      #########################################
      # initialization code skipped
      #########################################

      # We will check cache if a transform has been requested
      if( defined( $params->_Width ) ||
          defined( $params->_Height ) ||
          defined( $params->_PixelShape ) ||
          ( uc($suffix) ne uc($source_suffix) ) ) {
    
        # Use MD5 to hash the filename
        require Digest::MD5;
    
        my $cache_name = Digest::MD5::md5_hex($path);
    
        my $string = "";
        $string .= "_H" . $params->_Height if defined($params->_Height);
        $string .= "_W" . $params->_Width if defined($params->_Width);
        $string .= "_S" . $params->_PixelShape if defined($params->_PixelShape);
    
        # Append transform data to filename
        $cache_name .= $string;
    
        my $cache_dir = $server->_ImageDir || '/tmp';
        my $cache_path = "$cache_dir/$cache_name.$suffix";
    
        # Send cached image directly, if it exists
        if( -f $cache_path ) {
    
          $path = $cache_path;
    
        # Otherwise, use Image::Magick to perform requested transforms
        } else {
    
          require Image::Magick;
    
          my $width = $self->_SourceWidth;
          my $height = $self->_SourceHeight;
          my $ratio = 1;

          #########################################
          # Calculation of scaling skipped
          #########################################

          my $img = new Image::Magick;
          $img->Read($self->_Path);
    
          if( $ratio ) {
          $img->Scale(width => $width * $ratio,
                      height => $height * $ratio);
        }
    
          $img->Write(filename => $cache_path);
          $path = $cache_path;
    
        }
    
      }

      #########################################
      # Return of binary data skipped
      #########################################
    }

So, we can rotate easily enough, right? Well, yes and no. We have the rotation in the request, but it turns out that there's no stateful storage of it, so the rotation parameter is always 90. It's easy to get the small image to rotate by 90 degrees, but impossible to get it to rotate more or un-rotate except by going out and back in. Ugh.

In a nutshell, what I ended up doing was two-fold. First, when a rotation is requested, the main image is rotated and stored in the cache. From then on, the cached main image is considered as the source image, rather than the original image on disk.

Then any scaled images (which is all of them, an untransformed image is never displayed directly by the TiVo) are created or pulled from the cache as necessary - and the date of the cached thumbnails are always compared to that of the source image (be it in the cache or out on disk). That way, if the image is rotated, all thumbnails are also equivalently rotated - and the rotation is persistant, so that a second rotation of 90 degrees nets a 180 degree rotation.

Took a number of iterations to get it just right, but it is now pretty cool. Though I had to figure out some of the Storable stuff as well, because it turns out that it stores a descriptor of the image in the cache, the aspect ratio got all hosed up if the cached descriptor wasn't updated as well.

Cool. So now I can look at all the images from our digital cameras on the TiVo. Not a bad deal. Next, figuring out how to create MP3s from CDs.[4]

Anyways, I have a ton of CDs, and it turns out that ripping MP3s is amazingly easy. My RedHat 7.3 system comes with cdparanoia, which happily pulls a track from a disk and saves it as a huge .wav file, and it took me minutes to download and build LAME, which merrily converts said .wav files into MP3s. Easy enough. Except that this deleting the /tmp cache every time I put a new file in is getting old.

Next I had an aborted attempt to support .wav files. CPAN loaded the Audio::Wav module quickly enough, and I updated the class hierarchy to support WAV as well as MP3. Then I loaded one and ... well ...

You ever try dumping a binary to the audio device on your machine? If so, then you know what it sounded like. Clearly either the TiVo doesn't *really* support WAV files (no docs on it, but it WAS in the TiVo.pm module), or I loaded it wrong .. and since I was reading it just fine with Audio::Wav, oh well.

Then it crashed the TiVo. Hell, I didn't even know the TiVo COULD crash - "HOLY CRAP", I'm thinking, "I broke the TiVo." Well, it recovered quickly, and I hope to never see the GSOD (Gray Screen of Death) again (It's interesting - I've seen this referred to a number of times as the "Green Screen of Death", but it sure as hell looks gray to me. Or then again, maybe I'm not seeing the Green Screen of Death, I'm seeing the Gray Screen of Boot. Dunno).

The WAV support was quickly removed.

So, next ... fixing the cache.

This took a little bit of trickiness. I'm no Perl guru, so a couple of things that I tried that didn't work mystified me, but in the end I got a working system for that too. The cache is measured in two ways - first, if the cached file is older than the source file, the item is expired and reloaded. Second, if the item in the cache is older than an absolute limit (I set it to 48 hours for no particular reason) the item will be refreshed. Now it picks up new files when they're dropped in the directory, because the directory itself has its date updated, and the entry is recreated.

There's still two outstanding things I will tackle one of these days. The first is that fool beacon so that I don't have to wait 25 seconds every time I go to the Music & Photos page. I downloaded a silly PHP shell script that was supposed to do it, but it didn't, and I have even less patience with PHP than I do with Perl. It's some wacky UDP thing on port 2190 and haven't yet been motivated enough to write it from scratch. One of these days I'll install the fool Windoze version and snoop the network to see what the real packet is.

Secondly, I'd like to put a limit on the cache size. I figure I can quickly calculate the disk usage of the cache directory (it's a flat cache, it doesn't mirror the source structure at all), and then while it's greater than the max allowed size, delete the oldest item there. But I'll do that another day.

For those of you interested, here's the original TiVo.pm module, and my modified version of the same.

  % ls -l TiVo*
  -r--r--r--    1 toolset  xruler      37319 Mar 20 12:55 TiVo-orig.pm
  -r--r--r--    1 toolset  xruler      46996 Sep 26 01:21 TiVo.pm

    #
    # Listen: Allows you to bind Apache to specific IP addresses and/or
    # ports, in addition to the default. See also the 
    # directive.
    #
    #Listen 3000
    #Listen 12.34.56.78:80
    Listen 80
    Listen 8101
    
    #
    # Port: The port to which the standalone server listens. For
    # ports < 1023, you will need httpd to be run as root initially.
    #
    Port 80
    

    [Sun Sep 21 15:55:08 2003] [crit] (98)Address already in use:
        make_sock: could not bind to port 80