Content Caching Mechanisms

As we were seeking to develop WACS 0.9.1, we decided to confront one of the ongoing dilemmas inherant in the dual use scenario we've always had for WACS. Are we an archive tool for collectors or a content management system for web site developers? Generally on the download side of things we'd stuck to our original model as an archive tool, avoiding duplication of data at a cost of greater processor activity and content delivery delays when downloads were requested. With the new tools and features of Wacs 0.9.1, we now provide a choice where none existed before. You can now choose to cache content for quicker and more accurate downloads, and in doing so improve the download experience at the cost of using significantly more disc space.

This emphasis on choice is important as we have not changed the defaults at all - we've just made new tools available to provide an option that didn't exist before. You need to use the tools to create and manage your content cache if you wish to reap the benefits.

How The Caching Works

The main changes to allow for content and thumbnail caching involve configuring values for the content cache areas within the WACS configuration file. If these values have non-empty values, they will be checked for the existence of a cache prior to falling back to using the conventional cgi-bin program normally used by WACS.

The login and logout programs will update the .htaccess files in the appropriate toplevel of the content cache tree. There can be up to three areas in here: images, videos and thumbnails depending upon the configuration. The simplest of these is the thumbnails which is automatically used if the server configuration section attribute serverturl is set to a suitable value. If no cached version of the thumbnail is available, the wacsthumb application will create one for the next time. Use of these depends on your application making use of the thumblink WACS API function when you add a thumbnail to your page.

For image sets, the caching process works a little differently - Nothing is automatically cached - you have to manually decide to cache a given set using the the section called “The wacscachectl command” command. This command will create a suitable .zip file for the set on demand.

% wacscachectl --create 123
[...]
%

At the simplest level, it works in much the same way for standard videos except that generally the stored video clip is exactly what we want to present to the user (no need to make a zip file) and so in this case the wacscachectl command will create an appropriate symbolic link to the video file within the stored video tree. The system does however provide for rather more complex features if required, but you do not need to use them if you don't want to.

[Note]Note

We've started using the long-present but thus far unused sdownload field in the set record for the name of the download file for the set. The new wacsdnlreg command allows you to specify this when creating the download record, or if the material has been downloaded from another site the name used there will be preserved. The name can be set on older records and changed using the Set Information Manager app (wacsinfomgr).

A recent change to the Wacs Placement Manager (wacsplacemgr ) has added a tick box that invokes wacscachectl with the appropriate options for a video file from the web interface when desired. This will be improved and expanded in future releases, but it has already made the process of creating a download cache much simpler.

The More Complex Features Of Caching

A lot of commercial sites out there have got into the habit of dividing up their videos into multiple parts in order to keep the download sizes more manageable. The same occasionally also happens for image sets, particularly amongst those sites who offer particularly high resolution images for download. When a set has been marked up with continuation parts, a whole new aspect of the content cache comes into play.

The WACS content cache system includes the concept of having both raw and cooked versions of the same set available. In a video, the cooked version of the primary set would be a compilation of all the parts, while the raw version would be just that - the unchanged first part of the video itself. In many of the standard WACS apps, this distinction is understood and the link to the video clip is appropriate to the context in which it is presented. The main links go to the cooked compilation of all the parts, while the numbered parts would link to the raw versions of the respective videos.

Where a video is multipart, the wacscachectl command will look in your process area for the necessary combilation video. Your process area is adjacent to the unpack area used in the unpacking and placement of photo set and video files within the download area on your server. The WACS system includes a tool to make the appropriate compilation video called wacsvidcomb for WACS VIDeo COMBiner. We'll discuss this tool a little further in a momnet, but lets just review what we have so far in the caching system.

Let's start with the very simple case of a zip file for a single self-contained photo set:

Example 13.1. Layout of Cache Files

     set123/
            Sarah_Set01_RedDress.zip

Next let's consider a simple video clip that is fully self contained - here the symbolic link points to the real video file sarah124-solo-1a.wmv within the main /video area but we have used the sdownload field to rename the downloaded file:

     set124/
	    Sarah_Movie01_PinkBikini.wmv
                     -> /video/blondes/Sarah/sarah124-solo-1a.wmv

For our third example, we've got a much more complex scenario where this set number is the first part of a three part movie of which we have made a compilation using the wacsvidcomb application. Here we have both raw and cooked versions:

     set125/
            cooked/
                   SarahAndJane_Movie01_KitchenCuties.wmv
            raw/
                   SarahAndJane_Movie01-part1.wmv
                       -> /video/blondes/Sarah/sarah-jane-gg1-1a.wmv

Using wacsvidcomb to make compilations

The wacsvidcomb command retrieves data from the Wacs set record and uses it to create the compilation video by way of the mencoder application that is part of the Linux mplayer video player project. Before the wacsvidcomb program is used, it's a good idea to check that it's mplayer companion can find all the necessary codecs to play the types of video files you plan on converting. They share a common architecture and codec library, so it is a good indication.

At it's simplest level, wacsvidcomb can simply make the compilation using the clips that make up the video sequence as recorded in the database. You must give it the set number of the Primary (first) video in the sequence in order to work. It will produce the resulting compilation, often in better than real time, and place it in your process area within the WACS download directory structure - exactly where a subsequent run of wacscachectl on the same set number would look.

However many of these multipart videos feature 18USC2257 declaration slates on the front of each part of the video that would make an annoying and unnecessary interuption on the compilation video. To this end there is a field within the sets database called sskipfr for skip frame. You can now set this value for video files using the wacsinfomgr as from WACS 0.9.1. Unfortunately it's not quite as simple as you might hope due to the way in which movie files on computers are constructed.

If the movie were uncompressed (which would make it HUGE), then you would specify the number of frames to skip, so if it was a 30 frame per second video, and the slate lasted six seconds, you would put a value of 180 into sskipfr. Simple enough. However it's very rarely that easy and the majority of video compression algorithms start from a special complete frame called an I-frame and then follow it with only the modifications to it. The cut can only be made at one of these I-frames and so the counting is based on the number of I-frames rather than the number of actual frames.

Unfortunately compression systems vary wildly in how frequently they force an I-frame; some every second, some every two seconds and some (like WMV) only on a major scene change. Generally the skip feature of mencoder counts in I-frames and trial and error is the only real way to find the correct value to put into the sskipfr value for any given video clip. Of course you can always do the edit manually with any one of a multitude of video editing applications and place the result into your process directory for wacscachectl to collect when creating the content cache for the set.