WACS: The Download System

Contents

Overview

This is one of the most important features of Wacs - it's ability to scan websites automatically for new sets by your favourite models and to automatically download them without your having to spend hours with your web browser starting off slow downloads of big zip files or movies. On a Unix/Linux box, this can be triggered during the night using the cron command and have the sets downloaded for you ready when you want to view them.

The chkmodel command

At it's simplest, you merely run the chkmodel command with the model number of the model you wish it to check on, and it will contact each site for which you have a subscription for and work out what new material by her there is available. It will add download records for each new set or video that it finds. It also updates the model checked date in the idmap table, and if it finds new stuff for a model marks the changed date in the idmap table entry too.

Of course in order to do this, you will need to add in details of your account for any existing vendor, or add details of a new vendor you wish to download from. For an existing supported vendor, you will merely need to call up the vendor manager, wacsvendmgr; select the preloaded vendor details and modify the database entry. This will mean changing subscribed to "Yes" and adding your username and password - the subscribed until field is purely informational at the moment, but may be useful to you. Once this is done, click on the [Update Database] button and the changes will be made for you.

The refresh command

The refresh command is a front end to the chkmodel command which goes through all the idmap entries for currently subscribed-to web sites and then calls chkmodel as appropriate. It takes one optional command line argument which causes it to check all models (either -a or --all depending on which style you prefer). At present it has a fairly simplistic algorithmn for assessing whether a model is currently active or not - if there's been more than a month during which her page has been checked and nothing new found, she is considered dormant. Unless the -a option is used, dormant models are not rechecked daily.

If you wish to do an automatic update system using crontab, you probably want to call refresh normally once per night, and refresh -a once per week at a suitably quiet time.

The getarc command

The getarc command reads through the outstanding download requests and picks a small number of them to go after. It downloads each in turn and places them into the download area of the wacs system, in a sub-directory for the source web site, and then sub-directory of that called either zips for image file zips, or videos for video files. Within these it creates a further sub-directory of the model's name, an underscore and her reference number to avoid confusion with multiple model of the same name. Inside this directory, it places the files it has downloaded.

Notes on configuring the Downloader

If you're using a pre-defined site record for the Vendor site you wish to download from, you should have an easy time; just enable the site, add your password, and then add a few model details using the Wacs Model Manager and it should all start working. If you're creating a new Vendor site description, it can be rather more difficult. The first thing you need to understand is how to set the Image and Video download id types to the correct values (vidtimg and vidtvid in the vendor schema). The actions take place in two places - first the chkmodel script must actually find the desired zip or video file name and create a download record requesting it be fetched. Second, the getarc script must actually perform the necessary retrieval and saving actions.

The following table details how the various values currently work:

Image Sets
1Default - just get the URL specified in the download record modulo the addition of the specified image server URL if the download record URL doesn't start with http://
Video Clips
1Default - just get the URL specified in the download record modulo the addition of the specified video server URL if the download record URL doesn't start with http://
2Adds the model id (usually the ikey) on the front to avoid clashes with duplicate file names.

Next Section: Configuration
Back to WACS Documentation Index