AutoSearch --VERSION AutoSearch --help AutoSearch --man

AutoSearch is amenable to be run as a cron job because all the input parameters are saved in the web pages. AutoSearch can act as a automated query agent for a particular search. The output files are designed to be a set of web pages to easily display the results set with a web browser.

Example:

    AutoSearch -n 'LSAM Replication'
        -s '"lsam replication"'
        -e AltaVista
        replication_query

This query (which should be all on one line) creates a directory replication_query and fills it with the fascinating output of the AltaVista query on "lsam replication", with pages titled AltaVista query on "lsam replication", with pages titled ``LSAM Replication''. (Note the quoting: the single quotes in '"lsam replication"' are for the shell, the double quotes are for AltaVista to search for the phrase rather than the separate words.)

A more complicated example:

    AutoSearch -n 'External Links to LSAM'
        -s '(link:www.isi.edu/~lsam) -url:isi.edu'
        -e AltaVista::AdvancedWeb
        -o coolness=hot

This query does an advanced AltaVista search and specifies the (hypothetical) ``coolness'' option to the search engine.

qid

The query identifer specifies the directory in which all the files that relate to this query and search results will live. It can be an absolute path, or a relative path from cwd. If the directory does not exist, it will be created and a new search started.

--stats

Show search statistics: the query string, number of hits, number of filtered hits, filter string, number of suspended (deleted) hits, previous set size, current set size, etc.

-v or --verbose

Verbose: output additional messages and warnings.

-n or --qn or --queryname

Specify the query name. The query name is used as a heading in the web pages, therefore it should be a 'nice' looking version of the query string.

-s or --qs or --querystring

Specify the query string. The query string is the character string which will be submitted to the search engine. You may include special characters to group or to qualify the search.

-e or --engine

Specify the search engine. The query string will be submitted to the user specified search engine.

In many cases there are specialized versions of search engines. For example, AltaVista::AdvancedWeb and AltaVista::News allow more powerful and Usenet searches. See AltaVista or the man page for your search engine for details about specialized variations.

--listnewurls

In addition to all the normal file maintenance, print all new URLs to STDOUT, one per line.

-o or --options

Specify the query options. The query options will be submitted to the user search engine with the query string. This feature permits modification of the query string for a specific search engine or option. More than one query option may be specified.

Example: "-o what=news" causes AltaVista to search Usenet. Although this works, the preferred mechanism in this case would be "-e AltaVista::News" or "-e AltaVista::AdvancedNews". Options are intended for internal or expert use.

-f or --uf or --urlfilter

This option specifies a regular expression which will be compared against the URLs of any results; if they match the case-insensitive regular expression, they will be removed from the hit set.

Example: "-f '.*\.isi\.edu'" avoids all of ISI's web pages.

--cleanup i

Delete all traces of query results from more than i days ago. If --cleanup is given, all other options other than the qid will be ignored.

--cmdline

Reconstruct the complete command line (AutoSearch and all its arguments) that was used to create the query results. Command line will be shown on STDERR. If --cmdline is given, all other options other than the qid will be ignored.

--mail user@address or -m user@address

After search is complete, send email to that user, listing the NEW results. Email is HTML format. Requires the Email::Send and related modules. If you send email through an SMTP server, you must set environment variable SMTPSERVER to your server name or IP address. If your SMTP server requires password, you must set environment variables SMTPUSERNAME and SMTPPASSWORD. If you send email via sendmail, you should set environment variable SENDMAIL if the sendmail executable is not in the path.

--emailfrom user@address

If your outgoing mail server rejects email from certain users, you can use this argument to set the From: header.

--userid bbunny

If the search engine requires a login/password (e.g. Ebay::Completed), use this.

--password Carr0t5

If the search engine requires a login/password (e.g. Ebay::Mature), use this.

To organize these results, each search result is placed in a query information directory (qid). The directory becomes the search results 'handle', an easy way to track a set of results. Thus a qid of "/usr/local/htdocs/lsam/autosearch/load_balancing" might locate the results on your web server at ".

Inside the qid directory you will find files relating to this query. The primary file is index.html, which reflects the latest search results. Every not-filtered hit for every search is stored in index.html. When a hit is no longer found by the search engine it a removed from index.html. As new results for a search are returned from the search engine they are placed in index.html.

At the bottom of index.html, there is a heading ``Weekly Search Results'', which is updated each time the search is submitted (see ``AUTOMATED SEARCHING''). The list of search runs is stored in reverse chronological order. Runs which provide no new information are identified with

        No Unique Results found for search on <date>

        Web search results for search on <date>