forward - arbitrary tcp socket forwarding over a secure/encrypted channel.

copydir, mirrordir - copy, mirror directory trees via a minimal set of changes, locally or over FTP, or over a secure tcp connection.

recursdir - recurse through local or remote directories to command/find files or create tar files.

mirrordir [-c | --copy-mode | --recurs-mode ] -[abBCdDFGhklMmNOopRrstvVX] src [src ...] dest

copydir -[abBCdeFGhklMmNOopRrstvVX] src [src ...] dest

recursdir -[abBCdeFGhklMmNOopRrstvVX] src [src ...]

pslogin [--key-size bits] [mc://][username@]hostname[:portnumber][ /path]

But usually just mirrordir [--exclude path] control mirror copydir src [src ...] dest recursdir src [src ...] [-C program] pslogin [username@]hostname[:portnumber]

mirrordir is a set of useful utilities for manipulating and mirroring directories. Included is also the command pslogin - an alternative to ssh(1), and forward(1) for forwarding arbitrary TCP socket connections over encrypted secure channels.

mirrordir copies files that are different between the directories control and mirror to the directory mirror. Files whose modification times or sizes differ are copied. File permissions, ownerships, modification times, access times (only if --access-times is used), sticky bits, and device types are duplicated. Symlinks are duplicated without any translation. Symlink modification and access times (of the symlink itself, not the file it points to) are not preserved. Hard linked files are merely copied. Creation times cannot be set with Unix as far as I can see.

mirrordir is a DANGEROUS command because files or directories that exist in mirror that don't exist in control are deleted. If control is entirely empty, then all files and directories in mirror will be deleted. If mirror is entirely empty, then all files and directories in control will be copied.

In short, mirrordir forces mirror to be an exact replica of the directory tree control in every possible detail suitable for purposes of timed backup. It naturally descends into subdirectories to all their depths. mirrordir tries to be as efficient as possible by making the minimal set of changes necessary to mirror the directory.

Access time duplication is not usually required and creates unnecessary load. Hence it is given as an option.

The directory control is left untouched. If --restore-access is given then access times are reset to their original with each read.

If the the --strict-locking option is on, files in control that are copied are locked for `shared reading'. This will ensure, if another process is busy writing to that file, that the file is not copied in its incomplete or corrupted state.

Usually mirrordir will not exit, but will give error messages to stderr to report any problems, and then will continue.

The directory mirror or dest must exist, even if it is empty.

Before erasing all the files in a directory, mirrordir checks for the file *--keep-me (where * is zero or one characters). If this file is present it will abort with an error message. Hence such a file can be created in all directories that you are fearful of being recursively erased.

copydir is equivalent to mirrordir -ck --no-erase-directories ... (although -c implies -k anyway), so copydir is very much like a rigorous version of cp(1) where filenames can also be URLs, and only outdated files are replaced. Use copydir instead of mirrordir for most file transfers. Only use mirrordir, when you really want to delete things.

recursdir is a further program that does nothing but descend into the directories on the command line. It is equivalent to mirrordir --recurs-mode ... It was born after the -C option was added, and can be used as a more rigorous version of find(1) and can also pack all the files it finds into a tar file.

pslogin is yet a further program which has almost nothing to do with the previous three. It envokes a secure login session using secure-mcserv. It is equivalent to mirrordir --login-mode --secure ... pslogin should be called logindir. See --login-mode below.

forward is yet a further program which has almost nothing to do with first three. It can do forwarding of arbitrary services over a secure channel. See forward(1) for details.

The importance of this package is that you can use URL's instead of normal filenames, and hence manipulate files over a network. The URL types currently supported are ftp:// and mc:// (http:// is not a filesystem and therefore is not supported). mc:// is the Midnight Commander filesystem and is served by the secure-mcserv daemon. It has the advantage of serving cryptographically strong secure file transfers and logins.

You can also use glob expressions in filenames for the recursdir and copydir commands. These will be recursively expanded.

--help

Print out detailed help, then exit.

--verbose

Specifies verbose output of file modifications made to mirror. This option can be given multiple times for greater verbosity. Output is written to stdout.

--restore-access

Restore the access times of control with each read.

--access-times

Duplicate even the access times of control.

--always-write

Force rewrite of every file regardless of whether they appear to be identical or not.

--recurs-mode

This is set by default with recursdir. Listed directories are read recursively and nothing is done to them. This option is useful with -C in order to execute shell commands and search for files. Note the system(), exec() and popen() functions available to the C interpretor.

--login-mode

This option is set by default with pslogin. This approximates rlogin(1) using secure-mcserv as a server. It is a secure encrypted connection intended as a replacement to ssh(1). pslogin implies --secure as well. When using this option or when using pslogin, only one path must be present on the command-line. The path is of the form [mc://][username@]hostname[:portnumber][ /path]. Immediately after logging in, a cd /path will be executed at the shell prompt. To determine if a prompt is available, pslogin searches for a #, $ or > character. Should these not be found, pslogin blocks indefinately. To avoid this behaviour, specify /path as exactly /. This will leave you in your login directory. Alternatively, change your shell prompt to include one of these characters.

--copy-mode

This is set by default with copydir. Approximate the behaviour of cp(1) as regards source and destination files. This option implies --keep-files. It overwrites any existing file or directory with the same name as a source file or directory, but does not delete files or directories unnecessarily. Multiple source files or directories may be given. The destination path must be a directory.

--no-erase-directories

With this option, if a target directory conflicts with a source file, and the directory is not empty, then an error message will be printed and the program will abort. This is the default behaviour for copydir.

--erase-directories

With this option, if a target directory conflicts with a source file, then the target directory will be recursively deleted. This is the default behaviour for mirrordir.

--allow-empty-ftp-dirs

Some ftp servers do not produce a . or .. directory. This makes it appear as though you did not have permission to read the directory. This option overrides this by assuming that such completely empty directories are merely empty and do not have errors. If you get unable to open directory: *: Permission denied errors, then you can use this option. This option is enabled by default (see next).

--no-allow-empty-ftp-dirs

Because we now cd to the directory to check for permission to access it, the default behaviour is to allow empty directories.

--only-delete

Do not make any changes to mirror that will cause the total space occupied by mirror to enlarge. This is a useful option when backing up onto a drive that has limited space, where changes have been made to control that might cause mirror to grow larger during the duration of the transfer. Running mirrordir once with this option and then once normally will ensure that the available space is not overrun.

-i, --ignore-next-exclude

This option dictates that the next --exclude- type option must completely overlook those files regardless of their existence or non-existence in the mirror directory. It has the effect of a return value of IGNORE for the --exclude-script option below. This can be used to cause certain files to never be modified, for example if you would like /etc/named.boot to never be modified, use mirrordir /mnt/1 /mnt/2 -i --exclude /mnt/1/etc/named.boot. Note that all paths must be specified in the control directory and not the mirror directory, hence --exclude /mnt/2/etc/named.boot won't work. This has the idiotic behaviour that if you want to avoid erasing a file, you have to have that file present in the control directory, even if it has zero length.

--exclude path

Exclude file or directory path. A large number of excluded paths on the command-line will slow performance. An excluded path will be removed from the mirror tree if it already exists, in the same way as any absent directory or file. Use the --ignore option to ignore keep rather than delete these files.

If you have a long list of files to exclude, use the --exclude-from option.

--exclude-glob glob

Exclude file or directory names matching glob style expression glob. Matches the file without its full path. For shells, the expression should be enclosed in appropriate quotes to prevent substitutions.

--exclude-regexp regex

Exclude full pathnames of files or directories matching regex. For shells, the expression should be enclosed in appropriate quotes to prevent substitutions.

--exclude-script [expr|file]

Cause execution of the script expr for each file before doing anything with that file. The script is a C style statement block, terminating with a `return expression;'. The value of expression can be one of INCLUDE, EXCLUDE, UNKNOWN or IGNORE (see -i above) to explain what is to be done with that file. If expr does not contain a semi-colon (;) then it is assumed to be the name of a file - which is then loaded. In either case, the text is byte compiled into reverse-polish notation for fast execution. This option can be specified multiple times and the scripts will be executed in order until a script returns something other than UNKNOWN. A return value of UNKNOWN allows the remaining --exclude- options on the command-line to take effect.

If you find that the interpretor incorrectly reports errors or segfaults, please report the breaking script to me.

The scripting language itself is a subset of the full C programming language. For example, the following is a valid script:

/* PATH is the full name of the file including
   its path, DIR is the directory, excluding
   the trailing slash (/), CWD is the current
   directory, and depth() returns the number
   of forward slashes (/) less one. */
if (depth (DIR) - depth (CWD) > 3) {
    printf ("%s: excluded\n", PATH);
    return EXCLUDE;
} else
    return INCLUDE;

The scripting language does not support the assignment operator, and hence does not support user defined variables.

The following predefined macros are available. Note that the expansion of the macros applies equally well to directories as to files.

FILE

current file without its path

NAME

file name without its path or extension or trailing dot (.)

EXTENSION

file extension without its leading dot (.)

DIR

directory without file-name or trailing slash (/)

PATH

full file name with path

CWD

current working directory

TIME

current time in seconds

All of the logical, arithmetic and bitwise C operators are supported. These are ( ) >= <= > < != == && || ! - + * / % & ^ and have the same meanings and precedences as in C.

The following further predefined macros are available. Each returns an integer (type long int in C). These are based on a C lstat (or stat if --follow-symlinks is used) on the file. See (2) for a detailed explanation.

stat.st_dev - device stat.st_ino - inode stat.st_mode - permissions stat.st_nlink - number of hard links stat.st_uid - user id of owner stat.st_gid - group id of owner stat.st_rdev - device type stat.st_size - file size in bytes stat.st_blksize - block-size for file-system I/O stat.st_blocks - number of blocks allocate stat.st_atime - time of last accessed in seconds stat.st_mtime - time of last modification in seconds stat.st_ctime - time of creation

The following functions return boolean values:

strncmp(string1, string2, integer);

returns an integer less than, equal to, or greater than zero if string1 is found, respectively, to be less than, to match, or be greater than string2.

glob(glob, string);

returns zero if string matches glob expression glob. Try to use only one glob expression in your code for efficiency of the underlying implementation.

regexp(regexp, string);

returns zero if string matches regular expression regexp. Try to use only one regular expression in your code for efficiency of the underlying implementation.

strstr(string1, string2);

returns the first occurance of string2 in string1 up to the length of string1, or zero if it did not occur.

The following functions also return a boolean value and are analogous to the corresponding macros explained in (2). They return non-zero if the specified condition is true.

S_ISLNK(integer); - file is a sym-link S_ISREG(integer); - file is a regular file S_ISDIR(integer); - file is a directory S_ISCHR(integer); - file is a character device S_ISBLK(integer); - file is a block device S_ISFIFO(integer); - file is an fifo S_ISSOCK(integer); - file is a socket

The following functions manipulate strings:

strcat(string1, string2);

returns the concatenation of string1 with string2. Note that the + operator also concatenates strings.

depth(string);

returns one less than the number of forward slashes (/) in string.

printf(format, ...);

behaves like (3) with an important exception: only long int format specifiers should be used. The behaviour of anything that results in conversion of other than a long int is undefined. For example, use "%ld" instead of "%d". This function prints to stdout.

The following functions do system calls:

system(command);

executes /bin/sh -c command, but unlike the C version, it returns the exit code of the command. I.e. it executes a single line of shell script, command.

exec(argv0, argv1, ...);

executes process argv0 with arguments argv1.... argv0 must be a full path name. This is faster than system because it need not envoke sh.

popen([string, ] shell_command);

like system, but returns the output of shell_command as a string. If string is given, this writes string into the standard input of shell_command and return zero on success.

The following further integer constants are available and are analogous to the macros defined in stat.h and explained in (2).

S_IFMT S_IFSOCK S_IFLNK S_IFREG S_IFBLK S_IFDIR S_IFCHR S_IFIFO S_ISUID S_ISGID S_ISVTX S_IRWXU S_IRUSR S_IWUSR S_IXUSR S_IRWXG S_IRGRP S_IWGRP S_IXGRP S_IRWXO S_IROTH S_IWOTH S_IXOTH

One of the the following constants should be returned using the return keyword, and imply to the caller as follows. If nothing is returned, the return value is assumed to be UNKNOWN.

UNKNOWN

didn't know what to do, continue with other --exclude- options

INCLUDE

include the file

IGNORE

do nothing with the file regardless of its existence or non-existence in the mirror directory

EXCLUDE

consider the file to be non-existent in the control directory and hence must be removed from the mirror directory (this does not override the option --keep-files)

The following perform flow control analogous to C:

The if clause causes statement1, statement2, etc. to be executed if integer is true (i.e. non-zero), or otherwise causes statement1, statement2, etc. to be executed. The else {...} part is optional.

    if (integer) {
        statement1;
        statement2;
        .
        .
        .
    } else {
        statementA;
        statementB;
        .
        .
        .
    }

The return clause gives a value back to mirrordir and causes the script to exit:

    return expression;

The exit function cause mirrordir to exit with the specified exit code.

    exit(integer);

C scripts would typically be used to exclude types of files. Note that this is an excessive implementation of a scripting language, and all the features are not meant for general use. A typical script will, for example, do nothing more than cause device files to be excluded:

if (S_ISSOCK(stat.st_mode) || S_ISFIFO(stat.st_mode)
         || S_ISBLK(stat.st_mode) || S_ISCHR(stat.st_mode)) {
    return EXCLUDE;
} else
    return INCLUDE;

C scripts can also be used to search for files with the --recurs-mode option (same as the recursdir command):

/* removes all core files */
/* This example has been moved to the EXAMPLES section. */

--exclude-from file

Exclude from a list of paths listed in the file file. Empty lines and comment lines (with a # as the first character of the line) are ignored. This list of files is sorted and binary searched, so if you have lots of filenames to exclude, it is best to include them here for performance. This option can be specified multiple times with different files. BUG: the last path in file must end with a newline.

--backup-extension level

Create backups of files before deleting or replacing them. extension is a C style format string e.g. .ORIG.%d (be careful of shell substitutions with the %). level is the highest number of revision to keep. extension is appended to the filename, the oldest file having the highest number.

--backup-outdate sec

Delete backup files older than sec seconds.

--nice num

Be nice to other processes by sleeping occasionally. num is a small integer. --nice causes the process to sleep for as long as it is active, times a factor of num. Hence a value of 1 will (very roughly) double the time it takes to do a copy, and a value of 3 will quadruple the time. This can be used where one would like timed backups to place less load on the CPU. --nice may not be available on your system.

--no-chmod

Normally the permissions of files are set. If you have restricted access and cannot change permissions, than this can be used to disable setting of permissions.

--no-chown

Normally the ownerships of files are set. If you have restricted access and cannot change ownerships, than this can be used to disable setting of ownerships.

--mtime-threshold sec

This is the deviation in mtime that is allowed for a file before it is overwritten. If you have mirrored an ftp site, the mtimes are accurate to within a minute only, thereafter an nfs mirror will cause every file to be copied: you can then use --mtime-threshold 60 to fix this.

--time-offset [[+]|-][H]H[:MM]

This sets the time offset of any vfs (i.e. non-local) directory. For instance, I am 8 hours east of New York, so when I mirror from New York, I use --time-offset -08:00.

--test-only, --dry-run

Do not make any changes. If used with --verbose, this will show what changes would be made. This is an effective way of comparing directory trees. This is untested - i.e. I don't know whether this option actually makes any changes or not!

--skip-symlinks

Symlinks are treated as though they were not read - hence if they are found in the mirror directory then they are deleted.

--keep-files

Don't remove files from mirror, even if they don't exist in control. This makes mirrordir somewhat like cp(1).

--no-hard-links

mirrordir mirrors hard-links properly unless this option is set, in which case hard-links are copied as regular files.

--follow-symlinks

mirrordir mirrors symlinks properly unless this option is set, in which case symlinks are copied as regular files. Useful for mirroring the Debian tree. Note that this follows symlinks in both the control and mirror directories, hence if symlinks already exist in the mirror directory, these will be left as symlinks. Note that --follow-symlinks implies --no-hard-links also.

--strict-locking

Create shared read locks on files as they are read. This prevents clashes especially when copying files from the mail directory - mail programs will be trying to write to those files at the same time as mirrordir is trying to to read from them. This option has no effect with virtual file systems.

--max-bytes [[num[k|M|G]]|num]

Maximum number of bytes to write before giving the message, `filled up all blocks - first file/dir not mirrored: path', which is written to stdout. The remaining files are deleted from mirror, but in their listed order - hence it is possible that the archive will grow greater than num while mirrordir is still running. You should make allowance for this eventuality by making num smaller than the maximum available space. Also note that some filesystems will give an error message `No space left on device' before the device is completely full. You can restart mirrordir with the option --starting-file path and hence continue with this file on another volume. This enables mirrordir to back up across different devices. num can be appended by k, M or G (case insensitive) to specify kilobytes, megabytes or gigabytes respectively. If any single file is greater than this number, then an error message will be given. See also --block-size.

--password password

Specify the password for FTP and mc:// connections. The anonymous password defaults to your login name @ your local machine name. You will be prompted for other login passwords. As usual, you are warned that including passwords in scripts is a security risk. It is much better to put the password in your ~/.netrc file and then not use the --no-netrc option; see man ftp for details.

--password-exact password]

Don't prepend a - to the anonymous password. With ftp anonymous passwords, a - is usually prepended to the password string. I don't know why the Midnight Commander vfs did this, but one user had problems with it, hence this option is given to send the password exactly as specified by password.

--test-login

When using --login-mode or pslogin, you may want to test access non-interactively (eg. in a shell script). To do this you can run pslogin with this option and then check its exit status. secure-mcserv uses this to verify if the user can login on the password server.

--no-warn-first-login

The first time you try a secure connection to a machine, no public key exists on the local machine. There is hence no security against a man-in-the-middle attack. A warning to this effect is printed and the user is prompted if they want to continue. This option disables this warning and goes ahead regardless.

--read-password-from-stdin

Instead of specifying the password on the command line, you can write it into the command via stdin. This is not the same as typing the invisible password as one normally does, because it can be used even if there is no terminal. This is useful for usage within other programs using, say (3). secure-mcserv uses this to verify if the user can login on the password server.

--netrc

Scan the ~/.netrc file. By default, this option is on.

--no-netrc

Turn off reading of ~/.netrc file.

--proxy-host host

Set the proxy for ftp downloads. Don't know how or if this works. Consult the mc(1) man page for info in proxy support.

--secure

(This feature is BETA) I have implemented a secure socket layer for mirrordir. It is enabled with this option and applies to connections to secure-mcserv (i.e. using mc:// type URLs). The secure socket library consists of libdiffie.a and a header file diffie-socket.h. If you include diffie-socket.h after you include sys/socket.h you can recompile the resulting program with all normal sockets turned into secure sockets. (This applies to any program that uses Unix socket calls, but is untested). The first service supporting this is secure-mcserv which compiles and installs by default. Hence you can use the mc:// file system with the --secure option, provided the remote host is running secure-mcserv (try secure-mcserv -h to for help). --secure uses a stream cipher (much more secure and faster than block ciphers like DES) with discrete logarithm key exchanges with public key server authentication (Diffie-Hellman and p-NEW schemes). For the full details, see the file diffie-socket.h in the source distribution. The default key size is 512 bits. Note that if you are using gcc you should compile mirrordir with the options -O3 -fomit-frame-pointer -s -Wall to speed key generation.

--key-size bits

The default key size is 512 bits. Sizes require primes to be generated and compiled into the file field.c, hence only those primes listed in field.c are supported, which are at the moment 512, 768, 1024 and 1536. 768 is the recommended value for medium security if you have slow computer. Otherwise 1536 is not an unreasonably paranoid size for long term security. 512 can be used if you are not worried about large corporations, well connected hackers, or governments snooping your connections. Note that the the stream cipher has a length of bits/2, which means that you have more chance of being hit by a meteor while cashing in at the state lottery, than cracking it. Be aware that a person can sniff your connection and then keep the output for as long as it takes to break it. In twenty years time a 1536 bit key will be considered small. Note also that the discrete logarithm problem (used here) is considered to be more difficult to solve than the factorisation problem, hence the key is effectively a little larger than for RSA. This is all my (rather uneducated) opinion.

--download-scripts

Mirrordir comes in two versions, an International version and a US version (see --version). The US version contains no encryption code whatsoever. Instead it downloads the needed algorithms from encrypt.obsidian.co.za (in South Africa). These are written in a fast, native, C-style interpreted language. There are four scripts: one for the server Diffie-Hellman key exchange, one for the client Diffie-Hellman key exchange, one for initialising the stream cipher and one for actually performing the encryption using the stream cipher. Mirrordir automatically downloads these scripts when you try to use the security features. The option --download-scripts however can be used to force a download at any time. The International version contains compiled-in versions of the stream cipher, hence only two scripts are used, which need never be downloaded since they come with the distribution. No speed penalty is incurred by having the Diffie-Hellman exchanges in scripts, however, you will notice a significant speed different when using the encryption scripts compared to compiled-in encryption.

--version

Prints out the version number as well as whether this is an International or US version of mirrordir. See --download-scripts.

-z, --gzip

Enable compression for mc:// connections. This actually envokes a compressed socket layer at a lower level than the encryption. Compression uses the libz library of gzip(1). The degree of compression is dynamically set to minimise transfer time. It will drop to no compression for fast ethernet connections, and will go to maximum compression for slow modem connections. The algorithm adjusts the compression level so that the time for a TCP write call is between 2% and 5% of the time for the same amount of data to be deflated (i.e. compressed).

--gzip-backups

Backups are usually just a copy of the file. With this option they are compressed and a .gz is added to the default extension. Note that if you specify your own extension using --backup-extension, then it must have a .gz at the end for compression to work.

--case-insensitive, --for-Robert-Seese

Ignores case in comparison of filenames and linknames. This option is useful when interacting with certain brain-dead operating systems. I am not sure if this option behaves correctly under all circumstances.

--to-lower

--to-upper

Convert all new filenames to upper or lower case. When used with --case-insensitive, it will apply only when creating new files. When used without --case-insensitive, all files, existing or not, will be converted to uppercase in the inefficient way of deleting the old file and then copying the new file. This option is useful when interacting with certain brain-dead operating systems. I am not sure if this options behave correctly under all circumstances.

--no-use-passive-connections

If you get an error message could not setup passive mode it is likely because you need to enable this option. I don't really understand what `passive' means, so don't ask me.

--tar-file filename

This is to be used only with recursdir. It creates a tar archive in the same format as GNU tar(1) and stores it in filename. Leading special prefixes and leading slashes are removed, i.e. ftp://machine/dir/file becomes dir/file. If the first character of the filename is a | character, the remaining text is considered as a command through which the output is to be piped. Hence a gzipped archive can be created for example with recursdir ftp://machine/dir --tar-file '| gzip -d > foo.tar.gz'

--tar-block-size N

Sets the block size for tar output to 512 * N. These are the units in which data is written to the archive. The default is 20. This is significant only when writing to block devices. This must not be confused with --block-size.

--block-size bytes

The default block size is 1024 bytes. File sizes are rounded up to the nearest block when calculating the total number of blocks consumed. If the actual block size is greater than assumed then it is possible for fewer blocks to be counted than are actually consumed during writing. Hence it is essential to specify a block size greater than or equal to the actual block size when using the option --max-bytes.

--strict-mtimes

When copying regular files, mirrordir normally only overwrites the mirror file if it is older than the control file. This option forces a copy if there is any discrepancy at all in the modified times of the files.

--no-mtimes

Copy files only if their sizes differ. Ignore the file modification time.

--ignore-size

Copy files depending on mtime, but ignore size differences.

--starting-file path

path can be a file or directory. Until path is read, files or directories will be processed as though they were excluded files (i.e. if they exist in mirror they will be deleted). Directories that contain path will be created if they don't exist. After path is read, files are mirrored as usual. path itself will also be mirrored. If path or any of its subdirectories are not present, then mirrordir exits immediately. This is the only time mirrordir exits prematurely. This is to prevent the entire filesystem from being erased on account of path never being found.

    mirrordir --verbose \
        ftp://lava.obsidian.co.za/pub/mirrordir \
            /home/mirrordir

    mirrordir --verbose /home/mirrordir \
        ftp://psheer@lava.obsidian.co.za/home/ftp/pub/mirrordir

You will notice (with --verbose) that mirrordir repeatedly tries to set modification and access times over ftp. I am going to leave these messages in to remind users of the limitations of the VFS type in use. These attempts do not appreciably detract from performance. You can use mc:// instead although performance is poor when uploading with this protocol. Downloading is always preferable to uploading.

In general you should NOT have an ftp upload in your cron jobs or use ftp upload to keep directories in sync. Always download from the other end when trying to keep directories in sync. Uploading is good for once off uploads only.