Creating a New Site

Select “Sites” from the left menu, then click “New”. This launches a wizard that takes you through the steps to create a new Site.

Managing Existing Sites

To change, update, or delete existing sites, browse to the Sites menu and click on a site to select it. Then right click and select an option, or click the “Delete” or “Edit” text inside the panel. “Edit” will open up the Site configuration where you can change location and other advanced settings.

Import/Export Sites

To import or export your Sites to an XML file, navigate to the Sites panel and click the “Import” or “Export” button as appropriate. Then select or enter a valid filename.

Site ID

Enter text to create a unique identifier. Choose something recognizable such as “New York Post-Processing”

Host

The location of the FileCatalyst Server (or FTP server) which HotFolder will connect to. Specify the the hostname or IP address (IPv4) of the server.

Port

The listener port of the remote FileCatalyst Server.

Enable SSL

Enable this option to allow HotFolder to establish an SSL-secured command channel (note: SSL must also be enabled on the server side). This overrides the default setting of using a cleartext channel for commands.

Enable Authentication

By default, HotFolder will use the “anonymous” account. To use an account provided by the administrator, you must select the Enable Authentication checkbox and fill out the username and password fields.

Servlet URL

An HTTP Servlet allows the FileCatalyst HotFolder to connect using an HTTP-based connection when UDP or FTP are not available. This functionality is an add-on feature licensed on the FileCatalyst Server. HTTP-based transfers cannot be accelerated with the FileCatalyst protocol, but two key benefits remain:

• Virtually guaranteed connectivity between endpoints: users do not need to configure their environment in order to transfer files.
• The reliability, security, and file transfer management capabilities of FileCatalyst technology are intact.

The value placed here should take the following format: http://serverIPorHostname:12480/servlet/ftpservlet

NOTE: The HotFolder will be making a connection to an FTP Servlet and the FTP Servlet will make a standard FTP connection to the server. As a result you will have differences in reported bandwidth between the HotFolder and the Server as there is a degree of separation between the two and they are indeed transferring at different rates. This also means that while data flowing to the FTP Servlet may indeed respect bandwidth restrictions, the FTP Servlet has no such built in throttling.

Note: Tab "Network / Bandwidth Characteristics" is available by enabling "Advanced View" on the HotFolder.

Bandwidth Characteristics

Upstream

The maximum possible upload speed between client and server. You may also change your maximum speed in the Bandwidth Scheduler; HotFolder will use the lesser of the two values.

Downstream

The maximum possible download speed between client and server. As with Upstream, Bandwidth Scheduler may also be used to set speeds; HotFolder will use the lesser of the two values.

Detect Bandwidth Characteristics

This utility attempts to measure the maximum possible upstream and downstream between client and server. The test may take several minutes and should be performed on a clean network to ensure accuracy.

Network Characteristics

Path MTU

The largest packet that may be transmitted without being fragmented into smaller packets. For example, if a network has an MTU of 1500 (typical for most networks) and the application transmits a packet of 3000 bytes, the packet would be broken into two smaller packets of 1500 bytes, transmitted, and recombined at the destination.

During an internet transmission, a packet may traverse several individual computer networks, all of which may have different MTU values. The Path MTU is the lowest MTU value on all networks between source and destination.

Average RTT

The average round trip time of a packet. RTT indicates the time required for a packet to be sent and acknowledgement to be received. In environments with high RTT, FileCatalyst will greatly outperform FTP.

Average Packet Loss

The number of packets that are lost in the network due to hardware/routing issues, oversized packets, or network congestion. Packet loss affects transfer speed; it is recommended that with higher packet loss, you set a lower packet size in the “Site Tuning” tab.

Detect Network Characteristics

This auto-detection utility will automatically determine the Path MTU, and run a test to obtain values for Average RTT and Packet Loss optimal for your connection. These values can then be used to optimize your unit and block size, as well as number of transmit threads being opened at any one time.

Note: Tab "Site Tuning" is available by enabling "Advanced View" on the HotFolder.

UDP Transfer Tuning

Block Size

Determines how much data will be sent by each individual transmitter thread to the receiver before waiting for acknowledgment. For example, if the block size is 10MB (10240000 bytes), then 10MB of data will be sent before proceeding to the next 10MB of data. The transmitter thread must receive an acknowledgment that a block is received before proceeding to the next block. The larger the block size, the better the throughput (due to fewer acknowledgments). Since block size works best when it is divisible by unit size, we will adjust this value to a value divisible by unit size, close the user defined value

Packet Size

The maximum UDP packet size which client applications will send to the server. This should be less than the MTU of your network (usually 1500).

Number of Sender Threads

To improve performance, FileCatalyst clients can begin sending new blocks of data before the previous block has been acknowledged. Sender Threads defines how many blocks can be started before waiting for all previous blocks to be acknowledged. This improves performance as it removes downtime during the acknowledgment phase. Memory consumption on the server and the client increases as this value is set higher. The exact amount of memory used is the product of the number of threads and the block size. Since overlap can be quite small (sometimes less than 500ms) the memory usage may appear to spike and drop immediately following acknowledgment.

Number of UDP Sender Sockets

If the bandwidth is above 500 Mbps and the supporting disk IO can support these speeds, it is recommended that client application specify UDP sender streams to be 2 or greater. This allows more data to be pushed out of the system, and provides higher maximum speeds.

Number of UDP Receiver Sockets

If the bandwidth is above 4 Gbps and the supporting disk IO can support these speeds, it is recommended that client application specify UDP receiver streams to be 2 or greater. This allows more data to be pulled from the system, and provides higher maximum speeds.

Optimize Based on Network Characteristics

This optimization tool takes the values entered or auto detected and calculates the optimal settings for your connection.

Note: Optimization is based on definitions in both the "Network / Bandwidth Characteristics" tab and the bandwidth scheduler. Before automatically optimizing, your connection speed should be properly configured in both.

TCP Transfer Tuning

Number of TCP Streams

In TCP mode, FileCatalyst can open one or more concurrent connections. This can resolve issues where TCP is hindered by high RTT. If the actual throughput of standard FTP is about 1/2 of available bandwidth, setting this value to 2 could optimize the connection. If standard FTP is only achieving 1/10th of actual available bandwidth, then setting this value to 10 could optimize the connection. Adding concurrent streams increases CPU consumption.

Network Timeout Settings

Connect Timeout

The length of time that HotFolder will persist in connecting to a Site. The default is 30 seconds; however, with extremely high RTT or packet loss this may need to be increased.

Read Timeout

The length of time that HotFolder will persist in reading a response from a Site after a command has been executed. The default is 30 seconds; however, with extremely high RTT or packet loss this may need to be increased. The timeout value applies only to commands that are expected to reply quickly.

Extended Read Timeout

The length of time HotFolder will persist in reading a response from a FileCatalyst Server after a command that is expected to reply more slowly. Examples of such commands would be MD5 on a very large file, or generating the deltas for a very large file. In cases where these commands take longer than the default 30 minutes, this value should be increased. The default is 1200 seconds or 30 minutes. The minimum value that may be applied for the Extended Read Timeout is 600 seconds, or 10 minutes.

Max Retries

The number of times HotFolder will attempt to reconnect to the server if the network connection fails.

Wait Retry

The length of time HotFolder will wait between retries if the network connection fails.

*Note* The browse functionality is unsupported in Linux.

Creating a New HotFolder

To create a new HotFolder location, browse to the HotFolders menu and click “New”. This will initialize the HotFolder wizard, and take you through the steps to create a new HotFolder on your system.

Managing Existing HotFolders

To change, update, or delete existing HotFolders, browse to the HotFolders menu and click on the HotFolder to select it. Then right click and select an option, or click the “Delete” or “Edit” text inside the panel. “Edit” button will open HotFolder configuration where you can change paths and names.

Import/Export HotFolders

To import or export your HotFolders to an XML file, browse to the HotFolder panel and click the “Import” or “Export” button as appropriate. Then select or enter a valid filename.

HotFolder ID

A unique ID used to define and identify this HotFolder.

HotFolder Location

The full directory path for the HotFolder. Edit the field or use the “Browse” button to select a valid path.

Disk Write IO Test

Performs a sequential write operation to the HotFolder directory. Specifying a small file size will likely demonstrate only system/hardware cache availability. Enter a larger file size to test and show true disk write speeds.

You may set a maximum timeout value (in seconds) to ensure that the IO test stops after a specified time. If the test is not completed, the last speed before stopping will be used and all IO resources will be released.

Note: The test will create a temporary file on the disk. Ensure enough space is available.

Task Schedule

The task schedule displays all of the current tasks in FileCatalyst HotFolder. It shows the HotFolder, Server, and direction (upload or download), as well as the Status (Idle, Transferring, Queued), and the date and time of the next execution.

Creating a New Task

You must have defined at least one Site and one HotFolder prior to adding a task. To create a new task from the scheduler menu, click the “Add New Task” button. You may follow the task wizard until completion, or click “Finish” partway through and continue creating the task manually. Once a task is configured, perform a test by right clicking on the task and selecting “Test Task Settings”.

Managing Existing Tasks

To edit, delete, or modify an existing task (including advanced settings not found in the initial creation wizard), browse to the scheduler menu and select the task you wish to work with. From there, either click the “edit” or “delete” buttons, or right click and select from the options on the sub-menu that appears. If a single task is selected, you may use this menu to edit, duplicate, delete, test, execute, cancel, disable, enable or view the activity of the selected task. If multiple tasks are selected, you may use the menu to duplicate, delete, execute, cancel, enable, or disable all of the tasks that are currently selected.

Note: Duplicated tasks are created disabled by default.

Execute a Task

To execute a task, either highlight a task and click the “Execute” button or right click the task and select “Execute”.

Cancel a Task

To cancel a task, either highlight a task and click the “Cancel” button or right click the task and select “Cancel”.

View Task Activity

To view a tasks activity, either highlight a task and click the “Activity” button or right click a task and select “View Activity”. A window will open displaying the current file being transferred, activity graph and logs for that specific transfer.

Pause/Resume Scheduler

Clicking “Pause Scheduler” will prevent any further scheduled task execution. However, any tasks in progress will complete their execution. You may resume the scheduler by clicking on the “Resume Scheduler” button.

Concurrent Tasks

The HotFolder can be configured to support execution of multiple tasks at a time. To do this, check the “Allow Concurrent Tasks” checkbox. To disable execution of concurrent tasks, uncheck the checkbox.

Import/Export Tasks

To import or export your Tasks to an XML file, browse to the Scheduler panel and click the appropriate import or export button. You will have to select or enter a file in which to load from or save the Tasks to.

Task Name

The name of the task.

HotFolder

The HotFolder used by this task.

Site

The Site used by this task

Transfer Direction

Specifies whether to upload from this HotFolder to the server, or download files from the server into this HotFolder.

Frequency Options

A set of five radio buttons allows you to select the frequency of this task:

User Triggered

The default "user triggered" option requires the user to open the HotFolder interface, select a task, and click "execute". This default prevents accidental execution of tasks while the user is still configuring their HotFolder, but is typically changed to one of the scheduling options below.

Once

Selecting this option will enable corresponding fields allowing you to specify a date and time for a single execution. The task remains in the scheduler, allowing you to specify a new date or to manually execute.

Every minute

This is the most common choice. The scheduled task will check every minute to see if a new file transfer is required.

Always On + File System Events

As soon as the task completes, it will repeat. Selecting this option will automatically check the “Monitor File System Events” box at the bottom of the page.

Note 1: The "Always On + File System Events" option is made available by enabling "Advanced View" on the HotFolder

NOTE 2: Using "File System Events" on network drives (Samba, CIFS, NFS, ATP) is not recommended. Any task transferring to or from mounted network drives should instead use a normal transfer schedule (ie: once a minute).

Scheduled Execution

Selecting this option enables a range of fields to create a repeated task. As the scheduled task is created, a plain-language representation of the task is updated at the top of the section.

After selecting the “Day(s)” for execution, you must select an interval. This may be:

• Once, at a particular time of day.
• Every X number of hours, at the specified minute past the hour.
• Every X number of minutes.
• Every X number of seconds.

Monitor File System Events

When this option is enabled, the task will first complete transfer of all files, and will then enter a “monitoring” state. When in the monitoring state, HotFolder will watch the file system at the source (local folder on uploads, and server folder on downloads) for events such as file creation, modification, deletion, and rename. When these events occur, the task will immediately transfer, delete, or rename the affected file(s).

Once the task enters monitoring mode, it will stay in this mode indefinitely or until it is canceled or an error occurs. The task is still considered to be running while in this state.

This option is useful when dealing with very large file sets that must be kept in sync. For a scheduled task, recursive listings, comparisons, and filtering determine which files have changed. On large file sets this process may takes several minutes and consume resources. However, with file system event monitoring enabled, recursive listings are not needed and resources will only be used when a file is changed.

Note on Monitoring Network storage: In many cases FileCatalyst is used to monitor directories that do not reside on the same physical machine but rather on a network attached storage. Depending on the OS and protocols used, the FileCatalyst software may not be notified of events that are triggered at a network location. For example, if a Linux machine is running FileCatalyst Server and has a user home directory pointing at a mounted directory (i.e. /mnt/network_dir), and changes are made to that directory from yet another machine pointing at that same directory, the events may not be registered.

Note: The "Monitor File System Events" checkbox is made available by enabling "Advanced View" on the HotFolder

Note: Because of how the multi-client manages itself, a multi-client transfer will identify as idle when Monitoring.

Protocol

This option allows you to specify the transfer mode you wish to use to connect and transfer files to/from the server. Options are UDP, TCP, HTTP (when Tunneling Servlet is enabled), and Auto Detect. By selecting Auto Detect, the HotFolder will cascade from UDP to multi-stream TCP to single-stream TCP and finally to HTTP in order to find the optimal mode.

Congestion Control

When Congestion Control is enabled, FileCatalyst will attempt to dynamically modify its transmission rate in accordance with changing network conditions. This is useful if the end to end link speed is not known, or if the network is also being used by other applications with which FileCatalyst would otherwise interfere. The congestion control mechanism is based on the RTT between the FileCatalyst Client and FileCatalyst Server machines. FileCatalyst will measure and average RTT value when no transfers are taking place, and use this as a benchmark to compare RTT values when the transfer starts and for the entire duration of the transfer. FileCatalyst will then increase/decrease speed depending on whether the current RTT is within a certain range above or below the initial RTT.

Start Rate

The rate at which FileCatalyst will begin transmitting data. It will then attempt to speed up the transmission rate until it either hits congestion (the RTT becomes too high compared to the initial RTT), or it reaches the target rate specified in the bandwidth scheduler.

Three options are available:

• Use Target: Attempt to transfer at maximum speed allocated by the task. Will slow down if congestion is detected, but initial rate may adversely effect network link if it cannot sustain the transfer speed.
• Specify: Manually specify a start rate for transfer speed. Recommend a maximum of 1/10th of the known line speed if this is not a dedicated line. You may also run a quick auto-detection test to determine line speed. Rates higher than the value of the bandwidth scheduler are ignored.
• Auto: Have the transfer automatically launch an auto-detect when the task starts. Best solution for single file transfers. Not recommended for multi-file session transfers, as each client will attempt to auto-detect transfer speed.

Congestion Control Strategy

RTT-based Strategy

RTT-based strategy works by establishing a baseline average RTT before the data starts to flow. Once the transmission begins, RTT is monitored continuously. While the RTT stays within a certain range of the baseline RTT, the speed of the transfer will be increased. Once the RTT begins to go above a certain range, the speed is decreased. How much the RTT is allowed to spike above the baseline average is controlled by the Congestion Control aggression setting FileCatalyst provides.

This type of congestion control works well on wireless or satellite links where there is packet loss from other sources besides congestion. TCP will slow down, for example, when a packet is lost due to interference or being too far from a cellular tower. When FileCatalyst is using the RTT based congestion control it ignores individual packet losses and focuses only on RTT. For these scenarios, FileCatalyst continues to maintain high speeds through the kinds of packet loss “hiccups” that would trigger decreases speeds under TCP.

Loss-based Strategy

There are circumstances in which this RTT based may not work properly. For example, when a router's queue is very small, the RTT may never spike when there is congestion. The RTT will remain low, and therefore FileCatalyst would continue to increase its transmission rate even when there is congestion present. For these scenarios, the only way to detect the congestion is using a packet loss approach. As outlined previously, packet loss may come from other sources besides congestion, so this mode is best used on terrestrial or land based networks where all packet loss is due to real congestion.

Loss-based congestion control mode reacts to packet loss by slowing down (just like TCP); however, FileCatalyst is not nearly as aggressive as TCP. The primary reason for using a UDP-based file transfer solution like FileCatalyst is because TCP is so aggressive in its congestion avoidance that it often ends up under-utilizing your link. The FileCatalyst loss-based congestion control algorithm was designed such that it is able to maximize link utilization while still avoiding congestion. Like the RTT-based congestion control, the loss-based mode can be tuned to be more or less aggressive. More aggressive settings allow for a higher percentage of packet loss before slowing down, while for more passive settings the opposite is true.

Rate-based Strategy (Labs)

There are circumstances in which both Loss and RTT will not work properly. If UDP packets are buffered in stacks instead of queues (last in first out vs. first in first out), this interferes with the RTT strategy as the RTT can skip along the top of the stack buffers since it is last in first out. Stack buffering interferes with the Loss-based strategy as loss detection is based on packet ordering and stacks reverse packet order.

Rate-based congestion control mode compares the rate we are sending packets to the rate we are receiving them. If we are receiving slower than we are sending, then we know that the network, at that time, cannot handle the current send rate, and we slow down.

Aggressiveness

This setting allows the threshold (or range) that is used when comparing the current RTT to the initial RTT to be increased or decreased. If aggression is increased, the current RTT will be allowed to increase to a higher level before it is considered to be congestion. Conversely, if the aggression is lowered, smaller increases in RTT will be considered as congestion and transmission rates will slow down. In this way, the congestion control may be tuned to the sensitivity of the network on which the transfers are being performed.

Auto-Resumes

With this option enabled, FileCatalyst HotFolder will automatically recover from failed network connections. This setting takes precedence over the transfer files with unique name option found within the incremental setting. If a file can be resumed and auto-resumes are enabled, the destination file will be resumed and a unique file will not be created at the destination location.

Verify File Integrity After Transfer

During all UDP transfers, as blocks of data are transferred, they are verified using MD5 against the same block in the source file. This is done on-the-fly, and incurs no delay after a transfer completes. This provides a very high level of protection against corruption. Note that FTP/TCP mode transfers are verified on-the-fly as well, but it is built into the TCP protocol. FileCatalyst also provides additional options for further protection against corruption by performing a full MD5 checksum after each transfer.

By checking the "Verify File Integrity After Transfer" option, integrity of the transferred file will be verified byte by byte on storage using MD5 after the transfer is complete. This option can be used in one of three modes:

Full

This will perform a byte by byte comparison of the entire file when the transfer is complete. Although this consumes less CPU on an ongoing basis, there is a delay between files being transferred as the checksum is calculated. If the files are large, it may cause delays of several seconds or even minutes depending on the speed of the I/O on the sending and receiving machines.

Concurrent

This will open up a separate session to perform checksums without interrupting the file transfer. For very fast transfers, disk I/O bottlenecks (competing between ongoing transfers and MD5 checksums) may result in lowered throughput.

Partial

Partial MD5 verification occurs after the file is completed, however it only compares certain portions of the file, and only at logical intervals that coincide with the transfer characteristics. This vastly decreases the time to perform the MD5 check. Note however that this check does not 100% guarantee that file. It is a trade off between speed and a high likelihood that the file is not corrupt.

Note: Partial MD5s are not available against servers that are older than 3.7. If partial MD5s are still selected at the time of transfer, then full MD5s will be used instead.

Force File Ownership (Download Only)

By selecting this option on a download task, all files transferred will have their operating system user ownership set to the value specified.

Note 1: On Linux or Mac OSX, an additional field allows configuring group ownership.

Note 2: If the Maintain UID/GID system property is enabled, the user and group data will be updated to accept user and group identifiers. Please note that this functionality is only on Linux and Mac OSX operating systems.

Note 3: If Keep File Permissions is enabled when Force File Ownership is enabled, the "Keep File Permission" option is automatically deselected.

Keep File Modification Timestamp

If this item is selected, the modification date and time will be preserved when files are transferred to the server.

When multi-client is enabled, folders are also maintained. This behaviour is different from our legacy single client transfers. If you desire the legacy behaviour, it can be reverted via setting this system property: unlimited.fc.legacy_timestamp

Transfer with Temporary Name

If this option is enabled, FileCatalyst will always upload or download files using a temporary name, and rename it back to the original name when the transfer is complete and verified. FileCatalyst Server will not report these temporary files when other HotFolder or FTP clients request file lists. This prevents them from downloading partial files. You may choose to either prepend the ".fctmp." prefix, or append the ".tmp" suffix to the files until they are finished transferring.

Task Priority

This allows you to specify the priority of a task. By default, bandwidth is split evenly between all running tasks. Setting the priority of a task will increase/decrease the weight of a task when bandwidth allocation is performed.

Enable Multi-Client (multiple concurrent file transfers)

HotFolder allows a single transfer task to utilize more than one transfer connection in order to push files across concurrently.

Useful when transferring dynamic file sets (where data is being added in real-time to multiple files in a directory). This feature, when combined with progressive transfers, allows multiple growth files (log files, video transcoding, etc) to be sent from one directory.

Note that as you are utilizing more than one transfer connection to a server, this may potentially impact server-side license restrictions. It will also utilize more resources (memory, ports) as expected when more work is being done at a time.

Multi-Client does not currently support FCWeb. Selecting specifying an FCWeb server will automatically execute the task as a Single Client Transfer.

Automatically rescan and add new files to the running task

By default, the HotFolder scheduler will not re-execute a task if it is currently running (only one instance of the same task is allowed to run at a time).

Because multi-file sessions are ideally suited for transfers of growth files (ie: live streaming video) which may take an extended period of time to complete (2+ hour event), rescheduling allows you to rescan a directory even if the original transfer has not yet completed (live video event still on-going). New static files (or growth files) will get picked up by the task at scheduled intervals.

NOTE: Transfer cache is enabled and locked when this feature is used. Files will be sent only once when using this feature. Transfer cache can be manually cleared in the Fileset tab.

NOTE: When transferring a progressive file that grows much slower than our transfer rate, a situation occurs where we are always waiting for a file to grow before sending the next chunk. We call this chasing the tail. When this is happening a progressive file might not grow for a long enough time that we think it has finished growing and consider the transfer complete. With reschedule enabled, the next scan will pick up the file if it has indeed grown. Since we cannot know if the file has just grown, been replaced, or been edited, we need to treat it as a new transfer. This only affects our statistics displayed as we will increment the amount of data and number of files transferred. However this can be corrected increasing the time a progressive transfer will wait for the file to grow, see progressive transfers

Enable Auto Archiving of small files.

This option will group together small files (5MB or less in size) archiving them and sending them over as a single transfer. Use this feature if your transfers will contain many small files. The maximum size for an individual archive is 100MB, after which a new one will be created. Previous archives are transferred and extracted while new ones are being created.

Multi-Client changes from single client and limitations

Due to the nature of how parallel transferring works, it is highly recommended to use the AutoArchive of Small Files option in MultiManager. This will keep the overhead of HTTP POSTs to a minimum during a transfer with a larger data set.

Changes from single client

Multi-Client has an added layer of redundancy, if an individual client fails a transfer, that transfer is pushed back onto an internal transfer queue to be attempted by a different client a set number of times before actually failing. This means that in cases where Single Client failed consistently, multi-client may succeed. Multi-Client can use auto archiving, incremental and compression simultaneously. Small files will still be auto archived and use the compression settings, ignoring the incremental while the larger, non archived files, will use incremental and compression. Normally archiving and incrementals are mutually exclusive for single client transfers. Note that while multi-client is enabled, some compression options in the Data Minimization pane will be disabled as they are handled by the auto archiving algorithms.

Limitations

Multi-Client transfers are not compatible with FCWeb so multi-client settings are ignored for transfers to an FCWeb site.

Show Activity Details When Transfer Begins

When set, the Activity window will automatically be seen when the task is launched.

Keep File Permissions (OSX/Linux/Solaris Only)

When this item is checked, file permissions will be saved when files are transferred.

Enable Incremental

This option limits transfer to files that are new or have been modified. This works in conjunction with “Do nothing with source files” option on the Post Task tab to allow you to treat the HotFolder as a working directory with only your modified files being transferred.

Transfer Entire File

Transfer the entire file if the remote file and the source file do not match.

Transfer File Deltas

Transfer the portions of the file that have changed (the "deltas"). These changes are then merged in with the file on the target to recreate the source file.

Transfer Entire File with Unique Name

Transfer the entire file, but renamed with a unique identifier. The unique name will use the following pattern:

<filename>_X.<extension>

Where X is a unique integer value.

NOTE:: If auto-resumes are enabled within the current task, the transfer file with unique name incremental option will only occur if the destination file cannot be successfully resumed.

Transfer File Deltas with Unique Name

This option will only transfer the portions of the file that have changed. These changes are then merged in with the file on the target to recreate the source file, however it will be renamed such that is has a unique name. The unique name will use the following pattern:

<filename>_X.<extension>

Where X is a unique integer value.

NOTE: Incremental delta transfers are bypassed under circumstances where it is more beneficial to sending the file directly than through a delta process. This includes:

• If either files (source or destination) is smaller than 200KB in size.
• If the destination file is less than the square root of the source file size.
• If the minimum delta file will be larger than transferring the entire file.

When Enable Incremental is enabled, the user also has the option to remove MD5 checks and only check the size of the file.

There are possible side effects using deltas in conjunction with Progressive transfers noted in this knowledge base article here.

Enable Compression

Compressing files as they are being sent to the server may reduce transfer time. However, if all files are already in compressed formats, it will only serve to add additional overhead to the transfer. Once you understand the following options, you will know which method (if any) is right for your situation.

Two compression methods are available: Zip Deflater and LMZA.

Zip Deflator

Zip Deflator is the standard compression library utilized by FileCatalyst, and is used for on-the-fly compression and for single zip archive (bundles multiple files together before transferring).

LMZA

LMZA is an additional compression method available for on-the-fly compression (single archive is not supported), and allows a very high level of compression. The algorithm is very CPU and memory intensive for the sender. Profiling the transfer via HotFolder or Server visualization window and configuration of threads (block sender threads for transmitter, writer threads for receiver) may be required for achieving high speed transfers.

Compress files into single archive

When sending several small files, there is an overhead to initiate the transfer of each file. With a high RTT, this means that far more time is spent “setting up” the transfer compared to actual data transmission. Furthermore, due to small file sizes, the transfer task never reaches full speed.

With the “compress files into single archive” option enabled, HotFolder will put all the files into a single package, transfer to the server, then extract it. This eliminates set-up and teardown, and allows the transfer task to reach its maximum speed.

Note: Single archive transfers are not compatible with Multi-Client transfers and will be disabled if Multi-Client transfers are enabled.

Note: If progressive transfers are disabled while enabling archiving, the archive file (tar/zip) must first be created locally before being transmitted. This may cause a noticeable delay before the transfer begins, but the total time for the transfer should still be significantly less.

Note: The maximum size of an individual zip file is 4GB, as is the maximum size of any single file contained in the zip file. For file sets exceeding 4GB in total size, please use the “Archive size limit” feature (described in the next section) to limit the size of each zip file created to under 4GB. If the zip file size is over 4GB due to no limit or a single file over 4GB in size, the zip file will not be able to be unzipped and will cause an error.

Archive size limit

This feature will limit the size of the zip archives to the specified value (in MB) when possible. If a file set is too large to fit in a single zip of the specified size, multiple zip archives will be created. These zip archives are automatically extracted and cleaned up as they are transferred. At most 5 archives will be created at one time. Additional archives are generated only after the oldest file is deleted. This helps reduce disk space required. The following values may be set in fchf.conf to configure the autozip.

FC.hotfolder.config.autoarchive.filesize_threshold=5242880

This value configures the maximum file size to be zipped. It defaults to 5MB.

FC.hotfolder.config.autoarchive.zipfilesize_limit=104857600

This value configures the maximum file for a zip chunk. It defaults to 100MB.

Compression Level Settings

This slider will allow you to select the level of compression to be applied when compression is enabled. Depending on the types of files and bandwidth constraints, it may be desirable to use more or less compression. For example, on a high bandwidth connection where volume of data transfer is not measured or is otherwise not a concern, it may be faster to send the data either uncompressed or with low compression. However, on a low bandwidth satellite network it may be more optimal to use maximum compression.

Exclude Files

Any files in the HotFolder directory or server matching the given expression will be excluded from compression. You may specify multiple file filters delimited by a comma. This option also supports wildcards (*). For example, if you wish to ignore all Zip files, you would set this value to "*.zip". If you wish to ignore all Zip files, and MP3 files, you would set this field to "*.zip, *.mp3".

Note: The compression option is only available for either UDP or TCP based transfers. HTTP based transfers currently do not support this option.

Disabled

This option turns off progressive transfers. Files will not continue to transfer as they grow.

Basic Progressive

Files will transfer as they grow. This is the most basic of progressive features. This is our former "Enable Progressives" feature.

Advanced Progressive

If Advanced Progressive is enabled, one of the four sub-options (described below) must be selected. If one of these sub-options is not selected, the transfer will default to Basic Progressive. Note that in single client transfers only the "Transfer these files as they grow" sub-option will be permitted, as the other options are multi-client specific.

For each selected sub-option, an inclusive filter must be specified (e.g. "*.mxf" for mxf files) to indicate which file types the functionality should be applied to. Files that do not match this filter will be transferred as per Basic Progressive. A wait time (in seconds) may also be specified for each sub-option to modify how long the client waits to check if a file has grown.

Transfer these files as they grow:

These transfers will check for files matching this particular filter within the specified number of seconds. If no changes to the file have occurred within the set time, the file growth will be considered complete. This option is useful for files that grow in intervals with pauses, or if a hard drive is heavily taxed and the actual growth of the file is slower than normal.

Wait for file growth to cease:

These transfers will check for files matching this particular filter within the specified number of seconds. If no changes to the file have occurred within the set time, the file growth will be considered complete. The transfer will only happen after the file growth is considered complete. This option ensures only complete files are being transferred and not ones that are still growing.

Re-Transfer Headers/Footers when complete:

These transfers will wait for a specified amount of time, then the header and footer bytes (as configured) will then be sent and patched into the destination file. This option is useful when the file behaviour is specifically known to change the headers and footers during file growth, it is a low impact, focused delta transfer. The default byte limit of header and footer is 1MB and it can be configured (using FC.hotfolder.config.max.headerfooter.bytes) in fchf.conf.

Re-Transfer Deltas when complete:

These transfers will wait for a specified amount of time, then the entire file will be checked for deltas. This option is useful in files where the growth behaviour is unknown but atypical. Checking the entire file has considerably more overhead than the "re-transfer headers/footers"; however, this option works well with a broader range of media files.

Limitations

Firstly, it is important to note that if the process that is growing the file locks said file (as is the case with Windows File Copy), progressive transfers will not work and the file will not transfer until it is unlocked.

This feature is designed to handle different types of file growth and/or change during the course of a transfer. Some files grow in a predictable pattern, such as tail-appending log files, while other files may grow and/or change in ways that require more advanced options, such as slow growth files or media files with multiple parts (e.g. multiple audio channels or audio/video files).

Due to the fact that the way a file grows is completely out of our control, if you have growth files, you must configure these options to properly represent that growth or the results can be undefined. For example, if you have a progressively growing file that changes once every 10 seconds but you set up a timeout delay for only 3 seconds, this can fail in a number of different ways depending on when the new data is appended, how it is appended and where we are in the transfer process.

Finally, since some of the dynamic file options will re-transfer all or parts of a file after it is completed, if After Transfer is selected under Verify File Integrity, the verification is moved to after the above options execute.

Remote Folder

By specifying a Remote Folder name, a folder by this name will be created or found on the FileCatalyst Server and all files will be transferred to and from that folder.

Enable Dynamic Folders

Selecting this will force the HotFolder to dynamically replace tags in the Remote Folder path when uploading. It supports the following tags:

• <%HOSTNAME%> is replaced with the hostname of the machine running the HotFolder.
• <%TIME%> is replaced with the time in the format: HH.mm.ss
• <%DATE%> is replaced with the date in the format: yyyy.MM.DD

File Filter

FileCatalyst HotFolder tasks allow you to define a filter which can be used to include or exclude particular files from the transfer. This filter is applied at the beginning of the transfer and can be configured through a wide range of available options that allow you to tune your fileset as needed.

Filter Phrase:

This option allows you to define a text phase that will filter any files in the HotFolder directory or on the Server that match the expression. To disable file filters entirely, leave this option blank.

Filter Mode:

This option allows you to define the type of filter that you wish to use. The available filter modes are as follows:

• Exclude Filter: Will exclude any files that match the given wildcard expression. The format for the expression is a comma-delimited list of files names, using a standard wildcard (“*”) character.

• Include Filter: Will only include files that match the given wildcard expression. The format for the expression is a comma-delimited list of files names, using a standard wildcard (“*”) character.

• Regular Expression Filter: This option allows you to build a complex regular expression to select which file(s) you want to include in a transfer.

  For additional documentation to how Java implements regular expressions, the following sources may be used:

  • https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html
  • http://www.sitepoint.com/article/java-regex-api-explained/

  Construct	Matches
  x	The character x
  \\	The backslash character
  \0n	The character with octal value 0n (0 <= n <= 7)
  \0nn	The character with octal value 0nn (0 <= n <= 7)
  \0mnn	The character with octal value 0mnn (0 <= m <= 3, 0 <= n <= 7)
  \xhh	The character with hexadecimal value 0xhh
  \uhhhh	The character with hexadecimal value 0xhhhh
  \t	The tab character ('\u0009')
  \n	The newline (line feed) character ('\u000A')
  \r	The carriage-return character ('\u000D')
  \f	The form-feed character ('\u000C')
  \a	The alert (bell) character ('\u0007')
  \e	The escape character ('\u001B')
  \cx	The control character corresponding to x
  Character classes
  [abc]	a, b, or c (simple class)
  [^abc]	Any character except a, b, or c (negation)
  [a-zA-Z]	a through z or A through Z, inclusive (range)
  [a-d[m-p]]	a through d, or m through p: [a-dm-p] (union)
  [a-z&&[def]]	d, e, or f (intersection)
  [a-z&&[^bc]]	a through z, except for b and c: [ad-z] (subtraction)
  [a-z&&[^m-p]]	a through z, and not m through p: [a-lq-z](subtraction)
  Predefined character classes
  .	Any character (may or may not match line terminators)
  \d	A digit: [0-9]
  \D	A non-digit: [^0-9]
  \s	A white space character: [ \t\n\x0B\f\r]
  \S	A non-whitespace character: [^\s]
  \w	A word character: [a-zA-Z_0-9]
  \W	A non-word character: [^\w]
  POSIX character classes (US-ASCII only)
  \p{Lower}	A lower-case alphabetic character: [a-z]
  \p{Upper}	An upper-case alphabetic character:[A-Z]
  \p{ASCII}	All ASCII:[\x00-\x7F]
  \p{Alpha}	An alphabetic character:[\p{Lower}\p{Upper}]
  \p{Digit}	A decimal digit: [0-9]
  \p{Alnum}	An alphanumeric character:[\p{Alpha}\p{Digit}]
  \p{Punct}	Punctuation: One of !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~
  \p{Graph}	A visible character: [\p{Alnum}\p{Punct}]
  \p{Print}	A printable character: [\p{Graph}]
  \p{Blank}	A space or a tab: [ \t]
  \p{Cntrl}	A control character: [\x00-\x1F\x7F]
  \p{XDigit}	A hexadecimal digit: [0-9a-fA-F]
  \p{Space}	A whitespace character: [ \t\n\x0B\f\r]
  Classes for Unicode blocks and categories
  \p{InGreek}	A character in the Greek block (simple block)
  \p{Lu}	An uppercase letter (simple category)
  \p{Sc}	A currency symbol
  \P{InGreek}	Any character except one in the Greek block (negation)
  [\p{L}&&[^\p{Lu}]]	Any letter except an uppercase letter (subtraction)
  Boundary matchers
  ^	The beginning of a line
  $	The end of a line
  \b	A word boundary
  \B	A non-word boundary
  \A	The beginning of the input
  \G	The end of the previous match
  \Z	The end of the input but for the final terminator, if any
  \z	The end of the input
  Greedy quantifiers
  X?	X, once or not at all
  X*	X, zero or more times
  X+	X, one or more times
  X{n}	X, exactly n times
  X{n,}	X, at least n times
  X{n,m}	X, at least n but not more than m times
  Reluctant quantifiers
  X??	X, once or not at all
  X*?	X, zero or more times
  X+?	X, one or more times
  X{n}?	X, exactly n times
  X{n,}?	X, at least n times
  X{n,m}?	X, at least n but not more than m times
  Possessive quantifiers
  X?+	X, once or not at all
  X*+	X, zero or more times
  X++	X, one or more times
  X{n}+	X, exactly n times
  X{n,}+	X, at least n times
  X{n,m}+	X, at least n but not more than m times
  Logical operators
  XY	X followed by Y
  X|Y	Either X or Y
  (X)	X, as a capturing group

Filter Target

This option allows you to specify the type of item that the file filter will apply to. The currently available options are as follows:

• Directory and file name: Applies the file filter to both file names and directory names.

  • Note 1: If a directory does not match the current filter settings, its contents will not be searched for other files that would have otherwise matched.
• File name: Applies the file filter to only file names. All directories that are found by the filter are automatically explored for other files that may match the current filter settings

• File path: Applies the file filter to the file path for a given file.

  • Note 1: Path-based filtering is relative to the source location currently defined by the transfer's current direction. If the task has an upload direction, then all path-based filter checks will be made relative to the current HotFolder location. If the task is a download, then all path-based filter calls will be made within the context of the currently defined remote folder. If no remote folder is currently defined, then the path based filter calls will be relative to the Server's home directory.
  • Note 2: If the task is a download task and the filter phrase starts with a leading slash, then all path based filter checks will be made within the context of the site's established home directory.

Filter Examples

Here are some example settings that can be used to help filter out necessary files in a transfer:

• Example 1:
  Filter Settings:
  

  Mode: Exclude
  Target: File name
  Phrase: *.pdf

  Result of settings:
  Single phrase filter that transfers all files that do not end with the '.pdf' extension

• Example 2:
  Filter Settings:
  

  Mode: Exclude
  Target: File path
  Phrase: folder1/**/*.docx,folder3/**/*.png

  Result of settings:
  Two phrase filter that transfers all files expect '.docx' files found in folder1, and '.png' files found in folder3

• Example 3:
  Filter Settings:
  

  Mode: Include
  Target: File name
  Phrase: *.mov,*.mp4

  Result of settings:
  Two phrase filter that transfers all files that end with the '.mov' or '.mp4' extensions

• Example 4:
  Filter Settings:
  

  Mode: Include
  Target: File path
  Phrase: **/videoProduction/**/4K Footage/**/*.mkv

  Result of settings:
  Single phrase filter that transfers all '.mkv' files that exist within the '4K Footage' sub-folders contained inside the 'videoProduction' parent folder.

• Example 5:
  Filter Settings:
  

  Mode: Reg Expr
  Target: File name
  Phrase: ".*?\.dat$"
  Regular Expression Definitions:
  

  Expression Section	Description
  ".*?"	Any character, 0, 1, or multiple times with reluctant matching
  "\."	Followed by a period (you must escape the dot character or it will be treated as a wildcard)
  "dat"	The exact characters 'dat' all lowercase
  "$"	Must be at end of the line

  Result of settings:
  Filter that transfer only files that end with ".dat". Windows wildcard equivalent of "*.dat"

• Example 6:
  Filter Settings:
  

  Mode: Reg Expr
  Target: File name
  Phrase: ".*?[0-9]+?.*?\.(mp3|wav)$"
  Regular Expression Definitions:
  

  Expression Section	Description
  ".*?"	Any character, 0, 1, or multiple times with reluctant matching
  "[0-9]+?"	Any numerical digit one or more times with reluctant matching
  ".*?"	Any character, 0, 1, or multiple times with reluctant matching
  "\."	Followed by a period (you must escape the dot character or it will be treated as a wildcard)
  "(mp3|wav)"	Characters may match exactly "mp3" OR "wav"
  "$"	Must be at end of the line

  Result of settings:
  Filter that transfer sound files (.mp3 or .wav) with numbers in the title

• Example 7:
  Filter Settings:
  

  Mode: Reg Expr
  Target: File name
  Phrase: "^.*\.(?!tmp$)[^.]+$"
  Regular Expression Definitions:
  

  Expression Section	Description
  "^.*"	Matches starting characters, 0, 1, or multiple times.
  "\."	Followed by a period (you must escape the dot character or it will be treated as a wildcard)
  "(?!tmp$)"	Negative lookahead that excludes files with the extension "tmp"
  "[^.]+"	Match everything not eliminated by the previous negative lookahead.
  "$"	Must be at end of the line

  Result of settings:
  Filter that transfer everything except temporary files ending in ".tmp"

• Example 8:
  Filter Settings:
  

  Mode: Reg Expr
  Target: File path
  Phrase: "^.*\.(?!(html|js|png)$)[^.]+$"
  Regular Expression Definitions:
  

  Expression Section	Description
  "^.*"	Matches starting characters, 0, 1, or multiple times.
  "\."	Followed by a period (you must escape the dot character or it will be treated as a wildcard)
  "(?!(html|js|png)$)"	Negative lookahead that excludes extensions that match "html", "js", or "png"
  "[^.]+"	Match everything not eliminated by the previous negative lookahead.
  "$"	Must be at end of the line

  Result of settings:
  Filter that transfers everything except files ending in ".html", ".js", or ".png"

Transfer Files Based on Size

This option allows you to specify whether the HotFolder should transfer all files or only files within the specified size limits.

Transfer Files Newer than X days

This option allows you to specify that only files newer than the given value (in days) will be transferred. This is based on the current modification time of the file, not when it was placed into the folder.

Filter Test

The filter tester will allow you to view the files that will be transferred with the given filters before applying them to the task. This is especially useful when experimenting with complicated regular expressions. Using the "Test" button will populate the text field with all of the files that will be allowed under the given filter, "Apply" will save your settings to the "Fileset" panel but will still need to be applied via the "Fileset" "Apply" button.

Note that this list will be sorted alphabetically and case sensitive for lists of less than 50,000 items. To maintain performance, lists of over 50,000 items will revert to the default sorting of the source operating system.

Transfer Cache

The file transfer cache is a history of all the files you have uploaded/downloaded from the server for this task. This is used to indicate whether a file has changed on the server and has been flagged for upload or download again. File changes are tracked using the size and the last modified timestamp. These values are updated when a change is detected and the file is redownloaded. Thus, even if you remove the files from the local directory, they will not be downloaded again unless the cache is cleared. When this cache is deleted, it will re-download all files from the server. In the case of upload tasks, only files that have been changed locally will be uploaded again.

Note: Transfer cache is enabled and locked if multi-file transfers is selected and requeueing of running task is allowed.

The cache database directory is saved in the cache directory as .fcdb by default. To save it somewhere else, specify the path in fchf.conf file with the following parameter key:

FC.hotfolder.config.file.cache.root=CACHEPATH

where CACHEPATH is replaced with the path of your target directory.

Clear Cache

Clears the current transfer cache entries. Files will be re-transferred even if they have previously been transferred.

Generate Cache

Generates the transfer cache without performing a transfer. This option is useful if both source and destination are already in sync. Cache generation occurs upon next task execution.

Remove File

When clicked, the user will have the ability to select a file for removal from the transfer cache. A browse dialog is presented showing the files in the source folder for this task. When a file is selected, if it is present in the transfer cache, it will be removed. If there was an error removing the file from the cache (i.e. it was not present), an error will be shown.

Note: When connecting to HotFolder service via remote admin, and attempting to remove a file from the transfer cache in an upload task, the full path of the file to be removed must be entered manually.

Always transfer directory structure (even if empty)

Empty directories are not transferred unless this is selected. If a filter hides all files in a directory, it is considered empty.

Transfer in priority sequence

This will apply a order the sequence of file transfers.

Oldest

Transfers files by last modified from oldest to newest.

Newest

Transfers files by last modified from newest to oldest.

Largest

Transfers files by size from largest to smallest.

Smallest

Transfers files by size from smallest to largest.

Synchronization Options

Delete files on destination not existing on source.

This option forces the source (HotFolder on uploads, Server on downloads) to make sure that both directories contain exactly the same fileset. This includes deletion of files which do not appear on the source.

File details (send details for Upload tasks, get details for Download tasks).

These flags are for a special configuration where two HotFolders want to synchronize files with each other with a server in between acting as a transit point (small temporary cache, not permanent file storage).

By enabling this feature, the source HotFolder will forward file listing information from the source in addition to the files themselves, allowing the destination HotFolder to download both the files and the listing and take appropriate actions.

The following options should be enabled to get this configuration setup:

On the Source HotFolder:

• "Transfer Cache" should be enabled (ensuring the same file is not sent up twice on the server)
• "Send details for Remote HotFolder synchronization" should be enabled.

On the Destination HotFolder:

• "Get remote HotFolder synchronization details" should be enabled.
• In Post Task tab, "Delete source file(s)" should be enabled.

This ensures that the destination HotFolder remain in-synch (Source HotFolder acting as the master repository), and that even through a Server acting as an intermediary, file creating and deletion events will be synchronized to the destination HotFolder.

Delete files on Destination that do not exist at source

When this item is checked, any files that exists on the destination that does not exist at the source will be deleted. In this way, the destination folder will always be kept in sync with the source folder because any new files added will be set to the destination, while anything removed will be deleted.

File Priority Settings

If this option is enabled, FileCatalyst allows you to specify which files should be transferred first. FileCatalyst will make a best effort attempt to transfer in the given priority with no significant impacton performance.

After a Transfer is Complete

Allows the user to select what to do with the files after they have been transferred. You can choose either to do nothing, move to a sent folder (Uploads only), or delete them.

If you choose to move to a sent folder and do not specify a location, the application will generate a folder named "Sent" in the root of the specified HotFolder. If a relative path such as "foo" is specified, the HotFolder will again be used as the parent folder. If an absolute path is specifed ("C:\" or "\\abc" in Windows, or starts with "/" in Linux/Solaris/MacOSX), the files will be saved to the specified location. If the folders do not exist, they will be created.

Note 1: If a file or folder is excluded from transferring (For example, a file filter has been configured to exclude the file from the transfer), it will not be moved or deleted if either of the 'Move source file(s) to "Sent" directory' or 'Delete source file(s)' options have been selected.

Note 2: When connecting to HotFolder service via remote admin, and attempting to set a 'Sent Directory' in an upload task, the path for the sent directory must be entered manually.

Post URL

To allow integration with web applications, HotFolder can perform an HTTP POST to a URL or launch a local script following the completion of a task.

If this field is set to a URL, the contents of the HTTP POST will include the following parameters:

• "f"- contains a | (pipe) delimited list of the full remote paths that were transferred up until a cancelation. Files never transferred will not appear in this list.
• "lf" - contains a | (pipe) delimited list of the full local paths that were transferred or schedule to be transferred
• "s" - contains a | (pipe) delimited list of the sizes of each file
• "status" - Status of each file transferred. "0" = File transfer was not attempted, "1" = File transfer was successful, "2" = File transfer was cancelled, "3" = File transfer failed with an error. For example, if the transfer included 5 files, and all files were transferred successfully, you would see status=11111. If the transfer was cancelled during the 3rd file, you would see status=11200.
• "allfiles"- contains a | (pipe) delimited list of the full paths that were transferred or schedule to be transferred, this list has a 1 to 1 relationship with the status value where each status value corresponds to each value in allfiles

If this field is set to a script, the script will be called upon completion of the task. There are 4 arguments that can be passed to the script by FileCatalyst.

• <remotefiles> - contains a | (pipe) delimited list of the full remote paths that were transferred or schedule to be transferred
• <localfiles> - contains a | (pipe) delimited list of the full local paths that were transferred or schedule to be transferred
• <sizes> - contains a | (pipe) delimited list of the sizes of each file
• <statuscodes> - Status of each file transferred. "0" = File transfer was not attempted, "1" = File transfer was successful, "2" = File transfer was cancelled, "3" = File transfer failed with an error. For example, if the transfer included 5 files, and all files were transferred successfully, you would see status=11111. If the transfer was cancelled during the 3rd file, you would see status=11200.
• <allfiles>- contains a | (pipe) delimited list of the full paths that were transferred or schedule to be transferred, this list has a 1 to 1 relationship with the status value where each status value corresponds to each value in allfiles

As an example, you may set the Script to:

/home/user/script.sh <remotefiles> <localfiles> <sizes> <statuscodes> <allfiles>

and FileCatalyst will automatically insert the proper values into the command line before execution.On Windows you may execute a batch file like this:

cmd /c start c:\\file.bat <remotefiles> <localfiles> <sizes> <statuscodes> <allfiles>

Note 1: If the Post URL field is set to a script, and the script is called upon completion of a task, the task will block until the script fully completes. This completion includes the originating script and any sub-processes that the script creates when it executes.

Enable E-Mail Alerts

If this is enabled and the client specifies to send an e-mail on completion of a transfer, then an e-mail will be sent to the address provided whenever a transfer is finished.

Note: E-mail messages will only be sent if the Server has properly configured SMTP settings, and has enabled the e-mail alerts setting. If you wish to have e-mail alerts for your task, please contact your Server administrator to verify that the Server's SMTP settings are correct, and that the e-mail alerts option is enabled.

E-Mail Address

Email destination: If Server Administrator is selected, the email will be sent to the administator address configured on the Server. Otherwise you can sellect Custom Address and enter an email address. If you wish to send email to multiple addresses, they must be separated with a semicolon(;).

Send E-Mail on Successful Transfer

Indicates whether or not to send an e-mail on successful transfers.

Custom Message Text on Successful Transfer

This message is sent as the e-mail notification on a successful transfer.

Send E-Mail on Transfer Error

Indicates whether or not to send an e-mail when a transfer error occurs.

Custom Message Text on Transfer Error

This message is sent as the e-mail notification when a transfer error occurs.

Note: With the introduction of Alarms to HotFolder, it is recommended that FileCatalyst Central be instead used for capturing transfer faults. Network outages or user authentication errors are not handled by Task e-mail notifications.

Why Manage Your Bandwidth?

Your network is used for email, applications, web site access and any other form of corporate communication. All of these tools utilize and require a percentage of your overall bandwidth. If each of these tasks does not have sufficient bandwidth, they will begin to feel slow and unresponsive.

FileCatalyst is capable of maximizing usage of your allocated bandwidth. For example, if your organization has a 1 Gbps link, FileCatalyst will be able to transfer at a full 1 Gbps. However, this may not be practical during peak periods of the day when FileCatalyst may need to share available bandwidth with other applications.

Users themselves may manage and schedule their bandwidth usage by using the Bandwidth scheduler (see below). The user may even specify periods during which there should be no transfer activity by choosing "No Transfers" for the selected time.

Selecting an Appropriate Rate

The bandwidth scheduler allows users to limit or cap the level of bandwidth usage for a period of time. Example: if you have a 1 Gbps connection, you may wish to allocate 500 Mbps of data transfer during the day and leave the remaining 500 Mbps for office use. However, at 6:00 PM you may wish to fully allocate 95% or 950 Mbps of bandwidth for data transfer. It is up to the users and the FileCatalyst administrator(s) to create an appropriate schedule.

Note: Congestion Control also automates the FileCatalyst transfer rate as networks become congested. As additional network traffic comes on-line and creates congestion, FileCatalyst will slow down or throttle back the transfer rate. It will remain in this state until the network is no longer congested. At which point the transfer will speed up until it resumes its set rate. This is a selectable feature which is enabled by editing individual tasks.

When configuring the FileCatalyst HotFolder to achieve high speed levels (500Mbps - 10Gbps for a single connection), several optimizations must be done in order to make sure the system is able to utilize multiple cores of a machine. These include configuring the number of reader threads per connection, packet processors (used for decoding AES packets on the receiver side), and the number of block writers.

# Reader threads

Number of threads PER CONNECTION spawned to read blocks off the disk and place them into the sender queue ready for transport. On the HotFolder, setting this value will only affect uploads. Most systems perform optimally when you configure only 1 thread to perform reading. There are two scenarios where this value should be moved up: when IO tests have shown that adding threads increases throughput for your storage device, or when utilizing high CPU load (more sender threads than CPU cores, or utilizing high CPU options such as AES encryption or LMZA compression), a single thread doing the block reads may be pushed down in priority and be CPU starved, causing a slowdown in transfer speeds. Multiple threads may help both of these scenarios.

Reader block size

By default, the block size used for each read operation is the same size specified by the client in the network block size. This can be adjusted for improved performance. Please run the advanced IO testing tool to see what works best for your system.

Packet processors

Packet processors are responsible for taking packets off the network socket, decrypting them, and preparing them for the block writers. When using AES at speeds greater than 200Mbps, it may be required to increase the number of packet processors on the receiving end to ensure multiple CPU's are utilized to decrypt the incoming data.

On the HotFolder, setting this value will only affect downloads. If uploading data to the server, similar settings should be set on the server applications.

# of Writer threads

Number of threads PER CONNECTION spawned to write blocks off the disk. On the HotFolder, setting this value will only affect downloads. Writer threads in FileCatalyst perform the following tasks:

• Uncompress data block (if using on-the-fly compression)
• Perform MD5 checksum on block (if using on-the-fly Verification)
• Write block data to disk

For systems using LMZA compression or possibly high speed transfers (> 1Gbps), setting multiple block writers may allow higher performance for single high speed connections.

Writer block size

By default, the block size used for each write operation is the same size specified by the client in the network block size. This can be adjusted for improved performance. Please run the advanced IO testing tool to see what works best for your system.

These values should be modified only after running through some performance IO testing.

Progress (all files)

A progress bar indicating the percentage complete of all files in this task. Progress is also represented numerically to the right of the bar.

Progress (current file)

A progress bar indicating the percentage complete of the current file. Progress is also represented numerically to the right of the bar.

Time Elapsed

How long the transfers have taken so far.

ETA (all files)

The approximate time remaining for the transfer of all files in this task.

ETA (current file)

The approximate time remaining for the current file.

Filename

The current file being transferred/verified. The current file number of the total transfer batch is shown in parentheses.

Transfer Rate

If a transfer is currently in progress, this will show you the average transfer rate in Kbps.

Current RTT

Latency observed on the link (round trip time, in ms)

Packet Loss

The current file being transferred/verified. The current file number of the total transfer batch is shown in parentheses.