Introduction

FileCatalyst HotFolder

FileCatalyst HotFolder software is a desktop application that is installed on the end user's machine and allows you to schedule automated file transfer tasks.

FileCatalyst HotFolder help documentation is available here.

Web Administration

Although FileCatalyst HotFolder is a desktop application, it can be managed remotely either with a Java-based Remote Administration tool or via this web-based tool.

This guide is intended to explain the FileCatalyst HotFolder's Web Administration tool, walking the user through the interface in order to show how to monitor and use FileCatalyst HotFolder.

User Interface (UI) Overview and Conventions

The interface uses a common “vocabulary” of interactions. A quick look at the application's conventions should allow most users to get right into using the product as a whole.

Main Navigation

Administrative and configuration categories are found in the main navigation. When the viewport (ie. browser window) meets a minimum size requirement, this navigation is always visible as a vertical sidebar beside the main content area.

When the viewport is below a certain size threshold, the navigation becomes hidden in an off canvas menu, accessible by clicking the "hamburger" menu widget:

Main Content

As the user navigates the sidebar, the content in the main area changes, exposing the tools related to the navigation item.

Within the main content area are a few additional control types to be aware of:

Sub-navigation

Drill down further into more specific configuration and information options by clicking through the sub-navigation items. In the large view, these appear as tabs:

In the small view, these become "accordion" panels. Clicking one of the collapsed accordions will expand it:

Item Tools

Certain views contain tables of data. These tables typically represent lists of editable items, such as lists of users present on the system. At the end of each row is an icon which opens a pop-up menu enabling actions such as editing or deleting the item.

Toolbar

Actions are often available in a toolbar above the table. Some of these are actions which are global to the table, such as creating a "new" item (eg. a new HotFolder). Others are redundant versions of the same actions available in the "Item Tools", but applicable to either single or multiple selections (Item Tools are for a single item). Tables that offer search functionality will also provide a search box in this toolbar.

In the upper-right corner of the application is a "gear cog" icon with a dropdown menu, which opens to show additional functionality:

Allows the administrator to sign out of the application and clear their credentials from the browser.

Note: It is possible to set FileCatalyst HotFolder to not request credentials for remote administration. In this case, the Sign Out button is replaced by a Refresh Session button, which clears the current session and refreshes the page.

Allows the administrator to visually pause updates to the UI. The application will be functioning as usual in the background, but the current view has its state paused. Activities which refresh data (such as sorting, page navigation, and editing individual items) will cause the lastest version of the data to appear. While paused, an indicator appears in the main application, and this menu option becomes "Resume UI"

Opens this help document.

This invokes a pop-up dialog with information about installation environment and version.

In addition to a Close button, the dialog also contains the Diagnostic Report button.

Diagnostic Report should only be pressed if requested by the support team, at which time it will create a package of files to be used during troubleshooting. No additional visual feedback is presented to the user after pressing the button.

Creating and Managing Sites

The HotFolder application can be configured to connect to any number of user accounts on any number of FileCatalyst Servers. Each such connection is considered a site regardless of which (or how many) FileCatalyst Servers are used. In other words, a "site" is defined by both its location (IP or domain) and the user account.

Select "Sites" from the main navigation to retrieve the Sites table and tools:

Adding a New Site

Click "New" above the Sites table in order to launch a modal dialog for adding a new site:

Complete the required fields and click "Apply" to add the new site.

  • Site ID: An arbitrary word or phrase allowing the user to easily remember the site.
  • Host: Hostname (domain or IP) for the FileCatalyst Server.
  • Port: The listener port of the remote FileCatalyst Server.
  • Use SSL: Enable this option to allow HotFolder to establish an SSL-secured command channel (note: SSL must also be enabled on the server side).
  • Enable Authentication: By default, HotFolder will use the "anonymous" account. To use an account provided by the administrator, you must select the Enable Authentication checkbox and fill out the username and password fields.
  • Username: Supply a username if "Enable Authentication" is selected.
  • Password: Supply a password if "Enable Authentication" is selected.
  • HTTP Transfer URL: An HTTP Servlet allows the FileCatalyst HotFolder to connect using an HTTP-based connection when UDP or FTP are not available. This functionality is an add-on feature licensed on the FileCatalyst Server. HTTP-based transfers cannot be accelerated with the FileCatalyst protocol, but two key benefits remain:
    1. Virtually guaranteed connectivity between endpoints: users do not need to configure their environment in order to transfer files.
    2. The reliability, security, and file transfer management capabilities of FileCatalyst technology are intact.
    The value placed here should take the following format and is provided to the HotFolder end-user by the FileCatalyt Server administrator: http://{server IP or Hostname}:12480/servlet/ftpservlet

Editing a Site / Advanced Settings

Select a site and click "Edit", or choose "Edit" from a site row's Item Tools. A new modal dialog will appear containing three tabs: "General Settings", "Advanced Network / Bandwidth Chracteristics", and "Advanced Site Tuning".

General Settings

These fields are an exact echo of those available when creating a new site. Use this tab to modify values for existing sites.

Advanced Network / Bandwidth Characteristics

These fields allow you to tune this particular connection to FileCatalyst Server in terms of the connection. Modifying these fields (and in particular, Network Characteristics) is considered an advanced operation and should be done with the guidance of an appropriate technical resource.

  • Upstream: The maximum possible upload speed between client and server. If a scheduled task has a maximum speed, HotFolder will use the lesser of the two values.
  • Downstream: The maximum possible download speed between client and server. As with Upstream, the scheduler may also be used to set speeds; HotFolder will use the lesser of the two values.
  • Path MTU: The largest packet that may be transmitted without being fragmented into smaller packets. For example, if a network has an MTU of 1500 (typical for most networks) and the application transmits a packet of 3000 bytes, the packet would be broken into two smaller packets of 1500 bytes, transmitted, and recombined at the destination.
    During an internet transmission, a packet may traverse several individual computer networks, all of which may have different MTU values. The Path MTU is the lowest MTU value on all networks between source and destination.
  • Average RTT: The average round trip time of a packet. RTT indicates the time required for a packet to be sent and acknowledgement to be received. In environments with high RTT, FileCatalyst will greatly outperform FTP.
  • Average Packet Loss: The number of packets that are lost in the network due to hardware/routing issues, oversized packets, or network congestion. Packet loss affects transfer speed; it is recommended that with higher packet loss, you set a lower packet size in the "Advanced Site Tuning" tab.

Advanced Site Tuning

These fields allow you to tune this particular connection to FileCatalyst Server in terms of the transfer's characteristics. Modifying these fields is considered an advanced operation and should be done with the guidance of an appropriate technical resource.

  • Block Size: Determines how much data will be sent by each individual transmitter thread to the receiver before waiting for acknowledgment. For example, if the block size is 10MB (10240000 bytes), then 10MB of data will be sent before proceeding to the next 10MB of data. The transmitter thread must receive an acknowledgment that a block is received before proceeding to the next block. The larger the block size, the better the throughput (due to fewer acknowledgments).
  • Packet Size: The maximum UDP packet size which client applications will send to the server. This should be less than the MTU of your network (usually 1500).
  • Sender Threads: To improve performance, FileCatalyst clients can begin sending new blocks of data before the previous block has been acknowledged. Sender Threads defines how many blocks can be started before waiting for all previous blocks to be acknowledged. This improves performance as it removes downtime during the acknowledgment phase. Memory consumption on the server and the client increases as this value is set higher. The exact amount of memory used is the product of the number of threads and the block size. Since overlap can be quite small (sometimes less than 500ms) the memory usage may appear to spike and drop immediately following acknowledgment.
  • UDP Sender Sockets: If the bandwidth is above 500 Mbps and the supporting disk IO can support these speeds, it is recommended that client application specify UDP sender streams to be 2 or greater. This allows more data to be pushed out of the system, and provides higher maximum speeds.
  • UDP Receiver Sockets: If the bandwidth is above 4 Gbps and the supporting disk IO can support these speeds, it is recommended that client application specify UDP receiver streams to be 2 or greater. This allows more data to be pulled from the system, and provides higher maximum speeds.
  • TCP Streams: In TCP mode, FileCatalyst can open one or more concurrent connections. This can resolve issues where TCP is hindered by high RTT. If the actual throughput of standard FTP is about 1/2 of available bandwidth, setting this value to 2 could optimize the connection. If standard FTP is only achieving 1/10th of actual available bandwidth, then setting this value to 10 could optimize the connection. Adding concurrent streams increases CPU consumption.
  • Connect Timeout: The length of time that HotFolder will persist in connecting to a Site. The default is 30 seconds; however, with extremely high RTT or packet loss this may need to be increased.
  • Read Timeout: The length of time that HotFolder will persist in reading a response from a Site after a command has been executed. The default is 30 seconds; however, with extremely high RTT or packet loss this may need to be increased. The timeout value applies only to commands that are expected to reply quickly.
  • Extended Read Timeout: The length of time HotFolder will persist in reading a response from a FileCatalyst Server after a command that is expected to reply more slowly. Examples of such commands would be MD5 on a very large file, or generating the deltas for a very large file. In cases where these commands take longer than the default 30 minutes, this value should be increased. The default is 1200 seconds or 30 minutes. The minimum value that may be applied for the Extended Read Timeout is 600 seconds, or 10 minutes.
  • Max Retries: The number of times HotFolder will attempt to reconnect to the server if the network connection fails.
  • Wait Retry: The length of time HotFolder will wait between retries if the network connection fails.

Adding A Filecatalyst Workflow Site

Any transfer task using a Filecatalyst Workflow site will not only transfer files to the server, but also send out an email notification with a download link. The recipient may then download the files by following this link. The Filecatalyst Workflow application needs to be licensed before HotFolder can connect to it this way. Contact FileCatalyst support or sales for further information. Configuration of Filecatalyst Workflow is beyond the scope of this document.

Setting up the HotFolder follows the same principle as setting up any other site except you choose the Filecatalyst Workflow option under the Site Type. The user name and password should match the username and password that has been configured in Filecatalyst Workflow.

The Site will verify itself during the creation process and provide you with errors if there are any issues. Once created you will be able to send files using the HotFolder Task FileCatalyst Workflow tab (shown below).

Setting a Filecatalyst Workflow Task

In the above example, you will be able to specify a recipient email address and short note, in addition to being able to configure the rest of the task.

Deleting a Site

From a site row's Item Tools, choose "Delete". Alternatively, select the row(s) to be deleted and press the "Delete" button in the toolbar.

Creating and Managing HotFolders

The HotFolder application can be configured to use any number of directories as "HotFolders", which are used as either sources or destinations for file transfer. Users may create new directories for this purpose, or use directories already on their system.

Select "HotFolders" from the main navigation to retrieve the HotFolders table and tools:

Adding a New HotFolder

Click "New" above the HotFolders table in order to launch a modal dialog for adding a new HotFolder:

Since a "HotFolder" is simply a directory and a unique identifier, the process requires only two fields:

  • HotFolder ID: A unique ID used to define and identify this HotFolder.
  • Location: The full directory path for the HotFolder. Pressing "..." will launch an explorer from which an user can select a path to be used as location.

While adding a HotFolder, you may also test the read and write access speed by clicking the "Test Access Rate" button, producing a test dialog:

Test Access Rate

This test is meant to help the user understand the read and write capabilities of the storage (for example, a disk array) for the path associated with the HotFolder. The test is measured against the maximum configured bandwidth, taken from the Bandwidth Scheduler at the time of test execution. There are three things to look for in the test:

  1. The block size which achieved best results. This will help inform choices when configuring tasks. There is a final result at the top of the interface, and a series of sorted boxes below showing individual test results.
  2. The average rate across all tests. This will help the user understand the overall performance of the attached storage.
  3. The access speed relative to the configured line speed. The user will be able to see at a glance if their bottleneck is the storage or the line speed. In a well-configured environment, access rate should be faster than line speed.

Click "Start" to begin a series of tests, which will be sorted as results are updated.

Click "Cancel Test" to stop testing and display the current results.

Note: The test creates temporary files on the disk. Ensure that enough space is available for the test.

Editing a HotFolder

Select a HotFolder and click "Edit", or choose "Edit" from a HotFolder row's Item Tools. A new modal dialog will appear containing the same two editable fields and a "Test Access Rate" button described in “Adding a New HotFolder”. Modify the fields to make changes to the selected HotFolder.

Deleting a HotFolder

From a HotFolder row's Item Tools, choose "Delete". Alternatively, select the row(s) to be deleted and press the "Delete" button in the toolbar.

Creating and Managing Transfers

Task Overview

Task Schedule

The task schedule displays all configured tasks in FileCatalyst HotFolder. It shows the HotFolder, Site Name, and Direction (upload or download), as well as the Status (Idle, Transferring, Queued) and the date/time of the next execution.

Creating a New Task

Once you have defined at least one “Site” and one “HotFolder”, you may create a task. To do so, click the “+ New” button found in the toolbar above the schedule table. The Create Task dialog will appear. The form is pre-populated with default settings, with “Task Name” being the only required field. Once the task is configured, click “Create Task” to save it.

Managing Existing Tasks

To edit, delete, duplicate, disable or enable an existing task, or to view its activity, first select it on the schedule table. Next, click on the corresponding action in the toolbar. You can also click on the row's tool icon and select from the options on the sub-menu that appears. If multiple tasks are selected, you may use the menu from the toolbar to delete, execute, cancel, enable, or disable all of the tasks that are currently selected.

Execute a Task

To execute a task, highlight a task and click the “Execute” button.

Cancel a Task

To cancel a task, highlight a task and click the “Cancel” button.

Pause / Resume Scheduler

Clicking “Pause Scheduler” will prevent any further scheduled task execution. However, any tasks in progress will complete their execution. You may resume the scheduler by clicking the same button, which is now labeled, “Scheduler PAUSED - press to resume”.

Concurrent Tasks

The HotFolder can be configured to support execution of multiple tasks at a time. To do this, check the “Allow concurrent tasks” checkbox. To disable execution of concurrent tasks, uncheck the checkbox.

Task Activity View

Select a task on the schedule table, and press the the “Activity” button or select “Activity” from the row's tool menu. This will open the "Activity" view in a new tab or window.

Note: Your browser may block these kinds of “popups”; if pressing the button does not show a new tab, please check you browser settings and pop-up blocker add-ons.

Title

Title of the activity view. It will show the task name, HotFolder used on the task, direction, and the FC Server Site ID involved in the transfer.

General Task Information

This table contains current information about the task. If the transfer is not in progress the table will be in an idle state. During transfer, it shows:

  • Progress (in terms of number of files sent)
  • ETA (estimated time of arrival)
  • Elapsed Time
  • Status (either a progress in terms of percent, or a description of current activity such as "Calculating MD5")
  • Bytes Transferred
  • Average Transfer Rate (for single-client) or Total Transfer Rate (for multi-client) for the current excecution
  • Current RTT (latency)
  • Average Packet Loss (for single-client) or Total Packet Loss (for multi-client) for the current execution

This header of the table also contains buttons to allow editing, starting or stopping the task in the same manner found in the section “Managing Existing Taks”.

Current File / Clients

This table contains information about current file transfers: the “current file” when single-client, or a list of active “clients” when multi-client. The available columns change depending on whether your task is set for single or multi-client usage.

Current File (Single Client):

  • Filename
  • ETA (estimated time of arrival)
  • Elapased Time since the file transfer or client started
  • Status (current activity such as "Calculating MD5")
  • Progress %
  • Transfer Rate (current)
  • Packet Loss (current)

Clients (Multi-Client):

  • Worker ID
  • Processor *
  • Filename
  • Status
  • Progress %
  • Transferred (amount of data)
  • File ETA
  • Packet Loss
  • Transfer Rate

* A Processor is a section of task execution that is independent from the transfer itself. For example, checking for file growth in the "Wait for file growth to cease" Advanced Progressive transfer option.

Note: If you are seeing a a multiple clients transfer the Show Overall Graphs and Logs will apear. When it is clicked the overall information for the task is displayed.

Graph, Visualization and Logs

Select an appropriate tab in order to change the information displayed below. “Visualization” is only available for single transfer threads. In other words, it is considered always-on for single-client tasks; for multi-client, an individual client must be selected from the “Clients” table above.

“Graph” (seen above) contains a line graph of effective and current transfer rate (in kbps):

“Visualization” is bundle of information representing resource usage for Source and Destination:

“Logs” shows the current logs for the task. Logs can be filtered by using the search form at the top of this container:

  • Search: Clicking Search will open a new tab that displays the current log filtered for the provided keyword. You can match case should you wish.
  • Scroll Lock: Enabling Scroll Lock will prevent the screen from scrolling to the latest text added to the text area.
  • Retrieve: will retrieve a user provided number of lines from the logs available on the server.
  • Copy to Clipboard: copies the text from the current log window to the clipboard.
  • Clear: clears all of the text in the current text area.

File Priority on Activity View

If the task is a multi-client transfer, a link will be presented beside the label of the “All files” (general information) table, on top-left corner. This link is only active when the task is running and it opens a dialog which allows you to move a file's transfer priority to the top of the queue. Since the top file in the queue is always the next file to be transferred, this feature allows you to manually intervene and ensure that important files are transferred first.

The dialog itself may contain only a subset of files queued for transfer; the “Filter” field helps the user narrows this down when looking for a particular file. Once the file in question appears in the list, the user can click to select it, then click the “Increase Priority of Selected File” button below. This button moves the selected file to the top of the transfer queue.

“Refresh List” will deselect files, remove any filters, and refresh the list.

Files that are already prioritized are show with their file path in bold text. Prioritized files may not appear on the list for two reasons: they are already transferred, or they are not in the current filtered list.

Task Settings

Press the “Edit” button to reach these settings (The “General” tab will be visible by default).

Task Name

The name of the task.

Select HotFolder

Specify the HotFolder used by this task.

Select Site

Specify which Site is used by this task

Select Direction

Specifies whether to upload from this HotFolder to the server, or download files from the server to this HotFolder.

Schedule

From the “Edit Task” sialog, select the “Schedule” tab.

Edit Task Schedule

Frequency Options

A set of five radio buttons allows you to select the frequency of this task:

User Triggered

The default “user triggered” option requires the user to open the Scheduler, select a task, and click "execute". This default prevents accidental execution of tasks while the user is still configuring their HotFolder, but is typically changed to one of the scheduling options below.

Once

Selecting this option will enable corresponding fields allowing you to specify a date and time for a single execution. The task remains in the scheduler, allowing you to specify a new date or to manually execute.

Every minute

This is the most common choice. The scheduled task will check every minute to see if a new file transfer is required.

Always On + File System Events

As soon as the task completes, it will repeat. Essentially, the transfer filter is being monitored continuously to see if a new file transfer must be initiated. Selecting this option will also automatically check the “Monitor File System Events” box at the bottom of the page.

Note 1: The "Always On + File System Events" option is made available by enabling "Advanced View" on the HotFolder

Note 2: Using "File System Events" on network drives (Samba, CIFS, NFS, ATP) is not recommended. Any task transferring to or from mounted network drives should instead use a normal transfer schedule (ie: once a minute).

Scheduled Execution

Selecting this option enables a range of fields to create a repeated task. As the scheduled task is created, a plain-language representation of the task is updated at the top of the section.

After selecting the “Day(s)” for execution, you must select an interval. This may be:

  • Once, at a particular time of day.
  • Every X number of hours, at the specified minute past the hour.
  • Every X number of minutes.
  • Every X number of seconds.

Monitor File System Events

When this option is enabled, the task will first complete transfer of all files, and will then enter a “monitoring” state. When in the monitoring state, HotFolder will watch the file system at the source (local folder on uploads, and server folder on downloads) for events such as file creation, modification, deletion, and rename. When these events occur, the task will immediately transfer, delete, or rename the affected file(s).

Once the task enters monitoring mode, it will stay in this mode indefinitely or until it is canceled or an error occurs. The task is still considered to be running while in this state.

This option is useful when dealing with very large file sets that must be kept in sync. For a scheduled task, recursive listings, comparisons, and filtering determine which files have changed. On large file sets this process may takes several minutes and consume resources. However, with file system event monitoring enabled, recursive listings are not needed and resources will only be used when a file is changed.

Note on Monitoring Network storage: In many cases FileCatalyst is used to monitor directories that do not reside on the same physical machine but rather on a network attached storage. Depending on the OS and protocols used, the FileCatalyst software may not be notified of events that are triggered at a network location. For example, if a Linux machine is running FileCatalyst Server and has a user home directory pointing at a mounted directory (i.e. /mnt/network_dir), and changes are made to that directory from yet another machine pointing at that same directory, the events may not be registered.

Note 1: The "Monitor File System Events" checkbox is made available by enabling "Advanced View" on the HotFolder

Note 2: Because of how the multi-client manages itself, a multi-client transfer will identify as “Idle” when it is monitoring.

Connection

Edit Task Connection

Protocol

This option allows you to specify the transfer mode you wish to use to connect and transfer files to/from the server. Options are UDP, TCP, HTTP (when HTTP Servlet is enabled), and Auto Detect. By selecting Auto Detect, the HotFolder will cascade from UDP to multi-stream TCP to single-stream TCP and finally to HTTP in order to find the optimal mode.

Auto-Resumes

With this option enabled, FileCatalyst HotFolder will automatically recover from failed network connections. This setting takes precedence over the “Transfer Files With Unique Name” options found within the incremental settings. If a file can be resumed and auto-resumes are enabled, the destination file will be resumed and a unique file will not be created at the destination location.

Congestion Control

When Congestion Control is enabled, FileCatalyst will attempt to dynamically modify its transmission rate in accordance with changing network conditions. This is useful if the end to end link speed is not known, or if the network is also being used by other applications with which FileCatalyst would otherwise interfere.

Start Rate

The rate at which FileCatalyst will begin transmitting data. It will then attempt to speed up the transmission rate until it either hits congestion, or it reaches the target rate specified in the bandwidth scheduler.

Three options are available:

  • Use Target: Attempt to transfer at maximum speed allocated by the task. Will slow down if congestion is detected, but initial rate may adversely effect network link if it cannot sustain the transfer speed.
  • Specify: Manually specify a start rate for transfer speed. We recommend a maximum of 1/10th of the known line speed if this is not a dedicated line.
  • Auto: Have the transfer automatically launch an auto-detect when the task starts. This is the best solution for single file transfers. It is not recommended for multi-file session transfers, as each client will attempt to auto-detect transfer speed.

Congestion Control Strategy

RTT-based Strategy

RTT-based strategy works by establishing a baseline average RTT before the data starts to flow. Once the transmission begins, RTT is monitored continuously. While the RTT stays within a certain range of the baseline RTT, the speed of the transfer will be increased. Once the RTT begins to go above a certain range, the speed is decreased. How much the RTT is allowed to spike above the baseline average is controlled by the Congestion Control aggression setting FileCatalyst provides.

This type of congestion control works well on wireless or satellite links where there is packet loss from other sources besides congestion. TCP will slow down, for example, when a packet is lost due to interference or being too far from a cellular tower. When FileCatalyst is using the RTT based congestion control it ignores individual packet losses and focuses only on RTT. For these scenarios, FileCatalyst continues to maintain high speeds through the kinds of packet loss ”hiccups” that would trigger decreases speeds under TCP.

Loss-based Strategy

There are circumstances in which the RTT-based strategy may not work properly. For example, when a router's queue is very small, the RTT may never spike when there is congestion. The RTT will remain low, and therefore FileCatalyst would continue to increase its transmission rate even when there is congestion present. For these scenarios, the only way to detect the congestion is using a packet loss approach. As outlined previously, packet loss may come from other sources besides congestion, so this mode is best used on terrestrial (land-based) networks where all packet loss is due to real congestion.

Loss-based congestion control mode reacts to packet loss by slowing down (just like TCP); however, FileCatalyst is not nearly as aggressive as TCP. The primary reason for using a UDP-based file transfer solution like FileCatalyst is because TCP is so aggressive in its congestion avoidance that it often ends up under-utilizing your link. The FileCatalyst loss-based congestion control algorithm was designed such that it is able to maximize link utilization while still avoiding congestion. Like the RTT-based congestion control, the loss-based mode can be tuned to be more or less aggressive. More aggressive settings allow for a higher percentage of packet loss before slowing down, while for more passive settings the opposite is true.

Aggressiveness

This setting allows the threshold (or range) that is used when comparing the current RTT to the initial RTT to be increased or decreased. If aggression is increased, the current RTT will be allowed to increase to a higher level before it is considered to be congestion. Conversely, if the aggression is lowered, smaller increases in RTT will be considered as congestion and transmission rates will slow down. In this way, the congestion control may be tuned to the sensitivity of the network on which the transfers are being performed.

Transfer

Edit Task Transfer

Verify File Integrity After Transfer

During all UDP transfers, as blocks of data are transferred, they are verified using MD5 against the same block in the source file. This is done on the fly, and incurs no delay after a transfer completes. This provides a very high level of protection against corruption. Note that FTP/TCP mode transfers are verified on the fly as well, but it is built into the TCP protocol. FileCatalyst provides the additional option to perform a full MD5 checksum after each transfer instead of on the fly.

By checking the "Verify File Integrity After Transfer" option, integrity of the transferred file will be verified byte by byte on storage using MD5 after the transfer is complete. This option can be used in one of three modes:

Full

This will perform a byte by byte comparison of the entire file when the transfer is complete. Although this consumes less CPU on an ongoing basis, there is a delay between files being transferred as the checksum is calculated. If the files are large, it may cause delays of several seconds or even minutes depending on the speed of the I/O on the sending and receiving machines.

Concurrent

This will open up a separate session to perform checksums without interrupting the file transfer. For very fast transfers, disk I/O bottlenecks (competing between ongoing transfers and MD5 checksums) may result in lowered throughput.

Partial

Partial MD5 verification occurs after the file is completed, however it only compares certain portions of the file, and only at logical intervals that coincide with the transfer characteristics. This vastly decreases the time to perform the MD5 check. Note, however, that although the file is almost certainly intact, the Partial check cannot absolutely guarantee with 100% certainty.

Note: Partial MD5s are not available against servers older than 3.7. If partial MD5s are still selected at the time of transfer, then full MD5s will be used instead.

Keep File Modification Timestamp

If this item is selected, the modification date and time will be preserved when files are transferred to the server.

When multi-client is enabled, folders are also maintained. This behaviour is different from our legacy single client transfers. If you desire the legacy behaviour, it can be reverted via setting this system property: unlimited.fc.legacy_timestamp

Force File Ownership

By selecting this option on a download task, all files transferred will have their operating system user ownership set to the value specified.

Note: On Linux or Mac OSX, an additional field allows configuring group ownership.

Note: If the Maintain UID/GID system property is enabled, the user and group data will be updated to accept user and group identifiers. Please note that this functionality is only on Linux and Mac OSX operating systems.

Note: If Keep File Permissions is enabled when Force File Ownership is enabled, the "Keep File Permission" option is automatically deselected.

Transfer with Temporary Name

If this option is enabled, FileCatalyst will always upload or download files using a temporary name, and rename it back to the original name when the transfer is complete and verified. FileCatalyst Server will not report these temporary files when other HotFolder or FTP clients request file lists. This prevents them from downloading partial files. You may choose to either prepend the ".fctmp." prefix, or append the ".tmp" suffix to the files until they are finished transferring.

Task Priority

This allows you to specify the priority of a task. By default, bandwidth is split evenly between all running tasks. Setting the priority of a task will increase/decrease the weight of a task when bandwidth allocation is performed.

Enable Multi-Client (multiple concurrent file transfers)

HotFolder allows a single transfer task to utilize more than one transfer connection in order to push files across concurrently.

Multi-Client is particularly useful when transferring dynamic file sets (where data is being added in real-time to multiple files in a directory). When combined with progressive transfers, multi-client allows multiple growth files (log files, video transcoding, etc) to be sent from one directory.

NOTE: as you are utilizing more than one transfer connection to a server, this may potentially impact server-side license restrictions. As one might expect, Multi-Client utilize more resources (memory, ports) as more clients are added.

Multi-Client does not currently support Filecatalyst Workflow. Selecting specifying a Filecatalyst Workflow server will automatically execute the task as a single client transfer.

Automatically rescan and add new files to the running task

By default, the HotFolder scheduler will not re-execute a task if it is currently running (only one instance of the same task is allowed to run at a time).

Because multi-file sessions are ideally suited for transfers of growth files (ie: live streaming video) which may take an extended period of time to complete (2+ hour event), rescheduling allows you to rescan a directory even if the original transfer has not yet completed (live video event still on-going). New static files (or growth files) will get picked up by the task at scheduled intervals.

NOTE: Transfer cache is enabled and locked when this feature is used. Files will be sent only once when using this feature. Transfer cache can be manually cleared in the Fileset tab.

NOTE: When transferring a progressive file that grows much slower than our transfer rate, a situation occurs where we are always waiting for a file to grow before sending the next chunk. We call this chasing the tail. When this is happening a progressive file might not grow for a long enough time that we think it has finished growing and consider the transfer complete. With reschedule enabled, the next scan will pick up the file if it has indeed grown. Since we cannot know if the file has just grown, been replaced, or been edited, we need to treat it as a new transfer. This only affects our statistics displayed as we will increment the amount of data and number of files transferred. However this can be corrected increasing the time a progressive transfer will wait for the file to grow (see progressive transfers)

Enable Auto Archiving of small files.

This option will group together small files (10 MB or less in size) archiving them and sending them over as a single transfer. Use this feature if your transfers will contain many small files. The maximum size for an individual archive is 100MB, after which a new one will be created. Previous archives are transferred and extracted while new ones are being created.

Keep File Permissions (OSX/Linux/Solaris Only)

When this item is checked, file permissions will be saved when files are transferred.

Multi-Client changes from single client and limitations

Due to the nature of how parallel transferring works, it is highly recommended to use the AutoArchive of Small Files option in Multi-Client. This will keep the overhead of HTTP POSTs to a minimum during a transfer with a larger data set.

Changes from single client

Multi-Client has an added layer of redundancy: if an individual client fails a transfer, that transfer is pushed back onto an internal transfer queue to be attempted by a different client a set number of times before actually failing. This means that in cases where Single Client failed consistently, multi-client may succeed. Multi-Client can use auto archiving, incremental and compression simultaneously. Small files will still be auto archived and use the compression settings, ignoring the incremental while the larger, non archived files, will use incremental and compression. Normally archiving and incrementals are mutually exclusive for single client transfers. Note that while multi-client is enabled, some compression options in the Data Minimization pane will be disabled as they are handled by the auto archiving algorithms.

Limitations

Multi-Client transfers are not compatible with Filecatalyst Workflow so multi-client settings are ignored for transfers to a Filecatalyst Workflow site.

Data Minimization

Enable Incremental

Incremental tasks limit transfer to files that are new or have been modified. This works in conjunction with “Do nothing with source files” option on the Post Task tab to allow you to treat the HotFolder as a working directory with only your modified files being transferred. With incremental enabled, the user also has the option to remove MD5 checks and only check the size of the file. Incremental options are as follows:

Transfer Entire File

Transfer the entire file if the remote file and the source file do not match.

Transfer File Deltas

Transfer the portions of the file that have changed (the "deltas"). These changes are then merged with the file on the target to recreate the source file.

Transfer Entire File with Unique Name

Transfer the entire file, with the target filename using a unique identifier. The unique name will use the following pattern:

<filename>_X.<extension>

Where X is a unique integer value.

NOTE:: If auto-resumes are enabled within the current task, a new file with unique name is created only if the destination file cannot be successfully resumed.

Transfer File Deltas with Unique Name

This option will only transfer the portions of the file that have changed. These changes are then merged with the file on the target to recreate the source file. The final file will be renamed such that is has a unique name using the following pattern:

<filename>_X.<extension>

Where X is a unique integer value.

NOTE: Incremental delta transfers are bypassed under circumstances where it is more beneficial to send the file normally. Deltas are bypassed if:

  • either file (source or destination) is smaller than 200KB in size
  • the destination file size is less than the square root of the source file size
  • the minimum delta file will be larger than transferring the entire file

There are possible side effects using deltas in conjunction with Progressive transfers noted in this knowledge base article here.

Enable Compression

Compressing files as they are being sent to the server may reduce transfer time. However, if all files are already in compressed formats, it will only serve to add additional overhead to the transfer. Once you understand the compression options, you will know which method (if any) is right for your situation.

Compression Method

Two compression methods are available: Zip Deflater and LMZA.

  • Zip Deflator: Zip Deflator is the standard compression library utilized by FileCatalyst, and is used for on-the-fly compression and for single zip archive (bundles multiple files together before transferring).
  • LMZA: LMZA is an additional compression method available for on-the-fly compression (single archive is not supported), and allows a very high level of compression. The algorithm is very CPU and memory intensive for the sender. Profiling the transfer via HotFolder or Server visualization window and configuration of threads (block sender threads for transmitter, writer threads for receiver) may be required for achieving high speed transfers.
Single Archive

When sending several small files, there is an overhead to initiate the transfer of each file. With a high RTT, this means that far more time is spent “setting up” the transfer compared to actual data transmission. Furthermore, due to small file sizes, the transfer task never reaches full speed.

With the “Single Archive” option enabled, HotFolder will put all the files into a single package, transfer to the server, then extract it. This consolidates set-up and teardown, and allows the transfer task to reach its maximum speed.

NOTE: Single archive transfers are not compatible with Multi-Client transfers and will be disabled if Multi-Client transfers are enabled.

NOTE: If progressive transfers are disabled while enabling archiving, the archive file (tar/zip) must first be created locally before being transmitted. This may cause a noticeable delay before the transfer begins, but the total time for the transfer should still be significantly less.

NOTE: The maximum size of an individual zip file is 4GB, as is the maximum size of any single file contained in the zip file. For file sets exceeding 4GB in total size, please use the “Archive size limit” feature (described in the next section) to limit the size of each zip file created to under 4GB. If the zip file size is over 4GB due to no limit or a single file over 4GB in size, the zip file will not be able to be unzipped and will cause an error.

Archive size limit

This feature will limit the size of the zip archives to the specified value (in MB) when possible. If a file set is too large to fit into a single zip of the specified size, multiple zip archives will be created. These zip archives are automatically extracted and cleaned up as they are transferred. At most 5 archives will be created at one time. Additional archives are generated only after the oldest file is deleted. This helps reduce disk space required.

Compression Level Settings

This slider will allow you to select the level of compression to be applied when compression is enabled. Depending on the types of files and bandwidth constraints, it may be desirable to use more or less compression. For example, on a high bandwidth connection on which volume of data transfer is not measured or is otherwise not a concern, it may be faster to send the data either uncompressed or with low compression. However, on a low bandwidth satellite network it may be more optimal to use maximum compression.

Exclude Files

Any files in the HotFolder directory or server matching the given expression will be excluded from compression. You may specify multiple file filters delimited by a comma. This option also supports wildcards (*). For example, if you wish to ignore all Zip files, you would set this value to "*.zip". If you wish to ignore all Zip files, and MP3 files, you would set this field to "*.zip, *.mp3".

NOTE: The compression option is only available for either UDP or TCP based transfers. HTTP based transfers currently do not support this option.

Dynamic Files

Edit Task Connection

Disabled

This option turns off progressive transfers. Files will not continue to transfer as they grow.

Basic Progressive

Files will transfer as they grow. This is the most basic of progressive features. This is our former "Enable Progressives" feature.

Advanced Progressive

If Advanced Progressive is enabled, one of the four sub-options (described below) must be selected. If one of these sub-options is not selected, the transfer will default to Basic Progressive. Note that in single client transfers only the "Transfer these files as they grow" sub-option will be permitted, as the other options are multi-client specific.

For each selected sub-option, an inclusive filter must be specified (e.g. "*.mxf" for mxf files) to indicate which file types the functionality should be applied to. Files that do not match this filter will be transferred as per Basic Progressive. A wait time (in seconds) may also be specified for each sub-option to modify how long the client waits to check if a file has grown.

Transfer these files as they grow:

These transfers will check for files matching this particular filter within the specified number of seconds. If no changes to the file have occurred within the set time, the file growth will be considered complete. This option is useful for files that grow in intervals with pauses, or if a hard drive is heavily taxed and the actual growth of the file is slower than normal.

Wait for file growth to cease:

These transfers will check for files matching this particular filter within the specified number of seconds. If no changes to the file have occurred within the set time, the file growth will be considered complete. The transfer will only happen after the file growth is considered complete. This option ensures only complete files are being transferred and not ones that are still growing.

Re-Transfer Headers/Footers when complete:

These transfers will wait for a specified amount of time, then the header and footer bytes (as configured) will then be sent and patched into the destination file. This option is useful when the file behaviour is specifically known to change the headers and footers during file growth. It is a low impact, focused delta transfer.

Re-Transfer Deltas when complete:

These transfers will wait for a specified amount of time, then the entire file will be checked for deltas. This option is useful in files where the growth behaviour is unknown but atypical. Checking the entire file has considerably more overhead than the "re-transfer headers/footers"; however, this option works well with a broader range of media files. The default byte limit of header and footer is 1MB and it can be configured (using FC.hotfolder.config.max.headerfooter.bytes) in fchf.conf.

Limitations

It is important to note that if the process growing the file locks said file (as is the case with Windows File Copy), progressive transfers will not work and the file will not transfer until the lock is released.

This feature is designed to handle different types of file growth and/or change during the course of a transfer. Some files grow in a predictable pattern, such as tail-appending log files, while other files may grow and/or change in ways that require more advanced options, such as slow growth files or media files with multiple parts (e.g. multiple audio channels or audio/video files).

Due to the fact that the way a file grows is completely out of our control, if you have growth files, you must configure these options to properly represent that growth or the results can be undefined. For example, if you have a progressively growing file that changes once every 10 seconds but you set up a timeout delay for only 3 seconds, this can fail in a number of different ways depending on when the new data is appended, how it is appended and where we are in the transfer process.

Since some of the dynamic file options will re-transfer all or parts of a file after it is completed, if After Transfer is selected under Verify File Integrity, the verification is moved to after the above options execute.

Fileset

Remote Folder

By specifying a Remote Folder name, a folder by this name will be created or found on the FileCatalyst Server and all files will be transferred to and from that folder.

Enable Dynamic Folders

Selecting this will force the HotFolder to dynamically replace tags in the Remote Folder path when uploading. It supports the following tags:

  • <%HOSTNAME%> is replaced with the hostname of the machine running the HotFolder.
  • <%TIME%> is replaced with the time in the format: HH.mm.ss
  • <%DATE%> is replaced with the date in the format: yyyy.MM.DD

File Filter

FileCatalyst HotFolder tasks allow you to define a filter which can be used to include or exclude particular files from the transfer. This filter is applied at the beginning of the transfer and can be configured through a wide range of available options that allow you to tune your fileset as needed.

Filter Phrase:

This option allows you to define a text phase that will filter any files in the HotFolder directory or on the Server that match the expression. To disable file filters entirely, leave this option blank.

Filter Mode:

This option allows you to define the type of filter that you wish to use. The available filter modes are as follows:

  • Exclude Filter: Will exclude any files that match the given wildcard expression. The format for the expression is a comma-delimited list of files names, using a standard wildcard (“*”) character.

  • Include Filter: Will only include files that match the given wildcard expression. The format for the expression is a comma-delimited list of files names, using a standard wildcard (“*”) character.

  • Regular Expression Filter: This option allows you to build a complex regular expression to select which file(s) you want to include in a transfer.

    For additional documentation to how Java implements regular expressions, the following sources may be used:

    Construct Matches
    x The character x
    \\ The backslash character
    \0n The character with octal value 0n (0 <= n <= 7)
    \0nn The character with octal value 0nn (0 <= n <= 7)
    \0mnn The character with octal value 0mnn (0 <= m <= 3, 0 <= n <= 7)
    \xhh The character with hexadecimal value 0xhh
    \uhhhh The character with hexadecimal value 0xhhhh
    \t The tab character ('\u0009')
    \n The newline (line feed) character ('\u000A')
    \r The carriage-return character ('\u000D')
    \f The form-feed character ('\u000C')
    \a The alert (bell) character ('\u0007')
    \e The escape character ('\u001B')
    \cx The control character corresponding to x
     
    Character classes
    [abc] a, b, or c (simple class)
    [^abc] Any character except a, b, or c (negation)
    [a-zA-Z] a through z or A through Z, inclusive (range)
    [a-d[m-p]] a through d, or m through p: [a-dm-p] (union)
    [a-z&&[def]] d, e, or f (intersection)
    [a-z&&[^bc]] a through z, except for b and c: [ad-z] (subtraction)
    [a-z&&[^m-p]] a through z, and not m through p: [a-lq-z](subtraction)
     
    Predefined character classes
    . Any character (may or may not match line terminators)
    \d A digit: [0-9]
    \D A non-digit: [^0-9]
    \s A white space character: [ \t\n\x0B\f\r]
    \S A non-whitespace character: [^\s]
    \w A word character: [a-zA-Z_0-9]
    \W A non-word character: [^\w]
     
    POSIX character classes (US-ASCII only)
    \p{Lower} A lower-case alphabetic character: [a-z]
    \p{Upper} An upper-case alphabetic character:[A-Z]
    \p{ASCII} All ASCII:[\x00-\x7F]
    \p{Alpha} An alphabetic character:[\p{Lower}\p{Upper}]
    \p{Digit} A decimal digit: [0-9]
    \p{Alnum} An alphanumeric character:[\p{Alpha}\p{Digit}]
    \p{Punct} Punctuation: One of !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~
    \p{Graph} A visible character: [\p{Alnum}\p{Punct}]
    \p{Print} A printable character: [\p{Graph}]
    \p{Blank} A space or a tab: [ \t]
    \p{Cntrl} A control character: [\x00-\x1F\x7F]
    \p{XDigit} A hexadecimal digit: [0-9a-fA-F]
    \p{Space} A whitespace character: [ \t\n\x0B\f\r]
     
    Classes for Unicode blocks and categories
    \p{InGreek} A character in the Greek block (simple block)
    \p{Lu} An uppercase letter (simple category)
    \p{Sc} A currency symbol
    \P{InGreek} Any character except one in the Greek block (negation)
    [\p{L}&&[^\p{Lu}]]  Any letter except an uppercase letter (subtraction)
     
    Boundary matchers
    ^ The beginning of a line
    $ The end of a line
    \b A word boundary
    \B A non-word boundary
    \A The beginning of the input
    \G The end of the previous match
    \Z The end of the input but for the final terminator, if any
    \z The end of the input
     
    Greedy quantifiers
    X? X, once or not at all
    X* X, zero or more times
    X+ X, one or more times
    X{n} X, exactly n times
    X{n,} X, at least n times
    X{n,m} X, at least n but not more than m times
     
    Reluctant quantifiers
    X?? X, once or not at all
    X*? X, zero or more times
    X+? X, one or more times
    X{n}? X, exactly n times
    X{n,}? X, at least n times
    X{n,m}? X, at least n but not more than m times
     
    Possessive quantifiers
    X?+ X, once or not at all
    X*+ X, zero or more times
    X++ X, one or more times
    X{n}+ X, exactly n times
    X{n,}+ X, at least n times
    X{n,m}+ X, at least n but not more than m times
     
    Logical operators
    XY X followed by Y
    X|Y Either X or Y
    (X) X, as a capturing group
Filter Examples

Here are some example settings that can be used to help filter out necessary files in a transfer:

  1. Mode: Exclude
    Target: File name
    Phrase: *.pdf
    Result of settings:
    Single phrase filter that transfers all files that do not end with the '.pdf' extension
  2. Mode: Exclude
    Target: File path
    Phrase: folder1/**/*.docx,folder3/**/*.png
    Result of settings:
    Two phrase filter that transfers all files expect '.docx' files found in folder1, and '.png' files found in folder3
  3. Mode: Include
    Target: File name
    Phrase: *.mov,*.mp4
    Result of settings:
    Two phrase filter that transfers all files that end with the '.mov' or '.mp4' extensions
  4. Mode: Include
    Target: File path
    Phrase: **/videoProduction/**/4K Footage/**/*.mkv
    Result of settings:
    Single phrase filter that transfers all '.mkv' files that exist within the '4K Footage' sub-folders contained inside the 'videoProduction' parent folder.
  5. Mode: Reg Expr
    Target: File name
    Phrase: ".*?\.dat$"
    Regular Expression Definitions:

    Expression Section Description
    ".*?" Any character, 0, 1, or multiple times with reluctant matching
    "\." Followed by a period (you must escape the dot character or it will be treated as a wildcard)
    "dat" The exact characters 'dat' all lowercase
    "$" Must be at end of the line
    Result of settings:
    Filter that transfer only files that end with ".dat". Windows wildcard equivalent of "*.dat"
  6. Mode: Reg Expr
    Target: File name
    Phrase: ".*?[0-9]+?.*?\.(mp3|wav)$"
    Regular Expression Definitions:

    Expression Section Description
    ".*?" Any character, 0, 1, or multiple times with reluctant matching
    "[0-9]+?" Any numerical digit one or more times with reluctant matching
    ".*?" Any character, 0, 1, or multiple times with reluctant matching
    "\." Followed by a period (you must escape the dot character or it will be treated as a wildcard)
    "(mp3|wav)" Characters may match exactly "mp3" OR "wav"
    "$" Must be at end of the line
    Result of settings:
    Filter that transfer sound files (.mp3 or .wav) with numbers in the title
  7. Mode: Reg Expr
    Target: File name
    Phrase: "^.*\.(?!tmp$)[^.]+$"
    Regular Expression Definitions:

    Expression Section Description
    "^.*" Matches starting characters, 0, 1, or multiple times.
    "\." Followed by a period (you must escape the dot character or it will be treated as a wildcard)
    "(?!tmp$)" Negative lookahead that excludes files with the extension "tmp"
    "[^.]+" Match everything not eliminated by the previous negative lookahead.
    "$" Must be at end of the line
    Result of settings:
    Filter that transfer everything except temporary files ending in ".tmp"
  8. Mode: Reg Expr
    Target: File path
    Phrase: "^.*\.(?!(html|js|png)$)[^.]+$"
    Regular Expression Definitions:

    Expression Section Description
    "^.*" Matches starting characters, 0, 1, or multiple times.
    "\." Followed by a period (you must escape the dot character or it will be treated as a wildcard)
    "(?!(html|js|png)$)" Negative lookahead that excludes extensions that match "html", "js", or "png"
    "[^.]+" Match everything not eliminated by the previous negative lookahead.
    "$" Must be at end of the line
    Result of settings:
    Filter that transfers everything except files ending in ".html", ".js", or ".png"
Filter Target

This option allows you to specify the type of item that the file filter will apply to. The currently available options are as follows:

  • Directory and file name: Applies the file filter to both file names and directory names.
    • Note 1: If a directory does not match the current filter settings, its contents will not be searched for other files that would have otherwise matched.
  • File name: Applies the file filter to only file names. All directories that are found by the filter are automatically explored for other files that may match the current filter settings
  • File path: Applies the file filter to the file path for a given file.

    • Note 1: Path-based filtering is relative to the source location currently defined by the transfer's current direction. If the task has an upload direction, then all path-based filter checks will be made relative to the current HotFolder location. If the task is a download, then all path-based filter calls will be made within the context of the currently defined remote folder. If no remote folder is currently defined, then the path based filter calls will be relative to the Server's home directory.
    • Note 2: If the task is a download task and the filter phrase starts with a leading slash, then all path based filter checks will be made within the context of the site's established home directory.
Transfer Files Based on Size

This option allows you to specify whether the HotFolder should transfer all files or only files within the specified size limits.

Transfer Files Newer than X days

This option allows you to specify that only files newer than the given value (in days) will be transferred. This is based on the current modification time of the file, not when it was placed into the folder.

Filter Test

The filter tester will allow you to view the files that will be transferred with the given filters before applying them to the task, this is especially useful when experimenting with complicated regular expressions. Using the "Test" button will populate the text field with all of the files that will be allowed under the given filter, "Apply" will save your settings to the "Fileset" panel but will still need to be applied via the "Fileset" "Apply" button.

Transfer Cache

The file transfer cache is a history of all the files you have uploaded/downloaded from the server for this task. This is used to indicate whether a file has changed on the server and has been flagged for upload or download again. File changes are tracked using the size and the last modified timestamp. These values are updated when a change is detected and the file is redownloaded. Thus, even if you remove the files from the local directory, they will not be downloaded again unless the cache is cleared. When this cache is deleted, it will re-download all files from the server. In the case of upload tasks, only files that have been changed locally will be uploaded again.

Note: Transfer cache is enabled and locked if multi-file transfers is selected and requeueing of running task is allowed.

The cache database directory is saved in the install directory as .fcdb by default. To save it somewhere else, specify the path in fchf.conf file with the following parameter key:

FC.config.derby.root=CACHEPATH

where CACHEPATH is replaced with the path of your target directory.

Clear Cache

Clears the current transfer cache entries. Files will be re-transferred even if they have previously been transferred.

Generate Cache

Generates the transfer cache without performing a transfer. This option is useful if both source and destination are already in sync. Cache generation occurs upon next task execution.

Remove File

When clicked, the user will have the ability to select a file for removal from the transfer cache. A browse dialog is presented showing the files in the source folder for this task. When a file is selected, if it is present in the transfer cache, it will be removed. If there was an error removing the file from the cache (i.e. it was not present), an error will be shown.

Note: When connecting to HotFolder service via remote admin, and attempting to remove a file from the transfer cache in an upload task, the full path of the file to be removed must be entered manually.

Always transfer directory structure (even if empty)

Empty directories are not transferred unless this is selected. If a filter hides all files in a directory, it is considered empty.

Transfer in priority sequence

This will apply a order the sequence of file transfers.

Oldest

Transfers files by last modified from oldest to newest.

Newest

Transfers files by last modified from newest to oldest.

Largest

Transfers files by size from largest to smallest.

Smallest

Transfers files by size from smallest to largest.

Synchronization Options

Delete files on destination not existing on source.

This option forces the source (HotFolder on uploads, Server on downloads) to make sure that both directories contain exactly the same fileset. This includes deletion of files which do not appear on the source.

File details (send details for Upload tasks, get details for Download tasks).

These flags are for a special configuration where two HotFolders want to synchronize files with each other with a server in between acting as a transit point (small temporary cache, not permanent file storage).

By enabling this feature, the source HotFolder will forward file listing information from the source in addition to the files themselves, allowing the destination HotFolder to download both the files and the listing and take appropriate actions.

The following options should be enabled to get this configuration setup:

On the Source HotFolder:

  • "Transfer Cache" should be enabled (ensuring the same file is not sent up twice on the server)
  • "Send details for Remote HotFolder synchronization" should be enabled.

On the Destination HotFolder:

  • "Get remote HotFolder synchronization details" should be enabled.
  • In Post Task tab, "Delete source file(s)" should be enabled.

This ensures that the destination HotFolder remain in-synch (Source HotFolder acting as the master repository), and that even through a Server acting as an intermediary, file creating and deletion events will be synchronized to the destination HotFolder.

Delete files on Destination that do not exist at source

When this item is checked, any files that exists on the destination that does not exist at the source will be deleted. In this way, the destination folder will always be kept in sync with the source folder because any new files added will be set to the destination, while anything removed will be deleted.

File Priority Settings

If this option is enabled, FileCatalyst allows you to specify which files should be transferred first. Depending on the number of files to be transferred, this setting may cause a delay before the actual transfer begins. This is a result of the file being sorted into the proper sequence.

Post Task

After a Transfer is Complete

Allows the user to select what to do with the files after they have been transferred. You can choose either to do nothing, move to a folder called "Sent" under the HotFolder directory (Uploads only), or delete them.

Post URL

To allow integration with web applications, HotFolder can perform an HTTP POST to a URL or launch a local script following the completion of a task.

If this field is set to a URL, the contents of the HTTP POST will include the following parameters:

  • "f"- contains a | (pipe) delimited list of the full remote paths that were transferred up until a cancelation. Files never transferred will not appear in this list.
  • "lf" - contains a | (pipe) delimited list of the full local paths that were transferred or schedule to be transferred
  • "s" - contains a | (pipe) delimited list of the sizes of each file
  • "status" - Status of each file transferred. "0" = File transfer was not attempted, "1" = File transfer was successful, "2" = File transfer was cancelled, "3" = File transfer failed with an error. For example, if the transfer included 5 files, and all files were transferred successfully, you would see status=11111. If the transfer was cancelled during the 3rd file, you would see status=11200.
  • "allfiles"- contains a | (pipe) delimited list of the full paths that were transferred or schedule to be transferred, this list has a 1 to 1 relationship with the status value where each status value corresponds to each value in allfiles

If this field is set to a script, the script will be called upon completion of the task. There are 4 arguments that can be passed to the script by FileCatalyst.

  • <remotefiles> - contains a | (pipe) delimited list of the full remote paths that were transferred or schedule to be transferred
  • <localfiles> - contains a | (pipe) delimited list of the full local paths that were transferred or schedule to be transferred
  • <sizes> - contains a | (pipe) delimited list of the sizes of each file
  • <statuscodes> - Status of each file transferred. "0" = File transfer was not attempted, "1" = File transfer was successful, "2" = File transfer was cancelled, "3" = File transfer failed with an error. For example, if the transfer included 5 files, and all files were transferred successfully, you would see status=11111. If the transfer was cancelled during the 3rd file, you would see status=11200.
  • <allfiles>- contains a | (pipe) delimited list of the full paths that were transferred or schedule to be transferred, this list has a 1 to 1 relationship with the status value where each status value corresponds to each value in allfiles

As an example, you may set the Script to:

/home/user/script.sh <remotefiles> <localfiles> <sizes> <statuscodes> <allfiles>

and FileCatalyst will automatically insert the proper values into the command line before execution.On Windows you may execute a batch file like this:

cmd /c start c:\\file.bat <remotefiles> <localfiles> <sizes> <statuscodes> <allfiles>

Note 1: If the Post URL field is set to a script, and the script is called upon completion of a task, the task will block until the script fully completes. This completion includes the originating script and any sub-processes that the script creates when it executes.

Enable E-Mail Alerts

If this is enabled and the client specifies to send an e-mail on completion of a transfer, then an e-mail will be sent to the address provided whenever a transfer is finished.

Note: E-mail messages will only be sent if the Server has properly configured SMTP settings, and has enabled the e-mail alerts setting. If you wish to have e-mail alerts for your task, please contact your Server administrator to verify that the Server's SMTP settings are correct, and that the e-mail alerts option is enabled.

E-Mail Address

Email destination: If Server Administrator is selected, the email will be sent to the administator address configured on the Server. Otherwise you can sellect Custom Address and enter an email address. If you wish to send email to multiple addresses, they must be separated with a semicolon(;).

Send E-Mail on Successful Transfer

Indicates whether or not to send an e-mail on successful transfers.

Custom Message Text on Successful Transfer

This message is sent as the e-mail notification on a successful transfer.

Send E-Mail on Transfer Error

Indicates whether or not to send an e-mail when a transfer error occurs.

Custom Message Text on Transfer Error

This message is sent as the e-mail notification when a transfer error occurs.

Note: With the introduction of Alarms to HotFolder, it is recommended that FileCatalyst Central be instead used for capturing transfer faults. Network outages or user authentication errors are not handled by Task e-mail notifications.

Bandwidth Management

Overview

Why Manage Your Bandwidth?

Your network is used for email, applications, web site access and any other form of corporate communication. All of these tools utilize and require a percentage of your overall bandwidth. If these tasks do not have sufficient bandwidth, they will begin to feel slow and unresponsive.

FileCatalyst is capable of maximizing usage of your allocated bandwidth. For example, if your organization has a T3 line, FileCatalyst is able to transfer at 45Mbps. However, this may not be practical during peak periods of the day when FileCatalyst may need to share available bandwidth with other applications.

Users themselves may manage and schedule their bandwidth usage by using the Bandwidth scheduler (see below). The user may even specify periods during which there should be no transfer activity by choosing "No Transfers" for the selected time.

Selecting an Appropriate Rate

The bandwidth scheduler allows users to limit or cap the level of bandwidth usage for a period of time. Example: if you have a T1 connection, you may wish to allocate 500 kbps of data transfer during the day and leave the remaining 1044 kbps for office use. However, at 6:00 PM you may wish to fully allocate 95% or 1467 kbps of bandwidth for data transfer. It is up to the users and the FileCatalyst administrator(s) to create an appropriate schedule.

Note:Congestion Control also automates the FileCatalyst transfer rate as networks become congested. As additional network traffic comes on-line and creates congestion, FileCatalyst will slow down or throttle back the transfer rate. It will remain in this state until the network is no longer congested. At which point the transfer will speed up until it resumes its set rate. This is a selectable feature which is enabled by editing individual tasks.

Bandwidth Scheduler

The Bandwidth Scheduler is found on the "Bandwidth" view, accessed from the sidebar navigation:

Bandwidth

Some of the optimizations described above depend on the Bandwidth Scheduler. To change the rate at a time slot (or variety of time slots), select the cells corresponding to the date/time of day you wish to change, and either click Select Rate what will display a dialog to enter a custom values, or press the to select one of the preset values.

Administration

By default, HotFolder may be administered on its local machine. However, you may also allow the application to have remote administration configured with a few important options. Navigate to the "Administration" panel to change options:

Administration Panel

Port Number

Specifies the port used by remote users to connect to this HotFolder.

Login Required for Access

Specifies whether or not to allow connections from computers outside of 'localhost'. If enabled, a username and password must be specified and used to connect to HotFolder from any system outside of the local machine.

Access Settings

The access settings will allow internet access to the administration of the FileCatalyst HotFolder. The web port can be customized as needed (by default it is port 12580). Choose the IP format based on internal or external access to the administration URL. Enable Remote Connections must be selected for Web Access to be available.

Note 1: Ports for Administration Settings and Web Port need to be opened on the firewall if one is in place. If the ports are configured to be closed on 3 the firewall, then no data transfer will be possible. With no data transfer possible, no internet applications will be able to access the FileCatalyst HotFolder.

Note 2: If the Bind all Interfaces option is selected, the FileCatalyst HotFolder will accept all internet access requests that have an IP format listed within the available dropdown. If the option is not selected, then only access requests matching the specified IP will be accepted by the HotFolder application.

The 'Webroot' URL is the entry point for web access the administration of the FileCatalyst HotFolder.

Reporting

Overview

FileCatalyst provides well defined Data and Bandwidth CSV files that may be used by 3rd party applications to generate reports over defined periods of time. The Data file will contain a record for every individual file transfer that takes place including uploads and downloads.

The Bandwidth file will contain records that include transmit and receive rate as well as username and a time stamp. The frequency of these records is determined by adjusting the report snapshot interval. If there are no active transfers at the time of the snapshot, there will be no record written. If there is data in the snapshot, a record will be written for each individual user that is currently transferring, as well as a record that is the sum of all transmit and receive rates for all users.

Note: If only one snapshot exists for the selected period, no line will be drawn.

Settings & Run Report

Navigate to "Reporting" from the main navigation to see two related panels:

Settings and Execute reports Form

Click the checkbox to enable reporting and press Apply. Define the location for reports in the first field, followed by the Retention Period and the Snapshot interval. The Retention Period is used to define how long logs are kept for before they are purged. Report Snapshot Interval refers to the granularity of the bandwidth data information being captured. Shorter intervals provide the most information, but at the cost of system performance and report file size.

Select a type of report and a range of time. Click Run Report to generate a graph of the statistics requested. The blue line indicates the total receive rate of the HotFolder (files being downloaded from the server).

Report Result

Central Management

The settings on the "Central Mgt" panel allow a FileCatalyst HotFolder to connect to FileCatalyst Central:

Central Management Form

Remote monitoring

Check the "Enable remote monitoring" box, and then configure the information in the panels provided. The first panel allows you to set the Address, Port, User (username), and Password for the connection to FileCatalyst Central. The HotFolder panel allows the administrator to specify which information is provided to Central, including the IP/Domain, and an Alias which Central will use as the display name for the HotFolder.

Enabling connection to FileCatalyst Central will also initiate report generation, so that Central can download transfer activities to it's database.

Trigger Alarms on Transfer

In addition to Service Down alarms (which are on by default if Alarm Monitoring is enabled), HotFolder can optionally send three types of notifications depending on whether a task finishes successfully, is canceled, or has an error.

Monitor

Select "Monitor" from the main navigation in order to see the bandwidth usage of current transfers in near-realtime:

A drop-down menu allows you to monitor the usage of a particular task, or "Overall Usage" for aggregated bandwidth consumption.

Advanced

Select "Advanced" from the main navigation to reveal the two advanced configuration tabs, License and System Properties.

Proxy

HotFolder Proxy

Depending on your network configuration, it may be necessary to configure HotFolder to use a proxy for some or all of the protocols that it uses for transferring files. The Proxy tab allows you to configure how HotFolder accesses the Internet.

By default HotFolder uses the system defaults however, you can configure the application to not use proxies at all or to specify particular proxies for the protocols used by HotFolder. If you are manually configuring specific values for the different protocols, please check with your system administrator for the correct values.

Proxy information can be input here if you want HotFolder to do HTTP/HTTPS transfers through a proxy.

Note: the "No proxy for" field will accept a list of hosts for which the proxy will be bypassed. Mulitple hosts must be separated with a "|".

Miscellaneous

HotFolder Miscellaneous

This tab allows the user to modify a number of advanced and regular option.

Enable Advanced View

Toggles hiding some more advanced features from the application. Enabling this will show all features the HotFolder can be configured with.

Polling

HotFolder will periodically wake up and poll all Sites and HotFolder directories to make sure that they are on-line and available. Should a Site or HotFolder become unavailable (server not seen on the network, mount point not available), the application will let the user know via a warning label on the Site or HotFolder list.

SSL Configuration

FileCatalyst HotFolder has an embedded web server used for remote administration via a web browser. Enable this option to use SSL encryption (HTTPS) when loading the remote administration from your browser.

Disable Force Flush

By default, write operations automatically flush both data and meta data to file system for all IO calls. Disabling force flush allows Java to utilizes the OS file cache (RAM). This option is only visible in advanced mode.

System Properties

Used for debugging purposes to set advanced system properties. System properties are case sensitive. Please contact support before attempting to use these features.

To add a new system property, click the "New System Property" button. A pair of fields will appear, representing the key (or "name") and the "value" of the new system property. Click the checkmark button to finish adding the new system property.

Press the discard icon next to an existing property value field in order to remove it.

Some properties may require a restart for these values to take effect.

UDP Resources

When configuring the FileCatalyst HotFolder to achieve high speed levels (500Mbps - 10Gbps for a single connection), several optimizations must be done in order to make sure the system is able to utilize multiple cores of a machine. These include configuring the number of reader threads per connection, packet processors (used for decoding AES packets on the receiver side), and the number of block writers.

# Reader threads

Number of threads PER CONNECTION spawned to read blocks off the disk and place them into the sender queue ready for transport. On the HotFolder, setting this value will only affect uploads. Most systems perform optimally when you configure only 1 thread to perform reading. There are two scenarios where this value should be moved up: when IO tests have shown that adding threads increases throughput for your storage device, or when utilizing high CPU load (more sender threads than CPU cores, or utilizing high CPU options such as AES encryption or LMZA compression), a single thread doing the block reads may be pushed down in priority and be CPU starved, causing a slowdown in transfer speeds. Multiple threads may help both of these scenarios.

Reader block size

By default, the block size used for each read operation is the same size specified by the client in the network block size. This can be adjusted for improved performance. Please run the advanced IO testing tool to see what works best for your system.

Packet processors

Packet processors are responsible for taking packets off the network socket, decrypting them, and preparing them for the block writers. When using AES at speeds greater than 200Mbps, it may be required to increase the number of packet processors on the receiving end to ensure multiple CPU's are utilized to decrypt the incoming data.

On the HotFolder, setting this value will only affect downloads. If uploading data to the server, similar settings should be set on the server applications.

# of Writer threads

Number of threads PER CONNECTION spawned to write blocks off the disk. On the HotFolder, setting this value will only affect downloads. Writer threads in FileCatalyst perform the following tasks:

  • Uncompress data block (if using on-the-fly compression)
  • Perform MD5 checksum on block (if using on-the-fly Verification)
  • Write block data to disk

For systems using LMZA compression or possibly high speed transfers (> 1Gbps), setting multiple block writers may allow higher performance for single high speed connections.

Writer block size

By default, the block size used for each write operation is the same size specified by the client in the network block size. This can be adjusted for improved performance. Please run the advanced IO testing tool to see what works best for your system.

These values should be modified only after running through some performance IO testing.

Performance Tuning

The FileCatalyst Performance Tuning documentation can be found here

Support

Support System

Visit our support website at https://support.filecatalyst.com to view the knowledge base and to submit a ticket (Available 24/7).

If you have already submitted a Ticket and would like to send us your case files please go to https://support2u.filecatalyst.com to upload them. A valid Ticket ID will be required.

Support Information

For support contact information, support hours and live chat: Visit our website at https://support.filecatalyst.com