AWS Releases Open-Source File Mover Express for S3 Transfers

New Open-Source Transfer Tool

AWS has released File Mover Express, an open-source application licensed under Apache 2.0. This tool is designed for transferring large files between local systems and Amazon S3. It is available for macOS, Windows, and Linux operating systems.

Key Features and Capabilities

File Mover Express supports parallel transfers, automatic performance tuning, retries, and pause and resume functionality. It also includes file integrity verification and automatic uploads from monitored folders. The tool provides both a desktop graphical user interface (GUI) and a scriptable command-line interface (CLI).

The core of File Mover Express is a daemon-based transfer engine, which allows transfers to continue independently of the desktop session. This daemon can run on a workstation, data center server, or an Amazon EC2 instance, controlled remotely via an encrypted connection.

Addressing Industry Needs

The tool targets challenges faced by media teams and other users who regularly transfer large volumes of data, often terabytes, between on-premises storage and the cloud. Traditional transfer clients may interrupt transfers if a desktop session ends, while commercial alternatives can incur licensing costs or rely on proprietary protocols. File Mover Express supports both uploads and downloads, enabling teams to move files to S3 and retrieve them for various workflows.

Technical Implementation

File Mover Express is primarily developed in Go for its transfer engine and TypeScript with Angular for its graphical interface. It splits large files into chunks and uploads them in parallel using multipart uploads. The autotuning mode adjusts the number of parallel streams and chunk sizes to optimize throughput based on the available connection. Security features include file encryption in transit and at rest.