We download data simultaneously from 240 separate devices that are servers with IP addresses in the 10.x.x.x range. It is done with the FTP server in Microsoft IIS. Each of the devices uploads several hundred files per session. All the same type of file.
I am somewhat familiar with the concept and configuring the IIS and FTP. I use Filezilla now and then on single devices to retrieve data files.
What I cannot get specific information on, is how the server broadcasts the download command to all the devices ( servers ) at once. I realize you can use a wildcard preceding the .XXX file type like Download *.xxx.
Is there a script somewhere that contains the command sequence in the FTP setup? Other software initiates the download using the IIS and FTP. Does the external software have to pass the commands to the FTP server?
I don’t expect a detailed tutorial. But some hints on where to look would be great.
We mostly download the data, but we also upload configurations to the devices.
Each device is a type of industrial PC running Linux. The devices are loaded into racks connected to switches, connected to the server. Our server sends a command to all the devices to send all the files of a particular type, in a particular folder to the server.
What I am wondering about is what that command is. To initiate all those separate devices, with their different IP addresses to send the files. Or do you have to send a command addressed to each one?
The downloads proceed concurrently. A very heavy data stream. But we are setup to handle it.
I wonder if the OP is talking about multicasting. I tried to set up multicasting on a DeployStudio server a few years ago, but could never get it to work right.
I doubt it is anything fancy. FTP will support getting or putting multiple files in a command with mget/mput. Using a single command and a utility like curl you could get all the files from one machine. There are utilities that allow batch running of commands across all the machines in a cluster. Usually just configured with a file of host names. FTP is awfully old. Most people would be using scp or the like. Setting up appropriate credentials for secure access makes this both secure and easy to use.
I don’t think Filezilla uses any multicasting techniques. It probably just opens a normal FTP connection to each server. Is there someplace in Filezilla where you configured the list of servers to download from?
Is the process initiated on the Windows machine, the Windows machine issues a command to the Linux machines, then the Linux machines connect to the Windows machine over FTP to transfer files?
If so, and what you’re wondering is what command the Windows machine sends, it could be anything from a simple HTTP request to some custom network call. If the documentation on the software you’re using isn’t forthcoming, the easiest way to figure it out would be to inspect the traffic between the machines.
Thanks folks.
Following up with the info here I found that it just takes a particular format of the packet address. All devices on the network will process it. So no need to individually address and send a couple hundred commands.
The website about multi casting contained good leads to solve it.
Multicast and FTP are not things that really work in the same breath. FTP is a peer to peer protocol, and runs over TCP. Each transfer requires two connections, and commands and data are exchanged over TCP. None of this works with Multicast.
Multicast allows you to send the packets out and have them received by a set of recipients. But there is no component that has the recipients sending anything back to you (part from layered reliable multicast where acknowledge packets may be used.)
The OP’s scenario can be effected in two ways really. Either the central machine initiates an FTP connection with each device, and pulls the needed files, or the devices initiate a connection and put the files onto the central machine. In the former case the central machine will initiate connections on a one by one basis - but can have the requests run in parallel. There is nothing stopping you having multiple ftp sessions running at once. In the latter case he devices could run on a timer, and independently initiate a connection to the central machine, or you could have the central machine send out a request to each device to initiate a send back session. There is no pre-canned multicast service that would allow you to do this, but it is not hard to work out how to create one. You just need a listener on a known multicast receptive port to see the request and to then initiate the transfer process. The transfer proper would run over TCP.
As I noted earlier, there are however useful utilities designed for managing compute clusters that provide parallel execution across a set of machines of commands. They don’t multicast, but you get the basic effect. (They also provide for reliable execution, something you won’t get with a simple multicast.) They include parallel distributed file transfer. For instance : https://www.linux.com/news/parallel-ssh-execution-and-single-shell-control-them-all
IIS, classic FTP, filezilla doesn’t support multicast.
What does is
uftp … a multicast FTP
http://uftp-multicast.sourceforge.net/
I think the point of the question was for you to understand multicast, and not ask us how the application uses multicast.
Clearly multicast is for sending out the file once to 200 hosts, so that the bandwidth isn’t saturated the same way as sending 200 copies.
Maybe you can multicast out the command to tirgger the ftp back. If you already had multicast setup for push, you might avoid having to maintain a complicated fetch program when you can just send one command.
Having delved into it with the help of tips here. I see that the system can use broadcast and/or individually targeted commands. But probably sends an individual command to each one. There are instances when only one device out of many is the target of the command. But overall your responses have led me to resources that I did not know of. So many thanks. This may help me create some very helpful tools for my work.