Too many recursive listings

[Pete]

Using FTPS, when I first connect to Fastmail, Cirrus does a LIST -aR. At this point I assume that Cirrus has the complete folder structure in memory. So it seems wrong and inefficient that it repeats the LIST -aR command every time that I double click on a folder to look at the files inside a folder.

I don't use the MLSD option.

In general, I'm very pleased with the performance and responsiveness of Cirrus and its FTPS implementation, but I just want to make it as fast as possible.

(I'm using Build 441 on WinXP against the file storage at Fastmail.)

[Zoë]

Fastmail's FTP server ignores the -R list option.

[Pete]

Craig, thanks, then using LIST -R doesn't affect performance with Fastmail.

However, it also happens with Serv-U. The initial remote directory in my case is the root, and Cirrus gets a listing for the entire folder tree. Then, every time I double click on one of the folders in the root folder, Cirrus asks the server for another recursive listing (for that folder branch).

Similarly, when I do a folder compare against a Serv-U server, Cirrus is doing a recursive listing in *every* folder on the server.

[Chris]

Pete,

Thanks for posting about the -R problems with Serv-U.

Do you know what version of Serv-U is running on the problem server?

Are you using FTP or FTPS to connect to the Serv-U server?

What options do you have checked in the "Listings" tab of the FTP profiles dialog for the server?

[Pete]

Chris,

The FTP server is Serv-U 6.4.0.2 (free version) on a WinXP Home Edition computer. Cirrus is running on a different XP Home computer. This is with plain FTP.

Options in the "Listings" tab that are enabled:

[X] Show hidden
[X] Recursive

Link Resolution: Fast

Nothing else is enabled in this tab. Thanks for investigating this.

[Zoë]

Pete,

Cirrus won't (shouldn't) ask for a directory listing for any directories that have already been retrieved because of a higher recursive listing. Us asking for a recursive listing doesn't guarantee that the server will send one, but I tested Serv-U 6.3 and 6.4 and both support -R and neither includes an obvious way to disable that support. Can you send or post a copy of a log showing the incorrect behavior?

If you want to do any troubleshooting on your side, you could try loading a site both with and without the recursive option checked. If it is returning recursive listings and Cirrus isn't handling them having the option checked should be slower than having it unchecked, since Serv-U would have to do more to produce the listings.

[Pete]

Given the following folder structure on the remote FTP server:

...
Top-Level Folder
   a1
      b1
      b2
   a2
...

Cirrus generates the following in the log:

09-Dec-2007 17:11:08 Stat> Connected.
09-Dec-2007 17:11:08 Recv> 220 Serv-U FTP Server v6.4 for WinSock ready...
09-Dec-2007 17:11:08 Sent> USER Nancy
09-Dec-2007 17:11:08 Recv> 331 User name okay, need password.
09-Dec-2007 17:11:08 Sent> PASS ********
09-Dec-2007 17:11:08 Recv> 230 User logged in, proceed.
09-Dec-2007 17:11:08 Sent> FEAT
09-Dec-2007 17:11:08 Recv> 211-Extension supported
09-Dec-2007 17:11:09 Recv> CLNT
09-Dec-2007 17:11:09 MDTM
09-Dec-2007 17:11:09 MDTM YYYYMMDDHHMMSS[+-TZ];filename
09-Dec-2007 17:11:09 SIZE
09-Dec-2007 17:11:09 SITE PSWD;EXEC;SET;INDEX;ZONE;CHMOD;MSG
09-Dec-2007 17:11:09 REST STREAM
09-Dec-2007 17:11:09 XCRC filename;start;end
09-Dec-2007 17:11:09 MODE Z
09-Dec-2007 17:11:09 MLST Type*;Size*;Create;Modify*;Win32.ea*;
09-Dec-2007 17:11:09 211 End
09-Dec-2007 17:11:09 Sent> CLNT Cirrus v3.0
09-Dec-2007 17:11:09 Recv> 200 Noted.
09-Dec-2007 17:11:09 Sent> TYPE I
09-Dec-2007 17:11:09 Recv> 200 Type set to I.
09-Dec-2007 17:11:09 Sent> SYST
09-Dec-2007 17:11:09 Recv> 215 UNIX Type: L8
09-Dec-2007 17:11:09 Sent> SITE ZONE
09-Dec-2007 17:11:09 Recv> 210 UTC-300
09-Dec-2007 17:11:09 Sent> PWD
09-Dec-2007 17:11:09 Recv> 257 "/c:/Documents and Settings/Nancy/My Documents" is current directory.
09-Dec-2007 17:11:09 Sent> CWD /n/Top-Level Folder
09-Dec-2007 17:11:09 Recv> 250 Directory changed to /n:/Top-Level Folder
09-Dec-2007 17:11:09 Sent> TYPE A
09-Dec-2007 17:11:09 Recv> 200 Type set to A.
09-Dec-2007 17:11:09 Sent> PORT 192,168,0,7,17,115
09-Dec-2007 17:11:10 Recv> 200 PORT Command successful.
09-Dec-2007 17:11:10 Sent> LIST -aTR
09-Dec-2007 17:11:10 Recv> 150 Opening ASCII mode data connection for /bin/ls.
09-Dec-2007 17:11:10 Recv> 226 Transfer complete.
09-Dec-2007 17:11:10 Load comparison: ftp://[email protected]//n/Top-Level Folder <-> N:\Databases\Nancy\Personal\Top-Level Folder
09-Dec-2007 17:11:10 Sent> CWD /n/Top-Level Folder/a1
09-Dec-2007 17:11:10 Recv> 250 Directory changed to /n:/Top-Level Folder/a1
09-Dec-2007 17:11:10 Sent> PORT 192,168,0,7,17,117
09-Dec-2007 17:11:10 Recv> 200 PORT Command successful.
09-Dec-2007 17:11:10 Sent> LIST -aTR
09-Dec-2007 17:11:10 Recv> 150 Opening ASCII mode data connection for /bin/ls.
09-Dec-2007 17:11:10 Recv> 226 Transfer complete.
09-Dec-2007 17:11:11 Sent> CWD /n/Top-Level Folder/a2
09-Dec-2007 17:11:11 Recv> 250 Directory changed to /n:/Top-Level Folder/a2
09-Dec-2007 17:11:11 Sent> PORT 192,168,0,7,17,119
09-Dec-2007 17:11:11 Recv> 200 PORT Command successful.
09-Dec-2007 17:11:11 Sent> LIST -aTR
09-Dec-2007 17:11:11 Recv> 150 Opening ASCII mode data connection for /bin/ls.
09-Dec-2007 17:11:11 Recv> 226 Transfer complete.
09-Dec-2007 17:11:11 Sent> CWD /n/Top-Level Folder/a1/b1
09-Dec-2007 17:11:11 Recv> 250 Directory changed to /n:/Top-Level Folder/a1/b1
09-Dec-2007 17:11:11 Sent> PORT 192,168,0,7,17,121
09-Dec-2007 17:11:11 Recv> 200 PORT Command successful.
09-Dec-2007 17:11:11 Sent> LIST -aTR
09-Dec-2007 17:11:11 Recv> 150 Opening ASCII mode data connection for /bin/ls.
09-Dec-2007 17:11:12 Recv> 226 Transfer complete.
09-Dec-2007 17:11:12 Sent> CWD /n/Top-Level Folder/a1/b2
09-Dec-2007 17:11:12 Recv> 250 Directory changed to /n:/Top-Level Folder/a1/b2
09-Dec-2007 17:11:12 Sent> PORT 192,168,0,7,17,123
09-Dec-2007 17:11:12 Recv> 200 PORT Command successful.
09-Dec-2007 17:11:12 Sent> LIST -aTR
09-Dec-2007 17:11:12 Recv> 150 Opening ASCII mode data connection for /bin/ls.
09-Dec-2007 17:11:12 Recv> 226 Transfer complete.

The second strange thing about this (maybe unrelated) is that it's using the -T option to the LIST command even though that checkbox is not enabled in my FTP profile.

[Zoë]

Pete,

Can you try the same test with the new release. If Cirrus is working correctly it should stop putting the -R in the LIST command after the first one. If Cirrus detects that it's connected to a Serv-U server it will automatically use the -T option because we know it supports it.

[Pete]

Craig, thanks for the great improvements to recursive listings. I tried it with both Fastmail and Serv-U. Cirrus now only does one recursive listing for the entire tree (done once at the beginning of the folder compare). Also, I can now double click on a folder on the ftp side of Cirrus to see the files inside and Cirrus doesn't connect to the server again. Nice.

[Pete]

Edit: I know that Fastmail doesn't support recursive listings but I mentioned it because I confirmed that Cirrus now uses LIST -a instead of LIST -aR.

[Zoë]

Pete,

Just so I'm clear, recursive listings on your Serv-U server are working again and it's only asking for the listing of the top directory?

[Pete]

Yes, Craig, there is only one recursive listing with Serv-U and it happens at the top folder. As far as I can tell, it's working perfectly. Thanks.