Octave suddenly slow

[David Bateman]

Rob Mahurin wrote:
> Adding 10^4 empty files to a directory makes octave open a couple 
> seconds slower; adding 10^5 empty files makes it take eight minutes.  I 
> have heard of bugs in filesystems that cause this problem, but that 
> doesn't seem to be the case since "ls | wc" is fast in the shell.
> 
> The filesystem bug I remember (or maybe an ls bug? or both?) was an 
> inefficient sort in the code that lists the names of files in a 
> directory.  Poking through the code, I don't see anything like that.
> 
> Hmmmm ... dir_entry::read() seems to guess that it will live in a small 
> directory.  For the case I've outlined above, the attached patch should 
> reduce the number of calls to Array::resize() from ~1000 to ~10.  Not 
> tested, though.


The conclusive manner to check would be with gprof. But that requires a 
  static build with

CFLAGS="-g -pg -O2"
CXXFLAGS="-g -pg -O2"
FFLAGS="-g -pg -O2"
LDFLAGS="-g -pg -O2"
./configure --enable-static --disable-dl --disable-shared
I previously noticed that file_ops::tilde_expand consumed a lot of time 
in gprof at startup.

http://www.nabble.com/forum/ViewPost.jtp?post=12531612&framed=y

So perhaps the attached change to file_ops::tilde_expand might also 
help. For me with this change I see with the following with the command

time octave --eval quit

the wall times are

                 Normal   1e4 files    1.1e5 files
With Patch       0.760    1.059         8.414
Without Patch    1.116    2.801        20.786

So for me the speeds are significantly faster with my patch (note this 
is without any packages installed). I don't think your patch is quite 
right as you start with a zero sized array with len not equal to zero. I 
prefer a patch like

--- a/liboctave/dir-ops.cc
+++ b/liboctave/dir-ops.cc
@@ -87,7 +87,7 @@
  	    {
  	      if (count >= len)
  		{
-		  len += grow_size;
+		  len = (len == 0 ? grow_size : len * 2);
  		  dirlist.resize (len);
  		}

Adding this and my previous patch gives a startup time of

                 Normal   1e4 files    1.1e5 files
With Patches     0.831    1.025         3.516
Without Patches  1.116    2.801        20.786

So the two patches together give a significant improvement. John added a 
slighly different change to dir_entry::read just now so I only committed 
my change to tilde_expand..

Regards
David


-- 
David Bateman                                dbateman at dbateman.org
35 rue Gambetta                              +33 1 46 04 02 18 (Home)
92100 Boulogne-Billancourt FRANCE            +33 6 72 01 06 33 (Mob)
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: patch8832
Url: https://www-old.cae.wisc.edu/pipermail/bug-octave/attachments/20081118/39a678c1/attachment.ksh