Octave suddenly slow

[John W. Eaton]

On 18-Nov-2008, David Bateman wrote:

| Rob Mahurin wrote:
| > Adding 10^4 empty files to a directory makes octave open a couple 
| > seconds slower; adding 10^5 empty files makes it take eight minutes.  I 
| > have heard of bugs in filesystems that cause this problem, but that 
| > doesn't seem to be the case since "ls | wc" is fast in the shell.
| > 
| > The filesystem bug I remember (or maybe an ls bug? or both?) was an 
| > inefficient sort in the code that lists the names of files in a 
| > directory.  Poking through the code, I don't see anything like that.
| > 
| > Hmmmm ... dir_entry::read() seems to guess that it will live in a small 
| > directory.  For the case I've outlined above, the attached patch should 
| > reduce the number of calls to Array::resize() from ~1000 to ~10.  Not 
| > tested, though.
| 
| 
| The conclusive manner to check would be with gprof. But that requires a 
|   static build with
| 
| CFLAGS="-g -pg -O2"
| CXXFLAGS="-g -pg -O2"
| FFLAGS="-g -pg -O2"
| LDFLAGS="-g -pg -O2"
| ./configure --enable-static --disable-dl --disable-shared
| I previously noticed that file_ops::tilde_expand consumed a lot of time 
| in gprof at startup.
| 
| http://www.nabble.com/forum/ViewPost.jtp?post=12531612&framed=y
| 
| So perhaps the attached change to file_ops::tilde_expand might also 
| help. For me with this change I see with the following with the command
| 
| time octave --eval quit
| 
| the wall times are
| 
|                  Normal   1e4 files    1.1e5 files
| With Patch       0.760    1.059         8.414
| Without Patch    1.116    2.801        20.786
| 
| So for me the speeds are significantly faster with my patch (note this 
| is without any packages installed). I don't think your patch is quite 
| right as you start with a zero sized array with len not equal to zero. I 
| prefer a patch like
| 
| --- a/liboctave/dir-ops.cc
| +++ b/liboctave/dir-ops.cc
| @@ -87,7 +87,7 @@
|   	    {
|   	      if (count >= len)
|   		{
| -		  len += grow_size;
| +		  len = (len == 0 ? grow_size : len * 2);
|   		  dirlist.resize (len);
|   		}
| 
| Adding this and my previous patch gives a startup time of
| 
|                  Normal   1e4 files    1.1e5 files
| With Patches     0.831    1.025         3.516
| Without Patches  1.116    2.801        20.786
| 
| So the two patches together give a significant improvement. John added a 
| slighly different change to dir_entry::read just now so I only committed 
| my change to tilde_expand..

The big performance problem was that Octave would stat each file in
each directory in the load path that had a relative name, and this
would happen every time load_path::update was called (i.e., at each
prompt, call to cd, etc).  The stat calls were happening because
load_path::dir_info::update was always calling
load_path::dir_info::initialize for directories with relative names.
That's the conservative thing to do if you don't have any other
information, because a cd could have happened since the last check,
and if it has, the information needs to be updated.  The change below
should avoid the unnecessary calls to load_path::dir_info::initialize.

Thanks,

jwe


-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: diffs
Url: https://www-old.cae.wisc.edu/pipermail/bug-octave/attachments/20081117/9d41e13d/attachment.ksh