Format rat and "-1"

[David Bateman]

Ben Abbott wrote:
> On Tuesday, June 30, 2009, at 03:56PM, "David Bateman" <dbateman at dbateman.org> wrote:
>   
>> Eduardo Garcia wrote:
>>     
>>> Cc: Eduardo Garcia
>>> --------
>>> Bug report for Octave 3.2.0 configured for i386-apple-darwin8.11.1
>>>
>>> Description:
>>> -----------
>>>
>>>    When using "format rat" sometimes "-2147483648/-2147483648" is  
>>> displayed instead of "-1".
>>>    (Same bug is present in Octave 3.0.5 configured for i386-apple- 
>>> darwin8.9.1)
>>>
>>> Repeat-By:
>>> ---------
>>> BUG:
>>> octave-3.2.0:23> a=[1 2;3 5]
>>> octave-3.2.0:24> det(a)
>>> ans = -1.0000
>>> octave-3.2.0:25> format rat
>>> octave-3.2.0:26> det(a)
>>> ans = -2147483648/-2147483648
>>>
>>> BUT:
>>> octave-3.2.0:29> a=[1 2;1 1]
>>> octave-3.2.0:30> format
>>> octave-3.2.0:31> det(a)
>>> ans = -1
>>> octave-3.2.0:32> format rat
>>> octave-3.2.0:33> det(a)
>>> ans = -1
>>>   
>>>       
>> Interesting problem.... What to do for the format rat when the numerator 
>> and/or the denominator is outside the valid integer range.. A simpler 
>> test example is
>>
>> octave:1> format rat
>> octave:2> 1 - eps
>> ans = -2147483648/-2147483648
>>
>> It would be interesting to know what matlab display for the cases 1 - 
>> eps,  1 - 10*eps, 1 - 1e3*eps and maybe a couple of other similar values 
>> before looking at a proper fix.
>>
>> D.
>>     
>
> I'm at a loss to come up with a cute punch line ... but the results are below ;-)
>
>   
>>> format rat 
>>> 1 - eps
>>>       
>
> ans =
>
>        1       
>
>   
>>> 1 - 10*eps
>>>       
>
> ans =
>
>        1       
>
>   
>>> 1 - 1e3*eps
>>>       
>
> ans =
>
>        1       
>
> Ben
>
>
>   
Ok then the attached patch that I committed to savannah should fix 
this.. It should probably be applied to the 3.2 tree as well.

D.


-- 
David Bateman                                dbateman at dbateman.org
35 rue Gambetta                              +33 1 46 04 02 18 (Home)
92100 Boulogne-Billancourt FRANCE            +33 6 72 01 06 33 (Mob)

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: patch
Url: https://www-old.cae.wisc.edu/pipermail/bug-octave/attachments/20090701/d59adb45/attachment.ksh