Bug 75492 - FILEOPEN DATALOSS ODT Silent corruption of formulas in TABLE
Summary: FILEOPEN DATALOSS ODT Silent corruption of formulas in TABLE
Status: VERIFIED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Writer (show other bugs)
Version:
(earliest affected)
4.2.0.0.beta1
Hardware: Other All
: high critical
Assignee: Not Assigned
URL:
Whiteboard: target:4.3.0 target:4.2.3
Keywords: bibisected, regression
Depends on:
Blocks:
 
Reported: 2014-02-25 11:11 UTC by Mike Kaganski
Modified: 2015-12-17 07:50 UTC (History)
3 users (show)

See Also:
Crash report or crash signature:


Attachments
Bug doc (1.91 MB, application/vnd.oasis.opendocument.text)
2014-02-25 11:11 UTC, Mike Kaganski
Details
Simplified bugdoc (44.79 KB, application/vnd.oasis.opendocument.text)
2014-03-07 04:24 UTC, Mike Kaganski
Details
Bugdoc simplified to minimum (4.68 KB, application/vnd.oasis.opendocument.text)
2014-03-07 04:53 UTC, Mike Kaganski
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Mike Kaganski 2014-02-25 11:11:14 UTC
Created attachment 94712 [details]
Bug doc

When opening attached ODT with LO 4.2.0.0.beta1 and higher (including 4.2.1.1), all the tables look normal. But interacting with some of the tables, or e.g. printing document, corrupts their contents - the cells contents becomes ** Expression is faulty **.
The cells that exhibit this problem contain formulas that refer to other tables, as well as some variables. Actually, the formulas are corrupted on opening, but the cells initially show cached data, so the problem is only evident on recalculation.

To see the problem, open the attached document, scroll down to page 11 and hover over column B (or D) of Table32. You may notice that the formulas are already corrupt (contain something like "=qо*<Table26.?>*(1+k1)/1000" - note the question mark instead of cell address). Until clicked, the table looks normal. Clicking inside the table immidiately makes most cells contain "** Expression is faulty **".
Opening the same document with LO 4.1.5.3 and older shows that the cells contain correct formulas like "=qо*<Table26.D2>*(1+k1)/1000".
If this file is opened with 4.2.x, then simply saved, then the corruption becomes permanent, i.e. opening such saved ODT with 4.1.x and lower gives corrupted formulas.

Tested with 4.2.0.0.beta1 - 4.2.1.1 under Win7x64 and 4.2.1.1 under Ubuntu 13.10 64-bit.
Not reproducible under 4.1.5.3 and lower -> regression.

This is a critical bug, because:
1. It affects the native ODT format;
2. It corrupts data;
3. The corruption may happen unnoticed, if one opens the document, makes some modifications outside of problematic table, then saves document; the problem could only be noticed after reopening the already corrupted file;
4. It is a regression.
Comment 1 sophie 2014-02-25 16:16:14 UTC
Confirmed with Version: 4.2.1.1
Build ID: d7dbbd7842e6a58b0f521599204e827654e1fb8b Ubuntu 13.10 and not reproducible with 4.1.5.3 Debian 6. 
Set as New - Sophie
Comment 2 Björn Michaelsen 2014-02-28 02:00:56 UTC
# bad: [25428b1e953636f74986622c5df614f04c150ed1] source-hash-cb4e009c4539c535108021934e545194b35cad9d
# good: [f0f6c65eb764f0303f59c58d320e9b0d5a894377] source-hash-4b9740b4ec3987e1d4d2ad6d20b4dcf996a4fa2e
git bisect start 'latest' 'oldest'
# good: [a72833796a7e527d9efc9ca6d8fe9b579e469105] source-hash-1472b5f87314fe660ef1a7b254e51272669f12f6
git bisect good a72833796a7e527d9efc9ca6d8fe9b579e469105
# good: [b21386bf459ae47bd6e461ea94cea6a06729a1ff] source-hash-570fe620e9d573cfc9fc260e6518563c6a6c1a3c
git bisect good b21386bf459ae47bd6e461ea94cea6a06729a1ff
# bad: [091d742e82f2b4608690c697d14f846ffc9164c7] source-hash-349c91c8ec6afc1f5c8499529d559af34d115a76
git bisect bad 091d742e82f2b4608690c697d14f846ffc9164c7
# good: [8f9ca7d8ab2e98061706d2fae7501b75a902d93a] source-hash-3cf0b5cdb05e1d77610431b1b1328102bf05b602
git bisect good 8f9ca7d8ab2e98061706d2fae7501b75a902d93a
# bad: [957af3a493bd71de62b536a590a2630aa4a53a27] source-hash-5166f28e797aec47a8a48213203ebd6a7ee06302
git bisect bad 957af3a493bd71de62b536a590a2630aa4a53a27
# good: [7bd35cbe665503fc9fc769437a723e90b46b2a3a] source-hash-e4ebe80be51fb33545091aa4f0bbc0ea2fe674f0
git bisect good 7bd35cbe665503fc9fc769437a723e90b46b2a3a
# bad: [896c52dbcc24f55d6487750f2d72f0c010f62cff] source-hash-d1cbaee70d3f922937a1993914436c8fc899ebfc
git bisect bad 896c52dbcc24f55d6487750f2d72f0c010f62cff
# bad: [e9c4bed0b2622077e75004f7b8a50dd9a02fa584] source-hash-417d1c2b13cbd70300d2921b5667dfadc7e25895
git bisect bad e9c4bed0b2622077e75004f7b8a50dd9a02fa584
# first bad commit: [e9c4bed0b2622077e75004f7b8a50dd9a02fa584] source-hash-417d1c2b13cbd70300d2921b5667dfadc7e25895

So offending commit should be in this range:
http://cgit.freedesktop.org/libreoffice/core/log/?qt=range&q=e4ebe80be51fb33545091aa4f0bbc0ea2fe674f0..417d1c2b13cbd70300d2921b5667dfadc7e25895
Comment 3 Björn Michaelsen 2014-02-28 13:15:00 UTC
Steps to reproduce:
 - open attachment 94712 [details]
 - press F5 to open the navigator
 - select Tables->Table 32 in the navigator

expected behavior:
 - table contains lots of '0.0'

actual behavior:
 - table contains lots of 'expression is faulty'

# bad: [417d1c2b13cbd70300d2921b5667dfadc7e25895] Map shared formula from xls to formula groups, and share the tokens as well.
# good: [e4ebe80be51fb33545091aa4f0bbc0ea2fe674f0] fix an OUString conversion bug
git bisect start '417d1c2b13cbd70300d2921b5667dfadc7e25895' 'e4ebe80be51fb33545091aa4f0bbc0ea2fe674f0'
# good: [e0e276ab255fb75c0412436baa0974836278b9db] cellForRowAtIndexPath instead of cellForItemAtIndexPath for iOS5 support
git bisect good e0e276ab255fb75c0412436baa0974836278b9db
# bad: [7ed89ac22d704907471a0e8b5b8cb6135a82d3f4] Removed unused file
git bisect bad 7ed89ac22d704907471a0e8b5b8cb6135a82d3f4
# good: [64deb339a97c1977f363fa08fd2b7d9fcfe2e957] Implement getTables(). (firebird-sdbc)
git bisect good 64deb339a97c1977f363fa08fd2b7d9fcfe2e957
# bad: [86637135b809f94973a6b55084d8045bbd276ceb] String to OUString and some optimizations
git bisect bad 86637135b809f94973a6b55084d8045bbd276ceb
# good: [88d74f35c2ce7968fe3f57f34e25d09374164ecd] Create Catalog to deal with sdbcx details. (firebird-sdbc)
git bisect good 88d74f35c2ce7968fe3f57f34e25d09374164ecd
# good: [ec365165ba7f332df479422174899808e1ff4152] Implement refreshTables. (firebird-sdbc)
git bisect good ec365165ba7f332df479422174899808e1ff4152
# first bad commit: [86637135b809f94973a6b55084d8045bbd276ceb] String to OUString and some optimizations

So, first bad commit is:
commit 86637135b809f94973a6b55084d8045bbd276ceb
Author: Matteo Casalin <matteo.casalin@yahoo.com>
Date:   Mon Aug 12 18:29:41 2013 +0200

    String to OUString and some optimizations
    
    Change-Id: I9d93d6aa26b2c9d20f7be8d201051a51e8e4ce7a

http://cgit.freedesktop.org/libreoffice/core/commit/?id=86637135b809f94973a6b55084d8045bbd276ceb

thus CC'ing Matteo.
Comment 4 Björn Michaelsen 2014-03-06 12:24:43 UTC
Sooo, I had a look at this and cant quite figure out what is going wrong. To simplify the issue I reduced the document to "Table 26" and "Table 32", but then the references seem to work just fine, which leads me to suspect the document might be corrupted.

Can you please provide a simplified document (e.g. just containing 2 tables) that shows the issue? => NEEDINFO
Comment 5 Björn Michaelsen 2014-03-06 16:20:12 UTC
Indeed the document seems to be corrupt altogether in some kind of way: table "Таблица16" is listed in the navigator, but cannot be deleted. adjusting severity/priority as this seems to be specific to this (corrupted?) example document.
Comment 6 Mike Kaganski 2014-03-07 04:24:03 UTC
Created attachment 95281 [details]
Simplified bugdoc

(In reply to comment #5)
Björn, thank you for your interest,

> Indeed the document seems to be corrupt altogether in some kind of way:
> table "Таблица16" is listed in the navigator, but cannot be deleted.
This specific table isn't corrupt: it is a table in a page footer, that has some locked cells, that may be unlocked and then the table may be deleted.

(In reply to comment #4)
I attach the requested basic document. It is reduced to three tables with minimum of cells, last of them is seemingly not related to the first two, but removing it makes the bug disappear. The problematic formula has been simplified to just "=<Table26.C1>", all fields removed, etc.

I don't know if this table is corrupt. Please adjust the severity in case it's not a corruption but an internal LO problem (taking into account that the problem only started to appear lately).

Setting status back to NEW as I have provided the requested info & the bug is already confirmed.
Comment 7 Mike Kaganski 2014-03-07 04:53:30 UTC
Created attachment 95282 [details]
Bugdoc simplified to minimum

I have simplified this further. Specifically, I opened it as ZIP, removed styles.xml and settings.xml; extracted content.xml, opened it with text editor and removed styles and variables sections, so that only text is left.

Everything looks OK; no corrupt tables; yet problem still there!
Comment 8 Mike Kaganski 2014-03-07 07:03:44 UTC
Ok, I revert severity to high values, because the problem is not caused by a corruption in document.

Steps to reproduce:
1. Create a new empty text document.
2. Add first table, having at least two columns (for simplicity, let it be 2x1).
3. Add second table 1x1.
4. Add third table 1x1.
5. Rename first table (menu Table->Table properties->Table->Name) to "Table20".
6. Make sure that second table has name "Table2".
7. Add formula to last table (<Table3.A1>): "=Table20.B1"

Leave the cell, and it becomes ** Expression is faulty **.

Important parts here:
1. The name of reference table must begin with name of another, that goes AFTER it.
2. Reference cells are from cols other than A.

Björn, I suddenly see that you are not in CC list. I add you, so that you don't miss the info. Please excuse me if this is a wrong thing to do.
Comment 9 Commit Notification 2014-03-07 11:52:35 UTC
Bjoern Michaelsen committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=7af06afee44f13f7f2cd69854a0e40e9ede7a378

fdo#75492: table names can start the same way and still be different



The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 10 Björn Michaelsen 2014-03-07 11:55:57 UTC
(In reply to comment #8)
> Steps to reproduce:
> ...

Doh, that makes it kinda obvious ;) -- waiting for review of fix for 4-2 on https://gerrit.libreoffice.org/#/c/8495/
Comment 11 Mike Kaganski 2014-03-07 21:43:16 UTC
Thank you very much!
Could you please mention here a direct link to a daily containing the patch, when it's ready, so that I could test?
Comment 12 Commit Notification 2014-03-09 16:53:42 UTC
Bjoern Michaelsen committed a patch related to this issue.
It has been pushed to "libreoffice-4-2":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=3c0bc4edf5bde8e8494c2ef7e44105f9d90d85c3&h=libreoffice-4-2

fdo#75492: table names can start the same way and still be different


It will be available in LibreOffice 4.2.3.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 13 Mike Kaganski 2014-03-11 21:39:33 UTC
VERIFIED with Version: 4.2.3.0.0+
Build ID: d7cc3e7d8a093cfe300f52c0fe51aecd80a34fff
TinderBox: Win-x86@42, Branch:libreoffice-4-2, Time: 2014-03-11_09:31:29

Thank you Björn!
Comment 14 Robinson Tryon (qubit) 2015-12-17 07:50:44 UTC Comment hidden (obsolete)