View Issue Details [ Jump to Notes ] [ Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0010332CDashpublic2010-02-25 14:452010-04-16 13:36
ReporterRoman Shtylman 
Assigned ToJulien Jomier 
PrioritynormalSeveritymajorReproducibilityalways
StatusclosedResolutionno change required 
PlatformOSOS Version
Product Version1.6.4 
Target VersionFixed in Version1.6.4 
Summary0010332: cdash does not properly handle large xml files
DescriptionIf my project has lots of tests and produces lots of output because of it, cdash will fail to handle the entire xml file. The problem lies in the fact that the ctestparser.php file only does "$content = fread($filehandle, 8192);" (around line 44) once. This means that large files are truncated.

This is a regression from previous versions.
Additional InformationThis is easily fixed by adding the following under that line:

while (!feof($filehandler))
  $content .= fread($filehandler, 8192);

likewise just using the above should work as well without the first $content read since the while loop will handle it but I have not tested that specific case after I did the initial fix.
TagsNo tags attached.
Attached Filesxml file icon Test.xml [^] (9,929 bytes) 2010-02-25 15:22

 Relationships

  Notes
(0019621)
Julien Jomier (manager)
2010-02-25 14:50

 The while loop is setup line 202. Are you sure this is the issue? We have been dealing with large XML file without any issues. Which error are you getting?
(0019622)
Roman Shtylman (reporter)
2010-02-25 15:05

 From what I can see the file is parsed by xml_parse($parser, $content, false) on line 0000070:0000115. The problem I was having was that my test reports where not being submitted. I tried submitting them manually using curl (as well as usual ctest methods) and no error is returned. It adds the entry for the build but the test columns are blank. If I add my fix, and redo the curl submit the columns are populated as I expect.

The loop on 202 seems to happen after the file is parsed... I might be missing something there tho. What I do know is that the above fixed my problem :/ I can explore more to see what else happened if that is needed.
(0019623)
Julien Jomier (manager)
2010-02-25 15:09

 We use SAX parsing with parse the file by chunk. We parse it once to determine which XML handler should be allocated depending on the type of file (build, update, etc...). Could you attach your XML file so I can try it here and see if there is something wrong?
(0019624)
Roman Shtylman (reporter)
2010-02-25 15:24

 I have attached a Test.xml file which causes cdash to fail for me. I can run ctest -D Nightly just fine. It also says upload successful, but when viewing the dashboard the test columns are all blank as if no tests were run. The database entries for tests are all -1.
(0019627)
Julien Jomier (manager)
2010-02-25 15:39

 This is strange. I just tested your file and it works well for me. I'm wondering if their is an issue with the current parser.
(0019628)
Roman Shtylman (reporter)
2010-02-25 15:57

 That is interesting indeed. I even tested it against svn trunk ... with the same failed results.
(0019629)
Julien Jomier (manager)
2010-02-25 16:02

 I'm wondering if your PHP installation allows for streaming XML parsing, what kind of server (apache, PHP version, etc...) are you using?
(0019630)
Roman Shtylman (reporter)
2010-02-25 16:03

 php5.2.12
on apache2.2.14

maybe its an option in the php.ini? The thing is I didn't have this problem with cdash 1.4
(0019631)
Julien Jomier (manager)
2010-02-25 16:09

 Could you make sure that the backup directory is writeable by the apache/web server?
(0019632)
Roman Shtylman (reporter)
2010-02-25 16:15

 It is writable and I even see the xml files in there. What is even more interesting is that I see the *full* xml file in there.
(0019637)
Roman Shtylman (reporter)
2010-02-25 18:47

 stupid me... I had async upload turned on and this was causing some weird behavior... I disabled it and it seems to be fine now... sorry for the bug report.

 Issue History
Date Modified Username Field Change
2010-02-25 14:45 Roman Shtylman New Issue
2010-02-25 14:46 Julien Jomier Status new => assigned
2010-02-25 14:46 Julien Jomier Assigned To => Julien Jomier
2010-02-25 14:50 Julien Jomier Note Added: 0019621
2010-02-25 15:05 Roman Shtylman Note Added: 0019622
2010-02-25 15:09 Julien Jomier Note Added: 0019623
2010-02-25 15:22 Roman Shtylman File Added: Test.xml
2010-02-25 15:24 Roman Shtylman Note Added: 0019624
2010-02-25 15:39 Julien Jomier Note Added: 0019627
2010-02-25 15:57 Roman Shtylman Note Added: 0019628
2010-02-25 16:02 Julien Jomier Note Added: 0019629
2010-02-25 16:03 Roman Shtylman Note Added: 0019630
2010-02-25 16:09 Julien Jomier Note Added: 0019631
2010-02-25 16:15 Roman Shtylman Note Added: 0019632
2010-02-25 18:47 Roman Shtylman Note Added: 0019637
2010-02-25 22:31 Julien Jomier Status assigned => resolved
2010-02-25 22:31 Julien Jomier Fixed in Version => 1.6.4
2010-02-25 22:31 Julien Jomier Resolution open => no change required
2010-04-16 13:36 Julien Jomier Status resolved => closed

Copyright © 2000 - 2018 MantisBT Team