Re: [xml] xmlReader: Possibility for cloning an xmlTextReader (or multi-pass reading)

["Eric S. Eberhard" <eric vicsmba com>]

It is also relatively trivial to do it yourself ... the offset of the nodes in my experience are always the same for the same document.  So you can keep in memory the offset from the base node (not the address of the nodes which is not constant).  So after the first read you would have direct access to any data you wanted in subsequent passes. 

I would also question the need for this if (as it sounds) you would only be reading the document twice.  In the "old days" I agonized to write super efficient code -- and in the more recent years have found that two parses, one after the other, due to modern machines and cache ... are hardly noticeable even with extreme volume.  We tend to do a lot of smaller documents with libxml2 but we process in and out about 50,000 document per hour on a relatively modest computer (4 core IBM AIX).  Many are read over and over because it is lazier and easier to code that way and pressure to get things working is now exceeding pressure to make the ultimate performance code.

What I am saying is you might cringe at -- and instinctively hate (as I do) -- the idea of just reading it twice == but you might want to run some benchmarks and see if you really care or not.

Eric

On 3/31/2013 12:00 AM, Liam R E Quin wrote:

On Sat, 2013-03-30 at 08:02 +0100, Martin B. wrote:
[...]
  

It turns out however, that the subtree where the large data resides has 
to be read not in-order, but I have to collect some (small amount of) 
data before the other. 
    

Do you process the file only once, or many times, between times when it
changes?

If you process the large file multiple times, and you have to
re-engineer the code, you could consider something like dbxml, which
will parse the document once and can make an index; subsequence accesses
can be fairly fast because there's no need to read the file.

If you process the large file at most once every time it changes, using
an XQery engine like dbxml isn't so obvious a win: it can still help
with the out-of-order access in some cases, but the gains might not be
worth the engineering effort of reworking your code.

As Daniel has said, libxml is pretty fast at parsing, so another
strategy might be fetching only the parts you need into a new, smaller
XML document and then using XSLT.

Liam

  

-- 
Eric S. Eberhard
VICS
PO Box 3661
Camp Verde, AZ  86322

928-567-3727  work                      928-301-7537  cell

http://www.vicsmba.com/index.html             (our work)
http://www.vicsmba.com/ourpics/index.html     (fun pictures)