Re: [Tracker] tracker-extract crashes in GLib: gmem.c:168: failed to allocate 2330172 bytes

[Martyn Russell <martyn lanedo com>]

On 19/05/14 11:47, Ralph Böhme wrote:
> Hi Martyn

Hi Ralph,

> Am 19.05.2014 um 12:37 schrieb Martyn Russell <martyn lanedo com>:
> > On 12/05/14 14:55, Ralph Böhme wrote:
> > > Hi Martyn
> >
> > On another note, that is a lot more memory than I would expect to be trying to allocated and the files 
> > themselves aren't even that big.
>
> yeah, for that reason I initially assumed there's a memleak somewhere. The memory keeps increasing while the 
> process processes files until it crashes with ENOMEM.
>
> > Something is not right there, what function is trying to allocate that memory and for what variable?
>
> I don't have a debug build so I only have a SBT posted initially:
>
> fe3cfed2 g_logv   (fe4669d9, 4, fe46d604, fdb3ec3c, 238e3c) + 1d2
> fe3d00c2 g_log    (fe4669d9, 4, fe46d604, fe46830f, 238e3c, 1494f725) + 32
> fe3ce730 g_realloc (14833008, 238e3c, fdb3ec90, 14213008, fdb3ecec, 14213008) + 80
> fe4191ba _g_gnulib_vasnprintf (0, fdb3ed4c, fd0bc35f, fdb3edf8, 14833008, 4) + 66a
> fe41a333 _g_gnulib_vasprintf (fdb3edac, fd0bc35f, fdb3edf8, fe481a3c, fe481a3c, 8b15e68) + 33
> fe413bd4 g_vasprintf (fdb3edac, fd0bc35f, fdb3edf8, 14213008, 8b24120, ffffffff) + 34
> fe3ee0d6 g_string_append_vprintf (8168260, fd0bc35f, fdb3edf8, fd0cc588, 8b15e68) + 46
> fe3ee2fb g_string_append_printf (8168260, fd0bc35f, 14213008, 1d, 8b15e68, 0) + 2b
> fd0ac385 tracker_sparql_builder_object_string (8b15e68, 14632008, fdb3ee5c, fd0abcee, 8168260, fcad11bd) + b5
> fd0ac55c tracker_sparql_builder_object_unvalidated (8b15e68, 14632008, fcad11f0, 86c26c8, 100000, 50) + 10c

Yea you sent this before, I meant more which extractor but I think I 
have some idea here because of the text extraction in your previous 
emails :)

The log you originally sent:

  ...

  12 May 2014, 14:46:19: Tracker:   Read 65535 bytes from file, 16 
bytes remaining until configured threshold is reached
  12 May 2014, 14:46:19: Tracker:   Read 16 bytes from file, 0 bytes 
remaining until configured threshold is reached
  12 May 2014, 14:46:19: GLib: gmem.c:168: failed to allocate 2330172 bytes

I wonder if the text extractor is broken in some way and/or the GIO code 
is too on Solaris?

Can you put the file somewhere to test so I can try on my local box here?

Reading that many bytes from a 1.2Mb file isn't right either.

I wonder if you have some upper limit on memory allocations on the system?

The other thing is, the default maximum bytes to read from a text file 
is 1048576. So we shouldn't even be reaching 2330172 bytes. Certainly 
sounds like a buffer overrun somewhere (unless you have specific 
configuration on your system that I don't know of)?

> fcad106e tracker_extract_get_metadata (8b156e0, 80, 8065bee, 83cb408, 816b934, 816b920) + 10e
> 08058ca3 get_file_metadata (84c4650, 8b15660, 8056bd0, fe481a3c, fe414749, fe481a3c) + c3
> 080593f2 get_metadata (135d9230, 0, 0, fe48354c, 8140200, fe481a3c) + 52
>
> The crash always happens in this callchain.

Thanks,

--
Regards,
Martyn

Founder & Director @ Lanedo GmbH.
http://www.linkedin.com/in/martynrussell