[Gutsy] Possible memory leak in trackerd

Could you provide some steps to reproduce this bug? How could I go about corrupting the database? Thanks in advance.

Will make trackerd die and then reindex on next startup when corruption found

Cant really test it but fix should work in theory

Hello, i am expieriencing something similar -- i think.

heres the terminal output, before i cntrl-c'd it:

matthew@matthew-desktop:~$ trackerd

Tracker version 0.6.3 Copyright (c) 2005-2007 by Jamie McCracken (<email address hidden>)

This program is free software and comes without any warranty.
It is licensed under version 2 or later of the General Public License which can be viewed at http://www.gnu.org/licenses/gpl.txt

Initialising tracker...
Could not set idle IO priority...attempting best effort 7 priority
Throttle level is 0
ERROR: execution of prepared query GetServiceID failed due to database disk image is malformed with return code 11
ERROR: execution of prepared query GetServiceID failed due to database disk image is malformed with return code 11
ERROR: execution of prepared query GetServiceID failed due to database disk image is malformed with return code 11
ERROR: execution of prepared query GetServiceID failed due to database disk image is malformed with return code 11
ERROR: execution of prepared query CreateService failed due to database disk image is malformed with return code 11
ERROR: CreateService uri is /usr/local/share/applications
ERROR: could not get file id for /usr/local/share//applications - unable to continue indexing this file
ERROR: execution of prepared query GetServiceID failed due to database disk image is malformed with return code 11
ERROR: execution of prepared query CreateService failed due to database disk image is malformed with return code 11
ERROR: CreateService uri is /usr/share//applications/alacarte.desktop
ERROR: could not get file id for /usr/share//applications/alacarte.desktop - unable to continue indexing this file
ERROR: execution of prepared query GetServiceID failed due to database disk image is malformed with return code 11
ERROR: execution of prepared query CreateService failed due to database disk image is malformed with return code 11
ERROR: CreateService uri is /usr/share//applications/accessibility-keyboard.desktop
ERROR: could not get file id for /usr/share//applications/accessibility-keyboard.desktop - unable to continue indexing this file
ERROR: execution of prepared query GetServiceID failed due to database disk image is malformed with return code 11
ERROR: execution of prepared query CreateService failed due to database disk image is malformed with return code 11

...

It just goes on ad finitum.

This is a fresh install of the gutsy rc.
If you need anything more just let me know, i'll do my best to help out.

Matt

OK, well i realized there was an update, so i have now updated. No longer getting the error...

I have the same problem in Gutsy. Also with Pidgin logs and Evolution emails, see:

mime is text/plain for /home/semente/.purple/logs/msn/semente@*********.***/semente@*********.***/2007-10-18.143554-0200BRST.txt
file extension is txt
file /home/semente/.purple/logs/msn/semente@*********.***/semente@*********.***/2007-10-18.143554-0200BRST.txt is indexable
file /home/semente/.purple/logs/msn/semente@*********.***/semente@*********.***/2007-10-18.143554-0200BRST.txt has fulltext 1 with service Text
Indexing /home/semente/.purple/logs/msn/semente@*********.***/semente@*********.***/2007-10-18.143554-0200BRST.txt with service Text and mime text/plain (new)
service id for GaimConversations is 19 and sid is 84045 with mime text/plain
ERROR: execution of prepared query CreateService failed due to constraint failed with return code 19
ERROR: CreateService uri is /home/semente/.purple/logs/msn/semente@*********.***/semente@*********.***/2007-10-18.143554-0200BRST.txt
ERROR: could not get file id for /home/semente/.purple/logs/msn/semente@*********.***/semente@*********.***/2007-10-18.143554-0200BRST.txt - unable to continue indexing this file
Indexing /home/semente/.purple/logs/msn/semente@*********.***/semente@*********.*** with service Folders and mime Folder (new)
service id for Folders is 3 and sid is 84046 with mime Folder
ERROR: execution of prepared query CreateService failed due to constraint failed with return code 19
ERROR: CreateService uri is /home/semente/.purple/logs/msn/semente@*********.***/semente@*********.***
ERROR: could not get file id for /home/semente/.purple/logs/msn/semente@*********.***/semente@*********.*** - unable to continue indexing this file

trackerd is slow in my machine. uses 100% CPU, but I don't know if have relationship with the problem above.

I had a existing ~/.cache file and then upgraded to gutsy. I had to rename that file to something else so that the ~/.cache/ could be created. Before I did that, trackerd would eat all my CPU. Maybe a check to see if ~/.cache is a directory and is writeable would be a good idea, otherwise alert the user about the problem, and/or allow the user to pick a new location for the ~/.cache/ directory.

Also, before I renamed that file, running tracker-stats and tracker-status would produce worthless errors:

$ tracker-status
Tracker daemon's status is �Ë
.

$ tracker-stats

** (tracker-stats:25786): WARNING **: tracker-stats: an error has occured: Process /usr/bin/trackerd exited with status 0

I see this too, but I did not have thacker instaled before updating to 7.10.
Tracker is now using 420MB on my system, and it usualy keeps growing until the system hangs.

tracker-status
Tracker daemon's status is �Ë
.

tracker-stats

** (tracker-stats:13903): WARNING **: tracker-stats: an error has occured: Process /usr/bin/trackerd exited with status 0

Ubuntu 7.10, with all available updates installed.

the latter error indicates that you have an old trackerd running which you should kill (perhaps you logged off and on again?)

I had only one instance of trackerd running, the one using all the memory.
I don't think I loged in and out, as I'm the only one using this machine, and don't usually log out, but I've seen this abnormal behaviour several times.

Trackerd is indexing about 500GB of data, mostly music and video; could this cause this ?

I have now removed ".local/share/tracker/" and ".cache/tracker" and rebooted as it was impossible to use the machine for more than one day; will see how it goes.

Is there anything I can do to help debug this ? I like tracker a lot and would like to help if I can, as in the current state it is actively harmful to my system.

What is an appropriate ammount of memory for tracker to use ? Right now it uses over 250MB, and keep growing; is this normal ?

uptime 3:25
victor 8176 0.9 21.2 240728 219512 ? SNl 10:54 1:54 trackerd
Tracker daemon's status is Idle.

uptime 4:35
victor 8176 0.8 24.8 278676 256796 ? SNl 10:54 2:21 trackerd
Tracker daemon's status is Idle.

I just killed it on my system using 1.2GB of ram. How can I produce some logs for you?

I did some home-directory cleanup right after upgrading to Gutsy. I know that I deleted some ~/.directories like ~/.local, and ~/.cache, assuming programs would recreate whatever they need. The next day I awoke to trackerd using over 1GB of memory. Did the one cause the other? It's impossible for me to say yet.

Eric,

its very possible

mem leaks are fixed in svn for general indexing however I have not checked mass deletes for them - will do so soon...

Jamie, I have a clean Install of Ubuntu 7.10 with all updates, a 8GB home and 370 GB of other data. In 4 hours today trackerd was really close to killing my system.

With trackerd running:
             total used free shared buffers cached
Mem: 1035328 964824 70504 0 4048 321608
-/+ buffers/cache: 639168 396160
Swap: 2345480 629212 1716268

After killing trackerd:
             total used free shared buffers cached
Mem: 1035328 580892 454436 0 4632 336512
-/+ buffers/cache: 239748 795580
Swap: 2345480 229456 2116024

All the memory (1GB) is used, plus 600MB of swap.
There is obviously something very wrong here.

Is there anything we can do to help debug this ?

Victor,

AFAIK all the mem leaks are fixed in svn (I will release 0.6.4 this weekend and get it backported into gutsy). ).6.3 in gutsy has a few mem leaks but my patch to fix them was rejected as only blocker bug fixes were allowed post RC.

I and a few others tested the memory leak fixes a few weeks ago and it never grew much beyond 30MB or so.

There however may be a few corner cases where mem leaks still occur so we need to know what triggers them (other than the initial index)

Hey all,
I have a similar problem:

I installed Gutsy day before yesterday (a fresh install since upgrade was problematic) ... this morning turned on my machine and with a couple of minutes the system ground almost to a halt with trackerd using all my system resources. I killed it and restarted it from a terminal and the problem arrises again.

I am on a amd64 architecture with 4GB RAM.

here are the last few lines of outpuit
ERROR: metadata Audio:TrackPeakGain not found
ERROR: metadata Audio:TrackPeakGain not found
ERROR: metadata Audio:TrackPeakGain not found
ERROR: metadata Audio:TrackPeakGain not found
ERROR: failed to decode mail part "/home/stephen/.evolution/mail/imap/***@***ac.uk/folders/INBOX/1124.2"

seems to be related to the evolution plugin.. Im going to disable it and see if it fixes the problem

also included is a screeny of the system panel and output

I just killed a trackerd process using 1.7GiB of RAM (I have 3 GiB total). The computer has been running for just over 4 hours. I have also had to kill trackerd several times over the last week because it looked like it was leaking memory.

How can I help solve this/produce logs/further diagnose this?

I've been runing trunk for a week and it works well. Is there any chance to push the changes to updates ?

tracker (0.6.4-1ubuntu1) hardy; urgency=low

  * Merge with Debian, remaining Ubuntu changes:
    - debian/control:
      + Addhere to DebianMaintainerField spec.
      + Do not build-depend on universe dependencies:
        libunac1-dev, libqdbm-dev.
      + tracker depends on o3read instead of recommend it, so we have
        OOo indexing by default.
    - debian/rules:
      + Enable sqlite external db instead of qdbm.
    - debian/patches/02_no_kde_autostart.patch:
      + Do not autostart trackerd in Kde, as they have strigi.
    - debian/patches/03_no_initial_index_in_battery.patch:
      + Do not run the initial index if running on battery.
        Patch taken from upstream SVN, revision 1075:
        http://svn.gnome.org/viewvc/tracker?view=revision&revision=1075
    - debian/patches/04_fix_crash_index_name_is_null.patch:
      + Fix a crash when index name is null during merging.
        Patch taken from upstream SVN, revision 1076:
        http://svn.gnome.org/viewvc/tracker?view=revision&revision=1076
    - debian/patches/05_typo_audio_track_peak_gain_tag.patch:
      + Fix a typo in a tag metadata. LP: #145359
        Patch taken from upstream SVN, revision 1077:
        http://svn.gnome.org/viewvc/tracker?view=revision&revision=1077
    - debian/patches/06_trackerd_infinite_loop.patch:
      + Fix an infinite loop in trackerd if a second instance is
        launched. Patch taken from upstream SVN, revision 1079:
        http://svn.gnome.org/viewvc/tracker?view=revision&revision=1079

  * Bugs fixed in the new release:
    - LP: #130935. Added a notification area tool displaying trackerd's
      status, and allowing to search from there.
    - LP: #138331: prevent hangs which stopped tracker quitting smoothly.
    - LP: #147756: Fixed memory leaks.
    - LP: #159807: Stop indexing if disk is full.
    - LP: #164148: Unsafe tempfile usage.
    - LP: #148520: Check for overlapping watch dirs
    - LP: #132463: Always show full path of files in tracker-search-tool.
    - LP: #133402: Do not repeat 'Preferences' in the preferences title.
    - LP: #164412: Limit log size to 10MB.
    - LP: #150814: Detect and prevent database corruption.
    - LP: #160262: Fix evolution email opening for the deskbar handlers.
    - LP: #150030: Show applications in search results.

 -- Emilio Pozuelo Monfort <email address hidden> Fri, 14 Dec 2007 01:16:26 +0100

Tracker seems to be doing something similar in Jaunty. It is using all available cpu when paused. It doesn't affect system performance because it has a nice of 19 but it is annoying because it affects battery life and makes my laptop very hot. It seems to be stuck on a folder. it is always done 7949 of 7953 with 2 seconds left and 1 hour 22 minutes 46 seconds left. The only thing that changes is the paused status.