size of log files should be controlled

I saw this recently on the twitternet: http://twitter.com/muffinresearch/status/4142427805 which prompted this discussion.

bzr runs two log files. ~/.bzr.log, and .bzr.log.old. We cap them at a
certain size, and rotate log ->old when we exceed that size.

This works pretty well.

-Rob

maybe bound the max size and compress it?

We need to:
- lower the log level for syncdaemon
- make sure we compress old log files
- make sure we delete older log files

configuring logrotate and syncdaemon with the correct default values should be enough.

Ok, let's talk a bit about this :)

> We need to:
> - lower the log level for syncdaemon
So should we move the log level to INFO?
btw, this implies that we aren't going to be able to debug what's going on.

> - make sure we compress old log files
This can be done, I already wrote code that do this using python logging framework for the server side, so doing this in the client is easy

> - make sure we delete older log files
We are currently rotating old log files, on a daily basis, if this isn't working there is a bug in the syncdaemon logger module.
We can change this to rotate the log files based on it's size (also supported by the logging framework)

>
> configuring logrotate and syncdaemon with the correct default values should be enough.
logrotate isn't an option, as the syncdaemon is a per user process, and logrotate only have a system wide config. so we will need one logrotate config for eahc user...

Oh, forgot to mention that we are rotating the logs on each startup, and we keep the last 5 log files, logs older than that are discarded.

If we're rotating logs only on start-up, that's a problem. It could be days and lots of operations before restarting.

Steve: we are currently rotating old log files, on a daily basis, *and* on start-up.

My girlfriend had a notification that her /home directory was full. After checking Disk Usage Analyzer, I found her .cache/ubuntuone/log/syncdaemon.log.2009-10-04_14-28-55 file was 39.8gb's. This can cause serious file system problems on something that doesn't have a lot of hard disc space (for instance a netbook) and this is a very severe problem.

The log level needs to be turned way down in the next upload, and if we aren't already rotating logs they need to be rotated.

Elliot,
Just for the record.
The requested fix is to change the log level to ERROR

branch proposed and approved, that changes the log level to info and limit the log file size to 1MB.
Now the log file is rotated:
 - when it size is 1MB
 - daily
 - on each daemon start

And the backup cound is 5

This bug was fixed in the package ubuntuone-client - 1.0.0-0ubuntu1

---------------
ubuntuone-client (1.0.0-0ubuntu1) karmic; urgency=low

  * New upstream release.
    - Limit log messages, file size, and rotate more often (LP: #435137)
    - Rotate oauth-login.log and cap the file size (LP: #445514)
    - Remove pycurl usage in favor of urllib and better errors (LP: #411029)
    - Save the syncdaemon settings for bw limiting (LP: #418882)
    - Fix NoSuchDatabase when pairing desktopcouch (LP: #438411)
    - Fix IOError when launching preferences window (LP: #441039)
    - Update icons to remove Ubuntu logo, better names (LP: #434886)
    - Use ngettext for the possibly singular notification (LP: #449269)
    - Only create Places bookmark when first authorizing (LP: #397749)
    - Don't always recreate the bookmark (LP: #401211)
    - Handle showing emblems better (LP: #440839)
    - Remove the System->Preferences menu item (LP: #443342)
    - Remove NotOnlineError usage (LP: #404550)
    - Set a default share name in the Nautilus sharing UI (LP: #369488)
    - Simplify some state handling (LP: #420354)

 -- Rodney Dawes <email address hidden> Mon, 28 Sep 2009 18:15:00 -0400

I am having similar issue with ubuntuone-client, specifically with ubuntuone-syncd cpu usage spikes.
I have a single 2.7K file on ubuntuone. CPU usage jumps between 8%-40% continuously.
...
ubuntuone-client-1.0.3-0ubuntu1
...
python ubuntuone-client-diagnose.py
Checking your Ubuntu One client...
No issues were detected.
...
 ls -l .cache/ubuntuone/log/

-rw-r--r-- 1 user user 559362 2010-03-09 21:09 oauth-login.log
-rw-r--r-- 1 user user 1048445 2010-03-09 16:25 oauth-login.log.1
-rw-r--r-- 1 user user 1067 2010-03-09 23:04 syncdaemon-exceptions.log
-rw-r--r-- 1 user user 1067 2010-03-09 23:04 syncdaemon-exceptions.log.2010-03-09_23-04-14
-rw-r--r-- 1 user user 1067 2010-03-09 23:04 syncdaemon-exceptions.log.2010-03-09_23-04-15
-rw-r--r-- 1 user user 1067 2010-03-09 23:04 syncdaemon-exceptions.log.2010-03-09_23-04-17
-rw-r--r-- 1 user user 1067 2010-03-09 23:04 syncdaemon-exceptions.log.2010-03-09_23-04-18
-rw-r--r-- 1 user user 1067 2010-03-09 23:04 syncdaemon-exceptions.log.2010-03-09_23-04-19
-rw-r--r-- 1 user user 2606 2010-03-09 23:04 syncdaemon.log
-rw-r--r-- 1 user user 2606 2010-03-09 23:04 syncdaemon.log.2010-03-09_23-04-14
-rw-r--r-- 1 user user 2606 2010-03-09 23:04 syncdaemon.log.2010-03-09_23-04-15
-rw-r--r-- 1 user user 2606 2010-03-09 23:04 syncdaemon.log.2010-03-09_23-04-17
-rw-r--r-- 1 user user 2606 2010-03-09 23:04 syncdaemon.log.2010-03-09_23-04-18
-rw-r--r-- 1 user user 2606 2010-03-09 23:04 syncdaemon.log.2010-03-09_23-04-19
...
cat .cache/ubuntuone/log/syncdaemon-exceptions.log
2010-03-09 23:05:22,607 - ubuntuone.SyncDaemon.Main - ERROR - Local rescan finished with error: Traceback: <type 'exceptions.KeyError'>: ('', 'None')
/usr/lib/python2.6/dist-packages/twisted/internet/defer.py:328:_runCallbacks
/usr/lib/python2.6/dist-packages/ubuntuone/syncdaemon/local_rescan.py:139:_process_next_queue
/usr/lib/python2.6/dist-packages/twisted/internet/defer.py:243:callback
/usr/lib/python2.6/dist-packages/twisted/internet/defer.py:312:_startRunCallbacks
--- <exception caught here> ---
/usr/lib/python2.6/dist-packages/twisted/internet/defer.py:328:_runCallbacks
/usr/lib/python2.6/dist-packages/ubuntuone/syncdaemon/local_rescan.py:77:_process_trash
/usr/lib/python2.6/dist-packages/ubuntuone/syncdaemon/filesystem_manager.py:1055:get_iter_trash
/usr/lib/python2.6/UserDict.py:110:iteritems
/usr/lib/python2.6/dist-packages/ubuntuone/syncdaemon/file_shelf.py:201:__getitem__
/usr/lib/python2.6/dist-packages/ubuntuone/syncdaemon/file_shelf.py:132:__getitem__
/usr/lib/python2.6/dist-packages/ubuntuone/syncdaemon/file_shelf.py:124:_load_pickle

I was always wondering what spins up my hard disk every 2 minutes. Now I created a file named ~/.config/ubuntuone/logging.conf with the following contents:

[logging]
level = ERROR

Did have to restart ubuntuone sync to make the change being applied (u1sdtool -q; u1sdtool -c). But afterwards the problem was fixed for me.

PeterPall, I suspect you are hitting a completely different issue. Probably bug #765441.