Opened 13 years ago

Closed 12 years ago

#63 closed defect (fixed)

mondoarchive slow for large number of files, particukarly with attr package installed

Reported by: andree@… Owned by: bruno
Priority: normal Milestone: 2.2.1
Component: mondo Version: 2.0.9
Severity: normal Keywords:
Cc:

Description (last modified by bruno)

Inspired by this Debian bug: debianBTS(355751)? (please see also the discussion there), I've done some testing to quantify things a bit more:

I have created a directory with 400000 files with 10k of zeroes each with the following results:

tar uncompressed

real 13m50.752s

user 0m1.508s

sys 0m26.006s

tar bzip2

real 13m0.948s

user 3m4.540s

sys 0m19.233s

tar gzip

real 10m53.339s

user 0m45.419s

sys 0m18.317s

mondoarchive default compression, acl, NO attr

real 21m32.037s

user 2m22.993s

sys 8m10.711s

mondoarchive default compression, acl, attr

real 38m58.321s

user 8m21.735s

sys 28m49.708s

mondoarchive uncompressed, acl, NO attr

real 21m5.416s

user 0m23.753s

sys 4m13.988s

mondoarchive uncompressed, acl, attr

real 39m41.141s

user 6m20.996s

sys 26m47.292s

mondoarchive was called like this:

time                                                                   \
mondoarchive -OiNFW(0)s 4380m                                          \
             -E "/"                                                    \
             -I "/home/andree/shed/testing/MR_runtime/many_files_test" \
             -S "/home/mondorescue/scratch"                            \
             -T "/home/mondorescue/tmp"                                \
             -d "/home/mondorescue/iso"

All measurements where taken on an Athlon64 X2 4800+ with 2GB of RAM running amd64 Debian sid with 2.6.17 kernel.

The two main observations I make when looking at above figures are:

  • tar is at least twice as fast compared to mondoarchive.
  • With the attr package installed, i.e. with getfattr available, mondoarchive takes twice as long to archive the same set of files compared to when getfattr is not available.

Cheers, Andree

Change History (6)

comment:1 Changed 12 years ago by bruno

  • Milestone set to 2.2.1
  • Status changed from new to assigned

comment:2 Changed 12 years ago by bruno

  • Description modified (diff)

comment:3 Changed 12 years ago by bruno

  • Milestone changed from 2.2.1 to 2.2.2

comment:4 Changed 12 years ago by bruno

rev [944] improve this by not calling everytime find_home_of the 2 commands. Probably unsufficient.

comment:5 Changed 12 years ago by bruno

  • Milestone changed from 2.2.2 to 2.2.1

as of rev [948], the default for mondoarchive is to NOT save those attributes. That should give a boost to mondo. that could break compatibility so this may not be the end :-)

comment:6 Changed 12 years ago by bruno

  • Resolution set to fixed
  • Status changed from assigned to closed

Andree could you test the impact of my modif on your side. We also need to check the accuracy of backup with and without ACL/XATTR.

Note: See TracTickets for help on using tickets.