The throughput here for md5sum is above 600MB/s. It is a Perl 5 program using Archive::Zip::computeCRC32() to compute the crc32. To summarize the speed if we consider md5sum‘s speed as the baseline:Ĭrc32 is the fastest here. head -c 3G /dev/zero for chk in crc32 md5sum sha1sum do echo $chk time $chk test doneĦe7f6dca8def40df0b21f58e11c1a41c3e000285 test It can only support 3GB by the time I did this test. The system is the same one on which I did the previous tests. I did a little bit change to the suggested command to do checksum on a file under /dev/shm/ as crc32 does not accept input from STDIN. Pádraig comments that we can avoid the I/O and measure the computational cost. If you want higher speed, improve your I/O speed first till CPU is the bottleneck (CPU usage reaches 100%). The conclusion is that use any tools that work the best for you (you may need to be aware of the the collisions for these algorithms, check Simard’s comment) without worrying a lot about the speed (it still consumes time) on a relatively modern computer. The algorithms and the tools themselves are not yet the limitation. So, almost all time are on reading the file content. That’s not bad for a single magnetic disk I/O storage. Now, let’s check how much time is needed to read the file content out. Why there is no much differences? To compute the checksums, the tools need to read these files and do the computation. md5sum is the slowest but just a little bit slower. But it is just a tiny bit faster than sha1sum and md5sum. Sys 0m4.890s md5sum speed $ time md5sum wiki.txtĮ2e649030c795ffa9f33a99bcb39dde7 wiki.txtįrom the results, crc32 is the fasted. Sys 0m4.668s crc32 speed $ time crc32 wiki.txt ![]() Now, let’s see how does the three tools perform for computing the checksum of the file. rw-r-r- 1 zma zma 15G Jun 14 10:28 wiki.txt The performance File to be checsum’ed is a 15GB text file: $ ls -lha wiki.txt
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |