linux kernel - is system(const char *command) lead to cpu sys 100% -


i create 1 background thread b,and in func of b,

void func() {   system('gzip -f text-file'); // size of text-file 100m   xxx } 

i found sometime sys of 1 cpu(my server has more 1 cpu core) 100%. strace progress, found clone syscall consume more 3 seconds, execution time of gzip.

**17:46:04.545159** clone(child_stack=0, flags=clone_parent_settid|sigchld, parent_tidptr=0x418dba38) = 39169 **17:46:07.432385** wait4(39169, [{wifexited(s) && wexitstatus(s) == 0}], 0, null) = 39169 

so question is, 1. system('gzip -f text-file') lead 100% cpu sys ? 2. root cause

sys_clone without clone_mm full copy of virtual memory mapping parent process child process, according https://www.kernel.org/doc/gorman/html/understand/understand021.html

343      allocate new mm 348-350  copy parent mm , initialise process specific mm fields init_mm() 352-353  initialise mmu context architectures not automatically manage mmu 355-357  call dup_mmap() responsible copying vmas regions in use parent process 

vma count process 60gb in 2000 mmaps high, , dup_mm may take lot of time.

you want small external run (gzip), fork not best solution such large programs. copies of vma trashed doing exec: http://landley.net/writing/memory-faq.txt

for example, fork/exec combo creates transient virtual memory usage spikes, go away again without ever breaking copy on write status of of pages in forked page tables. if large process forks off smaller process, enormous physical memory demands threaten happen (as far overcommit concerned), never materialize.

so, can better to:

  • check vfork+exec pair (aka posix_spawn), suspend huge process small time, until child exec or `exit)
  • create separate helper process before doing 60gb of mmaps; communicate using pipes/sockets/ipc/anything. helper process small , sleep time on ipc. when needs gzip, asks helper run it.
  • or integrate compression program. gzip , bzip2 both has libraries, zlib , libbz2, , there wrappers in boost.

Comments

Popular posts from this blog

java - Intellij Synchronizing output directories .. -

git - Initial Commit: "fatal: could not create leading directories of ..." -