From c9b14e575935b3540e08f4467cf6588be2669710 Mon Sep 17 00:00:00 2001 From: Oleg Nesterov Date: Tue, 14 Dec 2010 23:40:11 +0100 Subject: exec: make argv/envp memory visible to oom-killer commit 3c77f845722158206a7209c45ccddc264d19319c upstream. Brad Spengler published a local memory-allocation DoS that evades the OOM-killer (though not the virtual memory RLIMIT): http://www.grsecurity.net/~spender/64bit_dos.c execve()->copy_strings() can allocate a lot of memory, but this is not visible to oom-killer, nobody can see the nascent bprm->mm and take it into account. With this patch get_arg_page() increments current's MM_ANONPAGES counter every time we allocate the new page for argv/envp. When do_execve() succeds or fails, we change this counter back. Technically this is not 100% correct, we can't know if the new page is swapped out and turn MM_ANONPAGES into MM_SWAPENTS, but I don't think this really matters and everything becomes correct once exec changes ->mm or fails. Compared to upstream: before 2.6.36 kernel, oom-killer's badness() takes mm->total_vm into account and nothing else. So acct_arg_size() has to play with this counter too. Reported-by: Brad Spengler Signed-off-by: Andi Kleen Reviewed-and-discussed-by: KOSAKI Motohiro Signed-off-by: Oleg Nesterov Signed-off-by: Linus Torvalds Signed-off-by: Greg Kroah-Hartman --- include/linux/binfmts.h | 1 + 1 file changed, 1 insertion(+) (limited to 'include') diff --git a/include/linux/binfmts.h b/include/linux/binfmts.h index c809e286d21..39798c65c23 100644 --- a/include/linux/binfmts.h +++ b/include/linux/binfmts.h @@ -29,6 +29,7 @@ struct linux_binprm{ char buf[BINPRM_BUF_SIZE]; #ifdef CONFIG_MMU struct vm_area_struct *vma; + unsigned long vma_pages; #else # define MAX_ARG_PAGES 32 struct page *page[MAX_ARG_PAGES]; -- cgit v1.2.3-18-g5258