fluencelabs/musl
1
0
Fork 0
mirror of https://github.com/fluencelabs/musl synced 2025-04-25 07:12:15 +00:00
Code Issues Projects Releases Wiki Activity
musl/src/malloc/malloc.c

552 lines
12 KiB
C
Raw Normal View History

namespace fixes for sys/mman.h
2011-04-20 15:55:58 -04:00
#define _GNU_SOURCE
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
#include <stdlib.h>
#include <string.h>
#include <limits.h>
#include <stdint.h>
#include <errno.h>
#include <sys/mman.h>
#include "libc.h"
#include "atomic.h"
#include "pthread_impl.h"
move malloc implementation types and macros to an internal header
2018-04-19 18:43:05 -04:00
#include "malloc_impl.h"
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
workaround gcc got-register-reload performance problems in malloc with this patch, the malloc in libc.so built with -Os is nearly the same speed as the one built with -O3. thus it solves the performance regression that resulted from removing the forced -O3 when building libc.so; now libc.so can be both small and fast.
2012-09-14 23:52:51 -04:00
#if defined(__GNUC__) && defined(__PIC__)
#define inline inline __attribute__((always_inline))
#endif
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
static struct {
make all objects used with atomic operations volatile the memory model we use internally for atomics permits plain loads of values which may be subject to concurrent modification without requiring that a special load function be used. since a compiler is free to make transformations that alter the number of loads or the way in which loads are performed, the compiler is theoretically free to break this usage. the most obvious concern is with atomic cas constructs: something of the form tmp=*p;a_cas(p,tmp,f(tmp)); could be transformed to a_cas(p,*p,f(*p)); where the latter is intended to show multiple loads of *p whose resulting values might fail to be equal; this would break the atomicity of the whole operation. but even more fundamental breakage is possible. with the changes being made now, objects that may be modified by atomics are modeled as volatile, and the atomic operations performed on them by other threads are modeled as asynchronous stores by hardware which happens to be acting on the request of another thread. such modeling of course does not itself address memory synchronization between cores/cpus, but that aspect was already handled. this all seems less than ideal, but it's the best we can do without mandating a C11 compiler and using the C11 model for atomics. in the case of pthread_once_t, the ABI type of the underlying object is not volatile-qualified. so we are assuming that accessing the object through a volatile-qualified lvalue via casts yields volatile access semantics. the language of the C standard is somewhat unclear on this matter, but this is an assumption the linux kernel also makes, and seems to be the correct interpretation of the standard.
2015-03-03 22:50:02 -05:00
volatile uint64_t binmap;
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
struct bin bins[64];
make all objects used with atomic operations volatile the memory model we use internally for atomics permits plain loads of values which may be subject to concurrent modification without requiring that a special load function be used. since a compiler is free to make transformations that alter the number of loads or the way in which loads are performed, the compiler is theoretically free to break this usage. the most obvious concern is with atomic cas constructs: something of the form tmp=*p;a_cas(p,tmp,f(tmp)); could be transformed to a_cas(p,*p,f(*p)); where the latter is intended to show multiple loads of *p whose resulting values might fail to be equal; this would break the atomicity of the whole operation. but even more fundamental breakage is possible. with the changes being made now, objects that may be modified by atomics are modeled as volatile, and the atomic operations performed on them by other threads are modeled as asynchronous stores by hardware which happens to be acting on the request of another thread. such modeling of course does not itself address memory synchronization between cores/cpus, but that aspect was already handled. this all seems less than ideal, but it's the best we can do without mandating a C11 compiler and using the C11 model for atomics. in the case of pthread_once_t, the ABI type of the underlying object is not volatile-qualified. so we are assuming that accessing the object through a volatile-qualified lvalue via casts yields volatile access semantics. the language of the C standard is somewhat unclear on this matter, but this is an assumption the linux kernel also makes, and seems to be the correct interpretation of the standard.
2015-03-03 22:50:02 -05:00
volatile int free_lock[2];
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
} mal;
reintroduce hardening against partially-replaced allocator commit 618b18c78e33acfe54a4434e91aa57b8e171df89 removed the previous detection and hardening since it was incorrect. commit 72141795d4edd17f88da192447395a48444afa10 already handled all that remained for hardening the static-linked case. in the dynamic-linked case, have the dynamic linker check whether malloc was replaced and make that information available. with these changes, the properties documented in commit c9f415d7ea2dace5bf77f6518b6afc36bb7a5732 are restored: if calloc is not provided, it will behave as malloc+memset, and any of the memalign-family functions not provided will fail with ENOMEM.
2018-04-19 22:19:29 -04:00
int __malloc_replaced;
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
/* Synchronization tools */
workaround gcc got-register-reload performance problems in malloc with this patch, the malloc in libc.so built with -Os is nearly the same speed as the one built with -O3. thus it solves the performance regression that resulted from removing the forced -O3 when building libc.so; now libc.so can be both small and fast.
2012-09-14 23:52:51 -04:00
static inline void lock(volatile int *lk)
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
{
fix potential deadlock bug in libc-internal locking logic if a multithreaded program became non-multithreaded (i.e. all other threads exited) while one thread held an internal lock, the remaining thread would fail to release the lock. the the program then became multithreaded again at a later time, any further attempts to obtain the lock would deadlock permanently. the underlying cause is that the value of libc.threads_minus_1 at unlock time might not match the value at lock time. one solution would be returning a flag to the caller indicating whether the lock was taken and needs to be unlocked, but there is a simpler solution: using the lock itself as such a flag. note that this flag is not needed anyway for correctness; if the lock is not held, the unlock code is harmless. however, the memory synchronization properties associated with a_store are costly on some archs, so it's best to avoid executing the unlock code when it is unnecessary.
2013-09-20 02:00:27 -04:00
if (libc.threads_minus_1)
while(a_swap(lk, 1)) __wait(lk, lk+1, 1, 1);
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
}
workaround gcc got-register-reload performance problems in malloc with this patch, the malloc in libc.so built with -Os is nearly the same speed as the one built with -O3. thus it solves the performance regression that resulted from removing the forced -O3 when building libc.so; now libc.so can be both small and fast.
2012-09-14 23:52:51 -04:00
static inline void unlock(volatile int *lk)
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
{
fix potential deadlock bug in libc-internal locking logic if a multithreaded program became non-multithreaded (i.e. all other threads exited) while one thread held an internal lock, the remaining thread would fail to release the lock. the the program then became multithreaded again at a later time, any further attempts to obtain the lock would deadlock permanently. the underlying cause is that the value of libc.threads_minus_1 at unlock time might not match the value at lock time. one solution would be returning a flag to the caller indicating whether the lock was taken and needs to be unlocked, but there is a simpler solution: using the lock itself as such a flag. note that this flag is not needed anyway for correctness; if the lock is not held, the unlock code is harmless. however, the memory synchronization properties associated with a_store are costly on some archs, so it's best to avoid executing the unlock code when it is unnecessary.
2013-09-20 02:00:27 -04:00
if (lk[0]) {
a_store(lk, 0);
if (lk[1]) __wake(lk, 1, 1);
}
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
}
workaround gcc got-register-reload performance problems in malloc with this patch, the malloc in libc.so built with -Os is nearly the same speed as the one built with -O3. thus it solves the performance regression that resulted from removing the forced -O3 when building libc.so; now libc.so can be both small and fast.
2012-09-14 23:52:51 -04:00
static inline void lock_bin(int i)
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
{
fix potential deadlock bug in libc-internal locking logic if a multithreaded program became non-multithreaded (i.e. all other threads exited) while one thread held an internal lock, the remaining thread would fail to release the lock. the the program then became multithreaded again at a later time, any further attempts to obtain the lock would deadlock permanently. the underlying cause is that the value of libc.threads_minus_1 at unlock time might not match the value at lock time. one solution would be returning a flag to the caller indicating whether the lock was taken and needs to be unlocked, but there is a simpler solution: using the lock itself as such a flag. note that this flag is not needed anyway for correctness; if the lock is not held, the unlock code is harmless. however, the memory synchronization properties associated with a_store are costly on some archs, so it's best to avoid executing the unlock code when it is unnecessary.
2013-09-20 02:00:27 -04:00
lock(mal.bins[i].lock);
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
if (!mal.bins[i].head)
mal.bins[i].head = mal.bins[i].tail = BIN_TO_CHUNK(i);
}
workaround gcc got-register-reload performance problems in malloc with this patch, the malloc in libc.so built with -Os is nearly the same speed as the one built with -O3. thus it solves the performance regression that resulted from removing the forced -O3 when building libc.so; now libc.so can be both small and fast.
2012-09-14 23:52:51 -04:00
static inline void unlock_bin(int i)
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
{
unlock(mal.bins[i].lock);
}
static int first_set(uint64_t x)
{
#if 1
return a_ctz_64(x);
#else
static const char debruijn64[64] = {
0, 1, 2, 53, 3, 7, 54, 27, 4, 38, 41, 8, 34, 55, 48, 28,
62, 5, 39, 46, 44, 42, 22, 9, 24, 35, 59, 56, 49, 18, 29, 11,
63, 52, 6, 26, 37, 40, 33, 47, 61, 45, 43, 21, 23, 58, 17, 10,
51, 25, 36, 32, 60, 20, 57, 16, 50, 31, 19, 15, 30, 14, 13, 12
};
static const char debruijn32[32] = {
0, 1, 23, 2, 29, 24, 19, 3, 30, 27, 25, 11, 20, 8, 4, 13,
31, 22, 28, 18, 26, 10, 7, 12, 21, 17, 9, 6, 16, 5, 15, 14
};
if (sizeof(long) < 8) {
uint32_t y = x;
if (!y) {
y = x>>32;
return 32 + debruijn32[(y&-y)*0x076be629 >> 27];
}
return debruijn32[(y&-y)*0x076be629 >> 27];
}
return debruijn64[(x&-x)*0x022fdd63cc95386dull >> 58];
#endif
}
use lookup table for malloc bin index instead of float conversion float conversion is slow and big on soft-float targets. The lookup table increases code size a bit on most hard float targets (and adds 60byte rodata), performance can be a bit slower because of position independent data access and cpu internal state dependence (cache, extra branches), but the overall effect should be minimal (common, small size allocations should be unaffected).
2016-12-17 15:03:24 +01:00
static const unsigned char bin_tab[60] = {
32,33,34,35,36,36,37,37,38,38,39,39,
40,40,40,40,41,41,41,41,42,42,42,42,43,43,43,43,
44,44,44,44,44,44,44,44,45,45,45,45,45,45,45,45,
46,46,46,46,46,46,46,46,47,47,47,47,47,47,47,47,
};
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
static int bin_index(size_t x)
{
x = x / SIZE_ALIGN - 1;
if (x <= 32) return x;
use lookup table for malloc bin index instead of float conversion float conversion is slow and big on soft-float targets. The lookup table increases code size a bit on most hard float targets (and adds 60byte rodata), performance can be a bit slower because of position independent data access and cpu internal state dependence (cache, extra branches), but the overall effect should be minimal (common, small size allocations should be unaffected).
2016-12-17 15:03:24 +01:00
if (x < 512) return bin_tab[x/8-4];
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
if (x > 0x1c00) return 63;
use lookup table for malloc bin index instead of float conversion float conversion is slow and big on soft-float targets. The lookup table increases code size a bit on most hard float targets (and adds 60byte rodata), performance can be a bit slower because of position independent data access and cpu internal state dependence (cache, extra branches), but the overall effect should be minimal (common, small size allocations should be unaffected).
2016-12-17 15:03:24 +01:00
return bin_tab[x/128-4] + 16;
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
}
static int bin_index_up(size_t x)
{
x = x / SIZE_ALIGN - 1;
if (x <= 32) return x;
use lookup table for malloc bin index instead of float conversion float conversion is slow and big on soft-float targets. The lookup table increases code size a bit on most hard float targets (and adds 60byte rodata), performance can be a bit slower because of position independent data access and cpu internal state dependence (cache, extra branches), but the overall effect should be minimal (common, small size allocations should be unaffected).
2016-12-17 15:03:24 +01:00
x--;
if (x < 512) return bin_tab[x/8-4] + 1;
Fix off-by-one error in bin_index_up
2018-10-24 13:42:00 +02:00
if (x > 0x1c00) return 64;
use lookup table for malloc bin index instead of float conversion float conversion is slow and big on soft-float targets. The lookup table increases code size a bit on most hard float targets (and adds 60byte rodata), performance can be a bit slower because of position independent data access and cpu internal state dependence (cache, extra branches), but the overall effect should be minimal (common, small size allocations should be unaffected).
2016-12-17 15:03:24 +01:00
return bin_tab[x/128-4] + 17;
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
}
#if 0
void __dump_heap(int x)
{
struct chunk *c;
int i;
for (c = (void *)mal.heap; CHUNK_SIZE(c); c = NEXT_CHUNK(c))
fprintf(stderr, "base %p size %zu (%d) flags %d/%d\n",
c, CHUNK_SIZE(c), bin_index(CHUNK_SIZE(c)),
eliminate OOB array hacks in malloc
2011-06-26 16:12:43 -04:00
c->csize & 15,
NEXT_CHUNK(c)->psize & 15);
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
for (i=0; i<64; i++) {
if (mal.bins[i].head != BIN_TO_CHUNK(i) && mal.bins[i].head) {
fprintf(stderr, "bin %d: %p\n", i, mal.bins[i].head);
if (!(mal.binmap & 1ULL<<i))
fprintf(stderr, "missing from binmap!\n");
} else if (mal.binmap & 1ULL<<i)
fprintf(stderr, "binmap wrongly contains %d!\n", i);
}
}
#endif
refactor malloc's expand_heap to share with __simple_malloc this extends the brk/stack collision protection added to full malloc in commit 276904c2f6bde3a31a24ebfa201482601d18b4f9 to also protect the __simple_malloc function used in static-linked programs that don't reference the free function. it also extends support for using mmap when brk fails, which full malloc got in commit 5446303328adf4b4e36d9fba21848e6feb55fab4, to __simple_malloc. since __simple_malloc may expand the heap by arbitrarily large increments, the stack collision detection is enhanced to detect interval overlap rather than just proximity of a single address to the stack. code size is increased a bit, but this is partly offset by the sharing of code between the two malloc implementations, which due to linking semantics, both get linked in a program that needs the full malloc with realloc/free support.
2015-06-14 01:59:02 +00:00
void *__expand_heap(size_t *);
in malloc, refuse to use brk if it grows into stack the linux/nommu fdpic ELF loader sets up the brk range to overlap entirely with the main thread's stack (but growing from opposite ends), so that the resulting failure mode for malloc is not to return a null pointer but to start returning pointers to memory that overlaps with the caller's stack. needless to say this extremely dangerous and makes brk unusable. since it's non-trivial to detect execution environments that might be affected by this kernel bug, and since the severity of the bug makes any sort of detection that might yield false-negatives unsafe, we instead check the proximity of the brk to the stack pointer each time the brk is to be expanded. both the main thread's stack (where the real known risk lies) and the calling thread's stack are checked. an arbitrary gap distance of 8 MB is imposed, chosen to be larger than linux default main-thread stack reservation sizes and larger than any reasonable stack configuration on nommu. the effeciveness of this patch relies on an assumption that the amount by which the brk is being grown is smaller than the gap limit, which is always true for malloc's use of brk. reliance on this assumption is why the check is being done in malloc-specific code and not in __brk.
2015-06-09 20:30:35 +00:00
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
static struct chunk *expand_heap(size_t n)
{
refactor malloc's expand_heap to share with __simple_malloc this extends the brk/stack collision protection added to full malloc in commit 276904c2f6bde3a31a24ebfa201482601d18b4f9 to also protect the __simple_malloc function used in static-linked programs that don't reference the free function. it also extends support for using mmap when brk fails, which full malloc got in commit 5446303328adf4b4e36d9fba21848e6feb55fab4, to __simple_malloc. since __simple_malloc may expand the heap by arbitrarily large increments, the stack collision detection is enhanced to detect interval overlap rather than just proximity of a single address to the stack. code size is increased a bit, but this is partly offset by the sharing of code between the two malloc implementations, which due to linking semantics, both get linked in a program that needs the full malloc with realloc/free support.
2015-06-14 01:59:02 +00:00
static int heap_lock[2];
static void *end;
void *p;
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
struct chunk *w;
refactor malloc's expand_heap to share with __simple_malloc this extends the brk/stack collision protection added to full malloc in commit 276904c2f6bde3a31a24ebfa201482601d18b4f9 to also protect the __simple_malloc function used in static-linked programs that don't reference the free function. it also extends support for using mmap when brk fails, which full malloc got in commit 5446303328adf4b4e36d9fba21848e6feb55fab4, to __simple_malloc. since __simple_malloc may expand the heap by arbitrarily large increments, the stack collision detection is enhanced to detect interval overlap rather than just proximity of a single address to the stack. code size is increased a bit, but this is partly offset by the sharing of code between the two malloc implementations, which due to linking semantics, both get linked in a program that needs the full malloc with realloc/free support.
2015-06-14 01:59:02 +00:00
/* The argument n already accounts for the caller's chunk
* overhead needs, but if the heap can't be extended in-place,
* we need room for an extra zero-sized sentinel chunk. */
n += SIZE_ALIGN;
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
refactor malloc's expand_heap to share with __simple_malloc this extends the brk/stack collision protection added to full malloc in commit 276904c2f6bde3a31a24ebfa201482601d18b4f9 to also protect the __simple_malloc function used in static-linked programs that don't reference the free function. it also extends support for using mmap when brk fails, which full malloc got in commit 5446303328adf4b4e36d9fba21848e6feb55fab4, to __simple_malloc. since __simple_malloc may expand the heap by arbitrarily large increments, the stack collision detection is enhanced to detect interval overlap rather than just proximity of a single address to the stack. code size is increased a bit, but this is partly offset by the sharing of code between the two malloc implementations, which due to linking semantics, both get linked in a program that needs the full malloc with realloc/free support.
2015-06-14 01:59:02 +00:00
lock(heap_lock);
p = __expand_heap(&n);
if (!p) {
unlock(heap_lock);
return 0;
fix init race that could lead to deadlock in malloc init code the malloc init code provided its own version of pthread_once type logic, including the exact same bug that was fixed in pthread_once in commit 0d0c2f40344640a2a6942dda156509593f51db5d. since this code is called adjacent to expand_heap, which takes a lock, there is no reason to have pthread_once-type initialization. simply moving the init code into the interval where expand_heap already holds its lock on the brk achieves the same result with much less synchronization logic, and allows the buggy code to be eliminated rather than just fixed.
2015-03-04 09:29:39 -05:00
}
refactor malloc's expand_heap to share with __simple_malloc this extends the brk/stack collision protection added to full malloc in commit 276904c2f6bde3a31a24ebfa201482601d18b4f9 to also protect the __simple_malloc function used in static-linked programs that don't reference the free function. it also extends support for using mmap when brk fails, which full malloc got in commit 5446303328adf4b4e36d9fba21848e6feb55fab4, to __simple_malloc. since __simple_malloc may expand the heap by arbitrarily large increments, the stack collision detection is enhanced to detect interval overlap rather than just proximity of a single address to the stack. code size is increased a bit, but this is partly offset by the sharing of code between the two malloc implementations, which due to linking semantics, both get linked in a program that needs the full malloc with realloc/free support.
2015-06-14 01:59:02 +00:00
/* If not just expanding existing space, we need to make a
* new sentinel chunk below the allocated space. */
if (p != end) {
/* Valid/safe because of the prologue increment. */
avoid malloc failure for small requests when brk can't be extended this issue mainly affects PIE binaries and execution of programs via direct invocation of the dynamic linker binary: depending on kernel behavior, in these cases the initial brk may be placed at at location where it cannot be extended, due to conflicting adjacent maps. when brk fails, mmap is used instead to expand the heap. in order to avoid expensive bookkeeping for managing fragmentation by merging these new heap regions, the minimum size for new heap regions increases exponentially in the number of regions. this limits the number of regions, and thereby the number of fixed fragmentation points, to a quantity which is logarithmic with respect to the size of virtual address space and thus negligible. the exponential growth is tuned so as to avoid expanding the heap by more than approximately 50% of its current total size.
2014-04-02 17:57:15 -04:00
n -= SIZE_ALIGN;
refactor malloc's expand_heap to share with __simple_malloc this extends the brk/stack collision protection added to full malloc in commit 276904c2f6bde3a31a24ebfa201482601d18b4f9 to also protect the __simple_malloc function used in static-linked programs that don't reference the free function. it also extends support for using mmap when brk fails, which full malloc got in commit 5446303328adf4b4e36d9fba21848e6feb55fab4, to __simple_malloc. since __simple_malloc may expand the heap by arbitrarily large increments, the stack collision detection is enhanced to detect interval overlap rather than just proximity of a single address to the stack. code size is increased a bit, but this is partly offset by the sharing of code between the two malloc implementations, which due to linking semantics, both get linked in a program that needs the full malloc with realloc/free support.
2015-06-14 01:59:02 +00:00
p = (char *)p + SIZE_ALIGN;
w = MEM_TO_CHUNK(p);
avoid malloc failure for small requests when brk can't be extended this issue mainly affects PIE binaries and execution of programs via direct invocation of the dynamic linker binary: depending on kernel behavior, in these cases the initial brk may be placed at at location where it cannot be extended, due to conflicting adjacent maps. when brk fails, mmap is used instead to expand the heap. in order to avoid expensive bookkeeping for managing fragmentation by merging these new heap regions, the minimum size for new heap regions increases exponentially in the number of regions. this limits the number of regions, and thereby the number of fixed fragmentation points, to a quantity which is logarithmic with respect to the size of virtual address space and thus negligible. the exponential growth is tuned so as to avoid expanding the heap by more than approximately 50% of its current total size.
2014-04-02 17:57:15 -04:00
w->psize = 0 | C_INUSE;
}
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
refactor malloc's expand_heap to share with __simple_malloc this extends the brk/stack collision protection added to full malloc in commit 276904c2f6bde3a31a24ebfa201482601d18b4f9 to also protect the __simple_malloc function used in static-linked programs that don't reference the free function. it also extends support for using mmap when brk fails, which full malloc got in commit 5446303328adf4b4e36d9fba21848e6feb55fab4, to __simple_malloc. since __simple_malloc may expand the heap by arbitrarily large increments, the stack collision detection is enhanced to detect interval overlap rather than just proximity of a single address to the stack. code size is increased a bit, but this is partly offset by the sharing of code between the two malloc implementations, which due to linking semantics, both get linked in a program that needs the full malloc with realloc/free support.
2015-06-14 01:59:02 +00:00
/* Record new heap end and fill in footer. */
end = (char *)p + n;
w = MEM_TO_CHUNK(end);
eliminate OOB array hacks in malloc
2011-06-26 16:12:43 -04:00
w->psize = n | C_INUSE;
w->csize = 0 | C_INUSE;
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
refactor malloc's expand_heap to share with __simple_malloc this extends the brk/stack collision protection added to full malloc in commit 276904c2f6bde3a31a24ebfa201482601d18b4f9 to also protect the __simple_malloc function used in static-linked programs that don't reference the free function. it also extends support for using mmap when brk fails, which full malloc got in commit 5446303328adf4b4e36d9fba21848e6feb55fab4, to __simple_malloc. since __simple_malloc may expand the heap by arbitrarily large increments, the stack collision detection is enhanced to detect interval overlap rather than just proximity of a single address to the stack. code size is increased a bit, but this is partly offset by the sharing of code between the two malloc implementations, which due to linking semantics, both get linked in a program that needs the full malloc with realloc/free support.
2015-06-14 01:59:02 +00:00
/* Fill in header, which may be new or may be replacing a
* zero-size sentinel header at the old end-of-heap. */
w = MEM_TO_CHUNK(p);
eliminate OOB array hacks in malloc
2011-06-26 16:12:43 -04:00
w->csize = n | C_INUSE;
refactor malloc's expand_heap to share with __simple_malloc this extends the brk/stack collision protection added to full malloc in commit 276904c2f6bde3a31a24ebfa201482601d18b4f9 to also protect the __simple_malloc function used in static-linked programs that don't reference the free function. it also extends support for using mmap when brk fails, which full malloc got in commit 5446303328adf4b4e36d9fba21848e6feb55fab4, to __simple_malloc. since __simple_malloc may expand the heap by arbitrarily large increments, the stack collision detection is enhanced to detect interval overlap rather than just proximity of a single address to the stack. code size is increased a bit, but this is partly offset by the sharing of code between the two malloc implementations, which due to linking semantics, both get linked in a program that needs the full malloc with realloc/free support.
2015-06-14 01:59:02 +00:00
unlock(heap_lock);
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
return w;
}
static int adjust_size(size_t *n)
{
/* Result of pointer difference must fit in ptrdiff_t. */
make malloc(0) return unique pointers rather than NULL this change is made with some reluctance, but i think it's for the best. correct programs must handle either behavior, so there is little advantage to having malloc(0) return NULL. and i managed to actually make the malloc code slightly smaller with this change.
2011-02-20 16:16:33 -05:00
if (*n-1 > PTRDIFF_MAX - SIZE_ALIGN - PAGE_SIZE) {
if (*n) {
errno = ENOMEM;
return -1;
} else {
*n = SIZE_ALIGN;
return 0;
}
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
}
*n = (*n + OVERHEAD + SIZE_ALIGN - 1) & SIZE_MASK;
return 0;
}
static void unbin(struct chunk *c, int i)
{
if (c->prev == c->next)
a_and_64(&mal.binmap, ~(1ULL<<i));
c->prev->next = c->next;
c->next->prev = c->prev;
eliminate OOB array hacks in malloc
2011-06-26 16:12:43 -04:00
c->csize |= C_INUSE;
NEXT_CHUNK(c)->psize |= C_INUSE;
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
}
static int alloc_fwd(struct chunk *c)
{
int i;
size_t k;
eliminate OOB array hacks in malloc
2011-06-26 16:12:43 -04:00
while (!((k=c->csize) & C_INUSE)) {
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
i = bin_index(k);
lock_bin(i);
eliminate OOB array hacks in malloc
2011-06-26 16:12:43 -04:00
if (c->csize == k) {
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
unbin(c, i);
unlock_bin(i);
return 1;
}
unlock_bin(i);
}
return 0;
}
static int alloc_rev(struct chunk *c)
{
int i;
size_t k;
eliminate OOB array hacks in malloc
2011-06-26 16:12:43 -04:00
while (!((k=c->psize) & C_INUSE)) {
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
i = bin_index(k);
lock_bin(i);
eliminate OOB array hacks in malloc
2011-06-26 16:12:43 -04:00
if (c->psize == k) {
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
unbin(PREV_CHUNK(c), i);
unlock_bin(i);
return 1;
}
unlock_bin(i);
}
return 0;
}
/* pretrim - trims a chunk _prior_ to removing it from its bin.
* Must be called with i as the ideal bin for size n, j the bin
* for the _free_ chunk self, and bin j locked. */
static int pretrim(struct chunk *self, size_t n, int i, int j)
{
size_t n1;
struct chunk *next, *split;
/* We cannot pretrim if it would require re-binning. */
if (j < 40) return 0;
if (j < i+3) {
if (j != 63) return 0;
n1 = CHUNK_SIZE(self);
if (n1-n <= MMAP_THRESHOLD) return 0;
} else {
n1 = CHUNK_SIZE(self);
}
if (bin_index(n1-n) != j) return 0;
next = NEXT_CHUNK(self);
split = (void *)((char *)self + n);
split->prev = self->prev;
split->next = self->next;
split->prev->next = split;
split->next->prev = split;
eliminate OOB array hacks in malloc
2011-06-26 16:12:43 -04:00
split->psize = n | C_INUSE;
split->csize = n1-n;
next->psize = n1-n;
self->csize = n | C_INUSE;
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
return 1;
}
static void trim(struct chunk *self, size_t n)
{
size_t n1 = CHUNK_SIZE(self);
struct chunk *next, *split;
if (n >= n1 - DONTCARE) return;
next = NEXT_CHUNK(self);
split = (void *)((char *)self + n);
eliminate OOB array hacks in malloc
2011-06-26 16:12:43 -04:00
split->psize = n | C_INUSE;
split->csize = n1-n | C_INUSE;
next->psize = n1-n | C_INUSE;
self->csize = n | C_INUSE;
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
return chunks split off by memalign using __bin_chunk instead of free this change serves multiple purposes: 1. it ensures that static linking of memalign-family functions will pull in the system malloc implementation, thereby causing link errors if an attempt is made to link the system memalign functions with a replacement malloc (incomplete allocator replacement). 2. it eliminates calls to free that are unpaired with allocations, which are confusing when setting breakpoints or tracing execution. as a bonus, making __bin_chunk external may discourage aggressive and unnecessary inlining of it.
2018-04-19 20:56:26 -04:00
__bin_chunk(split);
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
}
void *malloc(size_t n)
{
struct chunk *c;
int i, j;
make malloc(0) return unique pointers rather than NULL this change is made with some reluctance, but i think it's for the best. correct programs must handle either behavior, so there is little advantage to having malloc(0) return NULL. and i managed to actually make the malloc code slightly smaller with this change.
2011-02-20 16:16:33 -05:00
if (adjust_size(&n) < 0) return 0;
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
if (n > MMAP_THRESHOLD) {
fix rare but nasty under-allocation bug in malloc with large requests the bug appeared only with requests roughly 2*sizeof(size_t) to 4*sizeof(size_t) bytes smaller than a multiple of the page size, and only for requests large enough to be serviced by mmap instead of the normal heap. it was only ever observed on 64-bit machines but presumably could also affect 32-bit (albeit with a smaller window of opportunity).
2011-04-04 17:26:41 -04:00
size_t len = n + OVERHEAD + PAGE_SIZE - 1 & -PAGE_SIZE;
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
char *base = __mmap(0, len, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
if (base == (void *)-1) return 0;
eliminate OOB array hacks in malloc
2011-06-26 16:12:43 -04:00
c = (void *)(base + SIZE_ALIGN - OVERHEAD);
c->csize = len - (SIZE_ALIGN - OVERHEAD);
c->psize = SIZE_ALIGN - OVERHEAD;
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
return CHUNK_TO_MEM(c);
}
i = bin_index_up(n);
for (;;) {
uint64_t mask = mal.binmap & -(1ULL<<i);
if (!mask) {
c = expand_heap(n);
if (!c) return 0;
if (alloc_rev(c)) {
struct chunk *x = c;
c = PREV_CHUNK(c);
eliminate OOB array hacks in malloc
2011-06-26 16:12:43 -04:00
NEXT_CHUNK(x)->psize = c->csize =
x->csize + CHUNK_SIZE(c);
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
}
break;
}
j = first_set(mask);
lock_bin(j);
c = mal.bins[j].head;
remove useless check of bin match in malloc this re-check idiom seems to have been copied from the alloc_fwd and alloc_rev functions, which guess a bin based on non-synchronized memory access to adjacent chunk headers then need to confirm, after locking the bin, that the chunk is actually in the bin they locked. the check being removed, however, was being performed on a chunk obtained from the already-locked bin. there is no race to account for here; the check could only fail in the event of corrupt free lists, and even then it would not catch them but simply continue running. since the bin_index function is mildly expensive, it seems preferable to remove the check rather than trying to convert it into a useful consistency check. casual testing shows a 1-5% reduction in run time.
2015-03-04 10:48:00 -05:00
if (c != BIN_TO_CHUNK(j)) {
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
if (!pretrim(c, n, i, j)) unbin(c, j);
unlock_bin(j);
break;
}
unlock_bin(j);
}
/* Now patch up in case we over-allocated */
trim(c, n);
return CHUNK_TO_MEM(c);
}
optimize malloc0 Implementation of __malloc0 in malloc.c takes care to preserve zero pages by overwriting only non-zero data. However, malloc must have already modified auxiliary heap data just before and beyond the allocated region, so we know that edge pages need not be preserved. For allocations smaller than one page, pass them immediately to memset. Otherwise, use memset to handle partial pages at the head and tail of the allocation, and scan complete pages in the interior. Optimize the scanning loop by processing 16 bytes per iteration and handling rest of page via memset as soon as a non-zero byte is found.
2017-12-16 14:27:25 +03:00
static size_t mal0_clear(char *p, size_t pagesz, size_t n)
{
#ifdef __GNUC__
typedef uint64_t __attribute__((__may_alias__)) T;
#else
typedef unsigned char T;
#endif
char *pp = p + n;
size_t i = (uintptr_t)pp & (pagesz - 1);
for (;;) {
pp = memset(pp - i, 0, i);
if (pp - p < pagesz) return pp - p;
for (i = pagesz; i; i -= 2*sizeof(T), pp -= 2*sizeof(T))
if (((T *)pp)[-1] | ((T *)pp)[-2])
break;
}
}
allow interposition/replacement of allocator (malloc) replacement is subject to conditions on the replacement functions. they may only call functions which are async-signal-safe, as specified either by POSIX or as an implementation-defined extension. if any allocator functions are replaced, at least malloc, realloc, and free must be provided. if calloc is not provided, it will behave as malloc+memset. any of the memalign-family functions not provided will fail with ENOMEM. in order to implement the above properties, calloc and __memalign check that they are using their own malloc or free, respectively. choice to check malloc or free is based on considerations of supporting __simple_malloc. in order to make this work, calloc is split into separate versions for __simple_malloc and full malloc; commit ba819787ee93ceae94efd274f7849e317c1bff58 already did most of the split anyway, and completing it saves an extra call frame. previously, use of -Bsymbolic-functions made dynamic interposition impossible. now, we are using an explicit dynamic-list, so add allocator functions to the list. most are not referenced anyway, but all are added for completeness.
2018-04-17 18:36:19 -04:00
void *calloc(size_t m, size_t n)
fix calloc when __simple_malloc implementation is used previously, calloc's implementation encoded assumptions about the implementation of malloc, accessing a size_t word just prior to the allocated memory to determine if it was obtained by mmap to optimize out the zero-filling. when __simple_malloc is used (static linking a program with no realloc/free), it doesn't matter if the result of this check is wrong, since all allocations are zero-initialized anyway. but the access could be invalid if it crosses a page boundary or if the pointer is not sufficiently aligned, which can happen for very small allocations. this patch fixes the issue by moving the zero-fill logic into malloc.c with the full malloc, as a new function named __malloc0, which is provided by a weak alias to __simple_malloc (which always gives zero-filled memory) when the full malloc is not in use.
2015-06-22 18:50:09 +00:00
{
allow interposition/replacement of allocator (malloc) replacement is subject to conditions on the replacement functions. they may only call functions which are async-signal-safe, as specified either by POSIX or as an implementation-defined extension. if any allocator functions are replaced, at least malloc, realloc, and free must be provided. if calloc is not provided, it will behave as malloc+memset. any of the memalign-family functions not provided will fail with ENOMEM. in order to implement the above properties, calloc and __memalign check that they are using their own malloc or free, respectively. choice to check malloc or free is based on considerations of supporting __simple_malloc. in order to make this work, calloc is split into separate versions for __simple_malloc and full malloc; commit ba819787ee93ceae94efd274f7849e317c1bff58 already did most of the split anyway, and completing it saves an extra call frame. previously, use of -Bsymbolic-functions made dynamic interposition impossible. now, we are using an explicit dynamic-list, so add allocator functions to the list. most are not referenced anyway, but all are added for completeness.
2018-04-17 18:36:19 -04:00
if (n && m > (size_t)-1/n) {
errno = ENOMEM;
return 0;
}
n *= m;
fix calloc when __simple_malloc implementation is used previously, calloc's implementation encoded assumptions about the implementation of malloc, accessing a size_t word just prior to the allocated memory to determine if it was obtained by mmap to optimize out the zero-filling. when __simple_malloc is used (static linking a program with no realloc/free), it doesn't matter if the result of this check is wrong, since all allocations are zero-initialized anyway. but the access could be invalid if it crosses a page boundary or if the pointer is not sufficiently aligned, which can happen for very small allocations. this patch fixes the issue by moving the zero-fill logic into malloc.c with the full malloc, as a new function named __malloc0, which is provided by a weak alias to __simple_malloc (which always gives zero-filled memory) when the full malloc is not in use.
2015-06-22 18:50:09 +00:00
void *p = malloc(n);
reintroduce hardening against partially-replaced allocator commit 618b18c78e33acfe54a4434e91aa57b8e171df89 removed the previous detection and hardening since it was incorrect. commit 72141795d4edd17f88da192447395a48444afa10 already handled all that remained for hardening the static-linked case. in the dynamic-linked case, have the dynamic linker check whether malloc was replaced and make that information available. with these changes, the properties documented in commit c9f415d7ea2dace5bf77f6518b6afc36bb7a5732 are restored: if calloc is not provided, it will behave as malloc+memset, and any of the memalign-family functions not provided will fail with ENOMEM.
2018-04-19 22:19:29 -04:00
if (!p) return p;
if (!__malloc_replaced) {
if (IS_MMAPPED(MEM_TO_CHUNK(p)))
return p;
if (n >= PAGE_SIZE)
n = mal0_clear(p, PAGE_SIZE, n);
}
optimize malloc0 Implementation of __malloc0 in malloc.c takes care to preserve zero pages by overwriting only non-zero data. However, malloc must have already modified auxiliary heap data just before and beyond the allocated region, so we know that edge pages need not be preserved. For allocations smaller than one page, pass them immediately to memset. Otherwise, use memset to handle partial pages at the head and tail of the allocation, and scan complete pages in the interior. Optimize the scanning loop by processing 16 bytes per iteration and handling rest of page via memset as soon as a non-zero byte is found.
2017-12-16 14:27:25 +03:00
return memset(p, 0, n);
fix calloc when __simple_malloc implementation is used previously, calloc's implementation encoded assumptions about the implementation of malloc, accessing a size_t word just prior to the allocated memory to determine if it was obtained by mmap to optimize out the zero-filling. when __simple_malloc is used (static linking a program with no realloc/free), it doesn't matter if the result of this check is wrong, since all allocations are zero-initialized anyway. but the access could be invalid if it crosses a page boundary or if the pointer is not sufficiently aligned, which can happen for very small allocations. this patch fixes the issue by moving the zero-fill logic into malloc.c with the full malloc, as a new function named __malloc0, which is provided by a weak alias to __simple_malloc (which always gives zero-filled memory) when the full malloc is not in use.
2015-06-22 18:50:09 +00:00
}
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
void *realloc(void *p, size_t n)
{
struct chunk *self, *next;
size_t n0, n1;
void *new;
if (!p) return malloc(n);
if (adjust_size(&n) < 0) return 0;
self = MEM_TO_CHUNK(p);
n1 = n0 = CHUNK_SIZE(self);
if (IS_MMAPPED(self)) {
eliminate OOB array hacks in malloc
2011-06-26 16:12:43 -04:00
size_t extra = self->psize;
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
char *base = (char *)self - extra;
size_t oldlen = n0 + extra;
size_t newlen = n + extra;
very cheap double-free checks in malloc
2011-03-23 13:24:00 -04:00
/* Crash on realloc of freed chunk */
use new a_crash() asm to optimize double-free handler. gcc generates extremely bad code (7 byte immediate mov) for the old null pointer write approach. it should be generating something like "xor %eax,%eax ; mov %al,(%eax)". in any case, using a dedicated crashing opcode accomplishes the same thing in one byte.
2011-08-23 09:43:45 -04:00
if (extra & 1) a_crash();
malloc: fix an over-allocation bug Fix an instance where realloc code would overallocate by OVERHEAD bytes amount. Manually arrange for reuse of memcpy-free-return exit sequence.
2018-04-16 20:54:35 +03:00
if (newlen < PAGE_SIZE && (new = malloc(n-OVERHEAD))) {
n0 = n;
goto copy_free_ret;
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
}
newlen = (newlen + PAGE_SIZE-1) & -PAGE_SIZE;
if (oldlen == newlen) return p;
base = __mremap(base, oldlen, newlen, MREMAP_MAYMOVE);
if (base == (void *)-1)
handle mremap failure in realloc of mmap-serviced allocations mremap seems to always fail on nommu, and on some non-Linux implementations of the Linux syscall API, it at least fails to increase allocation size, and may fail to move (i.e. defragment) the existing mapping when shrinking it too. instead of failing realloc or leaving an over-sized allocation that may waste a large amount of memory, fallback to malloc-memcpy-free if mremap fails.
2017-06-15 12:54:40 -04:00
goto copy_realloc;
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
self = (void *)(base + extra);
eliminate OOB array hacks in malloc
2011-06-26 16:12:43 -04:00
self->csize = newlen - extra;
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
return CHUNK_TO_MEM(self);
}
next = NEXT_CHUNK(self);
harden realloc/free to detect simple overflows the sizes in the header and footer for a chunk should always match. if they don't, the program has definitely invoked undefined behavior, and the most likely cause is a simple overflow, either of a buffer in the block being freed or the one just below it. crashing here should not only improve security of buggy programs, but also aid in debugging, since the crash happens in a context where you have a pointer to the likely-overflowed buffer.
2013-07-19 20:00:11 -04:00
/* Crash on corrupted footer (likely from buffer overflow) */
if (next->psize != self->csize) a_crash();
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
/* Merge adjacent chunks if we need more space. This is not
* a waste of time even if we fail to get enough space, because our
* subsequent call to free would otherwise have to do the merge. */
if (n > n1 && alloc_fwd(next)) {
n1 += CHUNK_SIZE(next);
next = NEXT_CHUNK(next);
}
/* FIXME: find what's wrong here and reenable it..? */
if (0 && n > n1 && alloc_rev(self)) {
self = PREV_CHUNK(self);
n1 += CHUNK_SIZE(self);
}
eliminate OOB array hacks in malloc
2011-06-26 16:12:43 -04:00
self->csize = n1 | C_INUSE;
next->psize = n1 | C_INUSE;
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
/* If we got enough space, split off the excess and return */
if (n <= n1) {
//memmove(CHUNK_TO_MEM(self), p, n0-OVERHEAD);
trim(self, n);
return CHUNK_TO_MEM(self);
}
handle mremap failure in realloc of mmap-serviced allocations mremap seems to always fail on nommu, and on some non-Linux implementations of the Linux syscall API, it at least fails to increase allocation size, and may fail to move (i.e. defragment) the existing mapping when shrinking it too. instead of failing realloc or leaving an over-sized allocation that may waste a large amount of memory, fallback to malloc-memcpy-free if mremap fails.
2017-06-15 12:54:40 -04:00
copy_realloc:
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
/* As a last resort, allocate a new chunk and copy to it. */
new = malloc(n-OVERHEAD);
if (!new) return 0;
malloc: fix an over-allocation bug Fix an instance where realloc code would overallocate by OVERHEAD bytes amount. Manually arrange for reuse of memcpy-free-return exit sequence.
2018-04-16 20:54:35 +03:00
copy_free_ret:
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
memcpy(new, p, n0-OVERHEAD);
free(CHUNK_TO_MEM(self));
return new;
}
return chunks split off by memalign using __bin_chunk instead of free this change serves multiple purposes: 1. it ensures that static linking of memalign-family functions will pull in the system malloc implementation, thereby causing link errors if an attempt is made to link the system memalign functions with a replacement malloc (incomplete allocator replacement). 2. it eliminates calls to free that are unpaired with allocations, which are confusing when setting breakpoints or tracing execution. as a bonus, making __bin_chunk external may discourage aggressive and unnecessary inlining of it.
2018-04-19 20:56:26 -04:00
void __bin_chunk(struct chunk *self)
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
{
ldso, malloc: implement reclaim_gaps via __malloc_donate Split 'free' into unmap_chunk and bin_chunk, use the latter to introduce __malloc_donate and use it in reclaim_gaps instead of calling 'free'.
2018-04-16 20:54:36 +03:00
struct chunk *next = NEXT_CHUNK(self);
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
size_t final_size, new_size, size;
int reclaim=0;
int i;
final_size = new_size = CHUNK_SIZE(self);
harden realloc/free to detect simple overflows the sizes in the header and footer for a chunk should always match. if they don't, the program has definitely invoked undefined behavior, and the most likely cause is a simple overflow, either of a buffer in the block being freed or the one just below it. crashing here should not only improve security of buggy programs, but also aid in debugging, since the crash happens in a context where you have a pointer to the likely-overflowed buffer.
2013-07-19 20:00:11 -04:00
/* Crash on corrupted footer (likely from buffer overflow) */
if (next->psize != self->csize) a_crash();
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
for (;;) {
eliminate OOB array hacks in malloc
2011-06-26 16:12:43 -04:00
if (self->psize & next->csize & C_INUSE) {
self->csize = final_size | C_INUSE;
next->psize = final_size | C_INUSE;
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
i = bin_index(final_size);
lock_bin(i);
lock(mal.free_lock);
eliminate OOB array hacks in malloc
2011-06-26 16:12:43 -04:00
if (self->psize & next->csize & C_INUSE)
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
break;
unlock(mal.free_lock);
unlock_bin(i);
}
if (alloc_rev(self)) {
self = PREV_CHUNK(self);
size = CHUNK_SIZE(self);
final_size += size;
if (new_size+size > RECLAIM && (new_size+size^size) > size)
reclaim = 1;
}
if (alloc_fwd(next)) {
size = CHUNK_SIZE(next);
final_size += size;
if (new_size+size > RECLAIM && (new_size+size^size) > size)
reclaim = 1;
next = NEXT_CHUNK(next);
}
}
mitigate blow-up of heap size under malloc/free contention during calls to free, any free chunks adjacent to the chunk being freed are momentarily held in allocated state for the purpose of merging, possibly leaving little or no available free memory for other threads to allocate. under this condition, other threads will attempt to expand the heap rather than waiting to use memory that will soon be available. the race window where this happens is normally very small, but became huge when free chooses to use madvise to release unused physical memory, causing unbounded heap size growth. this patch drastically shrinks the race window for unwanted heap expansion by performing madvise with the bin lock held and marking the bin non-empty in the binmask before making the expensive madvise syscall. testing by Timo Teräs has shown this approach to be a suitable mitigation. more invasive changes to the synchronization between malloc and free would be needed to completely eliminate the problem. it's not clear whether such changes would improve or worsen typical-case performance, or whether this would be a worthwhile direction to take malloc development.
2015-08-07 19:19:49 +00:00
if (!(mal.binmap & 1ULL<<i))
a_or_64(&mal.binmap, 1ULL<<i);
eliminate OOB array hacks in malloc
2011-06-26 16:12:43 -04:00
self->csize = final_size;
next->psize = final_size;
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
unlock(mal.free_lock);
self->next = BIN_TO_CHUNK(i);
self->prev = mal.bins[i].tail;
self->next->prev = self;
self->prev->next = self;
mitigate blow-up of heap size under malloc/free contention during calls to free, any free chunks adjacent to the chunk being freed are momentarily held in allocated state for the purpose of merging, possibly leaving little or no available free memory for other threads to allocate. under this condition, other threads will attempt to expand the heap rather than waiting to use memory that will soon be available. the race window where this happens is normally very small, but became huge when free chooses to use madvise to release unused physical memory, causing unbounded heap size growth. this patch drastically shrinks the race window for unwanted heap expansion by performing madvise with the bin lock held and marking the bin non-empty in the binmask before making the expensive madvise syscall. testing by Timo Teräs has shown this approach to be a suitable mitigation. more invasive changes to the synchronization between malloc and free would be needed to completely eliminate the problem. it's not clear whether such changes would improve or worsen typical-case performance, or whether this would be a worthwhile direction to take malloc development.
2015-08-07 19:19:49 +00:00
/* Replace middle of large chunks with fresh zero pages */
if (reclaim) {
uintptr_t a = (uintptr_t)self + SIZE_ALIGN+PAGE_SIZE-1 & -PAGE_SIZE;
uintptr_t b = (uintptr_t)next - SIZE_ALIGN & -PAGE_SIZE;
#if 1
__madvise((void *)a, b-a, MADV_DONTNEED);
#else
__mmap((void *)a, b-a, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED, -1, 0);
#endif
}
initial check-in, version 0.5.0
2011-02-12 00:22:29 -05:00
unlock_bin(i);
}
ldso, malloc: implement reclaim_gaps via __malloc_donate Split 'free' into unmap_chunk and bin_chunk, use the latter to introduce __malloc_donate and use it in reclaim_gaps instead of calling 'free'.
2018-04-16 20:54:36 +03:00
static void unmap_chunk(struct chunk *self)
{
size_t extra = self->psize;
char *base = (char *)self - extra;
size_t len = CHUNK_SIZE(self) + extra;
/* Crash on double free */
if (extra & 1) a_crash();
__munmap(base, len);
}
void free(void *p)
{
if (!p) return;
struct chunk *self = MEM_TO_CHUNK(p);
if (IS_MMAPPED(self))
unmap_chunk(self);
else
return chunks split off by memalign using __bin_chunk instead of free this change serves multiple purposes: 1. it ensures that static linking of memalign-family functions will pull in the system malloc implementation, thereby causing link errors if an attempt is made to link the system memalign functions with a replacement malloc (incomplete allocator replacement). 2. it eliminates calls to free that are unpaired with allocations, which are confusing when setting breakpoints or tracing execution. as a bonus, making __bin_chunk external may discourage aggressive and unnecessary inlining of it.
2018-04-19 20:56:26 -04:00
__bin_chunk(self);
ldso, malloc: implement reclaim_gaps via __malloc_donate Split 'free' into unmap_chunk and bin_chunk, use the latter to introduce __malloc_donate and use it in reclaim_gaps instead of calling 'free'.
2018-04-16 20:54:36 +03:00
}
void __malloc_donate(char *start, char *end)
{
size_t align_start_up = (SIZE_ALIGN-1) & (-(uintptr_t)start - OVERHEAD);
size_t align_end_down = (SIZE_ALIGN-1) & (uintptr_t)end;
comment __malloc_donate overflow logic
2018-04-17 15:18:49 -04:00
/* Getting past this condition ensures that the padding for alignment
* and header overhead will not overflow and will leave a nonzero
* multiple of SIZE_ALIGN bytes between start and end. */
ldso, malloc: implement reclaim_gaps via __malloc_donate Split 'free' into unmap_chunk and bin_chunk, use the latter to introduce __malloc_donate and use it in reclaim_gaps instead of calling 'free'.
2018-04-16 20:54:36 +03:00
if (end - start <= OVERHEAD + align_start_up + align_end_down)
return;
start += align_start_up + OVERHEAD;
end -= align_end_down;
struct chunk *c = MEM_TO_CHUNK(start), *n = MEM_TO_CHUNK(end);
c->psize = n->csize = C_INUSE;
c->csize = n->psize = C_INUSE | (end-start);
return chunks split off by memalign using __bin_chunk instead of free this change serves multiple purposes: 1. it ensures that static linking of memalign-family functions will pull in the system malloc implementation, thereby causing link errors if an attempt is made to link the system memalign functions with a replacement malloc (incomplete allocator replacement). 2. it eliminates calls to free that are unpaired with allocations, which are confusing when setting breakpoints or tracing execution. as a bonus, making __bin_chunk external may discourage aggressive and unnecessary inlining of it.
2018-04-19 20:56:26 -04:00
__bin_chunk(c);
ldso, malloc: implement reclaim_gaps via __malloc_donate Split 'free' into unmap_chunk and bin_chunk, use the latter to introduce __malloc_donate and use it in reclaim_gaps instead of calling 'free'.
2018-04-16 20:54:36 +03:00
}
Reference in New Issue Copy Permalink