The general recommendation is to keep within one NUMA zone. Fine. But given current Nehalem CPUs and Intel chipsets, you can build a server both faster and cheaper with two sockets than with one socket, because then you can populate 6 sockets with 1.3GHz RAM versus 3. The non-local RAM access penalty for Nehalem, as measured with numademo, is only about 16-30%, which is about the gain you get by going from 1GHz to 1.3GHz RAM. And you only pay that penalty for non-local memory access, so on the whole, you win.
Let's say I have a Zimbra server with 24GB RAM and two NUMA nodes. Substantially all memory is used by Jetty and MySQL, which as far as I know never share memory. Is there any real cost to putting them in separate NUMA zones? Do any other processes share memory, including disk buffer cache?
I'm inclined to put LDAP and MySQL in zone 1, and everything else in zone 0. Reasonable?
Relevant tools: numactl --hardware, numactl --preferred=0 -N0, numademo, cat /sys/devices/system/node/node*/numastat