[SPARK-17984][YARN][Mesos][Deploy][WIP] Add support for NUMA aware feature

[quanfuw]

What changes were proposed in this pull request?

Add NUMA aware support for Yarn based deployment mode.
This patch optimizes the memory allocation, executors are bound to NUMA nodes in round-robin for a worker node so that memory allocation tries local NUMA node firstly and only when there is no enough memory in local NUMA node it tries remote ones.
Before this patch, Spark is NUMA unaware in which many remote memory allocations happen and the tremendous remote memory accesses impact performance a lot. We observed significant performance improvement during NUMA aware patch evaluation.

To Do:

1. Add support for NUMA node numbers' configuration and make testing.
2. Add NUMA aware support for Mesos based deployment mode and make testing.
3. Add NUMA aware support for Standalone deployment mode and make testing.

How was this patch tested?

We observed significant performance improvement during evaluation with BigBench. We are still making evaluation and more detailed results will be updated continuously.

Setup:
Cluster Topo: 1 Master + 4 Slaves (Spark on Yarn)
CPU: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz(72 Cores)
Memory: 128GB(2 NUMA Nodes)
NIC: 1x10Gb/Sec
Disk: Write -1.5GB/Sec, Read- 5GB/Sec
SW Version: Hadoop-5.7.0 + Spark-2.0.0

NUMA Introduction

As below diagram depicts, in UMA(Uniform Memory Access) model, processors share one bus. The contention on bus becomes very heavy when processer scales up. NUMA(Non-Uniform Memory Access) processer has a better scalability by dividing processors and memory blocks into nodes, nodes are interconnected with added bus.
For NUMA, the memory accessing to a remote node is much slower than accesing to local one, while, for UMA memory accessing to any nodes is uniform.

For more NUMA information, please refer to https://en.wikipedia.org/wiki/Non-uniform_memory_access.

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

This should be closed in favor of #15579 at least