HugePages Configurator

///HugePages Configurator

Brief Introduction To HugePages

Memory is managed in blocks known as pages. A standard page is 4096 bytes. 1MB of memory is equal to 256 pages; 1GB of memory is equal to 256,000 pages, etc. Those pages can be swapped to disk, etc. CPUs have a built-in memory management unit that contains a list of these pages, with each page referenced through a page table entry. Since the process address space are virtual, the CPU and the operating system have to remember which page belong to which process, and where it is stored. Obviously, the more pages you have, the more time it takes to find where the memory is mapped.

Most current CPU architectures support bigger pages (so the CPU/OS have less entries to look-up), those are named HugePages (on Linux), Super Pages (on BSD) or Large Pages (on Windows), but it all the same thing.

HugePages are a feature integrated into the Linux kernel 2.6. Enabling HugePages makes it possible for the operating system to support memory pages greater than the default (usually 4KB). Using very large page sizes can improve system performance by reducing the amount of system resources required to access page table entries. HugePages are useful for both 32-bit and 64-bit configurations. HugePage sizes vary from 2MB to 256MB, depending on the kernel version and the hardware architecture.

You should take into account that not every application can work with HugePages, and those which can, should be properly configured. In most cases using HugePages does make sense in relation to MySQL, PostgreSQL, Oracle, and JAVA.

Linux Kernel Parameters

root@svelab:~# grep ^Huge /proc/meminfo
HugePages_Total:    4096
HugePages_Free:     3079
HugePages_Rsvd:     2822
HugePages_Surp:        0
Hugepagesize:       2048 kB

HugePages_Total

Size of the pool of huge pages.

HugePages_Free

Number of huge pages in the pool that are not yet allocated.

HugePages_Rsvd

Number of huge pages for which a commitment to allocate from the pool has been made, but no allocation has yet been done. Reserved huge pages guarantee that an application will be able to allocate a huge page from the pool of huge pages at fault time.

HugePages_Surp

Number of huge pages in the pool above the value in /proc/sys/vm/nr_hugepages. The maximum number of surplus huge pages is controlled by /proc/sys/vm/nr_overcommit_hugepages.

Hugepagesize

Size of a single memory page.

HugePages Configurator

HugePages Configurator (hugecfg.sh) is the bash script which allows you to easily configure HugePages and show real-time values as well as settings in a config file. You do not need to memorize and calculate values for the Linux kernel parameters, this script will do all the needful for you.

Name: HugePages Configurator
Version: 0.2 (check for the latest version)
Language: bash
Platform: Linux Logo
Requirements: GNU awk, grep
Download

Install & Configure

Download

sudo bash wget https://svelab.com/download/hugecfg.zip -O /tmp/hugecfg.zip unzip /tmp/hugecfg.zip -d /usr/sbin/ chmod +x /usr/sbin/hugecfg.sh rm -f /tmp/hugecfg.zip exit

Configurable user variables

hpCFG

/etc/sysctl.d/hugepages.conf

Path to config. All kernel parameters will be saved to this file

hpGroup

hugepages_grp

Authorized group to use HugePages. Service accounts (for example, mysql, tomcat, etc.) must be members of this group to be allowed for using HugePages

hpGID

15000

Group ID of ‘hpGroup’. Change it to any value suitable for your OS or keep it as is

hpExtra

256

This amount of memory (in megabytes) will be added to the final amount of memory you want to allocate for HugePages. This variable is to simplify calculation of needs in the volume of HugePages

Usage

root@svelab:~# hugecfg.sh help
Command Description
hugecfg.sh help Show all command line arguments (see below)
hugecfg.sh Show real-time values (RAM, HugePages, Swap)
hugecfg.sh -s value Configure HugePages (where ‘value’ in megabytes):
hugecfg.sh -g Show HugePages config
cat /root/hugepages_reminder Show a reminder (will be created automatically after configuring HugePages)

Screenshots

Show all command line arguments
<pre class=””>root@svelab:~# hugecfg.sh help</pre>
<img class=”alignnone size-full wp-image-365″ src=”https://svelab.com/wp-content/uploads/2017/09/hugecfg_help.png” alt=”hugecfg.sh help” width=”1280″ height=”179″ />
Show real-time values
<pre class=””>root@svelab:~# hugecfg.sh</pre>
<img class=”alignnone size-full wp-image-366″ src=”https://svelab.com/wp-content/uploads/2017/09/hugecfg_none.png” alt=”hugecfg.sh” width=”1280″ height=”376″ />
Configure HugePages
<pre class=””>root@svelab:~# hugecfg.sh -s 16640</pre>
<img class=”alignnone size-full wp-image-367″ src=”https://svelab.com/wp-content/uploads/2017/09/hugecfg_set.png” alt=”hugecfg.set -s 16640″ width=”1280″ height=”221″ />
Show HugePages config
<pre class=””>root@svelab:~# hugecfg.sh -g</pre>
<img class=”alignnone size-full wp-image-364″ src=”https://svelab.com/wp-content/uploads/2017/09/hugecfg_get.png” alt=”hugecfg.sh -g” width=”1280″ height=”245″ />

Reminder

The script creates the file /root/hugepages_reminder as a reminder with the following text:

  1. The memory associated with HugePages cannot be swapped to disk.
  2. The memory associated with HugePages are marked as used.
  3. If an application cannot take the needed amount of HugePages it will use standard pages.
  4. MySQL: add parameter ‘large-pages’ under [mysql] section in /etc/mysql/my.cnf
  5. MySQL: HugePages must be greater than ‘innodb_buffer_pool_size’ + all buffers
  6. JAVA: add parameter CATALINA_OPTS=”$CATALINA_OPTS -XX:+UseLargePages”
  7. JAVA: HugePages must be greater than ‘-Xmx’ + heaps

Source Code

See Also

<a href=”https://svelab.com/2017/09/11/system-monitor/”>System Monitor</a>

Feedback

Contact us

Please, feel free to contact us if you have any questions or suggestions. Post a comment below if you want to report a bug.

By |2019-01-20T19:18:18+00:00August 27th, 2017|Linux, Scripts|0 Comments

Leave A Comment