Understanding and Optimizing Linux Swappiness

Understanding and optimizing swappiness is key to maximizing Linux performance and stability. This comprehensive guide covers everything you need to know, from swappiness fundamentals to benchmarking different values in production environments.

What is Swappiness on Linux?

The swappiness value controls the balance between mapped memory pages and executable code in physical RAM versus disk-based swap space. Specifically, this parameter configures how aggressively the Linux kernel will swap out active / recently used memory pages to disk.

Higher swappiness values prioritize transferring memory pages into swap, freeing up more RAM for active processes and file system cache. But excessive swapping can lead to performance problems. Lower values keep more data resident in physical memory, reducing swap I/O, but this also limits available RAM if workloads are memory intensive.

Finding the right balance is critical. Swappiness directly impacts a Linux system’s performance, stability and responsiveness.

How the Linux Kernel Handles Memory

When running processes need more memory than available RAM, Linux uses disk storage as additional virtual memory through a mechanism called swapping:

• Memory pages associated with inactive processes are swapped out to a dedicated swap partition or file
• This frees up physical RAM for other running applications and kernel functions
• When needed again later, swapped-out pages are retrieved back into memory

The component that handles swapping is termed the swap daemon or swappiness daemon. There are actually two daemons, kswapd for asynchronous swapping, and kswapd2 for higher priority swapping needs.

Key Memory Areas Related to Swappiness:

• RAM – used for mapped executable code and data needed actively by running processes
• File System Cache – portion retaining frequently accessed file system data, allowing faster reads next time
• Swap Space – dedicated disk area holding inactive pages that have been swapped out from RAM

Impacts of Swappiness Misconfiguration

Problems arise when swappiness is set incorrectly for a given workload:

• Low performance – excessive swapping creates disk I/O bottleneck due to constant paging in/out
• Thrashing – a surge in memory pages being written to disk and needing immediate access again, resulting in chaotic I/O
• High latency – requests waiting on memory pages retrieved from slow swap area
• Out Of Memory errors – without enough free RAM available due to undersized swap allocation

Tuning swappiness properly avoids these issues by optimizing virtual memory behavior.

When optimizing swappiness, the two core metrics to regularly assess are:

1. Current swappiness value
2. Current swap usage %

Check the active swappiness value with:

cat /proc/sys/vm/swappiness

Typical default is 60. We’ll dig into optimal values later.

Next, verify swap space usage with the free command:

free -m

Watch the swap usage value over time to see if it approaches maximum allocated space during peak memory usage. Consistently high utilization signals that additional swap space may be needed or swappiness should be adjusted.

For systems with extra RAM slots available, adding more physical memory can also help significantly.

How is Swappiness Calculated in Linux?

The kernel determines when to initiate swapping based on a series of calculations using the swappiness value.

The essential formula is:

Swapout Threshold = ((100 - swappiness Value) * free RAM) / 100

For example, using the common default swappiness of 60:

60 = 100 - Swappiness 
40 = (100 - 60) * free RAM / 100

So with 1GB free RAM:

0.4GB = (40 * 1GB) / 100

This means when free RAM drops below 400MB (40% of total free), swapping will start.

Key variables in play:

• Available RAM
• Number of file pages accessed per second
• Page fault rate
• Actual vs estimated next memory usage

Swapping intensifies as free RAM keeps decreasing or page fault rates spike – indicating processes need more memory than physically available. The specific swappiness value fine tunes this behavior.

Tools to monitor overall swap activity:

• vmstat
• sar

These reveal statistics about swap operations, vital for benchmarking.

Now let’s explore configuring swappiness properly.

Tuning the Swappiness Setting for Linux

The primary configuration file for swappiness is:

/proc/sys/vm/swappiness

Set or alter this file’s value to adjust swapping aggression from low (0-10) to high (90-100).

Temporarily Change Swappiness

Use sysctl for runtime changes lost on reboot:

sudo sysctl vm.swappiness=30

Permanent Change

Update /etc/sysctl.conf and reboot:

vm.swappiness=30

Impact of Different Swappiness Values:

• 0-10 = Extremely conservative swapping
• 10-40 = Rarely swap unless critical
• 60 = Default for most Linux distributions
• 80+ = Aggressive swapping
• 100 = Basically all unused pages get swapped

When to Adjust From Default?

• Memory starved systems = Lower value
• Ample RAM available = Higher value
• Mix of workloads = Tune individually instead

Optimizing Swappiness and Cache Settings Together

There is an intricate connection between cache management and swapping behavior in Linux. Optimizing them in tandem is key for maximum performance.

The Linux kernel contains memory regions dedicated to caching frequently accessed files to accelerate reads. This is known as the page cache or file system cache.

When data gets cached or reclaimed from cache depends on:

• File access patterns
• Cache eviction LRU algorithms
• System memory pressures
• The vfs_cache_pressure parameter

This cache pressure setting is essentially the counterpart to swappiness for file system cache management.

Metrics to assess page cache efficiency:

• Cache hit ratio – % of reads satisfied from cache vs needing disk fetch
• Cache miss ratio – 1 – (hit ratio)
• Dentries cached
• Inode cache usage

Tools to analyze page cache activity:

• vmstat
• slabtop
• drop_caches script

Tuning cache pressure alongside swappiness balances out RAM utilization. As one area gets reclaimed, it impacts the other.

RELATIONSHIP WITH ZONE_RECLAIM SETTINGS

The zone_reclaim_mode variable also interacts closely with the page cache and swappiness behaviors.

File system cache enters an area of memory called the pagecache zone. This zone can have relaxed or aggressive pressure thresholds just like swapping does.

Configuring zone_reclaim_mode properly is vital for balanced cache performance:

• 0 = relaxed cache reclaim
• 1 = normal cache reclaim
• 2 = always reclaim cache aggressively

There are additionally multiple mechanisms that Linux can use for cache and swap memory:

• frontswap
• swapcache
• zcache

Each approach has pros and cons depending on workload patterns.

When tuning swappiness, take a holistic view across memory subsystems – don’t just isolate swap management alone.

Monitor overall block I/O throughput and latency as well:

• Sequential read/write throughput to disk
• Random IOPS performance
• Average read/write latency per I/O op

If swapping activity rises, it may drag down disk performance in other areas.

Determining Needed Swap Space for Your Workloads

When right-sizing swap allocation, consider these influencing factors:

• Average memory utilization over time
• Peak memory usage under load spikes
• Memory demand growth rate projections

Common rules of thumb for swap sizing:

• 50-100% of total RAM
• For dynamic cloud workloads – allocate 1-2X total RAM

Regardless of above guidelines…

The best indicator is monitoring actual swap usage over an extended profile period:

• Observe min/max/average swap usage at different times
• Graph peak usage during cyclical daily/weekly/monthly cycles
• Extrapolate needs if adding memory-intensive features
• Add swap space if you start seeing it fully utilized at peaks

With cloud infrastructure, allocated disk space is flexible. Vertically scale swap space independently as workloads change.

For on-prem environments, allocate swap diligently upfront by sizing RAM needs accurately.

Additional swap configuration notes:

• You can have multiple swap files for more flexibility
• Kernel samepage merging deduplicates identical memory pages across swap
• zram modules compress swap contents in RAM for space savings

Benchmarking Swappiness Performance

Rigorously benchmarking how swappiness value changes impact performance is key before rolling out to production.

Follow a standardized methodology to enable apples-to-apples comparison across configurations:

1. Profile workloads

• Capture production workload attributes
• Memory, CPU, I/O usage over time
• Request volume and patterns
• Monitor during daily/weekly peak cycles

2. Simulate loaded state

• Use stress-ng to max memory utilization
• Configure load generator for production request volumes
• Issue reads/writes against test database

3. Adjust swappiness variable

• Initially use default 60 then test range from 0 to 100
• Consider increments of 10 for coarse changes
• Finer 5 value increments for further granularity

4. Run benchmarks

• Compare run time, latency, throughput at each
• Tailor workload performance KPIs based on environment type:
  • Backend Processing: operation execution runtime, IOPS, memory page faults
  • Web Servers: requests per second, latency, concurrent sessions
  • Databases: transactions/sec, query response time

5. Graph results

• Memory usage, cache hits, swapping activity
• Request latency, throughput, error rate over time
• Identify swappiness value where degraded performance observed

6. Overlay operational metrics

• Model production workload against benchmark dataset
• Validate if hardware configuration mirrors production environment

Repeated runs will provide a 360 view of optimal range for your system profiles.

From here, best practices applying swappiness tuning in live systems…

Optimizing Swappiness in Production Systems

Applying smart swappiness configurations in prod enables notable gains. Start with analyzed data – don’t blindly alter defaults without reason.

Capacity Plan

Give services adequate memory headroom by monitoring growth velocity. Hot add RAM before constant swapping becomes necessary.

Workload Isolation

Distinct apps or process types may need tailored swappiness:

• User-facing requests vs background batch
• Read-heavy vs write-heavy ops
• Session workload intensity
• High I/O load vs minimal I/O

Identify Memory Priority

Reserve guaranteed memory for critical processes first:

cgroups_memory_SwapInhibit = 1

Prevent specific daemons from getting swapped out.

Set Warning Triggers

Alert if swap usage reaches 90% for too long. Proactively add RAM before swapping gets excessive.

Horizontal Scaling

When supporting usage spikes – scale app instances out to more hosts. Provide headroom handling peak loads.

Apply these techniques to smooth Linux efficiency – then monitor metrics to validate improvements. Stay vigilant that workloads don’t outgrow initial swappiness optimizations over time.