2020ok  Directory of FREE Online Books and FREE eBooks

Free eBooks > Computers & Internet > Operating Systems > Unix > General > Understanding the Linux Virtual Memory Manager

Understanding the Linux Virtual Memory Manager

by Mel Gorman

Download Book
(Respecting the intellectual property of others is utmost important to us, we make every effort to make sure we only link to legitimate sites, such as those sites owned by authors and publishers. If you have any questions about these links, please contact us.)

link 1

About Book

Book Description

A programmer wanting to understand the workings of the Linux VM today literally has no choice but to study the kernel source code, line-by-line - an excruciatingly difficult and time-consuming task. This book dedicates itself to explaining, in detail, how the memory manager is implemented in Linux, thereby cutting down the time needed to understand it from many months to mere weeks. The Linux VM is the single most important component of the Linux kernel. The behavior of the VM affects all other kernel subsystems, and has a dramatic impact on overall system performance. This book is unique in that not only does it describe the Linux VM itself in unprecedented detail, it also includes the theoretical foundations for it which is of interest to both developers and students but has been omitted from recent Linux kernel related material. It is split into three parts. The first part begins with an introduction on how to approach reading the code of an open source project. It then provides a detailed description of the VM architecture with the aid of numerous diagrams and call graphs, which is suitable for people who need a clear understanding of how the VM functions. The second part is a detailed line-by-line description of the C source modules (source code commentary) that implement the VM in the Linux 2.4 kernel. The third part describes new features in the upcoming 2.6 kernel.

From the Back Cover

Finally, a comprehensive guide to the Linux VM!

VM's behavior affects every Linux kernel subsystem and dramatically impacts overall performance. But until now, there was only one way to understand VM: study the poorly documented source one line at a time. Now there's an easier, faster alternative. This book describes VM in unprecedented detail, presenting both theoretical foundations and a line-by-line source code commentary. It systematically covers everything from physical memory description to out-of-memory management. Coverage includes:

  • Linux VM 2.4 architecture in depth-with diagrams and call graphs
  • Physical memory description, page tables, address spaces, and memory allocation
  • High memory, swapping, shared memory, and much more
  • Expert guidance for analyzing the code of any open source project
  • New Linux 2.6 kernel features in every chapter

Well organized and superbly written, Understanding the Linux Virtual Memory Manager will be indispensable to every kernel programmer and researcher.


Complete VM Learning Lab! Contains the author's new toolkit for exploring VM, including a browsable version of kernel source, CodeViz call graph generator; and VMRegress for analyzing and benchmarking VM. Also includes all code commentary in HTML, PDF, and plain text formats.

About the Author

MEL GORMAN specializes in documenting open source software. He holds a MSc in Computer Science from the University of Limerick, Ireland, and has served as an instructor there. He has worked as a system administrator, applications developer, and consultant, and has been researching Linux memory management for more than two years. Currently he is an applications developer with J2EE technologies at IBM, Dublin.

Excerpt. © Reprinted by permission. All rights reserved.


Linux is developed with a stronger practical emphasis than a theoretical one. When new algorithms or changes to existing implementations are suggested, it is common to request code to match the argument. Many of the algorithms used in the Virtual Memory (VM) system were designed by theorists, but the implementations have now diverged considerably from the theory. In part, Linux does follow the traditional development cycle of design to implementation, but changes made in reaction to how the system behaved in the "real world" and intuitive decisions by developers are more common.

This means that the VM performs well in practice. However, very little VM documentation is available except for a few incomplete overviews on a small number of Web sites, except the Web site containing an earlier draft of this book, of course! This lack of documentation has led to the situation where the VM is fully understood only by a small number of core developers. New developers looking for information on how VM functions are generally told to read the source. Little or no information is available on the theoretical basis for the implementation. This requires that even a casual observer invest a large amount of time reading the code and studying the field of Memory Management.

This book gives a detailed tour of the Linux VM as implemented in 2.4.22 and gives a solid introduction of what to expect in 2.6. As well as discussing the implementation, the theory that Linux VM is based on will also be introduced. This is not intended to be a memory management theory book, but understanding why the VM is implemented in a particular fashion is often much simpler if the underlying basis is known in advance.

To complement the description, the appendices include a detailed code commentary on a significant percentage of the VM. This should drastically reduce the amount of time a developer or researcher needs to invest in understanding what is happening inside the Linux VM because VM implementations tend to follow similar code patterns even between major versions. This means that, with a solid understanding of the 2.4 VM, the later 2.5 development VMs and the 2.6 final release will be decipherable in a number of weeks.

The Intended Audience

Anyone interested in how the VM, a core kernel subsystem, works will find answers to many of their questions in this book. The VM, more than any other subsystem, affects the overall performance of the operating system. The VM is also one of the most poorly understood and badly documented subsystems in Linux, partially because there is, quite literally, so much of it. It is very difficult to isolate and understand individual parts of the code without first having a strong conceptual model of the whole VM, so this book intends to give a detailed description of what to expect before going to the source. This material should be of prime interest to new developers who want to adapt the VM to their needs and to readers who simply would like to know how the VM works. It also will benefit other subsystem developers who want to get the most from the VM when they interact with it and operating systems researchers looking for details on how memory management is implemented in a modern operating system. For others, who just want to learn more about a subsystem that is the focus of so much discussion, they will find an easy-to-read description of the VM functionality that covers all the details without the need to plow through source code.

However, it is assumed that the reader has read at least one general operating system book or one general Linux kernel-orientated book and has a general knowledge of C before tackling this book. Although every effort is made to make the material approachable, some prior knowledge of general operating systems is assumed.

Book Overview

In Chapter 1, we go into detail on how the source code may be managed and deciphered. Three tools are introduced that are used for analysis, easy browsing and management of code. The main tools are the Linux Cross Referencing (LXR) tool, which allows source code to be browsed as a Web page, and CodeViz, which was developed while researching this book, for generating call graphs. The last tool, PatchSet, is for managing kernels and the application of patches. Applying patches manually can be time consuming, and using version control software, such as Concurrent Versions Systems (CVS) (http://www.cvshome.org/) or BitKeeper (http://www.bitmover.com), is not always an option. With PatchSet, a simple specification file determines what source to use, what patches to apply and what kernel configuration to use.

In the subsequent chapters, each part of the Linux VM implementation is discussed in detail, such as how memory is described in an architecture-independent manner, how processes manage their memory, how the specific allocators work and so on. Each chapter will refer to other sources that describe the behavior of Linux, as well as covering in depth the implementation, the functions used and their call graphs so that the reader will have a clear view of how the code is structured. The end of each chapter has a "What's New" section, which introduces what to expect in the 2.6 VM.

The appendices are a code commentary of a significant percentage of the VM. They give a line-by-line description of some of the more complex aspects of the VM. The style of the VM tends to be reasonably consistent, even between major releases of the kernel, so an in-depth understanding of the 2.4 VM will be an invaluable aid to understanding the 2.6 kernel when it is released.

Companion CD

A companion CD is included with this book, and it is highly recommended the reader become familiar with it, especially as you progress more through the book and are using the code commentary. It is recommended that the CD is used with a GNU/Linux system, but it is not required.

The text of the book is contained on the CD in HTML, PDF and plain text formats so the reader can perform basic text searches if the index does not have the desired information. If you are reading the .rst edition of the book, you may notice small differences between the CD version and the paper version due to printing deadlines, but the differences are minor.

Almost all the tools used to research the book's material are contained on the CD. Each of the tools may be installed on virtually any GNU/Linux installation, references are included to available documentation and the project home sites, so you can check for further updates.

With many GNU/Linux installations, there is the additional bonus of being able to run a Web server directly from the CD. The server has been tested with Red Hat 7.3 and Debian Woody but should work with any distribution. The small Web site it provides at http://localhost:10080 offers a number of useful features:

  • A searchable index for functions that have a code commentary available. If a function is searched for that does not have a commentary, the browser will be automatically redirected to LXR.
  • A Web browsable copy of the Linux 2.4.22 source. This allows code to be browsed and identifiers to be searched for.
  • A live version of CodeViz, the tool used to generate call graphs for the book, is available. If you feel that the book's graphs are lacking some detail you want, generate them yourself.
  • The VMRegress, CodeViz and PatchSet packages, which are discussed in Chapter 1, are available in /cdrom/software. gcc-3.0.4 is also provided because it is required for building CodeViz.



PLEASE READ: All comments must be approved before appearing in the thread; time and space constraints prevent all comments from appearing. We will only approve comments that are directly related to the article, use appropriate language and are not attacking the comments of others.

Message (please, no HTML tags. Web addresses will be hyperlinked):

Related Free eBooks

Related Tags

DIGG This story   Save To Google   Save To Windows Live   Save To Del.icio.us   diigo it   Save To blinklist
Save To Furl   Save To Yahoo! My Web 2.0   Save To Blogmarks   Save To Shadows   Save To stumbleupon   Save To Reddit