The Linux Programming Interface

1. The Kernel: The Core of System Programming

The core operating system: The kernel.

Kernel's Role. The kernel is the heart of the operating system, managing the computer's resources like the CPU, memory, and devices. It acts as an intermediary between applications and the hardware, providing a consistent interface for programs to perform tasks. The kernel is responsible for process scheduling, memory management, file system access, and device interaction.

User vs. Kernel Mode. The kernel operates in a privileged mode, allowing it to access all memory and hardware resources. User programs run in a restricted mode, preventing them from directly accessing kernel memory or performing privileged operations. This separation ensures system stability and security.

System Calls. User programs interact with the kernel through system calls, which are controlled entry points into the kernel. These calls allow programs to request services from the kernel, such as file I/O, process creation, and network communication. The Linux system call API is the primary focus of this book.

2. File I/O: The Universal Interface

Universality of I/O.

Universal I/O Model. UNIX systems, including Linux, employ a universal I/O model, where the same system calls (open(), read(), write(), close(), etc.) are used to perform I/O on all types of files, including regular files, devices, pipes, and sockets. This abstraction simplifies programming and promotes code reusability.

File Descriptors. Open files are referred to using file descriptors, which are small integers. A process inherits three standard file descriptors: 0 (standard input), 1 (standard output), and 2 (standard error). These descriptors can be redirected to other files or devices.

Buffering. Both the kernel and the stdio library perform buffering of file I/O to improve performance. The kernel uses a buffer cache to store data in memory, reducing the number of disk accesses. The stdio library also uses buffers to reduce the number of system calls. Understanding buffering is crucial for writing efficient and reliable programs.

3. Processes: The Foundation of Execution

Processes and Programs.

Processes vs. Programs. A program is a file containing instructions and data, while a process is an instance of an executing program. The kernel manages processes, allocating resources such as CPU time, memory, and file descriptors. Each process has a unique process ID (PID) and a parent process ID (PPID).

Memory Layout. A process's virtual memory is divided into segments: text (program instructions), data (initialized global variables), heap (dynamically allocated memory), and stack (function call information). The kernel manages virtual memory, providing each process with its own isolated address space.

Process Creation and Termination. A new process is created using fork(), which duplicates the parent process. The child process can then use execve() to load and execute a new program. A process terminates using _exit() or exit(), and its parent can obtain its termination status using wait().

4. Memory Management: The Art of Allocation

Allocating Memory on the Heap.

Heap Allocation. Processes can dynamically allocate memory on the heap using functions like malloc() and free(). The heap is a region of memory that grows and shrinks as memory is allocated and deallocated. The kernel manages the heap by adjusting the program break.

Stack Allocation. The stack is a region of memory used to store local variables and function call information. The alloca() function allocates memory on the stack, but this memory is automatically deallocated when the function returns.

Memory Mappings. The mmap() system call creates a memory mapping, which can be used to map a file into memory or to create an anonymous memory region. Memory mappings can be shared between processes, providing a fast method of IPC.

5. Time and Scheduling: Controlling the Flow

Time.

Time Concepts. UNIX systems use calendar time (seconds since the Epoch) and process time (CPU time used by a process). The kernel maintains a software clock that measures time in units called jiffies.

Timers. The setitimer() and alarm() system calls establish interval timers that generate signals when they expire. These timers can be used to set timeouts on blocking operations.

Sleeping. The sleep() and nanosleep() functions suspend execution of a process for a specified interval. The POSIX clock_nanosleep() function provides a more precise method of sleeping, using a specified clock.

Process Scheduling. The kernel scheduler determines which processes get access to the CPU. Processes have a nice value that influences their priority. Realtime scheduling policies (SCHED_RR and SCHED_FIFO) provide more precise control over process scheduling.

6. Signals: Asynchronous Communication

Signals: Fundamental Concepts.

Signal Mechanism. Signals are a form of asynchronous notification to a process that an event has occurred. Signals can be generated by the kernel (e.g., hardware exceptions, terminal input), by another process, or by the process itself.

Signal Handling. A process can choose to ignore a signal, accept its default action (e.g., termination), or establish a signal handler, a function that is invoked when the signal is delivered.

Signal Masking. A process can block delivery of certain signals by adding them to its signal mask. Blocked signals remain pending until they are unblocked. Signals are not queued; if the same signal is generated multiple times while it is blocked, it is delivered only once.

Signal APIs. The signal() function is the older API for establishing signal handlers, but it is not portable. The sigaction() function is the preferred method for establishing signal handlers, since it provides more control and is more portable.

7. Threads: Concurrent Execution

Threads: Introduction.

Threads vs. Processes. Threads are a mechanism for concurrent execution within a single process. Threads share the same virtual memory, file descriptors, and signal dispositions, but each thread has its own stack and thread ID.

Pthreads API. The Pthreads API provides a standard set of functions for creating, terminating, and synchronizing threads. The pthread_create() function creates a new thread, and pthread_exit() terminates a thread. The pthread_join() function waits for a thread to terminate.

Thread Synchronization. Threads use mutexes and condition variables to synchronize access to shared resources. Mutexes provide exclusive access to a shared variable, while condition variables allow threads to wait for changes in the state of a shared variable.

Thread Safety. Thread-safe functions can be safely called from multiple threads at the same time. Non-thread-safe functions should be avoided in multithreaded programs. Thread-specific data and thread-local storage provide mechanisms for making non-thread-safe functions thread-safe without changing their interfaces.

8. Interprocess Communication: Sharing Data and Synchronization

Interprocess Communication Overview.

IPC Mechanisms. UNIX systems provide a variety of mechanisms for interprocess communication (IPC), including pipes, FIFOs, message queues, semaphores, shared memory, and sockets. These mechanisms can be used to exchange data and synchronize the actions of processes.

Data Transfer vs. Shared Memory. Data-transfer facilities (pipes, FIFOs, message queues, sockets) involve copying data between processes, while shared memory allows processes to directly access the same region of memory. Shared memory is faster, but requires explicit synchronization.

System V vs. POSIX IPC. System V IPC mechanisms (message queues, semaphores, shared memory) are older and more widely available, but their APIs are more complex and less consistent with the traditional UNIX I/O model. POSIX IPC mechanisms provide a simpler and more consistent API, but are not available on all UNIX implementations.

9. Sockets: Connecting Across Networks

Sockets: Introduction.

Sockets as IPC. Sockets are a versatile IPC mechanism that can be used for communication between processes on the same host (UNIX domain sockets) or between processes on different hosts connected via a network (Internet domain sockets).

Socket Types. Sockets come in two main types: stream sockets (SOCK_STREAM), which provide a reliable, bidirectional, byte-stream communication channel, and datagram sockets (SOCK_DGRAM), which provide unreliable, connectionless, message-oriented communication.

Socket Operations. The key socket system calls are socket() (create a socket), bind() (bind a socket to an address), listen() (mark a stream socket as passive), accept() (accept a connection on a listening stream socket), and connect() (connect to a peer socket).

Internet Domain Sockets. Internet domain sockets use IP addresses and port numbers to identify communication endpoints. The getaddrinfo() and getnameinfo() functions are used to convert between hostnames and service names and their corresponding numeric representations.

10. Terminals and Pseudoterminals: Interacting with the User

Terminals.

Terminal Attributes. Terminals have a range of attributes that control how they operate, including special characters, flags, and I/O modes. These attributes can be retrieved and modified using the tcgetattr() and tcsetattr() functions.

Terminal I/O Modes. Terminals can operate in canonical mode (line-at-a-time input) or noncanonical mode (character-at-a-time input). The terminal I/O mode is controlled by the terminal flags.

Pseudoterminals. Pseudoterminals (ptys) are a pair of connected virtual devices, a master and a slave. The slave device provides an interface that behaves like a terminal, while the master device provides a means of controlling the slave. Pseudoterminals are used in a variety of applications, including terminal windows and network login services.

Terminal Control. The ioctl() system call is used to perform a range of control operations on terminals, including setting terminal attributes, controlling the terminal line, and obtaining the terminal window size.

11. Security and Capabilities: Protecting the System

Writing Secure Privileged Programs.

Privileged Programs. Privileged programs have access to system resources that are not available to ordinary users. Such programs should be written with great care to avoid security vulnerabilities.

Least Privilege. Privileged programs should operate with the least privilege required to accomplish their tasks. This means dropping privileges when they are not needed and permanently dropping privileges when they will never again be required.

Input Validation. Privileged programs should carefully validate all inputs from untrusted sources, including command-line arguments, environment variables, and data from files and network connections.

Capabilities. The Linux capabilities scheme divides the traditional all-or-nothing superuser privilege into distinct units called capabilities. This allows a process to be granted only the privileges that it requires, thus reducing the potential for damage if the program is compromised.

12. Shared Libraries: Code Reusability and Efficiency

Fundamentals of Shared Libraries.

Object Libraries. Object libraries are files containing compiled object code for a set of functions. Static libraries are linked into an executable at compile time, while shared libraries are loaded at run time.

Shared Libraries. Shared libraries allow multiple processes to share the same copy of library code in memory, thus saving disk space and RAM. Shared libraries also allow library updates to be applied without requiring programs to be relinked.

Position-Independent Code. Shared libraries must be compiled using position-independent code (PIC), which allows the library code to be loaded at any address in memory.

Dynamic Linking. The dynamic linker is responsible for loading shared libraries at run time and resolving symbol references. The LD_LIBRARY_PATH environment variable can be used to specify additional directories in which the dynamic linker should search for shared libraries.

Symbol Visibility. Shared libraries should export only those symbols that are part of their public API. Version scripts can be used to control symbol visibility and to create versioned symbols.

Last updated: January 22, 2025