System Calls and Interrupts

A user program has exactly two legitimate ways into the kernel, traps and interrupts, and both of them go through a mode switch.

Dual mode and the protection boundary

An operating system does not let arbitrary code touch the hardware directly. Instructions that write to disk or change memory mappings must be executable only by the kernel. To enforce that, the central processing unit (CPU) provides dual mode.

Dual mode separates privilege with a single mode bit. A mode bit of 1 means user mode, 0 means kernel mode. Privileged instructions are blocked in user mode, and attempting one raises an exception.

One key constraint follows. The legitimate paths from user mode to kernel mode are fixed. A user program cannot set the mode bit to 0 on its own.

As the diagram shows, a user program enters the kernel only by raising a trap. The trap flips the mode bit to 0, and the kernel flips it back to 1 when the work is done. The entry point is also fixed to a handler the kernel registered in advance.

The three ways into kernel mode

Only three kinds of event cause the switch.

• System call: a user program requesting an operating system (OS) service. Entry happens through a trap.
• Interrupt: a hardware device signaling something, such as a timer expiring or input/output (I/O) completing.
• Exception: an error during execution, such as division by zero or a page fault.

All three end the same way. The CPU saves the current state and jumps to a designated handler in the kernel. The difference is whether the event is synchronous or asynchronous.

How a system call works

A system call is the programming interface through which a user program requests kernel services. Functions such as write and read sit on top of system calls.

Calling write on Linux x86-64 passes the number and arguments in agreed-upon registers. Normally this goes through a wrapper function in the C standard library.

#include <unistd.h>
 
int main(void) {
    const char msg[] = "hello\n";
    write(1, msg, 6);   // the glibc wrapper turns this into a syscall instruction
    return 0;
}

The wrapper function in the GNU C Library (glibc) takes this call and fills the registers. On x86-64, RAX holds the system call number and RDI/RSI/RDX hold the arguments. The same work written directly in assembly, without the wrapper, looks like this.

    mov rax, 1        ; system call number for write (Linux x86-64)
    mov rdi, 1        ; first argument: file descriptor stdout
    mov rsi, msg      ; second argument: buffer address
    mov rdx, 6        ; third argument: byte count
    syscall           ; trap, enter the kernel

Executing the syscall instruction after filling the registers raises a trap. The kernel uses the number in RAX to look up the system call dispatch table (sys_call_table) and branch to the matching function. For write, that is sys_write.

When the kernel function finishes, it puts the return value in RAX and returns to user mode. On success RAX holds a value of zero or more; on failure it holds a negative value. The glibc wrapper converts that negative value into -1 plus errno.

System calls versus ordinary function calls

An ordinary function call only changes the stack frame inside user mode. Same address space, same privilege, so the cost is small.

A system call is different. A trap flips the mode bit, the CPU state (registers, program counter) is saved on the kernel stack, and control enters the handler. All of that is undone on return. For the same amount of work, the overhead is therefore larger than a function call.

That cost is why ultra-low-latency systems try to reduce the number of system calls. Kernel bypass techniques process network packets without going through the kernel at all, which cuts system call overhead, user-kernel data copies, and interrupt cost together.

Classifying interrupts

An interrupt is a signal that makes the CPU stop what it is doing and handle something urgent first. It splits in two by origin.

Hardware interrupts come from external devices. Their timing is unpredictable, which makes them asynchronous. They divide further into maskable and non-maskable, since signals that must never be ignored, such as a power failure, cannot be masked.

Software interrupts are raised by program instructions. They occur exactly when the instruction executes, which makes them synchronous. Traps and exceptions belong here.

As the table shows, a system call falls under software interrupts (traps). In other words, a system call is a synchronous event that borrows the interrupt handling machinery wholesale.

Traps and interrupts are easy to confuse. A trap is a synchronous event the program raises deliberately, while a hardware interrupt is an asynchronous event that can arrive at any time. An exception is also synchronous, but unlike a trap it is an unintended error.

The interrupt handling sequence

When an interrupt arrives, the CPU handles it in a fixed order. The point is to store the current work safely and come back to exactly the same place.

The state-saving step puts the program counter (PC), the Program Status Word (PSW), and the registers on the stack. That bundle is called the context. Even if the interrupt handler overwrites registers, the restore step brings back the original values.

The vector table lookup decides where to jump. Every interrupt has a number, and that number indexes a table holding the address of the Interrupt Service Routine (ISR). The ISR is the kernel function that handles that particular interrupt.

The format of this table differs by CPU operating mode. On x86 the comparison looks like this.

The Interrupt Vector Table (IVT) is real mode's simple address table. Protected mode and x86-64 use the Interrupt Descriptor Table (IDT), whose gate descriptors carry privilege information alongside the address. That privilege information prevents user code from invoking an ISR at will.

Priority and nested interrupts

Several interrupts can arrive at once, which is why priorities exist. Non-maskable interrupts such as power failures and machine checks sit at the top. Below them come the timer, disk, network, and keyboard, with software interrupts (traps) at the bottom.

When another interrupt arrives during handling, there are two policies. One is to block new interrupts until the current ISR finishes. That is simple to implement, but urgent signals can be delayed.

The other is nested interrupts. If the new interrupt has higher priority, the current ISR is paused, the new ISR runs first, and then control returns. Most modern operating systems support this priority-based nesting.

Summary

System calls and interrupts share the same mechanism for crossing the boundary between user mode and kernel mode. A system call is a synchronous trap the program raises deliberately, while a hardware interrupt is an asynchronous signal arriving from outside. Both paths save the CPU state, jump to a designated handler, and restore the state on the way back. Understanding this structure explains why a system call costs more than a function call and how the kernel maintains its protection boundary.