Interrupts and CPU

Interrupts

An interrupt is a signal to the processor emitted by hardware or software indicating an event that needs immediate attention. An interrupt alerts the processor to a high-priority condition requiring the interruption of the current code the processor is executing. The processor responds by suspending its current activities, saving its state, and executing a function called an interrupt handler (or an interrupt service routine, ISR) to deal with the event. This interruption is temporary, and, after the interrupt handler finishes, the processor resumes normal activities.

There are two types of interrupts: hardware interrupts and software interrupts. Hardware interrupts are used by devices to communicate that they require attention from the operating system. For example, pressing a key on the keyboard or moving the mouse triggers hardware interrupts that cause the processor to read the keystroke or mouse position. Unlike the software type, hardware interrupts are asynchronous and can occur in the middle of instruction execution, requiring additional care in programming. The act of initiating a hardware interrupt is referred to as an interrupt request - IRQ ↪ IDT ↪ common_interrupt id on x86.

A software interrupt is caused either by an exceptional condition in the processor itself, or a special instruction in the instruction set which causes an interrupt when it is executed. The former is often called a trap (⚙️ do_trap id) or exception and is used for errors or events occurring during program execution that are exceptional enough that they cannot be handled within the program itself. For example, if the processor's arithmetic logic unit is commanded to divide a number by zero, this impossible demand will cause a divide-by-zero exception (⚙️ X86_TRAP_DE id), perhaps causing the computer to abandon the calculation or display an error message. Software interrupt instructions function similarly to subroutine calls and are used for a variety of purposes, such as to request services from low-level system software such as device drivers. For example, computers often use software interrupt instructions to communicate with the disk controller to request data be read or written to the disk.

Each interrupt has its own interrupt handler. The number of hardware interrupts is limited by the number of interrupt request (IRQ) lines to the processor, but there may be hundreds of different software interrupts.


⚲ API

/proc/interrupts
man 1 irqtop – utility to display kernel interrupt information
irqbalance – distribute hardware interrupts across processors on a multiprocessor system
There are many ways to request ISR, two of them
devm_request_threaded_irq id – preferable function to allocate an interrupt line for a managed device with a threaded ISR
request_irq id, free_irq id – old and common functions to add and remove a handler for an interrupt line
linux/interrupt.h inc – main interrupt support header
irqaction id – contains handler functions
linux/irq.h inc
irq_data id
linux/irqflags.h inc
irqs_disabled id
local_irq_save id ...
local_irq_disable id ...
linux/irqdesc.h inc
irq_desc id
linux/irqdomain.h inc
irq_domain id – hardware interrupt number translation object
irq_domain_get_irq_data id
linux/msi.h inc – Message Signaled Interrupts
msi_desc id
Structure of structures:
irq_desc id is container of
irq_data id
irq – interrupt number
irq_common_data id
list of irqaction id


⚙️ Internals

kernel/irq/settings.h src
kernel/irq src
kernel/irq/internals.h src
ls /sys/kernel/debug/irq/domains/
x86_vector_domain id, x86_vector_domain_ops id
irq_chip id
load_idt id – load Interrupt Descriptor Table
arch/x86/include/asm/idtentry.h src – interrupt entry/exit definitions
arch/x86/include/asm/hw_irq.h src
irq_entries_start id – hardware IRQ entry stubs
arch/x86/kernel/irq.c src
common_interrupt id – handles all normal device IRQs on x86

📖 References

IRQs doc
The irq_domain interrupt number mapping library doc
Linux generic IRQ handling doc
Message Signaled Interrupts: The MSI Driver Guide doc
Lock types and their rules doc
Hard IRQ Context doc
Interrupts


👁 Examples

dummy_irq_chip id – dummy interrupt chip implementation
lib/locking-selftest.c src


IRQ affinity

⚲ API

/proc/irq/default_smp_affinity
/proc/irq/*/smp_affinity and /proc/irq/*/smp_affinity_list


Common types and functions:

struct irq_affinity id – description for automatic irq affinity assignments, see devm_platform_get_irqs_affinity id
struct irq_affinity_desc id – interrupt affinity descriptor, see irq_update_affinity_desc id, irq_create_affinity_masks id
irq_set_affinity id
irq_get_affinity_mask id
irq_can_set_affinity id
irq_set_affinity_hint id
irqd_affinity_is_managed id
irq_data_get_affinity_mask id
irq_data_get_effective_affinity_mask id
irq_data_update_effective_affinity id
irq_set_affinity_notifier id
irq_affinity_notify id
irq_chip_set_affinity_parent id
irq_set_vcpu_affinity id


🛠️ Utilities

irqbalance – distributes hardware interrupts across CPUs


📖 References

SMP IRQ affinity doc
IRQ affinity, LF
managed_irq kernel parameter, @LKML
irqaffinity kernel parameter, @LKML


Non-maskable interrupts

⚲ API

linux/nmi.h inc
in_nmi id
touch_nmi_watchdog id
...
trace/events/nmi.h inc
arch/x86/include/asm/nmi.h src
register_nmi_handler id
unregister_nmi_handler id


⚙️ Internals

arch/x86/kernel/nmi.c src
arch/x86/kernel/nmi_selftest.c src


📖 References

NMI Trace Events doc


📚 Further reading

Non-maskable interrupt handler (NMI)


...

📚 Further reading about interrupts

IDT – Interrupt descriptor table
Tickless (Full dynticks) reduces timer interrupts overhead, CONFIG_NO_HZ_FULL id
LGDT/LIDT – Load Global/Interrupt Descriptor Table Register asm instruction

Deferred works

Scheduler context

kthread work

This framework simplifies the use of kernel kthreads. A kthread_work id item can be queued with kthread_queue_work id and flushed using kthread_flush_work id. All queued kthread_work items are processed by a dedicated kernel thread executing the kthread_worker_fn id function.

⚲ API

kthread_work id – contains kthread_work_func_t id to execute
kthread_init_work id
kthread_flush_work id
kthread_worker id – links a kthread_work and a task
kthread_run_worker id – creates and wakes a kthread worker
kthread_create_worker id
kthread_flush_worker id
kthread_destroy_worker id
kthread_queue_work id – queues a kthread_work on a kthread_worker


⚙️ Internals

__kthread_create_worker_on_node id
kthread_worker_fn id – executes work's function


👁 Example usages

watchdog_kworker id, pwq_release_worker id, pump_messages id

Threaded IRQ

⚲ API devm_request_threaded_irq id, request_threaded_irq id ISR should return IRQ_WAKE_THREAD to run thread function

⚙️ Internals

setup_irq_thread id, irq_thread id
kernel/irq/manage.c src

📖 References

request_threaded_irq doc


Work and workqueue

Generic async execution with shared worker pool.

⚲ API

PF_WQ_WORKER id – workqueue worker process flag
linux/workqueue.h inc, linux/workqueue_types.h inc
work_struct id, INIT_WORK id, schedule_work id,
delayed_work id, INIT_DELAYED_WORK id, schedule_delayed_work id, cancel_delayed_work_sync id
workqueue_struct id
alloc_workqueue id
destroy_workqueue id
queue_work id – queues work on a workqueue
show_all_workqueues id
show_one_workqueue id
system_power_efficient_wq id ...


👁 Example usage samples/ftrace/sample-trace-array.c src


⚙️ Internals

kernel/workqueue.c src
workqueue_init id
create_worker id
worker_thread id
process_one_work id
format_worker_id id – names kworker kthreads
pool_workqueue id
worker_pool id


📖 References

Concurrency Managed Workqueue doc

Interrupt context

linux/irq_work.h inc – framework for enqueueing and running callbacks from hardirq context
samples/trace_printk/trace-printk.c src

Timers

softirq timer

This timer is a softirq for periodical tasks with jiffies resolution

⚲ API

linux/timer.h inc
timer_list id, DEFINE_TIMER id, timer_setup id
mod_timer id — sets expiration time in jiffies.
del_timer id


⚙️ Internals

kernel/time/timer.c src
timer_bases id


👁 Examples

input_enable_softrepeat id and input_start_autorepeat id


📚 References

Time and timer routines doc
mod_timer_pending ... doc
High-resolution timer

⚲ API

/proc/timer_list
/proc/sys/kernel/timer_migration
linux/hrtimer_defs.h inc
linux/hrtimer.h inc
hrtimer id, hrtimer.function — callback
hrtimer_init id
hrtimer_setup id
hrtimer_start id — starts a timer with nanosecond resolution
hrtimer_cancel id


👁 Examples alarm_init id, watchdog_enable id


⚙️ Internals

CONFIG_HIGH_RES_TIMERS id
kernel/time/tick-internal.h src
hrtimer_bases id
kernel/time/hrtimer.c src
kernel/time/itimer.c src
kernel/time/timer_list.c src


📚 HR timers references

High-resolution timers doc
hrtimers - subsystem for high-resolution kernel timers doc
high resolution timers and dynamic ticks design notes doc
...

📚 Timers references

Timers doc
Better CPU selection for timer expiration

Tasklet

tasklet is a softirq, for time critical operations

⚲ API is deprecated in favor of threaded IRQs: devm_request_threaded_irq id

tasklet_struct id, tasklet_init id, tasklet_schedule id


⚙️ Internals: tasklet_action_common id HI_SOFTIRQ, TASKLET_SOFTIRQ


Softirq

softirq is internal system facility and should not be used directly. Use tasklet or threaded IRQs

⚲ API

linux/interrupt.h inc
cat /proc/softirqs
open_softirq id registers softirq_action id


⚙️ Internals

kernel/softirq.c src


📖 References

Introduction to deferred interrupts (Softirq, Tasklets and Workqueues)
Timers and time management
Deferred work, linux-kernel-labs
Chapter 7. Time, Delays, and Deferred Work

CPU specific

🖱️ GUI

tuna – program for tuning running processes


⚲ API

cat /proc/cpuinfo
/sys/devices/system/cpu/
/sys/devices/system/node/
/sys/cpu/
/sys/fs/cgroup/cpu/
grep -i cpu /proc/self/status
rdmsr – tool for reading CPU machine specific registers (MSR)
man 1 lscpu – display information about the CPU architecture
linux/arch_topology.h inc – arch specific cpu topology information
linux/cpu.h inc – generic cpu definition
linux/cpu_cooling.h inc
linux/cpu_pm.h inc
linux/cpufeature.h inc
arch/x86/include/asm/cpufeature.h src
linux/peci-cpu.h inc
linux/sched/cputime.h inc – cputime accounting APIs
linux/clk.h inc – interfaces for managing hardware clocks in device drivers


⚙️ Internals

drivers/base/cpu.c src
cpu_dev_init id


Cache

linux/cacheflush.h inc
arch/x86/include/asm/cacheflush.h src: clflush_cache_range id
linux/cache.h inc
arch/x86/include/asm/cache.h src


⚙️ Internals

arch/x86/mm/pat/set_memory.c src
arch/x86/kernel/cpu/mtrr/ src

📚 Further reading

MTRR – Memory type range register
CPU cache

This chapter is about multiprocessing and multi-core aspects of Linux kernel.

Key concepts and features of Linux SMP include:

  • Symmetry: In an SMP system, all processors are considered the same without hardware hierarchy in contradiction to use of coprocessors.
  • Load balancing: The Linux kernel employs load balancing mechanisms to distribute tasks evenly among available CPU cores. This prevents any one core from becoming overwhelmed while others remain underutilized.
  • Parallelism: SMP enables parallel processing, where multiple threads or processes can execute simultaneously on different CPU cores. This can significantly improve the execution speed of applications that are designed to take advantage of multiple threads.
  • Thread scheduling: The Linux kernel scheduler is responsible for determining which threads or processes run on which CPU cores and for how long. It aims to optimize performance by minimizing contention and maximizing CPU utilization.
  • Shared memory: In an SMP system, all CPU cores typically share the same physical memory space. This allows processes and threads running on different cores to communicate and share data more efficiently.
  • NUMA – Non-Uniform Memory Access: In larger SMP systems, memory access times might not be uniform due to the physical arrangement of memory banks and processors. Linux has mechanisms to handle NUMA architectures efficiently, allowing processes to be scheduled on CPUs closer to their associated memory.
  • Cache coherency: SMP systems require mechanisms to ensure that all CPU cores have consistent views of memory. Cache coherency protocols ensure that changes made to shared memory locations are correctly propagated to all cores.
  • Scalability: SMP systems can be scaled up to include more CPU cores, enhancing the overall computing power of the system. However, as the number of cores increases, challenges related to memory access, contention, and communication between cores may arise.
  • Kernel and user space: Linux applications running in user space can take advantage of SMP without needing to be aware of the underlying hardware details. The kernel handles the management of CPU cores and resource allocation.


⚲ API

ps -PLe – lists threads with processor that the thread last executed on (the third column PSR).
man 2 getcpu – determine CPU and NUMA node on which the calling thread is running
man 8 chcpu – configure CPUs
man 3 CPU_SET – macros for manipulating CPU sets
linux/smp.h inc
The most commonly used functions:
crash_smp_send_stop id – halts all CPUs except the calling one in a crash context
get_cpu id – disables preemption and returns the current processor ID
on_each_cpu id – runs a given function on all CPUs, possibly with synchronization
on_each_cpu_cond_mask id – conditionally runs a function on selected CPUs
on_each_cpu_mask id – executes a function on a specified set of CPUs
panic_smp_self_stop id – stops the local CPU during a panic while ensuring others halt
put_cpu id – re-enables preemption after a previous get_cpu call
raw_smp_processor_id id – returns the current CPU ID without preemption safety
setup_max_cpus id – sets up the maximum number of CPUs to be brought online
smp_call_func_t id – typedef for the function signature used in SMP function calls
smp_call_function id – invokes a function on all other CPUs asynchronously
smp_call_function_any id – runs a function on any available CPU in a given mask
smp_call_function_many id – sends a function call to a specified set of CPUs
smp_call_function_single id – sends a function to execute on a single target CPU
smp_call_function_single_async id – queues a function to run asynchronously on one CPU
smp_call_on_cpu id – executes a function on a specific CPU and waits for the result
smp_init id – initializes core SMP structures and state during boot
smp_prepare_boot_cpu id – prepares the boot CPU during early SMP initialization
smp_prepare_cpus id – prepares all CPUs for booting before secondary CPUs are started
smp_processor_id id – returns the ID of the current CPU with preemption checks
smp_send_reschedule id – sends a reschedule interrupt to a target CPU
smp_send_stop id – stops all other CPUs in response to critical events
wake_up_all_idle_cpus id – wakes all idle CPUs to ensure prompt task execution
linux/cpu.h inc
linux/group_cpus.h inc: group_cpus_evenly id – groups all CPUs evenly per NUMA/CPU locality
asm-generic/percpu.h inc
linux/percpu-defs.h inc – basic definitions for percpu areas
this_cpu_ptr id
linux/percpu.h inc
linux/percpu-refcount.h inc
linux/percpu-rwsem.h inc
linux/preempt.h inc
migrate_disable id, migrate_enable id
/sys/bus/cpu
per CPU local_lock
linux/topology.h inc – cpu_to_node id, numa_node_id id
arch/x86/include/asm/topology.h src
See also NUMA memory section


⚙️ Internals

boot_cpu_init id activates the first CPU
smp_prepare_cpus id initializes rest CPUs during boot
cpu_number id
CONFIG_SMP id
CONFIG_NUMA id
trace/events/percpu.h inc
IPI – Inter-processor interrupt
trace/events/ipi.h inc
kernel/irq/ipi.c src
ipi_send_single id, ipi_send_mask id ...
drivers/base/cpu.c src – CPU driver model subsystem support
kernel/cpu.c src
smpboot
linux/smpboot.h inc
kernel/smpboot.c src
arch/x86/kernel/smpboot.c src
lib/group_cpus.c src


🛠️ Utilities

irqbalance – distributes hardware interrupts across CPUs
man 8 numactl – controls NUMA policy for processes or shared memory


📖 References

Per-CPU Data doc
How CPU topology info is exported via sysfs doc


📚 Further reading

Functionalities of the scheduler TuneD plugin
tuned-adm – command line tool for switching between different tuning profiles


CPU affinity

Affinity refers to assigning a process or thread to specific CPU cores. This helps control which CPUs execute tasks, potentially improving performance by reducing data movement between cores. It can be managed using system calls or commands. Affinity can be represented as CPU bitmask: cpumask_t id or CPU affinity list: cpulist_parse id.


⚲ API

man 1 taskset – set or retrieve a process's CPU affinity
grep Cpus_allowed /proc/self/status
man 2 sched_setaffinity man 2 sched_getaffinity – set and get a thread's CPU affinity mask
↪ sched_setaffinity id
set_cpus_allowed_ptr id – common kernel function to change a task's affinity mask
linux/cpu_rmap.h inc – CPU affinity reverse-map support
linux/cpumask_types.h inc
struct cpumask, cpumask_t id – CPUs bitmap, can be very big
cpumask_var_t id – type for local cpumask variable, see alloc_cpumask_var id, free_cpumask_var id.
linux/cpumask.h inc – Cpumasks provide a bitmap suitable for representing the set of CPU's in a system, one bit position per CPU number
for_each_possible_cpu id
num_online_cpus id
cpumask_set_cpu id
cpumask_test_cpu id
for_each_cpu id


⚙️ Internals

cpus_mask id – affinity of task_struct id
cpus_allowed id – affinity of cpuset id


📚 Further reading

Processor affinity
Affinity mask

CPU hotplug

CPU hotplugging in Linux refers to the ability to dynamically add or remove CPUs from the system without needing a reboot. This feature is crucial in environments requiring high availability and resource flexibility, such as data centers, virtualized systems, and systems that use power management aggressively.


🗝️ Acronyms

BP – Bootstrap Processor
AP – Application Processor


⚲ API

/sys/devices/system/cpu/cpu*/online
/sys/devices/system/cpu/cpu*/hotplug/
linux/cpu.h inc
add_cpu id ...
linux/cpuhotplug.h inc
cpuhp_state id – CPU hotplug states
cpuhp_setup_state id ... – setups hotplug state callbacks
cpuhp_setup_state_multi id
cpuhp_setup_state_nocalls id
linux/cpuhplock.h inc – CPU hotplug locking
cpus_read_lock id ...
remove_cpu id ...


⚙️ Internals

kernel/cpu.c src
cpuhp_state id – from CPUHP_OFFLINE to CPUHP_AP_ACTIVE and CPUHP_ONLINE.
cpuhp_hp_states id
boot_cpu_hotplug_init id
cpuhp_threads_init id
... cpuhp_invoke_callback_range id ...
kernel/irq/cpuhotplug.c src
drivers/base/cpu.c src – CPU subsystem support
cpu_dev_init id
... cpu_subsys_online id
trace/events/cpuhp.h inc
linux/cpuhplock.h inc


👁️ Examples

torture_onoff id
tools/testing/selftests/sched_ext/hotplug.c src


📖 References

CPU hotplug in the Kernel doc
Introduction doc
Command Line Switches doc
CPU maps doc
Using CPU hotplug doc
The CPU hotplug coordination doc
The CPU hotplug API doc
CPU hotplug state machine doc
CPU online/offline operations doc
Allocating a state doc
Setup of a CPU hotplug state doc
Removal of a CPU hotplug state doc
Multi-Instance state instance management doc
Examples doc
Testing of hotplug states doc
Architecture’s requirements doc
User Space Notification doc
Kernel Inline Documentations Reference doc


📚 Further reading

stress-ng --cpu-online
CONFIG_CPU_HOTPLUG_STATE_CONTROL id – enables the ability to write incremental steps between "offline" and "online" states to the CPU's sysfs target file, allowing for more granular control of state transitions.
target_store id: cpu_up id/cpu_down id
cpuhotplug, cpuhp @LKML

CPU isolation

CPU isolation ensures that specific tasks run on dedicated CPUs, reducing contention and latency.

Housekeeping CPUs refer to the CPUs that are reserved for various system tasks. See hk_type id.

Isolated CPUs are dedicated to real-time applications, such as DPDK.

⚲ API

/sys/devices/system/cpu/isolated
/sys/devices/system/cpu/nohz_full
/sys/fs/cgroup/cpuset.cpus.isolated
/sys/fs/cgroup/.../cpuset.cpus.partition
linux/sched/isolation.h inc
hk_type id – housekeeping type
housekeeping_cpumask id – returns CPUs available for housekeeping of a given type
cpu_is_isolated id
linux/cpuset.h inc – cpuset interface
man 7 cpuset – confine processes to processor and memory node subsets


⚙️ Internals

CONFIG_CPU_ISOLATION id
kernel/sched/isolation.c src
housekeeping_init id, housekeeping_update id
CONFIG_CPUSETS id
kernel/cgroup/cpuset.c src
cpuset_init id
cpuset_init_smp id
isolated_cpus id
partition_xcpus_add id, partition_xcpus_del id


📖 References

CPU lists in command-line parameters doc
nohz_full clears housekeeping.cpumasks id for tick, wq, timer, rcu, misc, and kthread in housekeeping_nohz_full_setup id
isolcpus clears housekeeping.cpumasks id for domain (by default), nohz, and managed_irq in housekeeping_isolcpus_setup id
Housekeeping doc
NO_HZ: Reducing Scheduling-Clock Ticks doc
CPUSETS of cgroup v2 doc
CPUSETS of cgroup v1 doc


📚 Further reading

CPU Isolation state of the art, LPC'23
CPU Isolation
isolcpus @LKML
housekeeping @LKML
Explicitly Reserved CPU List, Kubernetes Documentation
CPU Partitioning
Scheduler Domains doc – the Scheduler balances CPUs (scheduling groups) within a sched domain
nohz_full @LKML

💾 Historical since v7.0

cpuset_cpu_is_isolated id – removed, use cpu_is_isolated id instead

Runtime code patching

The kernel patches its own machine code at boot or runtime to optimize hot paths. Static keys patch a NOP to a jump (or vice versa); static calls patch indirect calls to direct calls; alternatives replace instructions based on CPU features.

⚲ API

linux/jump_label.h inc – static keys
DEFINE_STATIC_KEY_FALSE id – define a key, initially false
static_branch_unlikely id – zero-cost branch check (NOP when key is false)
static_branch_enable id – patch all sites to take the branch (expensive, rare use)
linux/static_call.h inc – static calls
DEFINE_STATIC_CALL id – define a call site with initial target function
static_call id – invoke the patched direct call
static_call_update id – change target function, patch all sites
arch/x86/include/asm/alternative.h src – alternatives
ALTERNATIVE id – replace instructions at boot based on CPU features
arch/x86/include/asm/cpufeatures.h src – X86_FEATURE_* flags


⚙️ Internals

CONFIG_JUMP_LABEL id
kernel/jump_label.c src
arch/x86/kernel/jump_label.c src
CONFIG_HAVE_STATIC_CALL id
kernel/static_call_inline.c src
arch/x86/kernel/static_call.c src
arch/x86/include/asm/alternative.h src
arch/x86/kernel/alternative.c src


👁 Examples

housekeeping_overridden id
preempt_schedule id
tracepoints


📖 References

Static Keys doc
x86 CPU feature flags doc


Memory barriers (MB) are synchronization mechanisms used to ensure proper ordering of memory operations in a SMP environment. They play a crucial role in maintaining the consistency and correctness of data shared among different CPU cores or processors. MBs prevent unexpected and potentially harmful reordering of memory access instructions by the compiler or CPU, which can lead to data corruption and race conditions in a concurrent software system.

⚲ API

man 2 membarrier
asm-generic/barrier.h inc
mb id, rmb id, wmb id
smp_mb id, smp_rmb id, smp_wmb id


⚙️ Internals

arch/x86/include/asm/barrier.h src
kernel/sched/membarrier.c src


📖 References

Memory barriers doc


States

C-states and P-states are features in modern CPUs designed to improve energy efficiency.

🗝️ Acronyms

C-states – CPU(?) states, aka idle states
EPP – Energy Performance Preference doc
HWP – hardware-managed P-states, see hwp_only
P-states – Performance states, see CPU Power and frequency scaling


⚲ API

cpupower


📖 References

Working-State Power Management doc
https://lwn.net/Kernel/Index/#Power_management


Idle

C-states, power states:

C0 – the operating state.
C1 (aka Halt) – the processor is not executing instructions, but can return to an executing state instantaneously.
C2 (aka Stop-Clock) – the processor maintains all software-visible state, but may take longer to wake up.
C3 (aka Sleep) – takes longer to wake up.
...


⚲ API

turbostat --show CPU --quiet -n 1 --interval 0.1 --show sysfs
idle=
/dev/cpu_dma_latency – see set_cpu_dma_latency id
C-states interfaces:
/sys/devices/system/cpu/cpuidle/
/sys/devices/system/cpu/cpu*/cpuidle/
linux/cpuidle.h inc – a generic framework for CPU idle power management
intel_idle CPU Idle Time Management Driver doc
linux/pm_qos.h inc


⚙️ Internals

drivers/cpuidle src
kernel/power/qos.c src
cpu_latency_qos_miscdev id – implementation of /dev/cpu_dma_latency


📖 References

CPU Idle Time Management doc


📚 Further reading

https://lwn.net/Kernel/Index/#Power_management-cpuidle
PM Quality Of Service Interface doc

Power and frequency

P-states, performance states:

P0 – maximum power and frequency
Pn – less power and frequency
...


⚲ API

Reliable measurement of actual CPU frequency
turbostat --quiet --show CPU,Bzy_MHz -n 1 --interval 0.1
Bzy_MHz – average clock rate while the CPU was not idle (ie. in "c0" state).
intel_pstate=
Kernel Command Line Options for intel_pstate doc
cpufreq.default_governor=
P-states interfaces:
/sys/devices/system/cpu/cpufreq/
/sys/devices/system/cpu/cpu*/cpufreq/
scaling_cur_freq – unreliable assumption on CPU frequency
/sys/devices/system/cpu/intel_pstate/
linux/cpufreq.h inc
linux/sched/cpufreq.h inc – interface between cpufreq drivers and the scheduler


⚙️ Internals

drivers/cpufreq src
intel_pstate id
acpi_cpufreq_driver id
kernel/sched/cpufreq_schedutil.c src – implementation of cpufreq.default_governor=schedutil
arch/x86/kernel/cpu/intel_epb.c src – Intel Performance and Energy Bias Hint support


📖 References

CPU Performance Scaling doc
Device Frequency Scaling doc
CPUFreq Governor
CPUFreq - CPU frequency and voltage scaling doc
Intel Performance and Energy Bias Hint doc
intel_pstate CPU Performance Scaling Driver doc
General Information doc
Operation Modes doc
Active Mode doc
Passive Mode doc
Turbo P-states Support doc
Processor Support doc
Support for Hybrid Processors doc
Hybrid Processors with SMT doc
Capacity-Aware Scheduling Support doc
Energy-Aware Scheduling Support doc
User Space Interface in sysfs doc
Global Attributes doc
Interpretation of Policy Attributes doc
Coordination of P-State Limits doc
Energy vs Performance Hints doc
intel_pstate vs acpi-cpufreq doc
Kernel Command Line Options for intel_pstate doc
Diagnostics and Tuning doc
Trace Events doc
ftrace doc


📚 Further reading

https://lwn.net/Kernel/Index/#Power_management-Frequency_scaling
CPU frequency scaling
cpufreq ltp
How to use cpufrequtils
cpufreq-info
cpufreq-set
https://github.com/intel/power-optimization-library
https://github.com/intel/kubernetes-power-manager

Architectures

Linux CPU architectures refer to the different types of central processing units (CPUs) that are compatible with the Linux operating system. Linux is designed to run on a wide range of CPU architectures, which allows it to be utilized on various devices, from smartphones to servers and supercomputers. Each architecture has its own unique features, advantages, and design considerations.

Architectures are classified by family (e.g. x86, ARM), word or long int size (e.g. CONFIG_32BIT id, CONFIG_64BIT id).


Some functions with different implementations for different CPU architectures:

do_boot_cpu id > start_secondary id > cpu_init id
setup_arch id, start_thread id, get_current id, current id

⚲ API

BITS_PER_LONG id, __BITS_PER_LONG id,

⚙️ Arch internals

arch src
x86
CONFIG_X86 id
arch/x86 src
drivers/platform/x86 src
https://lwn.net/Kernel/Index/#Architectures-x86
man 1 perf-intel-pt – support for Intel Processor Trace within perf
ARM
CONFIG_ARM id
arch/arm src, ARM Architecture doc
https://lwn.net/Kernel/Index/#Architectures-ARM
arch/arm64 src, ARM64 Architecture doc
architecture-specific initialization


📖 References

CPU Architectures doc
x86-specific doc
x86_64 Support doc