> For the complete documentation index, see [llms.txt](https://huataihuang.gitbook.io/cloud-atlas-draft/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://huataihuang.gitbook.io/cloud-atlas-draft/os/linux/kernel/tracing/hrtimer_interrupt_took_xxxx_ns_local_softirq_pending.md).

# "hrtimer: interrupt took xxxx ns"和"NOHZ: local\_softirq\_pending 100"

在KVM平台运行的虚拟机，有时候突然有出现访问数据库的创建连接超时（500ms以上），但是从物理服务器看，系统负载并没有达到很高的程度。通过cgroup资源隔离，将底层vhost、存储以及vm所使用的CPU隔离及资源平衡后，确实得到了很大的改善。

但是，偶尔还是有个别vm出现抖动，看起来还有尚未彻底解决的因素影响了系统的稳定性。

## 排查

* 虚拟机日志检查

`dmesg`中可以发现有如下异常的日志：

```
NOHZ: local_softirq_pending 100
hrtimer: interrupt took 2310385 ns
NOHZ: local_softirq_pending 100
NOHZ: local_softirq_pending 100
NOHZ: local_softirq_pending 100
```

检查`/var/log/messages`可以看到：

```
Mar  7 22:38:41 exampleserver-1 kernel: NOHZ: local_softirq_pending 100
Mar 22 08:38:31 exampleserver-1 kernel: NOHZ: local_softirq_pending 100
Mar 25 09:22:12 exampleserver-1 kernel: NOHZ: local_softirq_pending 100
```

### 内核相关

[NOHZ: local\_softirq\_pending 100](https://forum.proxmox.com/threads/nohz-local_softirq_pending-100.11855/)提到了 `NOHZ: local_softirq_pending 100` 有可能是Red Hat Kernel 6.2.32 分支的bug。

在 <http://www.kubuntuforums.net/showthread.php?49223-Kernel-error-messsage-NOHZ-local_softirq_pending-100> 有一个解释：

`NOHZ`补丁包含了一个检查"处理器进入idle时候softirqs pending"的功能。这个BUG和`NOHZ`无关，只不过通过`NOHZ`补丁能够观察到这个现象：

* `t1`线程在CPU#0上运行softirq disabled代码，中断发生，出现了softirq，但是被推迟（deferred）(因为softirqs disabled)
* `t1`线程调用`cond_resched_softirq()`，通过`_local_bh_enable()`调用`schedule()`再次激活so
*

> `NOHZ`是只是显示了系统存在的问题，具体原因还是要进一步分析

在Red Hat官方文档中有一个 [VMware guest hangup and "NOHZ: local\_softirq\_pending 100" logged](https://access.redhat.com/solutions/1157563) 提示了类似情况


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://huataihuang.gitbook.io/cloud-atlas-draft/os/linux/kernel/tracing/hrtimer_interrupt_took_xxxx_ns_local_softirq_pending.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.