公司去年开始逐步在容器化应用中实现优雅关闭,我们可以认为正常的优雅停机包括两个部分:应用 & 平台。
1. 应用优雅退出,应用层需要做的任务是处理SIGTERM信号。应用自身需要实现优雅停机的处理逻辑,确保处理中的请求可以继续完成,资源得到有效的关闭释放,等等。
Spring boot (2.3.0及以上)官方文档:
Graceful shutdown is supported with all four embedded web servers (Jetty, Reactor Netty, Tomcat, and Undertow) and with both reactive and Servlet-based web applications. It occurs as part of closing the application context and is performed in the earliest phase of stopping SmartLifecycle beans. This stop processing uses a timeout which provides a grace period during which existing requests will be allowed to complete but no new requests will be permitted. The exact way in which new requests are not permitted varies depending on the web server that is being used. Jetty, Reactor Netty, and Tomcat will stop accepting requests at the network layer. Undertow will accept requests but respond immediately with a service unavailable (503) response.
1.1 配置To enable graceful shutdown, configure the server.shutdown property, as shown in the following example:
server.shutdown=graceful
To configure the timeout period, configure the spring.lifecycle.timeout-per-shutdown-phase property, as shown in the following example:
spring.lifecycle.timeout-per-shutdown-phase=20s
下面是 Yaml 配置:
# 开启优雅关闭
server:
shutdown: graceful
# 关闭的缓冲时间
spring:
lifecycle:
timeout-per-shutdown-phase: 10s
1.2 源码分析
优雅停机在stop SmarLifecycle bean 的初始阶段,作为关闭应用程序上下文的一部分发生:
public interface SmartLifecycle extends Lifecycle, Phased {
int DEFAULT_PHASE = 2147483647;
default boolean isAutoStartup() {
return true;
}
default void stop(Runnable callback) {
this.stop();
callback.run();
}
default int getPhase() {
return 2147483647;
}
}
public final class WebServerGracefulShutdownLifecycle implements SmartLifecycle {
@Override
public void stop(Runnable callback) {
this.running = false;
this.webServer.shutDownGracefully((result) -> callback.run());
}
}
public class TomcatWebServer implements WebServer {
private final GracefulShutdown gracefulShutdown;
public TomcatWebServer(Tomcat tomcat, boolean autoStart, Shutdown shutdown) {
Assert.notNull(tomcat, "Tomcat Server must not be null");
this.tomcat = tomcat;
this.autoStart = autoStart;
this.gracefulShutdown = (shutdown == Shutdown.GRACEFUL) ? new GracefulShutdown(tomcat) : null;
initialize();
}
@Override
public void shutDownGracefully(GracefulShutdownCallback callback) {
if (this.gracefulShutdown == null) {
callback.shutdownComplete(GracefulShutdownResult.IMMEDIATE);
return;
}
this.gracefulShutdown.shutDownGracefully(callback);
}
}
final class GracefulShutdown {
void shutDownGracefully(GracefulShutdownCallback callback) {
logger.info("Commencing graceful shutdown. Waiting for active requests to complete");
new Thread(() -> doShutdown(callback), "tomcat-shutdown").start();
}
private void doShutdown(GracefulShutdownCallback callback) {
List connectors = getConnectors();
connectors.forEach(this::close);
try {
for (Container host : this.tomcat.getEngine().findChildren()) {
for (Container context : host.findChildren()) {
while (isActive(context)) {
if (this.aborted) {
logger.info("Graceful shutdown aborted with one or more requests still active");
callback.shutdownComplete(GracefulShutdownResult.REQUESTS_ACTIVE);
return;
}
Thread.sleep(50);
}
}
}
}
catch (InterruptedException ex) {
Thread.currentThread().interrupt();
}
logger.info("Graceful shutdown complete");
callback.shutdownComplete(GracefulShutdownResult.IDLE);
}
}
public enum GracefulShutdownResult {
REQUESTS_ACTIVE,
IDLE,
IMMEDIATE;
}
REQUESTS_ACTIVE 说的是在缓冲期结束前连接保持活动状态,也就是虽然官方默认给你 30 秒的缓存时间来处理囤积请求,如果 30 秒之后还没处理完成,最后 Spring Boot 也会强制关闭应用。
所以需要注意的是,优雅关闭时一定要考虑当时的业务处理量,所设置的缓冲时间是否能处理完正在处理中的业务。
2. 平台平台层要能够将应用从负载均衡中去掉,确保应用不会再接受到新的请求连接,并且能够通知到应用要进行优雅停机处理。
容器终止流程
- Pod 被删除,状态置为 Terminating。
- kube-proxy 更新转发规则,将 Pod 从 service 的endpoint 列表中摘除掉,新的流量不再转发到该 Pod。
- 如果 Pod 配置了 preStop Hook ,将会执行。
- kubelet 对 Pod 中各个 container 发送 SIGTERM 信号以通知容器进程开始优雅停止。
- 等待容器进程完全停止,如果在 terminationGracePeriodSeconds 内 (默认 30s) 还未完全停止,就发送SIGKILL 信号强制杀死进程。 所有容器进程终止,清理 Pod 资源。
推荐优先使用exec格式。使用 shell 格式之后,程序会以 /bin/sh -c 的子命令启动,并且 shell 格式下不会传递任何信号(例如:SIGTERM )给程序。这也就导致,在 docker stop 容器的时候,以这种格式运行的程序捕捉不到发送的信号,也就谈不上优雅的关闭了。
Dockerfile 中的 ENTRYPOINT 和 CMD 指令,它们的主要功能是指定容器启动时执行的程序。
CMD 有三种格式:
- CMD ["executable","param1","param2"] (exec 格式, 推荐使用这种格式)
- CMD ["param1","param2"] (作为 ENTRYPOINT 指令参数)
- CMD command param1 param2 (shell 格式,默认 /bin/sh -c )
ENTRYPOINT 有两种格式:
- ENTRYPOINT ["executable", "param1", "param2"] (exec 格式,推荐优先使用这种格式)
- ENTRYPOINT command param1 param2 (shell 格式)
kubernetes文档指出,有些步骤是同时执行的。因此有可能会导致该Pod仍然列在服务的Endpoints中并仍然接收流量,而它已经收到SIGTERM并且已经停止,因此负载均衡器上可能会有一些Http 504。目前解决这个问题可以使用preStop Hook 在容器收到SIGTERM时sleep一段时间,以确终止期间的流量可以正确处理。
preStop-webhook
containers:
-name: ms-apps-openapi-service
image: artifactory.global.ABCcompany.com/ms-apps-openapi@iamgeversion
lifecycle:
preStop:
exec:
command: [ "sh", "-c", "sleep 10"]
调整优雅终止时间,terminationGracePeriodSeconds 默认是30s terminationGracePeriodSeconds 一定大于sleep的时间。preStop Hook并不会影响SIGTERM的处理,因此有可能preStopHook还没有执行完就收到SIGKILL导致容器强制退出。因此如果preStop Hook设置了n秒,需要设置terminationGracePeriodSeconds为terminationGracePeriodSeconds+n秒。
apiVersion: apps/v1
kind: Deployment
metadata:
name: ms-apps-openapi
namespace: api-industry
labels:
app: ms-apps-openapi
spec:
replicas: 3
selector:
matchLabels:
app: ms-apps-openapi
template:
spec:
hostname: ms-apps-openapi-service
terminationGracePeriodSeconds: 60



