kubernetes 部署和配置 ingress-nginx
在kubernetes高可用的集群下安装及配置ingress-nginx，可用于生产环境
1.0 基础工作
集群及dashboard的搭建工作，可以参考blog的其他文章
1.1 准备好一个k8s集群
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 Ready master 3d6h v1.17.2
master2 Ready master 3d6h v1.17.2
master3 Ready master 3d6h v1.17.2
node1 Ready <none> 3d6h v1.17.2
1.2 准备一个单独的外网nginx
集群外的nginx，用于演示
192.168.0.200
1.3 准备ingress-controller的docker镜像
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.29.0
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.29.0 quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.29.0
2.0 安装工作
2.1 安装nginx-ingress
国内打不开以下网站，需要准备梯子自行解决
$ wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
$ kubectl apply -f mandatory.yaml
官网文件，未做改动
apiVersion: v1
kind: Namespace
metadata:
name: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-configuration
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: tcp-services
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: udp-services
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: nginx-ingress-clusterrole
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- "extensions"
- "networking.k8s.io"
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
- "networking.k8s.io"
resources:
- ingresses/status
verbs:
- update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: nginx-ingress-role
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
resourceNames:
# Defaults to "<election-id>-<ingress-class>"
# Here: "<ingress-controller-leader>-<nginx>"
# This has to be adapted if you change either parameter
# when launching the nginx-ingress-controller.
- "ingress-controller-leader-nginx"
verbs:
- get
- update
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: nginx-ingress-role-nisa-binding
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: nginx-ingress-role
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: nginx-ingress-clusterrole-nisa-binding
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: nginx-ingress-clusterrole
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-ingress-controller
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
template:
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
annotations:
prometheus.io/port: "10254"
prometheus.io/scrape: "true"
spec:
# wait up to five minutes for the drain of connections
terminationGracePeriodSeconds: 300
serviceAccountName: nginx-ingress-serviceaccount
nodeSelector:
kubernetes.io/os: linux
containers:
- name: nginx-ingress-controller
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.29.0
args:
- /nginx-ingress-controller
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --publish-service=$(POD_NAMESPACE)/ingress-nginx
- --annotations-prefix=nginx.ingress.kubernetes.io
securityContext:
allowPrivilegeEscalation: true
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
# www-data -> 101
runAsUser: 101
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
protocol: TCP
- name: https
containerPort: 443
protocol: TCP
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
lifecycle:
preStop:
exec:
command:
- /wait-shutdown
---
apiVersion: v1
kind: LimitRange
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
limits:
- min:
memory: 90Mi
cpu: 100m
type: Container
2.2 安装NodePort Service
我采用了NodePort方式向外提供服务，
$ vi node-port-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
type: NodePort # NodePort方式
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
- name: https
port: 443
targetPort: 443
protocol: TCP
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
$ kubectl apply -f node-port-svc.yaml
2.3 验证是否安装成功
包括一个pod和一个svc
$ kubectl get pods -n ingress-nginx
NAME READY STATUS RESTARTS AGE
nginx-ingress-controller-6885bc7778-9c96t 1/1 Running 1 5h46m
$ kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx NodePort 10.96.140.105 <none> 80:30190/TCP,443:30564/TCP 5h46m
注意
向外提供的端口分别为：80->30190 443->30564
因为我外网还有一台nginx转发,向用户提供服务时，在nginx上使用https，但在k8s内网，使用http。
3.0 部署一个网站应用
3.1 先发布应用本身上关
$ vi app.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: default
name: home-web # 应用名
spec:
replicas: 1
selector:
matchLabels:
app: home-web
template:
metadata:
labels:
app: home-web
spec:
containers:
- name: home-web
image: registry.cn-shenzhen.aliyuncs.com/gk100/home-web:0.0.2 #使用的image，私有
ports:
- name: web
containerPort: 10002 # 镜像应用向外提供服务的端口
imagePullSecrets:
- name: aliyunsecret #私有镜像的secret
---
apiVersion: v1
kind: Service
metadata:
name: home-web-service # 应用对应的服务
namespace: default
spec:
selector:
app: home-web # 连接到应用
ports:
- name: web
port: 10002 # 服务的服务端口
protocol: TCP
$ kubectl apply -f app.yaml
3.2 验证应用
此应用还需要连接到ingress-nginx，完全走内网
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
home-web-6647df5f9c-kmvbd 1/1 Running 1 5h21m
$ sh kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
home-web-service ClusterIP 10.103.66.231 <none> 10002/TCP 5h22m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3d7h
3.2 为此应用添加一个ingress
$ vi home-web-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: home-web-ingress
namespace: default
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: test.api.zituo.net #主机
http:
paths:
- path: / # 地址
backend:
serviceName: home-web-service #连接3.1的服务名
servicePort: 10002 # 服务的端口
应用
$ kubectl apply -f home-web-ingress.yaml
验证
$ kubectl get ingress
home-web-ingress test.api.zituo.net 10.96.140.105 80 5h35m
4.0 外网nginx的配置
192.168.0.200的外网主机，不在k8s集群内
$ vi /etc/nginx/conf.d/test.api.zituo.net.conf
server {
server_name test.api.zituo.net;
listen 80;
add_header Strict-Transport-Security max-age=15768000;
add_header X-Content-Type-Options nosniff;
add_header X-powered-by "golang";
proxy_set_header Host $host;
proxy_read_timeout 3600s;
proxy_send_timeout 12s;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header REMOTE-HOST $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_buffering on;
proxy_buffer_size 128k;
proxy_buffers 8 1M;
proxy_busy_buffers_size 2M;
proxy_max_temp_file_size 1024m;
location / {
proxy_pass http://home;
}
}
upstream home {
server 192.168.0.85:30190; # 30190为ingress-nginx向外服务port
}
完成后的截图
在dashboard内，也能看到
