26.kubernetes(k8s)笔记 Ingress(一) Ingress-nginx

前言:

什么是Ingress？

官方的解释是:
Ingress 是对集群中服务的外部访问进行管理的 API 对象，典型的访问方式是 HTTP。
Ingress 可以提供负载均衡、SSL 终结和基于名称的虚拟托管。

Ingress简介
Ingress对象，实则就是对“反向代理”的一种抽象，简单的说就是一个全局的负载均衡器，可以通过访问URL定位到后端的Service
有了Ingress这个抽象，K8S就不需要关心Ingress的细节了，实际使用时，只需要选择一个具体的Ingress Controller部署就行了，业界常用的反向代理项目有：Nginx、HAProxy、Envoy、Traefik 都已经成为了K8S专门维护的Ingress Controller
一个Ingress对象的主要内容，就类似Nginx的配置文件描述，对应的转发规则就是ingressRule，
有了Ingress这个对象，用户就可以根据自己的需求选择Ingress Controller，例如，如果应用对代理服务的中断超级敏感，可以使用Treafik这样的Ingress Controller

Ingress工作在七层，Service工作在四层，当想要在Kubernetes里为应用进行TLS配置等HTTPS相关操作时，都必须通过Ingress来进行

ingress-nginx 简单的理解就是你原来需要改 Nginx 配置，然后配置各种域名对应哪个 Service，目前把这个动作抽象出来，变成一个 Ingress 对象，你可以用 yaml 创建，每次不要去改 Nginx 了，直接改 yaml 然后创建/更新就行了；那么问题来了：”Nginx 该怎么处理？”
Ingress Controller 这东西就是解决 “Nginx 的处理方式” 的；Ingress Controoler 通过与 Kubernetes API 交互，动态的去感知集群中 Ingress 规则变化，然后读取他，按照他自己模板生成一段 Nginx 配置，再写到 Nginx Pod 里，最后 reload 一下，工作流程如下图：
实际上Ingress也是Kubernetes API的标准资源类型之一，它实则就是一组基于DNS名称（host）或URL路径把请求转发到指定的Service资源的规则。用于将集群外部的请求流量转发到集群内部完成的服务发布。我们需要清楚的是，Ingress资源自身不能进行“流量穿透”，仅仅是一组规则的集合，这些集合规则还需要其他功能的辅助，列如监听某套接字，然后根据这些规则的匹配进行路由转发，这些能够为Ingress资源监听套接字并将流量转发的组件就是Ingress Controller
Ingress 两种路由方式
1.虚拟主机
2.URL 路径

Ingree-nginx部署

ingress-nginx 许多全局配置或修改默认配置都是通过annotations 注释来加载配置文件具体参数详细解释可
参考官方文档：

https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/

选择NodePort部署方式

https://kubernetes.github.io/ingress-nginx/deploy/

extensions/v1beta1 Ingress资源规范 1.22+版本后会被彻底弃用

apiVersion: extensions/v1betal #资源所属的API群组和版本
Kind: Ingress#资源类型标识符
metadata: #元数据
  name <string> #资源名称
  annotationsl #资源注解,v1betal使用下面的注解来指定要解析该资源的控制器类型
    kubernetes.io/ingress.class: <string> #适配的Ingress控制器类别
  namespace <string> #名称空间
spec:
  rules <[]Object> #Ingress规则列表;
  - host <string>  #虚拟主机的FQDN，支持“*"前缀通配，不支持IP，不支持指定端口
    http <object>
      paths<[]0bject>#虚拟主机PATH定义的列表，由path和backend组成
      - path <string> #流量匹配的HTTP PATH，必须以/开头
        pathType <string> #匹配机制，支持Exact、_Prefix和ImplementationSpecific
        backend <object> #匹配到的流量转发到的目标后端
          resource <Object> #引用的同一名称空间下的资源，与下面两个字段互斥
          serviceName <string>#引用的Service资源的名称
          servicePort <string># Service用于提供服务的端口
  tls <[]object> #TLS配置，用于指定上rules中定义的哪些host需要工作HTTPS模式
  - hosts <[]string>  #使用同一组证书的主机名称列表
    secretName <string>  #保存于数字证书和私钥信息的secret资源名称
  backend <object>   #默认backend的定义，可嵌套字段及使用格式跟rules字段中的一样
  ingressClassName <string> #ingress类名称，用于指定适配的控制器

v1 Ingress资源规范

apiVersion: networking.k8s.io/v1 #资源所属的API群组和版本
kind: Ingress #资源类型标识符
metadata: #元数据
  name <string> #资源名称
  annotations: #资源注解,vlbetal使用下面的注解来指定要解析该资源的控制器类型
    kubernetes.io/ingress.class:<string> #适配的Ingress控制器类别
  namespace <string>  #名称空间
spec:
  rules <[]object> #Ingress规则列表
  - host <string>  #虚拟主机的FQDN，支持“*"前缀通配，不支持IP，不支持指定端口
    http <object>
      paths <[]object>#虚拟主机PATH定义的列表，由path和backend组成
      - path <string> #流量匹配的HTTP PATH，必须以/开头
        pathType <string> #支持Exact、Prefix和ImplementationSpecific，必选
        backend <Object>#匹配到的流量转发到的目标后端
          resource <object>  #引用的同一名称空间下的资源，与下面两个字段互斥
          service <object> #关联的后端Service对象
            name <string> #后端Service的名称
            port bject> #后端Service上的端口对象
              name <string> #端口名称
              number <integer>  #端口号
  tls <[]object>  #TLS配置，用于指定上rules中定义的哪些host需要工作HTTPS模式
  - hosts <[]string>  #使用同一组证书的主机名称列表
    secretName <string> #保存于数字证书和私钥信息的secret资源名称
  backend <object>  #默认backend的定义，可嵌套字段及使用格式跟rules字段中的一样
  ingressClassName <string> #ingress类名称，用于指定适配的控制器

添加externalIPs:节点3 IP非必须步骤，方便记忆使用而不是记忆NodePort端口 NodePort与externalIPs可同时访问

[root@k8s-master Ingress]# vim deploy.yaml
...
---
# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    helm.sh/chart: ingress-nginx-4.0.1
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.0.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  type: NodePort
  externalIPs: [192.168.54.173]   #添加externalIPs字段 固定访问IP
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
      appProtocol: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
      appProtocol: https
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: controller
...

部署Ingress-nginx

[root@k8s-master Ingress]# kubectl apply -f deploy.yaml 
namespace/ingress-nginx unchanged
serviceaccount/ingress-nginx unchanged
configmap/ingress-nginx-controller configured
clusterrole.rbac.authorization.k8s.io/ingress-nginx unchanged
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx unchanged
role.rbac.authorization.k8s.io/ingress-nginx unchanged
rolebinding.rbac.authorization.k8s.io/ingress-nginx unchanged
service/ingress-nginx-controller-admission unchanged
service/ingress-nginx-controller configured
deployment.apps/ingress-nginx-controller configured
ingressclass.networking.k8s.io/nginx unchanged
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission configured
serviceaccount/ingress-nginx-admission unchanged
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
role.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
job.batch/ingress-nginx-admission-create unchanged
job.batch/ingress-nginx-admission-patch unchanged
[root@k8s-master Ingress]# kubectl get pod
NAME                              READY   STATUS    RESTARTS   AGE
etcd-operator-646cbffdb6-brbn6    1/1     Running   0          19h
example-etcd-cluster-5fb5d9d6n8   1/1     Running   0          49m
example-etcd-cluster-nc8pdgjrjr   1/1     Running   0          19h
example-etcd-cluster-svgdngq28k   1/1     Running   0          48m
[root@k8s-master Ingress]# kubectl get svc -n ingress-nginx
NAME                                 TYPE        CLUSTER-IP     EXTERNAL-IP      PORT(S)                      AGE
ingress-nginx-controller             NodePort    10.109.24.78   192.168.54.173   80:32493/TCP,443:30239/TCP   120m
ingress-nginx-controller-admission   ClusterIP   10.110.72.52   <none>           443/TCP                      120m

示例1：创建 Ingress-nginx虚拟主机

创建 Deployment 和与之对应的SVC

[root@k8s-master Ingress]# cat deployment-demo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-demo
  namespace: default
spec:
  replicas: 4
  selector:
    matchLabels:
      app: demoapp
      release: stable
  template:
    metadata:
      labels :
        app: demoapp
        release: stable
    spec:
      containers:
      - name: demoapp
        image: ikubernetes/demoapp:v1.1
        ports:
        - containerPort: 80
          name: http

---
apiVersion: v1
kind: Service
metadata:
  name: demoapp-deploy
  namespace: default
spec:
  selector:
    app: demoapp
    release: stable
  ports:
  - name: http
    port: 80
    targetPort: 80

创建 ingress-nginx

[root@k8s-master Ingress]# cat  demoapp-ingress.yaml 
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-demo
  annotations:
    kubernetes.io/ingress.class: "nginx"
  namespace: default
spec:
  rules:
  - host: www.ik8s.io  #虚拟主机
    http:
      paths:
      - path: /
        pathType: Prefix  #前缀匹配
        backend:
          service:
            name: demoapp-deploy
            port:
              number: 80

[root@k8s-master Ingress]# kubectl apply -f deployment-demo.yaml
[root@k8s-master Ingress]# kubectl apply -f demoapp-ingress.yaml

[root@k8s-master Ingress]# kubectl get svc -n ingress-nginx
NAME                                 TYPE        CLUSTER-IP     EXTERNAL-IP      PORT(S)                      AGE
ingress-nginx-controller             NodePort    10.109.24.78   192.168.54.173   80:32493/TCP,443:30239/TCP   5h50m
ingress-nginx-controller-admission   ClusterIP   10.110.72.52   <none>           443/TCP                      5h50m

[root@k8s-master Ingress]# kubectl get ingress
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAME           CLASS    HOSTS         ADDRESS         PORTS   AGE
ingress-demo   <none>   www.ik8s.io   192.168.4.171   80      3h11m

访问测试

[root@bigyong ~]# cat /etc/hosts   
# ::1       localhost localhost.localdomain localhost6 localhost6.localdomain6
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

...
192.168.54.173   www.ik8s.io   #写hosts

[root@bigyong ~]# curl 192.168.54.173  #直接访问只能到ingress-nginx 没到转发到后端
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
[root@bigyong ~]# curl -H  "Host:www.ik8s.io" 192.168.54.173  #访问成功
iKubernetes demoapp v1.1 !! ClientIP: 192.168.113.37, ServerName: deployment-demo-867c7d9d55-gw6qp, ServerIP: 192.168.113.39
[root@bigyong ~]# curl -H  "Host:www.ik8s.io" 192.168.54.173
iKubernetes demoapp v1.1 !! ClientIP: 192.168.113.37, ServerName: deployment-demo-867c7d9d55-9lnpq, ServerIP: 192.168.12.39!
[root@bigyong ~]# curl -H  "Host:www.ik8s.io" 192.168.54.173
iKubernetes demoapp v1.1 !! ClientIP: 192.168.113.37, ServerName: deployment-demo-867c7d9d55-2zcr5, ServerIP: 192.168.51.61!
[root@bigyong ~]# curl -H  "Host:www.ik8s.io" 192.168.54.173
iKubernetes demoapp v1.1 !! ClientIP: 192.168.113.37, ServerName: deployment-demo-867c7d9d55-9lnpq, ServerIP: 192.168.12.39!

示例2：创建TLS Ingress HTTPS

[root@k8s-master Ingress]# cat   demoapp-ingress.yaml 
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-demo
  annotations:
    kubernetes.io/ingress.class: "nginx"
  namespace: default
spec:
  rules:
  - host: www.ik8s.io
    http:
      paths:
      - path: /
        pathType: Prefix  #前缀匹配
        backend:
          service:
            name: demoapp-deploy
            port:
              number: 80
  tls:  #添加tls
  - hosts:
    - www.ik8s.io
    secretName: ik8s-tls
    
[root@k8s-master Ingress]# kubectl  apply -f demoapp-ingress.yaml 
ingress.networking.k8s.io/ingress-demo configured

[root@k8s-master Ingress]# kubectl describe ingress ingress-demo
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
Name:             ingress-demo
Namespace:        default
Address:          192.168.4.171
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
  ik8s-tls terminates www.ik8s.io
Rules:
  Host         Path  Backends
  ----         ----  --------
  www.ik8s.io  
               /   demoapp-deploy:80 (192.168.113.39:80,192.168.12.39:80)
Annotations:   kubernetes.io/ingress.class: nginx
Events:
  Type    Reason  Age                   From                      Message
  ----    ------  ----                  ----                      -------
  Normal  Sync    113s (x3 over 6h36m)  nginx-ingress-controller  Scheduled for sync

创建tls自签证书

[root@k8s-master Ingress]# (umask 077); openssl  genrsa -out tls.key 2048
Generating RSA private key, 2048 bit long modulus
............................+++
.......................+++
e is 65537 (0x10001)
[root@k8s-master Ingress]# openssl req -new -x509 -key tls.key -out tls.crt -subj "/CN=www.ik8s.io" -days 365
[root@k8s-master Ingress]# ls
demoapp-ingress.yaml    deployment-demo.yaml  tls.crt
deploy-externalIP.yaml  deploy.yaml           tls.key

创建Secret

[root@k8s-master Ingress]# kubectl create secret tls ik8s-tls --cert=./tls.crt --key=./tls.key
secret/ik8s-tls created

访问测试

[root@k8s-master Ingress]# curl -H  "Host:www.ik8s.io" 192.168.54.173   #308已经被重定向
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>

#通过https访问  提示证书无效不被信任
[root@k8s-master Ingress]# curl -H  "Host:www.ik8s.io" https://192.168.54.173 
curl: (60) Issuer certificate is invalid.
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn t adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you d like to turn off curl s verification of the certificate, use
 the -k (or --insecure) option.

忽略风险访问成功

[root@k8s-master Ingress]# curl -k -H  "Host:www.ik8s.io" https://192.168.54.173  
iKubernetes demoapp v1.1 !! ClientIP: 192.168.113.37, ServerName: deployment-demo-867c7d9d55-9lnpq, ServerIP: 192.168.12.39!

示例3：为dashboard 添加ingress

https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/

[root@k8s-master Ingress]# cat ingress-kubernetes-dashboard.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: dashboard
  annotations:
    kubernetes.io/ingress.class: "nginx" #指定控制器
    ingress.kubernetes.io/ssl-passthrough: "true"   #tcp代理 由于后端dashboard必须通过https访问  这里通过4层转发直接转发到后端Pod
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" #后端直接通过https协议访问
    nginx.ingress.kubernetes.io/rewrite-target: /$2 #重写标记 path:/dashboard(/|$)(.*) 这里的第2部分
  namespace: kubernetes-dashboard
spec:  #这里没有配置host 相当于*号通配所有主机 
  rules:
  - http:
      paths:
      - path: /dashboard(/|$)(.*)
        backend:
          serviceName: kubernetes-dashboard
          servicePort: 443

[root@k8s-master Ingress]# kubectl apply -f ingress-kubernetes-dashboard.yaml 
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
ingress.extensions/dashboard created

[root@k8s-master Ingress]# kubectl describe ingress dashboard -n kubernetes-dashboard
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
Name:             dashboard
Namespace:        kubernetes-dashboard
Address:          192.168.4.171
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           
              /dashboard(/|$)(.*)   kubernetes-dashboard:443 (192.168.51.64:8443)  #后端pod
Annotations:  ingress.kubernetes.io/ssl-passthrough: true
              kubernetes.io/ingress.class: nginx
              nginx.ingress.kubernetes.io/backend-protocol: HTTPS
              nginx.ingress.kubernetes.io/rewrite-target: /$2
Events:
  Type    Reason  Age                   From                      Message
  ----    ------  ----                  ----                      -------
  Normal  Sync    103s (x2 over 2m41s)  nginx-ingress-controller  Scheduled for sync

访问测试

注意：https://192.168.54.173/dashboard/ 必定要带上最后的/ $2为/后面部分不然无法访问

26.kubernetes(k8s)笔记 Ingress(一) Ingress-nginx

示例4：longhorn添加ingress-nginx basic认证

参考官方文档

https://longhorn.io/docs/1.2.0/deploy/accessing-the-ui/longhorn-ingress/

默认longhorn没有登录认证暴露到公网会存在风险第一添加basic认证

[root@k8s-master Ingress]# kubectl get pod -n longhorn-system 
NAME                                        READY   STATUS    RESTARTS   AGE
....
longhorn-manager-cc8sp                      1/1     Running   0          149m
longhorn-manager-fs5tx                      1/1     Running   2          149m
longhorn-manager-vwbzn                      1/1     Running   1          149m
longhorn-ui-79f8976fbf-c44ct                1/1     Running   1          149m
[root@k8s-master Ingress]# kubectl get svc  -n longhorn-system 
NAME                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)     AGE
csi-attacher        ClusterIP   10.108.190.205   <none>        12345/TCP   142m
csi-provisioner     ClusterIP   10.99.216.181    <none>        12345/TCP   142m
csi-resizer         ClusterIP   10.96.161.192    <none>        12345/TCP   141m
csi-snapshotter     ClusterIP   10.101.216.72    <none>        12345/TCP   141m
longhorn-backend    ClusterIP   10.108.67.31     <none>        9500/TCP    149m
longhorn-frontend   ClusterIP   10.107.71.176    <none>        80/TCP      149m

[root@k8s-master Ingress]# USER=user.com; PASSWORD=passwd.com; echo "${USER}:$(openssl passwd -stdin -apr1 <<< ${PASSWORD})" >> auth
[root@k8s-master Ingress]# ls
auth                  deploy-externalIP.yaml  deploy.yaml                        tls.crt
demoapp-ingress.yaml  deployment-demo.yaml    ingress-kubernetes-dashboard.yaml  tls.key

[root@k8s-master Ingress]# kubectl -n longhorn-system create secret generic basic-auth --from-file=authsecret/basic-auth created

[root@k8s-master Ingress]# cat longhorn-ui-ingress.yaml 
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: longhorn-ingress
  namespace: longhorn-system
  annotations:
    kubernetes.io/ingress.class: "nginx"   #指定控制器
    # type of authentication
    nginx.ingress.kubernetes.io/auth-type: basic  #认证类型
    # prevent the controller from redirecting (308) to HTTPS
    nginx.ingress.kubernetes.io/ssl-redirect:  false      #http通信
    # name of the secret that contains the user/password definitions
    nginx.ingress.kubernetes.io/auth-secret: basic-auth    #secret名称
    # message to display with an appropriate context why the authentication is required
    nginx.ingress.kubernetes.io/auth-realm:  Authentication Required     #输入提示
    # custom max body size for file uploading like backing image uploading
    nginx.ingress.kubernetes.io/proxy-body-size: 10000m
    nginx.ingress.kubernetes.io/rewrite-target: /$2 
spec:
  rules:
  - http:
      paths:
      - pathType: Prefix
        path: /longhorn(/|$)(.*)
        backend:
          service:
            name: longhorn-frontend
            port:
              number: 80

[root@k8s-master Ingress]# kubectl apply -f longhorn-ui-ingress.yaml 
ingress.networking.k8s.io/longhorn-ingress configured