在k8s中安装jenkins并实现动态生成 jenkins slave

安装jenkins
1、创建一个命名空间
$ kubectl create namespace kube-ops
2、为jenkins创建pvc（也可以使用存储类创建）

apiVersion: v1
kind: PersistentVolume
metadata:
  name: opspv
spec:
  capacity:
    storage: 2Gi
  accessModes:
  - ReadWriteMany
  persistentVolumeReclaimPolicy: Delete
  nfs:
    server: 192.168.1.244
    path: /data/k8s
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: opspvc
  namespace: kube-ops
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Gi

3、创建jenkins需要的rbac权限

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins2
  namespace: kube-ops
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: jenkins2
rules:
  - apiGroups: ["extensions", "apps"]
    resources: ["deployments"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get","list","watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: jenkins2
  namespace: kube-ops
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins2
subjects:
  - kind: ServiceAccount
    name: jenkins2
    namespace: kube-ops

也可以为ServiceAccoun绑定一个系统现有的 cluster-admin 集群角色权限
4、创建jenkins pod
$ docker pull docker.io/jenkins/jenkins:lts
$ docker pull cnych/jenkins:jnlp6

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: jenkins2
  namespace: kube-ops
spec:
  template:
    metadata:
      labels:
        app: jenkins2
    spec:
      terminationGracePeriodSeconds: 10
      serviceAccount: jenkins2
      containers:
      - name: jenkins
        image: docker.io/jenkins/jenkins:lts
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
          name: web
          protocol: TCP
        - containerPort: 50000
          name: agent
          protocol: TCP
        resources:
          limits:
            cpu: 1000m
            memory: 1Gi
          requests:
            cpu: 500m
            memory: 512Mi
        livenessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12
        readinessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12
        volumeMounts:
        - name: jenkinshome
          subPath: jenkins2
          mountPath: /var/jenkins_home
        env:
        - name: LIMITS_MEMORY
          valueFrom:
            resourceFieldRef:
              resource: limits.memory
              divisor: 1Mi
        - name: JAVA_OPTS
          value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvision
er.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai      
      securityContext:
        fsGroup: 1000
      volumes:
      - name: jenkinshome
        persistentVolumeClaim:
          claimName: opspvc
---
apiVersion: v1
kind: Service
metadata:
  name: jenkins2
  namespace: kube-ops
  labels:
    app: jenkins2
spec:
  selector:
    app: jenkins2
  type: NodePort
  ports:
  - name: web
    port: 8080
    targetPort: web
    nodePort: 30003
  - name: agent
    port: 50000
    targetPort: agent

$ kubectl apply -f jenkins2.yaml
$ kubectl get pod -n kube-ops
jenkins2-76644dbc9b-llcsp 0/1 Running 0 #不能正常启动
$ kubectl describe pod jenkins2-76644dbc9b-llcsp -n kube-ops
$ kubectl logs -f jenkins2-76644dbc9b-llcsp -n kube-ops
5、在nfs服务器上修改jenkins持久目录的权限并重新创建jenkins pod
$ chown -R 1000 /data/k8s/jenkins2 #在192.168.1.244上
$ kubectl delete -f jenkins2.yaml
$ kubectl apply -f jenkins2.yaml
$ kubectl get pod -n kube-ops
jenkins2-76644dbc9b-llcsp 1/1 Running 0
为什么是1000？
上述镜像的Dockerfile文件中定义的是：user=jenkins group=jenkins uid=1000 gid=1000
Dockerfile文件的地址：
https://github.com/jenkinsci/docker/blob/master/Dockerfile
ARG user=jenkins
ARG group=jenkins
ARG uid=1000
ARG gid=1000
ARG http_port=8080
ARG agent_port=50000
ARG JENKINS_HOME=/var/jenkins_home
$ kubectl get svc -n kube-ops
jenkins2 NodePort 10.105.121.176 <none> 8080:30003/TCP,50000:30936/TCP
http://192.168.1.243:30003
初始密码在nfs服务器上
$ cat /data/k8s/jenkins2/secrets/initialAdminPassword

在jenkins上创建kubernetes云
enkins Master 和 Jenkins Slave 以 Pod 形式运行在 Kubernetes 集群的 Node 上，Master 运行在其中一个节点，并且将其配置数据存储到一个 Volume 上去，Slave 运行在各个节点上，并且它不是一直处于运行状态，它会按照需求动态的创建并自动删除
这种方式的工作流程大致为：当 Jenkins Master 接受到 Build 请求时，会根据配置的 Label 动态创建一个运行在 Pod 中的 Jenkins Slave 并注册到 Master 上，当运行完 Job 后，这个 Slave 会被注销并且这个 Pod 也会自动删除，恢复到最初状态。
1、安装插件
安装kubernetes plugin， 点击 Manage Jenkins -> Manage Plugins -> Available -> Kubernetes plugin
2、增加kubernetes云
点击 Manage Jenkins —> Configure System —> (拖到最下方)Add a new cloud —> 选择 Kubernetes，然后填写 Kubernetes 和 Jenkins 配置信息----连接测试
name：kubernetes
Kubernetes 地址：https://kubernetes.default.svc.cluster.local
Kubernetes 命名空间：kube-ops
Jenkins 地址：http://jenkins2.kube-ops.svc.cluster.local:8080（jenkins2是svc）
3、添加pod模板
添加pod模板----Kubernetes Pod Template
名称：jnlp
命名空间：kube-ops
标签列表：dongyali-jnlp
4、添加容器模板
添加容器----Container Template
名称：jnlp
Docker 镜像：cnych/jenkins:jnlp6（Jenkins 版本在 2.176.x以下的镜像名字去掉6）
工作目录：/home/jenkins/agent
运行的命令：清空
命令参数：清空
5、添加两个卷
添加卷----Host Path Volume
主机路径：/var/run/docker.sock
挂载路径：/var/run/docker.sock
主机路径：/root/.kube
挂载路径：/root/.kube
6、可能需要配置ServiceAccount
$ kubectl get sa -n kube-ops
jenkins2 1 14h
点击添加卷下面的高级----Service Account----jenkins2
7、用shell测试 Kubernetes 动态生成 jenkins slave
新建任务----名字----自由风格
通用----勾选限制项目的运行节点----标签表达式：dongyali-jnlp
构建----执行shell----输入如下内容----保存立即构建

echo "测试 Kubernetes 动态生成 jenkins slave"
echo "==============docker in docker==========="
docker info
echo "=============kubectl============="
kubectl get pods

观察 Kubernetes 集群中 Pod 的变化：
$ kubectl get pod -n kube-ops
jenkins2-76644dbc9b-llcsp 1/1 Running 0 3h59m
jnlp-tl1km 1/1 Running 0 44s
当任务运行完毕，jnlp这个slave pod就会自动消失。
8、用pipeline测试 Kubernetes 动态生成 jenkins slave
新建任务----名字----流水线
在流水线脚本中输入如下内容----保存立即构建

node('dongyali-jnlp') {
    stage('Clone') {
      echo "1.Clone Stage"
    }
    stage('Test') {
      echo "2.Test Stage"
    }
    stage('Build') {
      echo "3.Build Stage"
    }
    stage('Deploy') {
      echo "4. Deploy Stage"
    }
}