storm中基于k8s的安装示例

发布时间:2021-11-12 11:29:47 作者:小新
来源:亿速云 阅读:233

这篇文章主要介绍了storm中基于k8s的安装示例,具有一定借鉴价值,感兴趣的朋友可以参考下,希望大家阅读完这篇文章之后大有收获,下面让小编带着大家一起了解一下。

首先我们先来编写一下Dockerfile

FROM ip:80/bigdata/centos7-jdk:1.8
#config storm env
ENV STORM_USER=storm \
    STORM_INSTALL_BASE=/home/frank/platform \
    STORM_HOME=/home/frank/platform/storm \
    STORM_CONF_DIR=/home/frank/conf/storm \
    STORM_DIR=/home/frank/hard_disk/storm \
    STORM_BLOBS=/home/frank/hard_disk/storm/blobs \
    STORM_NIMBUS=/home/frank/hard_disk/storm/nimbus \
    STORM_LOG_DIR=/home/frank/logs/storm \ 
    STORM_ROOT=/home/frank \
    STORM_LOG=/home/frank/logs
ENV PATH $PATH:$STORM_HOME/bin
#add FLINK user and make dirs
RUN set -x \
    && adduser "$STORM_USER" \
    && mkdir -p "$STORM_INSTALL_BASE" "$STORM_CONF_DIR" "$STORM_LOG" "$STORM_LOG_DIR" "$STORM_BLOBS" "$STORM_NIMBUS"
#install storm libs
ADD apache-storm-1.2.2.tar.gz  $STORM_INSTALL_BASE
RUN cd $STORM_INSTALL_BASE && ln -s -f apache-storm-1.2.2 storm && mkdir -p /home/storm

RUN chown -R "$STORM_USER:$STORM_USER" "$STORM_ROOT"
COPY docker-entrypoint.sh /home/storm
RUN chmod 755 /home/storm/docker-entrypoint.sh
USER storm
ENTRYPOINT ["/home/storm/docker-entrypoint.sh"]

其中的脚本如下:

#!/bin/sh
# If unspecified, the hostname of the container is taken as the JobManager address
ACTION_CMD="$1"
source /etc/profile 
export STORM_CONF_DIR=/home/frank/conf/storm
export STORM_CONF_FILE=/home/frank/conf/storm/storm.yaml
sed -i -e "s/storm.zookeeper.servers: \[localhost\]/storm.zookeeper.servers: \[${STORM_ZK_QUORUM}\]/" /home/frank/conf/storm/storm.yaml
sed -i -e "s/storm.zookeeper.root: \"\/storm\"/storm.zookeeper.root: \/${STORM_ZK_ROOT}/" /home/frank/conf/storm/storm.yaml
# if use cluster model, pod ${JOB_CLUSTER_NAME}-0,${JOB_CLUSTER_NAME}-1 as jobmanager
if [ ${ACTION_CMD} == "cluster" ]; then
  nimbuses=(${NIMBUS_HOSTS//,/ })
  ACTION_CMD="supervisor"
  for i in ${!nimbuses[@]}
  do
      if [ "$(hostname -s)" == "${nimbuses[i]}" ]; then
          NIMBUS_ADDRESS=${NIMBUS_ADDRESS:-$(hostname -f)}
          ACTION_CMD="nimbus"
          sed -i -e "s/nimbus.seeds: \[localhost\]/nimbus.seeds: \["${NIMBUS_ADDRESS}"\]/" /home/frank/conf/storm/storm.yaml
          echo "pod hostname match nimbus config host, change action to nimbus."
      fi
  done
fi
# if ha model, replace ha configuration
if [ ${ACTION_CMD} == "help" ]; then
    echo "Usage: $(basename "$0") (cluster|nimbus|supervisor|ui|logviewer|help)"
    exit 0
elif [ ${ACTION_CMD} == "nimbus" ]; then
    echo "Starting nimbus ui logviewer supervisor"
    nohup /home/frank/platform/storm/bin/storm nimbus >> /tmp/nimbus.log  2>&1 & \
    nohup /home/frank/platform/storm/bin/storm ui >> /tmp/ui.log 2>&1 & \
    /home/frank/platform/storm/bin/storm logviewer
elif [ ${ACTION_CMD} == "supervisor" ]; then
    echo "Starting supervisor"
    /home/frank/platform/storm/bin/storm supervisor 
fi

镜像构建命令如下:

docker build --network=host -f Dockerfile -t ip:80/bigdata/storm:1.2.2 .

下面我们提供一下service的创建文件

apiVersion: v1
kind: Service
metadata:
  name: storm-cluster
  labels:
    app: storm-cluster
  namespace: storm
spec:
  clusterIP: None
  ports:
    - port: 8080
      name: ui
  selector:
    app: storm-cluster

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: storm-cluster
  namespace: storm
spec:
  selector:
    matchLabels:
      app: storm-cluster
  serviceName: storm-cluster
  replicas: 4
  podManagementPolicy: Parallel
  template:
    metadata:
      labels:
        app: storm-cluster
    spec:
      terminationGracePeriodSeconds: 2
      containers:
      - name: storm-cluster
        imagePullPolicy: Always
        image: ip:80/bigdata/storm:1.2.2
        command: ["/bin/sleep"]
        args: ["30d"]
        volumeMounts:
          - name: storm-conf
            mountPath: /home/conf/storm
          - name: storm-log
            mountPath: /home/logs/storm
        resources:
          requests:
            memory: "4096Mi"
            cpu: 4
          limits:
            memory: "4096Mi"
            cpu: 4
        env:
        - name: NIMBUS_HOSTS
          value: "storm-cluster-2,storm-cluster-3"
        - name: STORM_ZK_QUORUM
          value: "ip"
        - name: STORM_ZK_ROOT
          value: "stormNew"
        ports:
        - containerPort: 6627
          name: nimbus
        - containerPort: 8000
          name: logviewer
        - containerPort: 8080
          name: storm-ui
        - containerPort: 6700
          name: sv0
        - containerPort: 6701
          name: sv1
        - containerPort: 6702
          name: sv2
        - containerPort: 6703
          name: sv3
        - containerPort: 22
          name: ssh
      volumes:
        - name: storm-conf
          configMap:
            name: storm-conf
        - name: storm-log
          hostPath:
            path: /stormtmp
            type: Directory
service的创建脚本如下:

kubectl create -f storm-svc.yaml

statefulSet的创建脚本如下:

kubectl create -f storm-statefulSet.yaml

这个创建脚本要注意如下问题:

command:/bin/sleep

程序需要手动启动

/home/vipshop/hard_disk/storm

drwxr-xr-x 2 storm storm 4096 Jul 12 12:20 blobs
drwxr-xr-x 2 storm storm 4096 Jul 12 12:20 nimbus
drwxrwxr-x 5 storm storm 4096 Jul 29 00:00 supervisor
这个hard_disk目录需要有storm权限

注:nimbus.thrift.max_buffer_size: 10485760

这个参数需要是2的倍数

感谢你能够认真阅读完这篇文章,希望小编分享的“storm中基于k8s的安装示例”这篇文章对大家有帮助,同时也希望大家多多支持亿速云,关注亿速云行业资讯频道,更多相关知识等着你来学习!

推荐阅读:
  1. storm的基本概念安装测试
  2. Storm安装

免责声明:本站发布的内容(图片、视频和文字)以原创、转载和分享为主,文章观点不代表本网站立场,如果涉及侵权请联系站长邮箱:is@yisu.com进行举报,并提供相关证据,一经查实,将立刻删除涉嫌侵权内容。

storm k8s

上一篇:JSON stringify()的不常见用法有哪些

下一篇:Django中的unittest应用是什么

相关阅读

您好,登录后才能下订单哦!

密码登录
登录注册
其他方式登录
点击 登录注册 即表示同意《亿速云用户服务条款》