Dako's Blog 大可的博客

在GCE上面部署RocketChat的经历

Dako -

先回答几个问题

  • 为什么不用Helm chart直接部署
    • Helm chart需要用到persistent volume以及persistent volume claim，这一部分是比单纯的用hostPath复杂不少的（又一个为什么k8s非常像是给云服务商量身定制，而不是对on-premise非常友好的原因）
  • 为什么不用persistent volume和persistent volume claim
    • 我只想试用一下，暂时不想太复杂。
    • 暂时懒得学。。
    • HA什么的, reliability什么的，暂时先想开点吧。
  • 为什么不直接用GKE
    • 我想把这套东西以后host在我自家的服务器（破笔记本）上面，所以GCE+Ubuntu 20.04才是正确的选择。

Components

RocketChat requires

  • A MongoDB ReplicaSet
    • So, we have a problem, Deployment or StatefulSet
    • And we will have some issues to initialize the DB
    • And as MongoDB is an dependency of RocketChat, we need the DB to expose a service and RocketChat must be able to access it.
  • A RocketChat deployment and service
  • Ingress

步骤

Namespace

还是建立一个rocketchat namespace吧。

kubectl create namespace rocketchat

Mongo DB RBAC

这一步是因为接下来的MongoDB部分需要一个MongoDB sidecar container，这个东西需要一些访问cluster本身的一些权限。因为我enable了rbac，所以需要预先设置一下。

这里最终的效果是让 MongoDB sidecar 能够通过 system:serviceaccount:rocketchat:default 访问到cluster的pods。这个全名是deploy mongoDB sidecar的时候error log里面的。但是下文中其实也只是定义了 default，这个全名中最后的一部分，没有定义过全名。

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: mongo-sidecar
rules:
  - apiGroups:
    - ""
    resources:
      - pods
      - services
      - endpoints
    verbs:
      - get
      - list
      - watch
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: default
  namespace: rocketchat
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: mongo-sidecar
subjects:
  - kind: ServiceAccount
    name: default
    namespace: rocketchat
roleRef:
  kind: ClusterRole
  name: mongo-sidecar
  apiGroup: rbac.authorization.k8s.io

Reference

Broken with kubernetes 1.6 with RBAC · Issue #44 · cvallance/mongo-k8s-sidecar

MongoDB

Should refer Kubernetes Self-Hosted for the difference between StatefulSet and Deployment

Here we are using Deployment

And we were NOT specifying a MongoDB password!

The MongoDB service is a ‘head-less’ service. So the clients of MongoDB can specify the urls to the MongoDB pods.

Refer to Kubernetes Self-Hosted for testing the DNS, so we know what URL should we use for MongoDB client.

The mongo-sidecar is quite important here. It initializes the MongoDB database, including replica sets (not the replica sets in the K8s context, just replica sets for within MongoDB), etc.

Otherwise, the RocketChat Deployment will complain replica set not ready to use.

# apiVersion: v1
# kind: Secret
# metadata:
  # namespace: rocketchat
  # name: mongo-db-cred
# type: Opaque
# # rocketchat
# data:
  # password: cm9ja2V0Y2hhdA==
---
apiVersion: apps/v1
kind: StatefulSet
# kind: Deployment
metadata:
  namespace: rocketchat
  labels:
    app: mongo-db
  name: mongo-db
spec:
  serviceName: "mongo"
  replicas: 1
  selector:
    matchLabels:
      app: mongo-db
  template:
    metadata:
      labels:
        app: mongo-db
    spec:
      nodeSelector:
        host_rocketchat: "true"
      containers:
      - name: mongo-db
        image: mongo:4.0
        command:
        - "mongod"
        - "--bind_ip_all"
        - "--smallfiles"
        - "--oplogSize"
        - "128"
        - "--replSet"
        - "rs0"
        - "--noprealloc"
        ports:
          - containerPort: 27017
        volumeMounts:
        - name: mongo-db-files
          mountPath: /data/db
        # - name: mongo-key
          # mountPath: "/etc/secrets-volume"
          # readOnly: true
      - name: mongo-sidecar
        image: cvallance/mongo-k8s-sidecar
        env:
          - name: MONGO_SIDECAR_POD_LABELS
            value: "app=mongo-db"
          - name: KUBE_NAMESPACE
            value: "rocketchat"
      restartPolicy: Always
      volumes:
      - name: mongo-db-files
        hostPath:
          path: /mongo-db-files
          type: DirectoryOrCreate
      # - name: mongo-key
        # secret:
          # defaultMode: 0400
          # secretName: mongo-db-cred
---
apiVersion: v1
kind: Service
metadata:
  namespace: rocketchat
  labels:
    app: mongo-db
  name: mongo
spec:
  ports:
  - name: "27017"
    port: 27017
    targetPort: 27017
  selector:
    app: mongo-db
  # type: ClusterIP
  clusterIP: None

Possible errors

  • Pay attention to the mongo-sidecar RBAC not setup properly, the system:serviceaccount:rocket:default can’t list resources
anthony@instance-1:~/rocket$ kubectl logs pod/mongo-db-0 mongo-sidecar -n rocketchat 

> mongo-k8s-sidecar@0.1.0 start /opt/cvallance/mongo-k8s-sidecar 
> forever src/index.js 

warn: --minUptime not set. Defaulting to: 1000ms 
warn: --spinSleepTime not set. Your script will exit if it does not stay up for at least 1000ms Using mongo port: 27017 Starting up mongo-k8s-sidecar The cluster domain 'cluster.local' was successfully verified. Error in workloop { [Error: [object Object]]
 message: { kind: 'Status', 
 apiVersion: 'v1', 
 metadata: {}, 
 status: 'Failure', 
 message: 'pods is forbidden: User "system:serviceaccount:rocketchat:default" cannot list resource "pods" in API group "" in the namespace "rocketchat"', 
 reason: 'Forbidden', 
 details: { kind: 'pods' }, code: 403 }, 
 statusCode: 403 
}

RocketChat

The service and ingress are just normal ones

Deployment 里面关于 MONGO_URL 和 MONGO_OPLOG_URL 的部分需要特别注意

OPLOG_URL需要有 ?replicaSet=rs0 这样的部分

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: rocketchat
  labels:
    app: rocketchat
  name: rocketchat
spec:
  selector:
    matchLabels:
      app: rocketchat
  replicas: 1
  template:
    metadata:
      labels:
        app: rocketchat
    spec:
      nodeSelector:
        host_rocketchat: "true"
      containers:
      - name: rocketchat
        image: rocketchat/rocket.chat:latest
        volumeMounts:
          - name: rocketchat-files
            mountPath: /app/uploads
        env:
        - name: ROOT_URL
          value: "<https://rocketchat.duckdns.org>"
        - name: MONGO_URL
          value: "mongodb://mongo-db-0.mongo.rocketchat.svc.cluster.local:27017/rocketchat"
        - name: MONGO_OPLOG_URL
          value: "mongodb://mongo-db-0.mongo.rocketchat.svc.cluster.local:27017/local?replicaSet=rs0"
        ports:
        - containerPort: 3000
      restartPolicy: Always
      volumes:
        - name: rocketchat-files
          hostPath:
            path: /rocketchat-files
            type: DirectoryOrCreate
---
apiVersion: v1
kind: Service
metadata:
  namespace: rocketchat
  labels:
    app: rocketchat
  name: rocketchat
spec:
  ports:
  - name: "3000"
    port: 3000
    targetPort: 3000
    protocol: TCP
  selector:
    app: rocketchat
  type: ClusterIP
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  namespace: rocketchat
  name: rocketchat
  annotations:
    kubernetes.io/tls-acme: "true"
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  tls:
  - hosts:
    - rocketchat.duckdns.org
    secretName: rocketchat.duckdns.org
  rules:
    - host: rocketchat.duckdns.org
      http:
        paths:
        - path: /
          backend:
            serviceName: rocketchat
            servicePort: 3000

Possible errors

  • MONGO_OPLOG_URL not set MongoError: not master and slaveOk=false reference: MongoError: not master and slaveOk=false · Issue #67 · RocketChat/Docker.Official.Image
  • MONGO_OPLOG_URL path does not end at local. rror: $MONGO_OPLOG_URL must be set to the 'local' database of a Mongo replica set Fix: check the MONGO_OPLOG_URL to be value: "mongodb://mongo-db-0.mongo.rocketchat.svc.cluster.local:27017/local?replicaSet=rs0"
  • Mongo replica set not set properly in MONGO_OPLOG_URL MongoError: no primary found in replicaset or invalid replica set name 需要在MONGO_OPLOG_URL后面加 ?replicaSet=rs0 Fix: check the MONGO_OPLOG_URL

这时应该就可以用了，访问 rocketchat.duckdns.org 应该就可以访问并进行相应的配置了。

似乎 kubectl logs 正常返回这个时候是空的