Prerequisites (Before Starting)

Before integrating OpenShift Data Foundation (ODF) with an external Ceph cluster, ensure the following conditions are met:

1. OpenShift Cluster:

• A functional OpenShift Container Platform (OCP) cluster. Version should be compatible with ODF (refer to ODF documentation).
• An OCP user account with cluster-admin privileges.
• OpenShift CLI (oc) installed and configured to connect to the target OCP cluster.
• Basic understanding of OCP networking, nodes, and Operator lifecycle management.

2. External Ceph Cluster:

• A standalone, operational Ceph storage cluster. Version must be supported by ODF (check ODF docs).
• Cluster in healthy state (HEALTH_OK or explicitly acknowledge HEALTH_WARN).
• Administrative access to Ceph cluster (e.g., execute sudo ceph ... commands).
• Familiarity with Ceph concepts: Monitors, OSDs, Pools, Users, FSID, CephFS.
• (If using CephFS) Metadata Server (MDS) daemon deployed and at least one in active state.

3. Network Configuration:

• OCP worker nodes running ODF CSI pods must have network access to all Ceph Monitors on ports 6789 (v1) and 3300 (v2).
• Depending on CSI driver settings, allow worker nodes to access OSD ports typically in range 6800-7300.
• Ensure firewalls between OCP nodes and Ceph cluster are properly configured.

4. ODF Operator Availability:

• Online Environment: Access to Red Hat Operator Catalog via OpenShift OperatorHub.
• Offline/Disconnected/Custom Environment:
• ODF Operator images and dependencies mirrored to private registry (e.g., Harbor).
• CatalogSource resource configured to point to private registry (e.g., cs-redhat-operator-index).
• ImageContentSourcePolicy configured if needed for private registry image pulls.

5. Required Information:

• Have Ceph cluster FSID and Monitor node IP addresses/port numbers ready.

6. Application Image (Optional):

• If following the Tomcat example, ensure harborprod.ny.shuanghui.net/tomcat/tomcat:9.0 is available in your Harbor registry or use an alternative accessible image.

Environment Details (Key Information)

This document captures essential details for the ODF-Ceph integration deployment:

• Integration Date: YYYY-MM-DD (e.g., 2025-04-11)
• OpenShift Container Platform (OCP):
• Cluster API URL: https://api.ocp4.ny.shuanghui.net:6443
• Web Console URL: https://console-openshift-console.apps.ocp4.ny.shuanghui.net
• OCP Version: 4.17.17
• ODF Installation Namespace: openshift-storage
• External Ceph Cluster:
• Ceph Version: 18.2.4
• Cluster FSID: 60500f06-15ec-11f0-9aa8-53fa8a1db53b
• Monitor Nodes: 192.168.100.106:3300,192.168.100.17:3300,192.168.100.108:3300
• Admin Node: 192.168.100.106 (used for Ceph commands)
• Ceph Nodes: 192.168.100.106, 107, 108
• OpenShift Data Foundation (ODF):
• ODF Operator Version: 4.17.6-rhodf
• StorageSystem CR Name: ocs-external-storagesystem
• StorageCluster CR Name: ocs-storagecluster-external
• RBD StorageClass: my-external-ceph-rbd
• CephFS StorageClass: my-external-cephfs

Integration Process

1. Install ODF Operator

Steps:

1. Access OpenShift Web Console.
2. Navigate to "Operators" -> "OperatorHub".
3. Ensure correct CatalogSource is selected (e.g., cs-redhat-operator-index).
4. Search for "OpenShift Data Foundation".
5. Install Operator into openshift-storage namespace.

Verification Commands:

# Optional: Clean up old CatalogSources
oc delete catalogsource certified-operators community-operators redhat-marketplace -n openshift-marketplace
# Disable default sources
oc patch operatorhub cluster --type merge -p '{"spec":{"disableAllDefaultSources": true}}'
# Check Operator installation status
oc get csv -n openshift-storage -w
# Verify Pods are running
oc get pods -n openshift-storage -w

2. Configure ODF with External Ceph

Generate Ceph Cluster Details:

# Run on Ceph admin node
python3 ceph-external-cluster-details-exporter.py \
  --rbd-data-pool-name my-rbd-pool \
  --cephfs-filesystem-name my-cephfs \
  --cephfs-data-pool-name my-cephfs-data \
  --cephfs-metadata-pool-name my-cephfs-metadata \
  --output ceph-cluster-details.json

Upload Cluster Details:

# Upload JSON file to OpenShift console
# Verify StorageClasses are created
oc get sc

3. Test RBD Storage

Create PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-rbd-test
  namespace: test-rbd
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: my-external-ceph-rbd

Apply and Check:

oc apply -f rbd-test-pvc.yaml -n test-rbd
oc get pvc my-rbd-test -n test-rbd -w

4. Test CephFS Storage

Create PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-cephfs-test
  namespace: test-cephfs
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: my-external-cephfs

Apply and Check:

oc apply -f cephfs-test-pvc.yaml -n test-cephfs
oc get pvc my-cephfs-test -n test-cephfs -w

5. Deploy Test Application (Tomcat Example)

Prepare Tomcat Image:

# Option 1: Direct pull and push
docker pull tomcat:9.0
docker login harborprod.ny.shuanghui.net
docker tag tomcat:9.0 harborprod.ny.shuanghui.net/tomcat/tomcat:9.0
docker push harborprod.ny.shuanghui.net/tomcat/tomcat:9.0

# Option 2: Offline import
docker manifest inspect tomcat:9.0
docker pull --platform=linux/amd64 tomcat:9.0
docker save -o tomcat_9.0_amd64.tar tomcat:9.0
# Transfer to Harbor node
docker load -i tomcat_9.0_amd64.tar
docker tag tomcat:9.0 harborprod.ny.shuanghui.net/tomcat/tomcat:9.0
docker push harborprod.ny.shuanghui.net/tomcat/tomcat:9.0

Create PVCs for Tomcat:

# For logs (RBD)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: tomcat-logs-pvc
  namespace: tomcat-app
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: my-external-ceph-rbd

# For shared data (CephFS)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: tomcat-shared-pvc
  namespace: tomcat-app
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  storageClassName: my-external-cephfs

Deploy Tomcat:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-deployment
  namespace: tomcat-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: tomcat
  template:
    metadata:
      labels:
        app: tomcat
    spec:
      containers:
        - name: tomcat
          image: harborprod.ny.shuanghui.net/tomcat/tomcat:9.0
          ports:
            - containerPort: 8080
          volumeMounts:
            - name: logs-volume
              mountPath: /usr/local/tomcat/logs
            - name: shared-volume
              mountPath: /shared_data
      volumes:
        - name: logs-volume
          persistentVolumeClaim:
            claimName: tomcat-logs-pvc
        - name: shared-volume
          persistentVolumeClaim:
            claimName: tomcat-shared-pvc

Apply and Monitor:

oc apply -f tomcat-deployment.yaml -n tomcat-app
oc get pods -n tomcat-app -w

Create Service and Route:

# Service
apiVersion: v1
kind: Service
metadata:
  name: tomcat-service
  namespace: tomcat-app
spec:
  ports:
    - port: 8080
  selector:
    app: tomcat

# Route
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: tomcat-route
  namespace: tomcat-app
spec:
  to:
    kind: Service
    name: tomcat-service
  port:
    targetPort: 8080

Access Application:

oc apply -f tomcat-service.yaml -n tomcat-app
oc apply -f tomcat-route.yaml -n tomcat-app
oc get route tomcat-route -n tomcat-app

Visit the provided route URL to access the Tomcat application.