背景介紹

需要訪問ceph創(chuàng)建subvolume卷，因為在ceph quincy版本rest api不支持subvolume的創(chuàng)建（reef版本已經(jīng)支持）。所以需要引入ceph的sdk庫進行subvlume的創(chuàng)建。我選擇的是go-ceph:github.com/ceph/go-ceph

前提條件

1. 在ceph已創(chuàng)建好volume和subvolumegroup

ceph fs volume create xxx
ceph fs subvolumegroup create <vol_name> <group_name>

2. 創(chuàng)建對接賬號

ceph auth get-or-create client.test mon "allow r" mds "allow rw" osd 'allow rwx pool=cephfs.testfs.data'
#生成密鑰對
ceph auth get client.test -o ceph.client.test.keyring
ceph auth print-key client.test> test.key

創(chuàng)建subvolume

創(chuàng)建go-ceph客戶端(參考csi代碼修改)

import (
    "fmt"
    "github.com/ceph/go-ceph/rados"
)
const (
    cephConfigRoot = "/etc/ceph"
    // CephConfigPath ceph configuration file.
    CephConfigPath = "/etc/ceph/ceph.conf"
    keyRing = "/etc/ceph/keyring"
)
func GetConn(monitors, user, keyfile string) (*rados.Conn, error) {
    // construct and connect a new rados.Conn
    args := []string{"--keyfile=" + keyfile}
    conn, err := rados.NewConnWithUser(user)
    if err != nil {
        return nil, fmt.Errorf("creating a new connection failed: %w", err)
    }
    err = conn.ParseCmdLineArgs(args)
    if err != nil {
        return nil, fmt.Errorf("parsing cmdline args (%v) failed: %w", args, err)
    }
    if err = conn.ReadConfigFile(CephConfigPath); err != nil {
        return nil, fmt.Errorf("failed to read config file %q: %w", CephConfigPath, err)
    }
    err = conn.Connect()
    if err != nil {
        return nil, fmt.Errorf("connecting failed: %w", err)
    }
    return conn, nil
}
conn, err := GetConn("", "slurm", "/etc/ceph/slurm.key")

創(chuàng)建ceph fs客戶端, 并創(chuàng)建subvolume

import ca "github.com/ceph/go-ceph/cephfs/admin"

fsadmin := ca.NewFromConn(conn)
if err := fsadmin.CreateSubVolume("testfs", "data", fmt.Sprintf("mount%d", rand.Intn(100)), &ca.SubVolumeOptions{Size: 10}); err != nil {
    fmt.Printf("create subvolume err, %v", err)
    eturn err
}
return nil

編譯

因為go-ceph 引入了ceph的c語言動態(tài)鏈接庫，需要找一臺linux機器，安裝對應(yīng)的動態(tài)鏈接庫之后才能正常編譯。ceph-go中已經(jīng)做了相關(guān)說明GitHub - ceph/go-ceph: Go bindings for Ceph :octopus: :octopus:

The code in go-ceph is purely a library module. Typically, one will import go-ceph in another Go based project. When building the code the native RADOS, RBD, & CephFS library and development headers are expected to be installed.
On debian based systems (apt) these may be:

libcephfs-dev librbd-dev librados-dev
On rpm based systems (dnf, yum, etc) these may be:

libcephfs-devel librbd-devel librados-devel
On MacOS you can use brew to install the libraries:

brew tap mulbc/ceph-client
brew install ceph-client