If you follow best practices for deployment and maintenance, Ceph becomes a much easier beast to tame and operate. Here’s a look at some of the most fundamental and useful Ceph commands we use on a day to day basis to manage our own internal Ceph clusters, and those of our customers.

1. status

First and foremost is ceph -s, or ceph status, which is typically the first command you’ll want to run on any Ceph cluster. The output consolidates many other command outputs into one single pane of glass that provides an instant view into cluster health, size, usage, activity, and any immediate issues that may be occuring.

HEALTH_OK is the one to look for; it’s an immediate sign that you can sleep at night, as opposed to HEALTH_WARN or HEALTH_ERR, which could indicate drive or node failure or worse.

Other key things to look for are how many OSDs you have in vs out, how many other services you have running, such as rgw or cephfs, and how they’re doing.

$ ceph -s
  cluster:
    id:     7c9d43ce-c945-449a-8a66-5f1407c7e47f
    health: HEALTH_OK
  services:
    mon: 1 daemons, quorum danny-mon (age 2h)
    mgr: danny-mon(active, since 2h)
    osd: 36 osds: 36 up (since 2h), 36 in (since 2h)
    rgw: 1 daemon active (danny-mgr)

  task status:

  data:
    pools:   6 pools, 2208 pgs
    objects: 187 objects, 1.2 KiB
    usage:   2.3 TiB used, 327 TiB / 330 TiB avail
    pgs:     2208 active+clean

2. osd tree

Next up is ceph osd tree, which provides a list of every OSD and also the class, weight, status, which node it’s in, and any reweight or priority. In the case of an OSD failure this is the first place you’ll want to look, as if you need to look at OSD logs or local node failure, this will send you in the right direction. OSDs are typically weighted against each other based on size, so a 1TB OSD will have twice the weight of a 500GB OSD, in order to ensure that the cluster is filling up the OSDs at an equal rate.

If there’s an issue with a particular OSD in your tree, or you are running a very large cluster and want to quickly check a single OSD’s details without grep-ing or scrolling through a wall of text first, you can also use osd find. This command will enable you to identify an OSD’s IP address, rack location and more with a single command.

$ ceph osd tree
ID CLASS WEIGHT    TYPE NAME        STATUS REWEIGHT PRI-AFF
-1       329.69476 root default
-3       109.89825     host danny-1
  0   hdd   9.15819         osd.0        up  1.00000 1.00000
  1   hdd   9.15819         osd.1        up  1.00000 1.00000
  2   hdd   9.15819         osd.2        up  1.00000 1.00000
  3   hdd   9.15819         osd.3        up  1.00000 1.00000
  4   hdd   9.15819         osd.4        up  1.00000 1.00000
  5   hdd   9.15819         osd.5        up  1.00000 1.00000
  6   hdd   9.15819         osd.6        up  1.00000 1.00000
-7       109.89825     host danny-2
12   hdd   9.15819         osd.12       up  1.00000 1.00000
13   hdd   9.15819         osd.13       up  1.00000 1.00000
14   hdd   9.15819         osd.14       up  1.00000 1.00000
15   hdd   9.15819         osd.15       up  1.00000 1.00000
16   hdd   9.15819         osd.16       up  1.00000 1.00000
17   hdd   9.15819         osd.17       up  1.00000 1.00000
-5       109.89825     host danny-3
24   hdd   9.15819         osd.24       up  1.00000 1.00000
25   hdd   9.15819         osd.25       up  1.00000 1.00000
26   hdd   9.15819         osd.26       up  1.00000 1.00000
27   hdd   9.15819         osd.27       up  1.00000 1.00000
28   hdd   9.15819         osd.28       up  1.00000 1.00000

$ ceph osd find 37
{
    "osd": 37,
    "ip": "172.16.4.68:6804/636",
    "crush_location": {
        "datacenter": "pa2.ssdr",
        "host": "lxc-ceph-main-front-osd-03.ssdr",
        "physical-host": "store-front-03.ssdr",
        "rack": "pa2-104.ssdr",
        "root": "ssdr"
    }
}

3. df

Similar to the *nix df command, that tells us how much space is free on most unix and linux systems, Ceph has its own df command, ceph df, which provides an overview and breakdown of the amount of storage we have in our cluster, how much is used vs how much is available, and how that breaks down across our pools and storage classes.

Filling a cluster to the brim is a very bad idea with Ceph – you should add more storage well before you get to the 90% mark, and ensure that you add it in a sensible way to allow for redistribution. This is particularly important if your cluster has lots of client activity on a regular basis.

$ ceph df
RAW STORAGE:
    CLASS     SIZE        AVAIL       USED        RAW USED     %RAW USED
    hdd       330 TiB     327 TiB     2.3 TiB      2.3 TiB          0.69
    TOTAL     330 TiB     327 TiB     2.3 TiB      2.3 TiB          0.69

POOLS:
    POOL                          ID     PGS     STORED    OBJECTS   USED        %USED   MAX AVAIL
    .rgw.root                      1       32    1.2 KiB         4   768 KiB       0     104 TiB
    default.rgw.control            2       32        0 B         8       0 B       0     104 TiB
    default.rgw.meta               3       32        0 B         0       0 B       0     104 TiB
    default.rgw.log                4       32        0 B       175       0 B       0     104 TiB
    default.rgw.buckets.index      5       32        0 B         0       0 B       0     104 TiB
    default.rgw.buckets.data       6     2048        0 B         0       0 B       0     104 TiB

4. osd pool ls detail

This is a useful one for getting a quick view of pools, but with a lot more information about their particular configuration. Ideally we need to know if a pool is erasure coded or triple-replicated, what crush rule we have in place, what the min_size is, how many placement groups are in a pool, and what application we’re using this particular pool for.

$ ceph osd pool ls detail
pool 1 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 64 flags hashpspool stripe_width 0 application rgw
pool 2 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 68 flags hashpspool stripe_width 0 application rgw
pool 3 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 73 flags hashpspool stripe_width 0 application rgw
pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 71 flags hashpspool stripe_width 0 application rgw
pool 5 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 76 flags hashpspool stripe_width 0 application rgw
pool 6 'default.rgw.buckets.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 2048 pgp_num 2048 autoscale_mode warn last_change 83 lfor 0/0/81 flags hashpspool stripe_width 0 application rgw

5. osd crush rule dump

At the heart of any Ceph cluster are the CRUSH rules. CRUSH is Ceph’s placement algorithm, and the rules help us define how we want to place data across the cluster – be it drives, nodes, racks, datacentres. For example if we need to mandate that we need at least one copy of data at each one of our sites for our image store, we’d assign a CRUSH rule to our image store pool that mandated that behaviour, regardless of how many nodes we may have on each side.

crush rule dump is a good way to quickly get a list of our crush rules and how we’ve defined them in the cluster. If we want to then make changes, we have a whole host of crush commands we can use to make modifications, or we can download and decompile the crush map to manually edit it, recompile it and push it back up to our cluster.

$ ceph osd crush rule dump
[
    {
        "rule_id": 0,
        "rule_name": "replicated_rule",
        "ruleset": 0,
        "type": 1,
        "min_size": 1,
        "max_size": 10,
        "steps": [
            {
                "op": "take",
                "item": -1,
                "item_name": "default"
            },
            {
                "op": "chooseleaf_firstn",
                "num": 0,
                "type": "host"
            },
            {
                "op": "emit"
            }
        ]
    }
]

6. versions

With a distributed cluster running in production, upgrading everything at once and praying for the best is clearly not the best approach. For this reason, each cluster-wide daemon in Ceph has its own version and can be upgraded independently. This means that we can upgrade daemons on a gradual basis and bring our cluster up to date with little or no disruption to service.

As long as we keep our versions somewhat close to one another, daemons with differing versions will work alongside each other perfectly happily. This does mean that we could potentially have hundreds of different daemons and respective versions to manage during an upgrade process. Enter ceph versions – a very easy way to get a look at how many instances of a daemon running a specific version are running.

$ ceph versions
{
    "mon": {
        "ceph version 14.2.15-2-g7407245e7b (7407245e7b329ac9d475f61e2cbf9f8c616505d6) nautilus (stable)": 1
    },
    "mgr": {
        "ceph version 14.2.15-2-g7407245e7b (7407245e7b329ac9d475f61e2cbf9f8c616505d6) nautilus (stable)": 1
    },
    "osd": {
        "ceph version 14.2.15-2-g7407245e7b (7407245e7b329ac9d475f61e2cbf9f8c616505d6) nautilus (stable)": 36
    },
    "mds": {},
    "rgw": {
        "ceph version 14.2.15-2-g7407245e7b (7407245e7b329ac9d475f61e2cbf9f8c616505d6) nautilus (stable)": 1
    },
    "overall": {
        "ceph version 14.2.15-2-g7407245e7b (7407245e7b329ac9d475f61e2cbf9f8c616505d6) nautilus (stable)": 39
    }
}

7. auth print-key

If we have lots of different clients using our cluster, we’ll need to get our keys off the cluster so they can authenticate. ceph auth print-key is a pretty handy way of quickly viewing any key, rather than fishing through configuration files. Another useful and related command is ceph auth list, which will show us a full list of all the authentication keys across the cluster for both clients and daemons, and what their respective capabilities are.

$ ceph auth print-key client.admin
AQDgrLhg3qY1ChAAzzZPHCw2tYz/o+2RkpaSIg==d

8. crash ls

Daemon crashed? There could be all sorts of reasons why this may have happened, but ceph crash ls is the first place we want to look. We’ll get an idea of what’s crashed and where, so we’ll be able to diagnose further. Often these will be minor warnings or easy to address errors, but crashes can also indicate more serious problems. Related useful commands are ceph crash info <id>, which gives more info on the crash ID in question, and ceph crash archive-all, which will archive all of our crashes if they’re warnings we’re not worried about, or issues that we’ve already dealt with.

$ ceph crash ls
1 daemons have recently crashed
osd.9 crashed on host danny-1 at 2021-03-06 07:28:12.665310Z

9. osd flags

There are a number of OSD flags that are incredibly useful. For a full list, see OSDMAP_FLAGS, but the most common ones are:

  • pauserd, pausewr – Read and Write requests will no longer be answered.
  • noout – Ceph won’t consider OSDs as out of the cluster in case the daemon fails for some reason.
  • nobackfill, norecover, norebalance – Recovery and rebalancing is disabled

We can see how to set these flags below with the ceph osd set command, and also how this impacts our health messaging. Another useful and related command is the ability to take out multiple OSDs with a simple bash expansion.

$ ceph osd out {7..11}
marked out osd.7\. marked out osd.8\. marked out osd.9\. marked out osd.10\. marked out osd.11.
$ ceph osd set noout
noout is set
$ ceph osd set nobackfill
nobackfill is set
$ ceph osd set norecover
norecover is set
$ ceph osd set norebalance
norebalance is set
$ ceph osd set nodown
nodown is set
$ ceph osd set pause
pauserd,pausewr is set
$ ceph health detail
HEALTH_WARN pauserd,pausewr,nodown,noout,nobackfill,norebalance,norecover flag(s) set
OSDMAP_FLAGS pauserd,pausewr,nodown,noout,nobackfill,norebalance,norecover flag(s) set

1. pg dump

All data is placed into Ceph, which provides an abstraction layer – a bit like data buckets (not S3 buckets) – for our storage, and allows the cluster to easily decide how to distribute data and best react to failures. It’s often useful to get a granular look at how our placement groups are mapped across our OSDs, or the other way around. We can do both with pg dump, and while many of the placement group commands can be very verbose and difficult to read, ceph pg dump osds does a good job of distilling this into a single pane.

$ ceph pg dump osds
dumped osds
OSD_STAT USED    AVAIL   USED_RAW TOTAL   HB_PEERS                                                                          PG_SUM PRIMARY_PG_SUM
31        70 GiB 9.1 TiB   71 GiB 9.2 TiB                     [0,1,2,3,4,5,6,8,9,12,13,14,15,16,17,18,19,20,21,22,23,30,32]    175             72
13        70 GiB 9.1 TiB   71 GiB 9.2 TiB             [0,1,2,3,4,5,6,7,8,9,10,11,12,14,24,25,26,27,28,29,30,31,32,33,34,35]    185             66
25        77 GiB 9.1 TiB   78 GiB 9.2 TiB                         [0,1,2,3,4,5,6,12,13,14,15,16,17,18,19,20,21,22,23,24,26]    180             64
32        83 GiB 9.1 TiB   84 GiB 9.2 TiB                       [0,1,2,3,4,5,6,7,12,13,14,15,16,17,18,19,20,21,22,23,31,33]    181             73
23       102 GiB 9.1 TiB  103 GiB 9.2 TiB                [0,1,2,3,4,5,6,7,8,9,10,11,22,24,25,26,27,28,29,30,31,32,33,34,35]    191             69
18        77 GiB 9.1 TiB   78 GiB 9.2 TiB             [0,1,2,3,4,5,6,7,8,9,10,11,17,19,24,25,26,27,28,29,30,31,32,33,34,35]    188             67
11        64 GiB 9.1 TiB   65 GiB 9.2 TiB                                                   [10,12,21,28,29,31,32,33,34,35]      0              0
8         90 GiB 9.1 TiB   91 GiB 9.2 TiB                                                       [1,2,7,9,14,15,21,27,30,33]      2              0
14        70 GiB 9.1 TiB   71 GiB 9.2 TiB             [0,1,2,3,4,5,6,7,8,9,10,11,13,15,24,25,26,27,28,29,30,31,32,33,34,35]    177             64
33        77 GiB 9.1 TiB   78 GiB 9.2 TiB                         [0,1,2,3,4,5,6,12,13,14,15,16,17,18,19,20,21,22,23,32,34]    187             80
3         89 GiB 9.1 TiB   90 GiB 9.2 TiB   [2,4,8,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35]    303             74
30        77 GiB 9.1 TiB   78 GiB 9.2 TiB                       [0,1,2,3,4,5,6,9,12,13,14,15,16,17,18,19,20,21,22,23,29,31]    179             76
15        71 GiB 9.1 TiB   72 GiB 9.2 TiB               [0,1,2,3,4,5,6,7,8,10,11,14,16,24,25,26,27,28,29,30,31,32,33,34,35]    178             72
7         70 GiB 9.1 TiB   71 GiB 9.2 TiB                                                     [6,8,15,17,30,31,32,33,34,35]      0              0
28        90 GiB 9.1 TiB   91 GiB 9.2 TiB                     [0,1,2,3,4,5,6,7,9,12,13,14,15,16,17,18,19,20,21,22,23,27,29]    188             73
16        77 GiB 9.1 TiB   78 GiB 9.2 TiB             [0,1,2,3,4,5,6,7,8,9,10,11,15,17,24,25,26,27,28,29,30,31,32,33,34,35]    183             66
1         77 GiB 9.1 TiB   78 GiB 9.2 TiB [0,2,8,9,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35]    324             70
26        77 GiB 9.1 TiB   78 GiB 9.2 TiB                         [0,1,2,3,4,5,6,12,13,14,15,16,17,18,19,20,21,22,23,25,27]    186             61
22        89 GiB 9.1 TiB   90 GiB 9.2 TiB                [0,1,2,3,4,5,6,7,8,9,11,21,23,24,25,26,27,28,29,30,31,32,33,34,35]    178             80
0        103 GiB 9.1 TiB  104 GiB 9.2 TiB       [1,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35]    308             83
5         70 GiB 9.1 TiB   71 GiB 9.2 TiB     [4,6,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35]    312             69
21        77 GiB 9.1 TiB   78 GiB 9.2 TiB             [0,1,2,3,4,5,6,7,8,9,10,11,20,22,24,25,26,27,28,29,30,31,32,33,34,35]    187             63
4         96 GiB 9.1 TiB   97 GiB 9.2 TiB  [3,5,10,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35]    305             77
34        96 GiB 9.1 TiB   97 GiB 9.2 TiB                     [0,1,2,3,4,5,6,8,9,12,13,14,15,16,17,18,19,20,21,22,23,33,35]    189             73
17        96 GiB 9.1 TiB   97 GiB 9.2 TiB             [0,1,2,3,4,5,6,7,8,9,10,11,16,18,24,25,26,27,28,29,30,31,32,33,34,35]    185             72
24        77 GiB 9.1 TiB   78 GiB 9.2 TiB                         [0,1,2,3,4,5,6,10,12,13,14,15,16,17,18,19,20,21,22,23,25]    186             73
10        76 GiB 9.1 TiB   77 GiB 9.2 TiB                                                     [4,9,11,15,17,18,25,29,34,35]      1              0
27        89 GiB 9.1 TiB   90 GiB 9.2 TiB                      [0,1,2,3,4,5,6,10,12,13,14,15,16,17,18,19,20,21,22,23,26,28]    185             75
2         77 GiB 9.1 TiB   78 GiB 9.2 TiB   [1,3,8,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35]    310             62
19        77 GiB 9.1 TiB   78 GiB 9.2 TiB             [0,1,2,3,4,5,6,7,8,9,10,11,18,20,24,25,26,27,28,29,30,31,32,33,34,35]    184             77
20        77 GiB 9.1 TiB   78 GiB 9.2 TiB             [0,1,2,3,4,5,6,7,8,9,10,11,19,21,24,25,26,27,28,29,30,31,32,33,34,35]    183             69
35        96 GiB 9.1 TiB   97 GiB 9.2 TiB                            [0,1,2,3,4,5,6,12,13,14,15,16,17,18,19,20,21,22,23,34]    187             78
9         77 GiB 9.1 TiB   78 GiB 9.2 TiB                                                     [1,8,10,12,13,16,21,23,32,35]      1              0
6         83 GiB 9.1 TiB   84 GiB 9.2 TiB     [5,7,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35]    323             58
12        89 GiB 9.1 TiB   90 GiB 9.2 TiB                  [0,1,2,3,4,5,6,8,9,10,11,13,24,25,26,27,28,29,30,31,32,33,34,35]    189             78
29        64 GiB 9.1 TiB   65 GiB 9.2 TiB                       [0,1,2,3,4,5,6,9,12,13,14,15,16,17,18,19,20,21,22,23,28,30]    185             74
sum      2.8 TiB 327 TiB  2.9 TiB 330 TiB

With these essential commands, you’re well-equipped to handle daily Ceph cluster management.

Just as kids learn how to add, subtract, divide and multiply on paper before being given the convenience of a calculator, it’s important for any Ceph administrator to understand these critical Ceph commands. But once they’re under your belt, then why not make cluster management even simpler and/or delegate simple management tasks to those less well versed in the team, with our robust private cloud platform, HyperCloud?