summaryrefslogtreecommitdiff
path: root/doc/guides/platform/octeontx2.rst
blob: 97c631f3bec6acfbda3e328cedd28cada9568795 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
..  SPDX-License-Identifier: BSD-3-Clause
    Copyright(c) 2019 Marvell International Ltd.

Marvell OCTEON TX2 Platform Guide
=================================

This document gives an overview of **Marvell OCTEON TX2** RVU H/W block,
packet flow and procedure to build DPDK on OCTEON TX2 platform.

More information about OCTEON TX2 SoC can be found at `Marvell Official Website
<https://www.marvell.com/embedded-processors/infrastructure-processors/>`_.

Supported OCTEON TX2 SoCs
-------------------------

- CN96xx
- CN93xx

OCTEON TX2 Resource Virtualization Unit architecture
----------------------------------------------------

The :numref:`figure_octeontx2_resource_virtualization` diagram depicts the
RVU architecture and a resource provisioning example.

.. _figure_octeontx2_resource_virtualization:

.. figure:: img/octeontx2_resource_virtualization.*

    OCTEON TX2 Resource virtualization architecture and provisioning example


Resource Virtualization Unit (RVU) on Marvell's OCTEON TX2 SoC maps HW
resources belonging to the network, crypto and other functional blocks onto
PCI-compatible physical and virtual functions.

Each functional block has multiple local functions (LFs) for
provisioning to different PCIe devices. RVU supports multiple PCIe SRIOV
physical functions (PFs) and virtual functions (VFs).

The :numref:`table_octeontx2_rvu_dpdk_mapping` shows the various local
functions (LFs) provided by the RVU and its functional mapping to
DPDK subsystem.

.. _table_octeontx2_rvu_dpdk_mapping:

.. table:: RVU managed functional blocks and its mapping to DPDK subsystem

   +---+-----+--------------------------------------------------------------+
   | # | LF  | DPDK subsystem mapping                                       |
   +===+=====+==============================================================+
   | 1 | NIX | rte_ethdev, rte_tm, rte_event_eth_[rt]x_adapter, rte_security|
   +---+-----+--------------------------------------------------------------+
   | 2 | NPA | rte_mempool                                                  |
   +---+-----+--------------------------------------------------------------+
   | 3 | NPC | rte_flow                                                     |
   +---+-----+--------------------------------------------------------------+
   | 4 | CPT | rte_cryptodev, rte_event_crypto_adapter                      |
   +---+-----+--------------------------------------------------------------+
   | 5 | SSO | rte_eventdev                                                 |
   +---+-----+--------------------------------------------------------------+
   | 6 | TIM | rte_event_timer_adapter                                      |
   +---+-----+--------------------------------------------------------------+

PF0 is called the administrative / admin function (AF) and has exclusive
privileges to provision RVU functional block's LFs to each of the PF/VF.

PF/VFs communicates with AF via a shared memory region (mailbox).Upon receiving
requests from PF/VF, AF does resource provisioning and other HW configuration.

AF is always attached to host, but PF/VFs may be used by host kernel itself,
or attached to VMs or to userspace applications like DPDK, etc. So, AF has to
handle provisioning/configuration requests sent by any device from any domain.

The AF driver does not receive or process any data.
It is only a configuration driver used in control path.

The :numref:`figure_octeontx2_resource_virtualization` diagram also shows a
resource provisioning example where,

1. PFx and PFx-VF0 bound to Linux netdev driver.
2. PFx-VF1 ethdev driver bound to the first DPDK application.
3. PFy ethdev driver, PFy-VF0 ethdev driver, PFz eventdev driver, PFm-VF0 cryptodev driver bound to the second DPDK application.

OCTEON TX2 packet flow
----------------------

The :numref:`figure_octeontx2_packet_flow_hw_accelerators` diagram depicts
the packet flow on OCTEON TX2 SoC in conjunction with use of various HW accelerators.

.. _figure_octeontx2_packet_flow_hw_accelerators:

.. figure:: img/octeontx2_packet_flow_hw_accelerators.*

    OCTEON TX2 packet flow in conjunction with use of HW accelerators

HW Offload Drivers
------------------

This section lists dataplane H/W block(s) available in OCTEON TX2 SoC.

#. **Ethdev Driver**
   See :doc:`../nics/octeontx2` for NIX Ethdev driver information.

#. **Mempool Driver**
   See :doc:`../mempool/octeontx2` for NPA mempool driver information.

#. **Event Device Driver**
   See :doc:`../eventdevs/octeontx2` for SSO event device driver information.

#. **DMA Rawdev Driver**
   See :doc:`../rawdevs/octeontx2_dma` for DMA driver information.

Procedure to Setup Platform
---------------------------

There are three main prerequisites for setting up DPDK on OCTEON TX2
compatible board:

1. **OCTEON TX2 Linux kernel driver**

   The dependent kernel drivers can be obtained from the
   `kernel.org <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/marvell/octeontx2>`_.

   Alternatively, the Marvell SDK also provides the required kernel drivers.

   Linux kernel should be configured with the following features enabled:

.. code-block:: console

        # 64K pages enabled for better performance
        CONFIG_ARM64_64K_PAGES=y
        CONFIG_ARM64_VA_BITS_48=y
        # huge pages support enabled
        CONFIG_HUGETLBFS=y
        CONFIG_HUGETLB_PAGE=y
        # VFIO enabled with TYPE1 IOMMU at minimum
        CONFIG_VFIO_IOMMU_TYPE1=y
        CONFIG_VFIO_VIRQFD=y
        CONFIG_VFIO=y
        CONFIG_VFIO_NOIOMMU=y
        CONFIG_VFIO_PCI=y
        CONFIG_VFIO_PCI_MMAP=y
        # SMMUv3 driver
        CONFIG_ARM_SMMU_V3=y
        # ARMv8.1 LSE atomics
        CONFIG_ARM64_LSE_ATOMICS=y
        # OCTEONTX2 drivers
        CONFIG_OCTEONTX2_MBOX=y
        CONFIG_OCTEONTX2_AF=y
        # Enable if netdev PF driver required
        CONFIG_OCTEONTX2_PF=y
        # Enable if netdev VF driver required
        CONFIG_OCTEONTX2_VF=y
        CONFIG_CRYPTO_DEV_OCTEONTX2_CPT=y
        # Enable if OCTEONTX2 DMA PF driver required
        CONFIG_OCTEONTX2_DPI_PF=n

2. **ARM64 Linux Tool Chain**

   For example, the *aarch64* Linaro Toolchain, which can be obtained from
   `here <https://releases.linaro.org/components/toolchain/binaries/7.4-2019.02/aarch64-linux-gnu/>`_.

   Alternatively, the Marvell SDK also provides GNU GCC toolchain, which is
   optimized for OCTEON TX2 CPU.

3. **Rootfile system**

   Any *aarch64* supporting filesystem may be used. For example,
   Ubuntu 15.10 (Wily) or 16.04 LTS (Xenial) userland which can be obtained
   from `<http://cdimage.ubuntu.com/ubuntu-base/releases/16.04/release/ubuntu-base-16.04.1-base-arm64.tar.gz>`_.

   Alternatively, the Marvell SDK provides the buildroot based root filesystem.
   The SDK includes all the above prerequisites necessary to bring up the OCTEON TX2 board.

- Follow the DPDK :doc:`../linux_gsg/index` to setup the basic DPDK environment.


Debugging Options
-----------------

.. _table_octeontx2_common_debug_options:

.. table:: OCTEON TX2 common debug options

   +---+------------+-------------------------------------------------------+
   | # | Component  | EAL log command                                       |
   +===+============+=======================================================+
   | 1 | Common     | --log-level='pmd\.octeontx2\.base,8'                  |
   +---+------------+-------------------------------------------------------+
   | 2 | Mailbox    | --log-level='pmd\.octeontx2\.mbox,8'                  |
   +---+------------+-------------------------------------------------------+

Debugfs support
~~~~~~~~~~~~~~~

The **OCTEON TX2 Linux kernel driver** provides support to dump RVU blocks
context or stats using debugfs.

Enable ``debugfs`` by:

1. Compile kernel with debugfs enabled, i.e ``CONFIG_DEBUGFS=y``.
2. Boot OCTEON TX2 with debugfs supported kernel.
3. Verify ``debugfs`` mounted by default "mount | grep -i debugfs" or mount it manually by using.

.. code-block:: console

       # mount -t debugfs none /sys/kernel/debug

Currently ``debugfs`` supports the following RVU blocks NIX, NPA, NPC, NDC,
SSO & CGX.

The file structure under ``/sys/kernel/debug`` is as follows

.. code-block:: console

        octeontx2/
        |-- cgx
        |   |-- cgx0
        |   |   '-- lmac0
        |   |       '-- stats
        |   |-- cgx1
        |   |   |-- lmac0
        |   |   |   '-- stats
        |   |   '-- lmac1
        |   |       '-- stats
        |   '-- cgx2
        |       '-- lmac0
        |           '-- stats
        |-- cpt
        |   |-- cpt_engines_info
        |   |-- cpt_engines_sts
        |   |-- cpt_err_info
        |   |-- cpt_lfs_info
        |   '-- cpt_pc
        |---- nix
        |   |-- cq_ctx
        |   |-- ndc_rx_cache
        |   |-- ndc_rx_hits_miss
        |   |-- ndc_tx_cache
        |   |-- ndc_tx_hits_miss
        |   |-- qsize
        |   |-- rq_ctx
        |   |-- sq_ctx
        |   '-- tx_stall_hwissue
        |-- npa
        |   |-- aura_ctx
        |   |-- ndc_cache
        |   |-- ndc_hits_miss
        |   |-- pool_ctx
        |   '-- qsize
        |-- npc
        |    |-- mcam_info
        |    '-- rx_miss_act_stats
        |-- rsrc_alloc
        '-- sso
             |-- hws
             |   '-- sso_hws_info
             '-- hwgrp
                 |-- sso_hwgrp_aq_thresh
                 |-- sso_hwgrp_iaq_walk
                 |-- sso_hwgrp_pc
                 |-- sso_hwgrp_free_list_walk
                 |-- sso_hwgrp_ient_walk
                 '-- sso_hwgrp_taq_walk

RVU block LF allocation:

.. code-block:: console

        cat /sys/kernel/debug/octeontx2/rsrc_alloc

        pcifunc    NPA    NIX    SSO GROUP    SSOWS    TIM    CPT
        PF1         0       0
        PF4                 1
        PF13                          0, 1     0, 1      0

CGX example usage:

.. code-block:: console

        cat /sys/kernel/debug/octeontx2/cgx/cgx2/lmac0/stats

        =======Link Status======
        Link is UP 40000 Mbps
        =======RX_STATS======
        Received packets: 0
        Octets of received packets: 0
        Received PAUSE packets: 0
        Received PAUSE and control packets: 0
        Filtered DMAC0 (NIX-bound) packets: 0
        Filtered DMAC0 (NIX-bound) octets: 0
        Packets dropped due to RX FIFO full: 0
        Octets dropped due to RX FIFO full: 0
        Error packets: 0
        Filtered DMAC1 (NCSI-bound) packets: 0
        Filtered DMAC1 (NCSI-bound) octets: 0
        NCSI-bound packets dropped: 0
        NCSI-bound octets dropped: 0
        =======TX_STATS======
        Packets dropped due to excessive collisions: 0
        Packets dropped due to excessive deferral: 0
        Multiple collisions before successful transmission: 0
        Single collisions before successful transmission: 0
        Total octets sent on the interface: 0
        Total frames sent on the interface: 0
        Packets sent with an octet count < 64: 0
        Packets sent with an octet count == 64: 0
        Packets sent with an octet count of 65127: 0
        Packets sent with an octet count of 128-255: 0
        Packets sent with an octet count of 256-511: 0
        Packets sent with an octet count of 512-1023: 0
        Packets sent with an octet count of 1024-1518: 0
        Packets sent with an octet count of > 1518: 0
        Packets sent to a broadcast DMAC: 0
        Packets sent to the multicast DMAC: 0
        Transmit underflow and were truncated: 0
        Control/PAUSE packets sent: 0

CPT example usage:

.. code-block:: console

        cat /sys/kernel/debug/octeontx2/cpt/cpt_pc

        CPT instruction requests   0
        CPT instruction latency    0
        CPT NCB read requests      0
        CPT NCB read latency       0
        CPT read requests caused by UC fills   0
        CPT active cycles pc       1395642
        CPT clock count pc         5579867595493

NIX example usage:

.. code-block:: console

        Usage: echo <nixlf> [cq number/all] > /sys/kernel/debug/octeontx2/nix/cq_ctx
               cat /sys/kernel/debug/octeontx2/nix/cq_ctx
        echo 0 0 > /sys/kernel/debug/octeontx2/nix/cq_ctx
        cat /sys/kernel/debug/octeontx2/nix/cq_ctx

        =====cq_ctx for nixlf:0 and qidx:0 is=====
        W0: base                        158ef1a00

        W1: wrptr                       0
        W1: avg_con                     0
        W1: cint_idx                    0
        W1: cq_err                      0
        W1: qint_idx                    0
        W1: bpid                        0
        W1: bp_ena                      0

        W2: update_time                 31043
        W2:avg_level                    255
        W2: head                        0
        W2:tail                         0

        W3: cq_err_int_ena              5
        W3:cq_err_int                   0
        W3: qsize                       4
        W3:caching                      1
        W3: substream                   0x000
        W3: ena                                 1
        W3: drop_ena                    1
        W3: drop                        64
        W3: bp                          0

NPA example usage:

.. code-block:: console

        Usage: echo <npalf> [pool number/all] > /sys/kernel/debug/octeontx2/npa/pool_ctx
               cat /sys/kernel/debug/octeontx2/npa/pool_ctx
        echo 0 0 > /sys/kernel/debug/octeontx2/npa/pool_ctx
        cat /sys/kernel/debug/octeontx2/npa/pool_ctx

        ======POOL : 0=======
        W0: Stack base          1375bff00
        W1: ena                 1
        W1: nat_align           1
        W1: stack_caching       1
        W1: stack_way_mask      0
        W1: buf_offset          1
        W1: buf_size            19
        W2: stack_max_pages     24315
        W2: stack_pages         24314
        W3: op_pc               267456
        W4: stack_offset        2
        W4: shift               5
        W4: avg_level           255
        W4: avg_con             0
        W4: fc_ena              0
        W4: fc_stype            0
        W4: fc_hyst_bits        0
        W4: fc_up_crossing      0
        W4: update_time         62993
        W5: fc_addr             0
        W6: ptr_start           1593adf00
        W7: ptr_end             180000000
        W8: err_int             0
        W8: err_int_ena         7
        W8: thresh_int          0
        W8: thresh_int_ena      0
        W8: thresh_up           0
        W8: thresh_qint_idx     0
        W8: err_qint_idx        0

NPC example usage:

.. code-block:: console

        cat /sys/kernel/debug/octeontx2/npc/mcam_info

        NPC MCAM info:
        RX keywidth    : 224bits
        TX keywidth    : 224bits

        MCAM entries   : 2048
        Reserved       : 158
        Available      : 1890

        MCAM counters  : 512
        Reserved       : 1
        Available      : 511

SSO example usage:

.. code-block:: console

        Usage: echo [<hws>/all] > /sys/kernel/debug/octeontx2/sso/hws/sso_hws_info
        echo 0 > /sys/kernel/debug/octeontx2/sso/hws/sso_hws_info

        ==================================================
        SSOW HWS[0] Arbitration State      0x0
        SSOW HWS[0] Guest Machine Control  0x0
        SSOW HWS[0] SET[0] Group Mask[0] 0xffffffffffffffff
        SSOW HWS[0] SET[0] Group Mask[1] 0xffffffffffffffff
        SSOW HWS[0] SET[0] Group Mask[2] 0xffffffffffffffff
        SSOW HWS[0] SET[0] Group Mask[3] 0xffffffffffffffff
        SSOW HWS[0] SET[1] Group Mask[0] 0xffffffffffffffff
        SSOW HWS[0] SET[1] Group Mask[1] 0xffffffffffffffff
        SSOW HWS[0] SET[1] Group Mask[2] 0xffffffffffffffff
        SSOW HWS[0] SET[1] Group Mask[3] 0xffffffffffffffff
        ==================================================

Compile DPDK
------------

DPDK may be compiled either natively on OCTEON TX2 platform or cross-compiled on
an x86 based platform.

Native Compilation
~~~~~~~~~~~~~~~~~~

make build
^^^^^^^^^^

.. code-block:: console

        make config T=arm64-octeontx2-linux-gcc
        make -j

The example applications can be compiled using the following:

.. code-block:: console

        cd <dpdk directory>
        export RTE_SDK=$PWD
        export RTE_TARGET=build
        cd examples/<application>
        make -j

meson build
^^^^^^^^^^^

.. code-block:: console

        meson build
        ninja -C build

Cross Compilation
~~~~~~~~~~~~~~~~~

Refer to :doc:`../linux_gsg/cross_build_dpdk_for_arm64` for generic arm64 details.

make build
^^^^^^^^^^

.. code-block:: console

        make config T=arm64-octeontx2-linux-gcc
        make -j CROSS=aarch64-marvell-linux-gnu- CONFIG_RTE_KNI_KMOD=n

meson build
^^^^^^^^^^^

.. code-block:: console

        meson build --cross-file config/arm/arm64_octeontx2_linux_gcc
        ninja -C build

.. note::

   By default, meson cross compilation uses ``aarch64-linux-gnu-gcc`` toolchain,
   if Marvell toolchain is available then it can be used by overriding the
   c, cpp, ar, strip ``binaries`` attributes to respective Marvell
   toolchain binaries in ``config/arm/arm64_octeontx2_linux_gcc`` file.