summaryrefslogtreecommitdiff
path: root/doc/guides/freebsd_gsg/build_dpdk.rst
blob: 24a9f879df885c6f9738e589f34c80b40a5d94b8 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
..  BSD LICENSE
    Copyright(c) 2010-2014 Intel Corporation. All rights reserved.
    All rights reserved.

    Redistribution and use in source and binary forms, with or without
    modification, are permitted provided that the following conditions
    are met:

    * Redistributions of source code must retain the above copyright
    notice, this list of conditions and the following disclaimer.
    * Redistributions in binary form must reproduce the above copyright
    notice, this list of conditions and the following disclaimer in
    the documentation and/or other materials provided with the
    distribution.
    * Neither the name of Intel Corporation nor the names of its
    contributors may be used to endorse or promote products derived
    from this software without specific prior written permission.

    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
    "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
    A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
    OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
    SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
    LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

.. _building_from_source:

Compiling the DPDK Target from Source
=====================================

System Requirements
-------------------

The DPDK and its applications require the GNU make system (gmake)
to build on FreeBSD. Optionally, gcc may also be used in place of clang
to build the DPDK, in which case it too must be installed prior to
compiling the DPDK. The installation of these tools is covered in this
section.

Compiling the DPDK requires the FreeBSD kernel sources, which should be
included during the installation of FreeBSD on the development platform.
The DPDK also requires the use of FreeBSD ports to compile and function.

To use the FreeBSD ports system, it is required to update and extract the FreeBSD
ports tree by issuing the following commands:

.. code-block:: console

    portsnap fetch
    portsnap extract

If the environment requires proxies for external communication, these can be set
using:

.. code-block:: console

    setenv http_proxy <my_proxy_host>:<port>
    setenv ftp_proxy <my_proxy_host>:<port>

The FreeBSD ports below need to be installed prior to building the DPDK.
In general these can be installed using the following set of commands::

   cd /usr/ports/<port_location>

   make config-recursive

   make install

   make clean

Each port location can be found using::

   whereis <port_name>

The ports required and their locations are as follows:

* dialog4ports: ``/usr/ports/ports-mgmt/dialog4ports``

* GNU make(gmake): ``/usr/ports/devel/gmake``

* coreutils: ``/usr/ports/sysutils/coreutils``

For compiling and using the DPDK with gcc, the compiler must be installed
from the ports collection:

* gcc: version 4.9 is recommended ``/usr/ports/lang/gcc49``.
  Ensure that ``CPU_OPTS`` is selected (default is OFF).

When running the make config-recursive command, a dialog may be presented to the
user. For the installation of the DPDK, the default options were used.

.. note::

    To avoid multiple dialogs being presented to the user during make install,
    it is advisable before running the make install command to re-run the
    make config-recursive command until no more dialogs are seen.


Install the DPDK and Browse Sources
-----------------------------------

First, uncompress the archive and move to the DPDK source directory:

.. code-block:: console

    unzip DPDK-<version>.zip
    cd DPDK-<version>

The DPDK is composed of several directories:

*   lib: Source code of DPDK libraries

*   app: Source code of DPDK applications (automatic tests)

*   examples: Source code of DPDK applications

*   config, tools, scripts, mk: Framework-related makefiles, scripts and configuration

Installation of the DPDK Target Environments
--------------------------------------------

The format of a DPDK target is::

   ARCH-MACHINE-EXECENV-TOOLCHAIN

Where:

* ``ARCH`` is: ``x86_64``

* ``MACHINE`` is: ``native``

* ``EXECENV`` is: ``bsdapp``

* ``TOOLCHAIN`` is: ``gcc`` | ``clang``

The configuration files for the DPDK targets can be found in the DPDK/config
directory in the form of::

    defconfig_ARCH-MACHINE-EXECENV-TOOLCHAIN

.. note::

   Configuration files are provided with the ``RTE_MACHINE`` optimization level set.
   Within the configuration files, the ``RTE_MACHINE`` configuration value is set
   to native, which means that the compiled software is tuned for the platform
   on which it is built.  For more information on this setting, and its
   possible values, see the *DPDK Programmers Guide*.

To make the target, use ``gmake install T=<target>``.

For example to compile for FreeBSD use:

.. code-block:: console

    gmake install T=x86_64-native-bsdapp-clang

.. note::

   If the compiler binary to be used does not correspond to that given in the
   TOOLCHAIN part of the target, the compiler command may need to be explicitly
   specified. For example, if compiling for gcc, where the gcc binary is called
   gcc4.9, the command would need to be ``gmake install T=<target> CC=gcc4.9``.

Browsing the Installed DPDK Environment Target
----------------------------------------------

Once a target is created, it contains all the libraries and header files for the
DPDK environment that are required to build customer applications.
In addition, the test and testpmd applications are built under the build/app
directory, which may be used for testing.  A kmod directory is also present that
contains the kernel modules to install.

.. _loading_contigmem:

Loading the DPDK contigmem Module
---------------------------------

To run a DPDK application, physically contiguous memory is required.
In the absence of non-transparent superpages, the included sources for the
contigmem kernel module provides the ability to present contiguous blocks of
memory for the DPDK to use. The contigmem module must be loaded into the
running kernel before any DPDK is run.  The module is found in the kmod
sub-directory of the DPDK target directory.

The amount of physically contiguous memory along with the number of physically
contiguous blocks to be reserved by the module can be set at runtime prior to
module loading using:

.. code-block:: console

    kenv hw.contigmem.num_buffers=n
    kenv hw.contigmem.buffer_size=m

The kernel environment variables can also be specified during boot by placing the
following in ``/boot/loader.conf``::

    hw.contigmem.num_buffers=n hw.contigmem.buffer_size=m

The variables can be inspected using the following command:

.. code-block:: console

    sysctl -a hw.contigmem

Where n is the number of blocks and m is the size in bytes of each area of
contiguous memory.  A default of two buffers of size 1073741824 bytes (1 Gigabyte)
each is set during module load if they are not specified in the environment.

The module can then be loaded using kldload (assuming that the current directory
is the DPDK target directory):

.. code-block:: console

    kldload ./kmod/contigmem.ko

It is advisable to include the loading of the contigmem module during the boot
process to avoid issues with potential memory fragmentation during later system
up time.  This can be achieved by copying the module to the ``/boot/kernel/``
directory and placing the following into ``/boot/loader.conf``::

    contigmem_load="YES"

.. note::

    The contigmem_load directive should be placed after any definitions of
    ``hw.contigmem.num_buffers`` and ``hw.contigmem.buffer_size`` if the default values
    are not to be used.

An error such as:

.. code-block:: console

    kldload: can't load ./x86_64-native-bsdapp-gcc/kmod/contigmem.ko:
             Exec format error

is generally attributed to not having enough contiguous memory
available and can be verified via dmesg or ``/var/log/messages``:

.. code-block:: console

    kernel: contigmalloc failed for buffer <n>

To avoid this error, reduce the number of buffers or the buffer size.

.. _loading_nic_uio:

Loading the DPDK nic_uio Module
-------------------------------

After loading the contigmem module, the ``nic_uio`` module must also be loaded into the
running kernel prior to running any DPDK application.  This module must
be loaded using the kldload command as shown below (assuming that the current
directory is the DPDK target directory).

.. code-block:: console

    kldload ./kmod/nic_uio.ko

.. note::

    If the ports to be used are currently bound to a existing kernel driver
    then the ``hw.nic_uio.bdfs sysctl`` value will need to be set before loading the
    module. Setting this value is described in the next section below.

Currently loaded modules can be seen by using the ``kldstat`` command and a module
can be removed from the running kernel by using ``kldunload <module_name>``.

To load the module during boot, copy the ``nic_uio`` module to ``/boot/kernel``
and place the following into ``/boot/loader.conf``::

    nic_uio_load="YES"

.. note::

    ``nic_uio_load="YES"`` must appear after the contigmem_load directive, if it exists.

By default, the ``nic_uio`` module will take ownership of network ports if they are
recognized DPDK devices and are not owned by another module. However, since
the FreeBSD kernel includes support, either built-in, or via a separate driver
module, for most network card devices, it is likely that the ports to be used are
already bound to a driver other than ``nic_uio``. The following sub-section describe
how to query and modify the device ownership of the ports to be used by
DPDK applications.

.. _binding_network_ports:

Binding Network Ports to the nic_uio Module
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Device ownership can be viewed using the pciconf -l command. The example below shows
four IntelĀ® 82599 network ports under ``if_ixgbe`` module ownership.

.. code-block:: console

    pciconf -l
    ix0@pci0:1:0:0: class=0x020000 card=0x00038086 chip=0x10fb8086 rev=0x01 hdr=0x00
    ix1@pci0:1:0:1: class=0x020000 card=0x00038086 chip=0x10fb8086 rev=0x01 hdr=0x00
    ix2@pci0:2:0:0: class=0x020000 card=0x00038086 chip=0x10fb8086 rev=0x01 hdr=0x00
    ix3@pci0:2:0:1: class=0x020000 card=0x00038086 chip=0x10fb8086 rev=0x01 hdr=0x00

The first column constitutes three components:

#. Device name: ``ixN``

#. Unit name: ``pci0``

#. Selector (Bus:Device:Function): ``1:0:0``

Where no driver is associated with a device, the device name will be ``none``.

By default, the FreeBSD kernel will include built-in drivers for the most common
devices; a kernel rebuild would normally be required to either remove the drivers
or configure them as loadable modules.

To avoid building a custom kernel, the ``nic_uio`` module can detach a network port
from its current device driver. This is achieved by setting the ``hw.nic_uio.bdfs``
kernel environment variable prior to loading ``nic_uio``, as follows::

    hw.nic_uio.bdfs="b:d:f,b:d:f,..."

Where a comma separated list of selectors is set, the list must not contain any
whitespace.

For example to re-bind ``ix2@pci0:2:0:0`` and ``ix3@pci0:2:0:1`` to the ``nic_uio`` module
upon loading, use the following command::

    kenv hw.nic_uio.bdfs="2:0:0,2:0:1"

The variable can also be specified during boot by placing the following into
``/boot/loader.conf``, before the previously-described ``nic_uio_load`` line - as
shown::

    hw.nic_uio.bdfs="2:0:0,2:0:1"
    nic_uio_load="YES"

Binding Network Ports Back to their Original Kernel Driver
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If the original driver for a network port has been compiled into the kernel,
it is necessary to reboot FreeBSD to restore the original device binding. Before
doing so, update or remove the ``hw.nic_uio.bdfs`` in ``/boot/loader.conf``.

If rebinding to a driver that is a loadable module, the network port binding can
be reset without rebooting. To do so, unload both the target kernel module and the
``nic_uio`` module, modify or clear the ``hw.nic_uio.bdfs`` kernel environment (kenv)
value, and reload the two drivers - first the original kernel driver, and then
the ``nic_uio driver``. Note: the latter does not need to be reloaded unless there are
ports that are still to be bound to it.

Example commands to perform these steps are shown below:

.. code-block:: console

    kldunload nic_uio
    kldunload <original_driver>

    # To clear the value completely:
    kenv -u hw.nic_uio.bdfs

    # To update the list of ports to bind:
    kenv hw.nic_uio.bdfs="b:d:f,b:d:f,..."

    kldload <original_driver>

    kldload nic_uio  # optional