joy/android_kernel_sony_sdm845
Watch
1
Fork
You've already forked android_kernel_sony_sdm845
 0
Code Issues Pull requests Projects Releases Wiki Activity

Soft RoCE driver

Browse source 
Soft RoCE (RXE) - The software RoCE driver

ib_rxe implements the RDMA transport and registers to the RDMA core
device as a kernel verbs provider. It also implements the packet IO
layer. On the other hand ib_rxe registers to the Linux netdev stack
as a udp encapsulating protocol, in that case RDMA, for sending and
receiving packets over any Ethernet device.  This yields a RDMA
transport over the UDP/Ethernet network layer forming a RoCEv2
compatible device.

The configuration procedure of the Soft RoCE drivers requires
binding to any existing Ethernet network device. This is done with
/sys interface.

A userspace Soft RoCE library (librxe) provides user applications
the ability to run with Soft RoCE devices.  The use of rxe verbs ins
user space requires the inclusion of librxe as a device specifics
plug-in to libibverbs. librxe is packaged separately.

Architecture:

     +-----------------------------------------------------------+
     |                          Application                      |
     +-----------------------------------------------------------+
                            +-----------------------------------+
                            |             libibverbs            |
User                        +-----------------------------------+
                            +----------------+ +----------------+
                            | librxe         | | HW RoCE lib    |
                            +----------------+ +----------------+
+---------------------------------------------------------------+
     +--------------+                           +------------+
     | Sockets      |                           | RDMA ULP   |
     +--------------+                           +------------+
     +--------------+                  +---------------------+
     | TCP/IP       |                  | ib_core             |
     +--------------+                  +---------------------+
                             +------------+ +----------------+
Kernel                       | ib_rxe     | | HW RoCE driver |
                             +------------+ +----------------+
     +------------------------------------+
     | NIC driver                         |
     +------------------------------------+

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     +-----------------------------------------------------------+
     |                          Application                      |
     +-----------------------------------------------------------+
                            +-----------------------------------+
                            |             libibverbs            |
User                        +-----------------------------------+
                            +----------------+ +----------------+
                            | librxe         | | HW RoCE lib    |
                            +----------------+ +----------------+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     +--------------+                           +------------+
     | Sockets      |                           | RDMA ULP   |
     +--------------+                           +------------+
     +--------------+                  +---------------------+
     | TCP/IP       |                  | ib_core             |
     +--------------+                  +---------------------+
                             +------------+ +----------------+
Kernel                       | ib_rxe     | | HW RoCE driver |
                             +------------+ +----------------+
     +------------------------------------+
     | NIC driver                         |
     +------------------------------------+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Soft RoCE resources:

[1[ https://github.com/SoftRoCE/librxe-dev librxe - source code in
Github
[2] https://github.com/SoftRoCE/rxe-dev/wiki/rxe-dev:-Home - Soft RoCE
Wiki page
[3] https://github.com/SoftRoCE/librxe-dev - Soft RoCE userspace library

Signed-off-by: Kamal Heib <kamalh@mellanox.com>
Signed-off-by: Amir Vadai <amirv@mellanox.com>
Signed-off-by: Moni Shoua <monis@mellanox.com>
Reviewed-by: Haggai Eran <haggaie@mellanox.com>
Signed-off-by: Doug Ledford <dledford@redhat.com>
This commit is contained in:
Moni Shoua 2016-06-16 16:45:23 +03:00 • committed by Doug Ledford
commit 8700e3e7c4
38 changed files with 13039 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

9
MAINTAINERS
View file

@ -7444,6 +7444,15 @@ W: http://www.mellanox.com
Q: http://patchwork.ozlabs.org/project/netdev/list/
F: drivers/net/ethernet/mellanox/mlxsw/
SOFT-ROCE DRIVER (rxe)
M: Moni Shoua <monis@mellanox.com>
L: linux-rdma@vger.kernel.org
S: Supported
W: https://github.com/SoftRoCE/rxe-dev/wiki/rxe-dev:-Home
Q: http://patchwork.kernel.org/project/linux-rdma/list/
F: drivers/infiniband/hw/rxe/
F: include/uapi/rdma/rdma_user_rxe.h
MEMBARRIER SUPPORT
M: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
M: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>

1
drivers/infiniband/Kconfig
View file

@ -84,6 +84,7 @@ source "drivers/infiniband/ulp/iser/Kconfig"
source "drivers/infiniband/ulp/isert/Kconfig"
source "drivers/infiniband/sw/rdmavt/Kconfig"
source "drivers/infiniband/sw/rxe/Kconfig"
source "drivers/infiniband/hw/hfi1/Kconfig"

1
drivers/infiniband/sw/Makefile
View file

@ -1 +1,2 @@
obj-$(CONFIG_INFINIBAND_RDMAVT) += rdmavt/
obj-$(CONFIG_RDMA_RXE) += rxe/

24
drivers/infiniband/sw/rxe/Kconfig Normal file
View file

@ -0,0 +1,24 @@
config RDMA_RXE
tristate "Software RDMA over Ethernet (RoCE) driver"
depends on INET && PCI && INFINIBAND
depends on NET_UDP_TUNNEL
---help---
This driver implements the InfiniBand RDMA transport over
the Linux network stack. It enables a system with a
standard Ethernet adapter to interoperate with a RoCE
adapter or with another system running the RXE driver.
Documentation on InfiniBand and RoCE can be downloaded at
www.infinibandta.org and www.openfabrics.org. (See also
siw which is a similar software driver for iWARP.)
The driver is split into two layers, one interfaces with the
Linux RDMA stack and implements a kernel or user space
verbs API. The user space verbs API requires a support
library named librxe which is loaded by the generic user
space verbs API, libibverbs. The other layer interfaces
with the Linux network stack at layer 3.
To configure and work with soft-RoCE driver please use the
following wiki page under "configure Soft-RoCE (RXE)" section:
https://github.com/SoftRoCE/rxe-dev/wiki/rxe-dev:-Home

24
drivers/infiniband/sw/rxe/Makefile Normal file
View file

@ -0,0 +1,24 @@
obj-$(CONFIG_RDMA_RXE) += rdma_rxe.o
rdma_rxe-y := \
rxe.o \
rxe_comp.o \
rxe_req.o \
rxe_resp.o \
rxe_recv.o \
rxe_pool.o \
rxe_queue.o \
rxe_verbs.o \
rxe_av.o \
rxe_srq.o \
rxe_qp.o \
rxe_cq.o \
rxe_mr.o \
rxe_dma.o \
rxe_opcode.o \
rxe_mmap.o \
rxe_icrc.o \
rxe_mcast.o \
rxe_task.o \
rxe_net.o \
rxe_sysfs.o

386
drivers/infiniband/sw/rxe/rxe.c Normal file
View file

@ -0,0 +1,386 @@
/*
* Copyright (c) 2016 Mellanox Technologies Ltd. All rights reserved.
* Copyright (c) 2015 System Fabric Works, Inc. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenIB.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
#include "rxe.h"
#include "rxe_loc.h"
MODULE_AUTHOR("Bob Pearson, Frank Zago, John Groves, Kamal Heib");
MODULE_DESCRIPTION("Soft RDMA transport");
MODULE_LICENSE("Dual BSD/GPL");
MODULE_VERSION("0.2");
/* free resources for all ports on a device */
static void rxe_cleanup_ports(struct rxe_dev *rxe)
{
kfree(rxe->port.pkey_tbl);
rxe->port.pkey_tbl = NULL;
}
/* free resources for a rxe device all objects created for this device must
* have been destroyed
*/
static void rxe_cleanup(struct rxe_dev *rxe)
{
rxe_pool_cleanup(&rxe->uc_pool);
rxe_pool_cleanup(&rxe->pd_pool);
rxe_pool_cleanup(&rxe->ah_pool);
rxe_pool_cleanup(&rxe->srq_pool);
rxe_pool_cleanup(&rxe->qp_pool);
rxe_pool_cleanup(&rxe->cq_pool);
rxe_pool_cleanup(&rxe->mr_pool);
rxe_pool_cleanup(&rxe->mw_pool);
rxe_pool_cleanup(&rxe->mc_grp_pool);
rxe_pool_cleanup(&rxe->mc_elem_pool);
rxe_cleanup_ports(rxe);
}
/* called when all references have been dropped */
void rxe_release(struct kref *kref)
{
struct rxe_dev *rxe = container_of(kref, struct rxe_dev, ref_cnt);
rxe_cleanup(rxe);
ib_dealloc_device(&rxe->ib_dev);
}
void rxe_dev_put(struct rxe_dev *rxe)
{
kref_put(&rxe->ref_cnt, rxe_release);
}
EXPORT_SYMBOL_GPL(rxe_dev_put);
/* initialize rxe device parameters */
static int rxe_init_device_param(struct rxe_dev *rxe)
{
rxe->max_inline_data = RXE_MAX_INLINE_DATA;
rxe->attr.fw_ver = RXE_FW_VER;
rxe->attr.max_mr_size = RXE_MAX_MR_SIZE;
rxe->attr.page_size_cap = RXE_PAGE_SIZE_CAP;
rxe->attr.vendor_id = RXE_VENDOR_ID;
rxe->attr.vendor_part_id = RXE_VENDOR_PART_ID;
rxe->attr.hw_ver = RXE_HW_VER;
rxe->attr.max_qp = RXE_MAX_QP;
rxe->attr.max_qp_wr = RXE_MAX_QP_WR;
rxe->attr.device_cap_flags = RXE_DEVICE_CAP_FLAGS;
rxe->attr.max_sge = RXE_MAX_SGE;
rxe->attr.max_sge_rd = RXE_MAX_SGE_RD;
rxe->attr.max_cq = RXE_MAX_CQ;
rxe->attr.max_cqe = (1 << RXE_MAX_LOG_CQE) - 1;
rxe->attr.max_mr = RXE_MAX_MR;
rxe->attr.max_pd = RXE_MAX_PD;
rxe->attr.max_qp_rd_atom = RXE_MAX_QP_RD_ATOM;
rxe->attr.max_ee_rd_atom = RXE_MAX_EE_RD_ATOM;
rxe->attr.max_res_rd_atom = RXE_MAX_RES_RD_ATOM;
rxe->attr.max_qp_init_rd_atom = RXE_MAX_QP_INIT_RD_ATOM;
rxe->attr.max_ee_init_rd_atom = RXE_MAX_EE_INIT_RD_ATOM;
rxe->attr.atomic_cap = RXE_ATOMIC_CAP;
rxe->attr.max_ee = RXE_MAX_EE;
rxe->attr.max_rdd = RXE_MAX_RDD;
rxe->attr.max_mw = RXE_MAX_MW;
rxe->attr.max_raw_ipv6_qp = RXE_MAX_RAW_IPV6_QP;
rxe->attr.max_raw_ethy_qp = RXE_MAX_RAW_ETHY_QP;
rxe->attr.max_mcast_grp = RXE_MAX_MCAST_GRP;
rxe->attr.max_mcast_qp_attach = RXE_MAX_MCAST_QP_ATTACH;
rxe->attr.max_total_mcast_qp_attach = RXE_MAX_TOT_MCAST_QP_ATTACH;
rxe->attr.max_ah = RXE_MAX_AH;
rxe->attr.max_fmr = RXE_MAX_FMR;
rxe->attr.max_map_per_fmr = RXE_MAX_MAP_PER_FMR;
rxe->attr.max_srq = RXE_MAX_SRQ;
rxe->attr.max_srq_wr = RXE_MAX_SRQ_WR;
rxe->attr.max_srq_sge = RXE_MAX_SRQ_SGE;
rxe->attr.max_fast_reg_page_list_len = RXE_MAX_FMR_PAGE_LIST_LEN;
rxe->attr.max_pkeys = RXE_MAX_PKEYS;
rxe->attr.local_ca_ack_delay = RXE_LOCAL_CA_ACK_DELAY;
rxe->max_ucontext = RXE_MAX_UCONTEXT;
return 0;
}
/* initialize port attributes */
static int rxe_init_port_param(struct rxe_port *port)
{
port->attr.state = RXE_PORT_STATE;
port->attr.max_mtu = RXE_PORT_MAX_MTU;
port->attr.active_mtu = RXE_PORT_ACTIVE_MTU;
port->attr.gid_tbl_len = RXE_PORT_GID_TBL_LEN;
port->attr.port_cap_flags = RXE_PORT_PORT_CAP_FLAGS;
port->attr.max_msg_sz = RXE_PORT_MAX_MSG_SZ;
port->attr.bad_pkey_cntr = RXE_PORT_BAD_PKEY_CNTR;
port->attr.qkey_viol_cntr = RXE_PORT_QKEY_VIOL_CNTR;
port->attr.pkey_tbl_len = RXE_PORT_PKEY_TBL_LEN;
port->attr.lid = RXE_PORT_LID;
port->attr.sm_lid = RXE_PORT_SM_LID;
port->attr.lmc = RXE_PORT_LMC;
port->attr.max_vl_num = RXE_PORT_MAX_VL_NUM;
port->attr.sm_sl = RXE_PORT_SM_SL;
port->attr.subnet_timeout = RXE_PORT_SUBNET_TIMEOUT;
port->attr.init_type_reply = RXE_PORT_INIT_TYPE_REPLY;
port->attr.active_width = RXE_PORT_ACTIVE_WIDTH;
port->attr.active_speed = RXE_PORT_ACTIVE_SPEED;
port->attr.phys_state = RXE_PORT_PHYS_STATE;
port->mtu_cap =
ib_mtu_enum_to_int(RXE_PORT_ACTIVE_MTU);
port->subnet_prefix = cpu_to_be64(RXE_PORT_SUBNET_PREFIX);
return 0;
}
/* initialize port state, note IB convention that HCA ports are always
* numbered from 1
*/
static int rxe_init_ports(struct rxe_dev *rxe)
{
struct rxe_port *port = &rxe->port;
rxe_init_port_param(port);
if (!port->attr.pkey_tbl_len || !port->attr.gid_tbl_len)
return -EINVAL;
port->pkey_tbl = kcalloc(port->attr.pkey_tbl_len,
sizeof(*port->pkey_tbl), GFP_KERNEL);
if (!port->pkey_tbl)
return -ENOMEM;
port->pkey_tbl[0] = 0xffff;
port->port_guid = rxe->ifc_ops->port_guid(rxe);
spin_lock_init(&port->port_lock);
return 0;
}
/* init pools of managed objects */
static int rxe_init_pools(struct rxe_dev *rxe)
{
int err;
err = rxe_pool_init(rxe, &rxe->uc_pool, RXE_TYPE_UC,
rxe->max_ucontext);
if (err)
goto err1;
err = rxe_pool_init(rxe, &rxe->pd_pool, RXE_TYPE_PD,
rxe->attr.max_pd);
if (err)
goto err2;
err = rxe_pool_init(rxe, &rxe->ah_pool, RXE_TYPE_AH,
rxe->attr.max_ah);
if (err)
goto err3;
err = rxe_pool_init(rxe, &rxe->srq_pool, RXE_TYPE_SRQ,
rxe->attr.max_srq);
if (err)
goto err4;
err = rxe_pool_init(rxe, &rxe->qp_pool, RXE_TYPE_QP,
rxe->attr.max_qp);
if (err)
goto err5;
err = rxe_pool_init(rxe, &rxe->cq_pool, RXE_TYPE_CQ,
rxe->attr.max_cq);
if (err)
goto err6;
err = rxe_pool_init(rxe, &rxe->mr_pool, RXE_TYPE_MR,
rxe->attr.max_mr);
if (err)
goto err7;
err = rxe_pool_init(rxe, &rxe->mw_pool, RXE_TYPE_MW,
rxe->attr.max_mw);
if (err)
goto err8;
err = rxe_pool_init(rxe, &rxe->mc_grp_pool, RXE_TYPE_MC_GRP,
rxe->attr.max_mcast_grp);
if (err)
goto err9;
err = rxe_pool_init(rxe, &rxe->mc_elem_pool, RXE_TYPE_MC_ELEM,
rxe->attr.max_total_mcast_qp_attach);
if (err)
goto err10;
return 0;
err10:
rxe_pool_cleanup(&rxe->mc_grp_pool);
err9:
rxe_pool_cleanup(&rxe->mw_pool);
err8:
rxe_pool_cleanup(&rxe->mr_pool);
err7:
rxe_pool_cleanup(&rxe->cq_pool);
err6:
rxe_pool_cleanup(&rxe->qp_pool);
err5:
rxe_pool_cleanup(&rxe->srq_pool);
err4:
rxe_pool_cleanup(&rxe->ah_pool);
err3:
rxe_pool_cleanup(&rxe->pd_pool);
err2:
rxe_pool_cleanup(&rxe->uc_pool);
err1:
return err;
}
/* initialize rxe device state */
static int rxe_init(struct rxe_dev *rxe)
{
int err;
/* init default device parameters */
rxe_init_device_param(rxe);
err = rxe_init_ports(rxe);
if (err)
goto err1;
err = rxe_init_pools(rxe);
if (err)
goto err2;
/* init pending mmap list */
spin_lock_init(&rxe->mmap_offset_lock);
spin_lock_init(&rxe->pending_lock);
INIT_LIST_HEAD(&rxe->pending_mmaps);
INIT_LIST_HEAD(&rxe->list);
mutex_init(&rxe->usdev_lock);
return 0;
err2:
rxe_cleanup_ports(rxe);
err1:
return err;
}
int rxe_set_mtu(struct rxe_dev *rxe, unsigned int ndev_mtu)
{
struct rxe_port *port = &rxe->port;
enum ib_mtu mtu;
mtu = eth_mtu_int_to_enum(ndev_mtu);
/* Make sure that new MTU in range */
mtu = mtu ? min_t(enum ib_mtu, mtu, RXE_PORT_MAX_MTU) : IB_MTU_256;
port->attr.active_mtu = mtu;
port->mtu_cap = ib_mtu_enum_to_int(mtu);
return 0;
}
EXPORT_SYMBOL(rxe_set_mtu);
/* called by ifc layer to create new rxe device.
* The caller should allocate memory for rxe by calling ib_alloc_device.
*/
int rxe_add(struct rxe_dev *rxe, unsigned int mtu)
{
int err;
kref_init(&rxe->ref_cnt);
err = rxe_init(rxe);
if (err)
goto err1;
err = rxe_set_mtu(rxe, mtu);
if (err)
goto err1;
err = rxe_register_device(rxe);
if (err)
goto err1;
return 0;
err1:
rxe_dev_put(rxe);
return err;
}
EXPORT_SYMBOL(rxe_add);
/* called by the ifc layer to remove a device */
void rxe_remove(struct rxe_dev *rxe)
{
rxe_unregister_device(rxe);
rxe_dev_put(rxe);
}
EXPORT_SYMBOL(rxe_remove);
static int __init rxe_module_init(void)
{
int err;
/* initialize slab caches for managed objects */
err = rxe_cache_init();
if (err) {
pr_err("rxe: unable to init object pools\n");
return err;
}
err = rxe_net_init();
if (err) {
pr_err("rxe: unable to init\n");
rxe_cache_exit();
return err;
}
pr_info("rxe: loaded\n");
return 0;
}
static void __exit rxe_module_exit(void)
{
rxe_remove_all();
rxe_net_exit();
rxe_cache_exit();
pr_info("rxe: unloaded\n");
}
module_init(rxe_module_init);
module_exit(rxe_module_exit);

77
drivers/infiniband/sw/rxe/rxe.h Normal file
View file

@ -0,0 +1,77 @@
/*
* Copyright (c) 2016 Mellanox Technologies Ltd. All rights reserved.
* Copyright (c) 2015 System Fabric Works, Inc. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenIB.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
#ifndef RXE_H
#define RXE_H
#include <linux/module.h>
#include <linux/skbuff.h>
#include <linux/crc32.h>
#include <rdma/ib_verbs.h>
#include <rdma/ib_user_verbs.h>
#include <rdma/ib_pack.h>
#include <rdma/ib_smi.h>
#include <rdma/ib_umem.h>
#include <rdma/ib_cache.h>
#include <rdma/ib_addr.h>
#include "rxe_net.h"
#include "rxe_opcode.h"
#include "rxe_hdr.h"
#include "rxe_param.h"
#include "rxe_verbs.h"
#define RXE_UVERBS_ABI_VERSION (1)
#define IB_PHYS_STATE_LINK_UP (5)
#define IB_PHYS_STATE_LINK_DOWN (3)
#define RXE_ROCE_V2_SPORT (0xc000)
int rxe_set_mtu(struct rxe_dev *rxe, unsigned int dev_mtu);
int rxe_add(struct rxe_dev *rxe, unsigned int mtu);
void rxe_remove(struct rxe_dev *rxe);
void rxe_remove_all(void);
int rxe_rcv(struct sk_buff *skb);
void rxe_dev_put(struct rxe_dev *rxe);
struct rxe_dev *net_to_rxe(struct net_device *ndev);
struct rxe_dev *get_rxe_by_name(const char* name);
void rxe_port_up(struct rxe_dev *rxe);
void rxe_port_down(struct rxe_dev *rxe);
#endif /* RXE_H */

98
drivers/infiniband/sw/rxe/rxe_av.c Normal file
View file

@ -0,0 +1,98 @@
/*
* Copyright (c) 2016 Mellanox Technologies Ltd. All rights reserved.
* Copyright (c) 2015 System Fabric Works, Inc. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenIB.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
#include "rxe.h"
#include "rxe_loc.h"
int rxe_av_chk_attr(struct rxe_dev *rxe, struct ib_ah_attr *attr)
{
struct rxe_port *port;
if (attr->port_num != 1) {
pr_info("rxe: invalid port_num = %d\n", attr->port_num);
return -EINVAL;
}
port = &rxe->port;
if (attr->ah_flags & IB_AH_GRH) {
if (attr->grh.sgid_index > port->attr.gid_tbl_len) {
pr_info("rxe: invalid sgid index = %d\n",
attr->grh.sgid_index);
return -EINVAL;
}
}
return 0;
}
int rxe_av_from_attr(struct rxe_dev *rxe, u8 port_num,
struct rxe_av *av, struct ib_ah_attr *attr)
{
memset(av, 0, sizeof(*av));
memcpy(&av->grh, &attr->grh, sizeof(attr->grh));
av->port_num = port_num;
return 0;
}
int rxe_av_to_attr(struct rxe_dev *rxe, struct rxe_av *av,
struct ib_ah_attr *attr)
{
memcpy(&attr->grh, &av->grh, sizeof(av->grh));
attr->port_num = av->port_num;
return 0;
}
int rxe_av_fill_ip_info(struct rxe_dev *rxe,
struct rxe_av *av,
struct ib_ah_attr *attr,
struct ib_gid_attr *sgid_attr,
union ib_gid *sgid)
{
rdma_gid2ip(&av->sgid_addr._sockaddr, sgid);
rdma_gid2ip(&av->dgid_addr._sockaddr, &attr->grh.dgid);
av->network_type = ib_gid_to_network_type(sgid_attr->gid_type, sgid);
return 0;
}
struct rxe_av *rxe_get_av(struct rxe_pkt_info *pkt)
{
if (!pkt || !pkt->qp)
return NULL;
if (qp_type(pkt->qp) == IB_QPT_RC || qp_type(pkt->qp) == IB_QPT_UC)
return &pkt->qp->pri_av;
return (pkt->wqe) ? &pkt->wqe->av : NULL;
}

734
drivers/infiniband/sw/rxe/rxe_comp.c Normal file
View file

@ -0,0 +1,734 @@
/*
* Copyright (c) 2016 Mellanox Technologies Ltd. All rights reserved.
* Copyright (c) 2015 System Fabric Works, Inc. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenIB.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
#include <linux/skbuff.h>
#include "rxe.h"
#include "rxe_loc.h"
#include "rxe_queue.h"
#include "rxe_task.h"
enum comp_state {
COMPST_GET_ACK,
COMPST_GET_WQE,
COMPST_COMP_WQE,
COMPST_COMP_ACK,
COMPST_CHECK_PSN,
COMPST_CHECK_ACK,
COMPST_READ,
COMPST_ATOMIC,
COMPST_WRITE_SEND,
COMPST_UPDATE_COMP,
COMPST_ERROR_RETRY,
COMPST_RNR_RETRY,
COMPST_ERROR,
COMPST_EXIT, /* We have an issue, and we want to rerun the completer */
COMPST_DONE, /* The completer finished successflly */
};
static char *comp_state_name[] = {
[COMPST_GET_ACK] = "GET ACK",
[COMPST_GET_WQE] = "GET WQE",
[COMPST_COMP_WQE] = "COMP WQE",
[COMPST_COMP_ACK] = "COMP ACK",
[COMPST_CHECK_PSN] = "CHECK PSN",
[COMPST_CHECK_ACK] = "CHECK ACK",
[COMPST_READ] = "READ",
[COMPST_ATOMIC] = "ATOMIC",
[COMPST_WRITE_SEND] = "WRITE/SEND",
[COMPST_UPDATE_COMP] = "UPDATE COMP",
[COMPST_ERROR_RETRY] = "ERROR RETRY",
[COMPST_RNR_RETRY] = "RNR RETRY",
[COMPST_ERROR] = "ERROR",
[COMPST_EXIT] = "EXIT",
[COMPST_DONE] = "DONE",
};
static unsigned long rnrnak_usec[32] = {
[IB_RNR_TIMER_655_36] = 655360,
[IB_RNR_TIMER_000_01] = 10,
[IB_RNR_TIMER_000_02] = 20,
[IB_RNR_TIMER_000_03] = 30,
[IB_RNR_TIMER_000_04] = 40,
[IB_RNR_TIMER_000_06] = 60,
[IB_RNR_TIMER_000_08] = 80,
[IB_RNR_TIMER_000_12] = 120,
[IB_RNR_TIMER_000_16] = 160,
[IB_RNR_TIMER_000_24] = 240,
[IB_RNR_TIMER_000_32] = 320,
[IB_RNR_TIMER_000_48] = 480,
[IB_RNR_TIMER_000_64] = 640,
[IB_RNR_TIMER_000_96] = 960,
[IB_RNR_TIMER_001_28] = 1280,
[IB_RNR_TIMER_001_92] = 1920,
[IB_RNR_TIMER_002_56] = 2560,
[IB_RNR_TIMER_003_84] = 3840,
[IB_RNR_TIMER_005_12] = 5120,
[IB_RNR_TIMER_007_68] = 7680,
[IB_RNR_TIMER_010_24] = 10240,
[IB_RNR_TIMER_015_36] = 15360,
[IB_RNR_TIMER_020_48] = 20480,
[IB_RNR_TIMER_030_72] = 30720,
[IB_RNR_TIMER_040_96] = 40960,
[IB_RNR_TIMER_061_44] = 61410,
[IB_RNR_TIMER_081_92] = 81920,
[IB_RNR_TIMER_122_88] = 122880,
[IB_RNR_TIMER_163_84] = 163840,
[IB_RNR_TIMER_245_76] = 245760,
[IB_RNR_TIMER_327_68] = 327680,
[IB_RNR_TIMER_491_52] = 491520,
};
static inline unsigned long rnrnak_jiffies(u8 timeout)
{
return max_t(unsigned long,
usecs_to_jiffies(rnrnak_usec[timeout]), 1);
}
static enum ib_wc_opcode wr_to_wc_opcode(enum ib_wr_opcode opcode)
{
switch (opcode) {
case IB_WR_RDMA_WRITE: return IB_WC_RDMA_WRITE;
case IB_WR_RDMA_WRITE_WITH_IMM: return IB_WC_RDMA_WRITE;
case IB_WR_SEND: return IB_WC_SEND;
case IB_WR_SEND_WITH_IMM: return IB_WC_SEND;
case IB_WR_RDMA_READ: return IB_WC_RDMA_READ;
case IB_WR_ATOMIC_CMP_AND_SWP: return IB_WC_COMP_SWAP;
case IB_WR_ATOMIC_FETCH_AND_ADD: return IB_WC_FETCH_ADD;
case IB_WR_LSO: return IB_WC_LSO;
case IB_WR_SEND_WITH_INV: return IB_WC_SEND;
case IB_WR_RDMA_READ_WITH_INV: return IB_WC_RDMA_READ;
case IB_WR_LOCAL_INV: return IB_WC_LOCAL_INV;
case IB_WR_REG_MR: return IB_WC_REG_MR;
default:
return 0xff;
}
}
void retransmit_timer(unsigned long data)
{
struct rxe_qp *qp = (struct rxe_qp *)data;
if (qp->valid) {
qp->comp.timeout = 1;
rxe_run_task(&qp->comp.task, 1);
}
}
void rxe_comp_queue_pkt(struct rxe_dev *rxe, struct rxe_qp *qp,
struct sk_buff *skb)
{
int must_sched;
skb_queue_tail(&qp->resp_pkts, skb);
must_sched = skb_queue_len(&qp->resp_pkts) > 1;
rxe_run_task(&qp->comp.task, must_sched);
}
static inline enum comp_state get_wqe(struct rxe_qp *qp,
struct rxe_pkt_info *pkt,
struct rxe_send_wqe **wqe_p)
{
struct rxe_send_wqe *wqe;
/* we come here whether or not we found a response packet to see if
* there are any posted WQEs
*/
wqe = queue_head(qp->sq.queue);
*wqe_p = wqe;
/* no WQE or requester has not started it yet */
if (!wqe || wqe->state == wqe_state_posted)
return pkt ? COMPST_DONE : COMPST_EXIT;
/* WQE does not require an ack */
if (wqe->state == wqe_state_done)
return COMPST_COMP_WQE;
/* WQE caused an error */
if (wqe->state == wqe_state_error)
return COMPST_ERROR;
/* we have a WQE, if we also have an ack check its PSN */
return pkt ? COMPST_CHECK_PSN : COMPST_EXIT;
}
static inline void reset_retry_counters(struct rxe_qp *qp)
{
qp->comp.retry_cnt = qp->attr.retry_cnt;
qp->comp.rnr_retry = qp->attr.rnr_retry;
}
static inline enum comp_state check_psn(struct rxe_qp *qp,
struct rxe_pkt_info *pkt,
struct rxe_send_wqe *wqe)
{
s32 diff;
/* check to see if response is past the oldest WQE. if it is, complete
* send/write or error read/atomic
*/
diff = psn_compare(pkt->psn, wqe->last_psn);
if (diff > 0) {
if (wqe->state == wqe_state_pending) {
if (wqe->mask & WR_ATOMIC_OR_READ_MASK)
return COMPST_ERROR_RETRY;
reset_retry_counters(qp);
return COMPST_COMP_WQE;
} else {
return COMPST_DONE;
}
}
/* compare response packet to expected response */
diff = psn_compare(pkt->psn, qp->comp.psn);
if (diff < 0) {
/* response is most likely a retried packet if it matches an
* uncompleted WQE go complete it else ignore it
*/
if (pkt->psn == wqe->last_psn)
return COMPST_COMP_ACK;
else
return COMPST_DONE;
} else if ((diff > 0) && (wqe->mask & WR_ATOMIC_OR_READ_MASK)) {
return COMPST_ERROR_RETRY;
} else {
return COMPST_CHECK_ACK;
}
}
static inline enum comp_state check_ack(struct rxe_qp *qp,
struct rxe_pkt_info *pkt,
struct rxe_send_wqe *wqe)
{
unsigned int mask = pkt->mask;
u8 syn;
/* Check the sequence only */
switch (qp->comp.opcode) {
case -1:
/* Will catch all *_ONLY cases. */
if (!(mask & RXE_START_MASK))
return COMPST_ERROR;
break;
case IB_OPCODE_RC_RDMA_READ_RESPONSE_FIRST:
case IB_OPCODE_RC_RDMA_READ_RESPONSE_MIDDLE:
if (pkt->opcode != IB_OPCODE_RC_RDMA_READ_RESPONSE_MIDDLE &&
pkt->opcode != IB_OPCODE_RC_RDMA_READ_RESPONSE_LAST) {
return COMPST_ERROR;
}
break;
default:
WARN_ON(1);
}
/* Check operation validity. */
switch (pkt->opcode) {
case IB_OPCODE_RC_RDMA_READ_RESPONSE_FIRST:
case IB_OPCODE_RC_RDMA_READ_RESPONSE_LAST:
case IB_OPCODE_RC_RDMA_READ_RESPONSE_ONLY:
syn = aeth_syn(pkt);
if ((syn & AETH_TYPE_MASK) != AETH_ACK)
return COMPST_ERROR;
/* Fall through (IB_OPCODE_RC_RDMA_READ_RESPONSE_MIDDLE
* doesn't have an AETH)
*/
case IB_OPCODE_RC_RDMA_READ_RESPONSE_MIDDLE:
if (wqe->wr.opcode != IB_WR_RDMA_READ &&
wqe->wr.opcode != IB_WR_RDMA_READ_WITH_INV) {
return COMPST_ERROR;
}
reset_retry_counters(qp);
return COMPST_READ;
case IB_OPCODE_RC_ATOMIC_ACKNOWLEDGE:
syn = aeth_syn(pkt);
if ((syn & AETH_TYPE_MASK) != AETH_ACK)
return COMPST_ERROR;
if (wqe->wr.opcode != IB_WR_ATOMIC_CMP_AND_SWP &&
wqe->wr.opcode != IB_WR_ATOMIC_FETCH_AND_ADD)
return COMPST_ERROR;
reset_retry_counters(qp);
return COMPST_ATOMIC;
case IB_OPCODE_RC_ACKNOWLEDGE:
syn = aeth_syn(pkt);
switch (syn & AETH_TYPE_MASK) {
case AETH_ACK:
reset_retry_counters(qp);
return COMPST_WRITE_SEND;
case AETH_RNR_NAK:
return COMPST_RNR_RETRY;
case AETH_NAK:
switch (syn) {
case AETH_NAK_PSN_SEQ_ERROR:
/* a nak implicitly acks all packets with psns
* before
*/
if (psn_compare(pkt->psn, qp->comp.psn) > 0) {
qp->comp.psn = pkt->psn;
if (qp->req.wait_psn) {
qp->req.wait_psn = 0;
rxe_run_task(&qp->req.task, 1);
}
}
return COMPST_ERROR_RETRY;
case AETH_NAK_INVALID_REQ:
wqe->status = IB_WC_REM_INV_REQ_ERR;
return COMPST_ERROR;
case AETH_NAK_REM_ACC_ERR:
wqe->status = IB_WC_REM_ACCESS_ERR;
return COMPST_ERROR;
case AETH_NAK_REM_OP_ERR:
wqe->status = IB_WC_REM_OP_ERR;
return COMPST_ERROR;
default:
pr_warn("unexpected nak %x\n", syn);
wqe->status = IB_WC_REM_OP_ERR;
return COMPST_ERROR;
}
default:
return COMPST_ERROR;
}
break;
default:
pr_warn("unexpected opcode\n");
}
return COMPST_ERROR;
}
static inline enum comp_state do_read(struct rxe_qp *qp,
struct rxe_pkt_info *pkt,
struct rxe_send_wqe *wqe)
{
struct rxe_dev *rxe = to_rdev(qp->ibqp.device);
int ret;
ret = copy_data(rxe, qp->pd, IB_ACCESS_LOCAL_WRITE,
&wqe->dma, payload_addr(pkt),
payload_size(pkt), to_mem_obj, NULL);
if (ret)
return COMPST_ERROR;
if (wqe->dma.resid == 0 && (pkt->mask & RXE_END_MASK))
return COMPST_COMP_ACK;
else
return COMPST_UPDATE_COMP;
}
static inline enum comp_state do_atomic(struct rxe_qp *qp,
struct rxe_pkt_info *pkt,
struct rxe_send_wqe *wqe)
{
struct rxe_dev *rxe = to_rdev(qp->ibqp.device);
int ret;
u64 atomic_orig = atmack_orig(pkt);
ret = copy_data(rxe, qp->pd, IB_ACCESS_LOCAL_WRITE,
&wqe->dma, &atomic_orig,
sizeof(u64), to_mem_obj, NULL);
if (ret)
return COMPST_ERROR;
else
return COMPST_COMP_ACK;
}
static void make_send_cqe(struct rxe_qp *qp, struct rxe_send_wqe *wqe,
struct rxe_cqe *cqe)
{
memset(cqe, 0, sizeof(*cqe));
if (!qp->is_user) {
struct ib_wc *wc = &cqe->ibwc;
wc->wr_id = wqe->wr.wr_id;
wc->status = wqe->status;
wc->opcode = wr_to_wc_opcode(wqe->wr.opcode);
if (wqe->wr.opcode == IB_WR_RDMA_WRITE_WITH_IMM ||
wqe->wr.opcode == IB_WR_SEND_WITH_IMM)
wc->wc_flags = IB_WC_WITH_IMM;
wc->byte_len = wqe->dma.length;
wc->qp = &qp->ibqp;
} else {
struct ib_uverbs_wc *uwc = &cqe->uibwc;
uwc->wr_id = wqe->wr.wr_id;
uwc->status = wqe->status;
uwc->opcode = wr_to_wc_opcode(wqe->wr.opcode);
if (wqe->wr.opcode == IB_WR_RDMA_WRITE_WITH_IMM ||
wqe->wr.opcode == IB_WR_SEND_WITH_IMM)
uwc->wc_flags = IB_WC_WITH_IMM;
uwc->byte_len = wqe->dma.length;
uwc->qp_num = qp->ibqp.qp_num;
}
}
static void do_complete(struct rxe_qp *qp, struct rxe_send_wqe *wqe)
{
struct rxe_cqe cqe;
if ((qp->sq_sig_type == IB_SIGNAL_ALL_WR) ||
(wqe->wr.send_flags & IB_SEND_SIGNALED) ||
(qp->req.state == QP_STATE_ERROR)) {
make_send_cqe(qp, wqe, &cqe);
rxe_cq_post(qp->scq, &cqe, 0);
}
advance_consumer(qp->sq.queue);
/*
* we completed something so let req run again
* if it is trying to fence
*/
if (qp->req.wait_fence) {
qp->req.wait_fence = 0;
rxe_run_task(&qp->req.task, 1);
}
}
static inline enum comp_state complete_ack(struct rxe_qp *qp,
struct rxe_pkt_info *pkt,
struct rxe_send_wqe *wqe)
{
unsigned long flags;
if (wqe->has_rd_atomic) {
wqe->has_rd_atomic = 0;
atomic_inc(&qp->req.rd_atomic);
if (qp->req.need_rd_atomic) {
qp->comp.timeout_retry = 0;
qp->req.need_rd_atomic = 0;
rxe_run_task(&qp->req.task, 1);
}
}
if (unlikely(qp->req.state == QP_STATE_DRAIN)) {
/* state_lock used by requester & completer */
spin_lock_irqsave(&qp->state_lock, flags);
if ((qp->req.state == QP_STATE_DRAIN) &&
(qp->comp.psn == qp->req.psn)) {
qp->req.state = QP_STATE_DRAINED;
spin_unlock_irqrestore(&qp->state_lock, flags);
if (qp->ibqp.event_handler) {
struct ib_event ev;
ev.device = qp->ibqp.device;
ev.element.qp = &qp->ibqp;
ev.event = IB_EVENT_SQ_DRAINED;
qp->ibqp.event_handler(&ev,
qp->ibqp.qp_context);
}
} else {
spin_unlock_irqrestore(&qp->state_lock, flags);
}
}
do_complete(qp, wqe);
if (psn_compare(pkt->psn, qp->comp.psn) >= 0)
return COMPST_UPDATE_COMP;
else
return COMPST_DONE;
}
static inline enum comp_state complete_wqe(struct rxe_qp *qp,
struct rxe_pkt_info *pkt,
struct rxe_send_wqe *wqe)
{
qp->comp.opcode = -1;
if (pkt) {
if (psn_compare(pkt->psn, qp->comp.psn) >= 0)
qp->comp.psn = (pkt->psn + 1) & BTH_PSN_MASK;
if (qp->req.wait_psn) {
qp->req.wait_psn = 0;
rxe_run_task(&qp->req.task, 1);
}
}
do_complete(qp, wqe);
return COMPST_GET_WQE;
}
int rxe_completer(void *arg)
{
struct rxe_qp *qp = (struct rxe_qp *)arg;
struct rxe_send_wqe *wqe = wqe;
struct sk_buff *skb = NULL;
struct rxe_pkt_info *pkt = NULL;
enum comp_state state;
if (!qp->valid) {
while ((skb = skb_dequeue(&qp->resp_pkts))) {
rxe_drop_ref(qp);
kfree_skb(skb);
}
skb = NULL;
pkt = NULL;
while (queue_head(qp->sq.queue))
advance_consumer(qp->sq.queue);
goto exit;
}
if (qp->req.state == QP_STATE_ERROR) {
while ((skb = skb_dequeue(&qp->resp_pkts))) {
rxe_drop_ref(qp);
kfree_skb(skb);
}
skb = NULL;
pkt = NULL;
while ((wqe = queue_head(qp->sq.queue))) {
wqe->status = IB_WC_WR_FLUSH_ERR;
do_complete(qp, wqe);
}
goto exit;
}
if (qp->req.state == QP_STATE_RESET) {
while ((skb = skb_dequeue(&qp->resp_pkts))) {
rxe_drop_ref(qp);
kfree_skb(skb);
}
skb = NULL;
pkt = NULL;
while (queue_head(qp->sq.queue))
advance_consumer(qp->sq.queue);
goto exit;
}
if (qp->comp.timeout) {
qp->comp.timeout_retry = 1;
qp->comp.timeout = 0;
} else {
qp->comp.timeout_retry = 0;
}
if (qp->req.need_retry)
goto exit;
state = COMPST_GET_ACK;
while (1) {
pr_debug("state = %s\n", comp_state_name[state]);
switch (state) {
case COMPST_GET_ACK:
skb = skb_dequeue(&qp->resp_pkts);
if (skb) {
pkt = SKB_TO_PKT(skb);
qp->comp.timeout_retry = 0;
}
state = COMPST_GET_WQE;
break;
case COMPST_GET_WQE:
state = get_wqe(qp, pkt, &wqe);
break;
case COMPST_CHECK_PSN:
state = check_psn(qp, pkt, wqe);
break;
case COMPST_CHECK_ACK:
state = check_ack(qp, pkt, wqe);
break;
case COMPST_READ:
state = do_read(qp, pkt, wqe);
break;
case COMPST_ATOMIC:
state = do_atomic(qp, pkt, wqe);
break;
case COMPST_WRITE_SEND:
if (wqe->state == wqe_state_pending &&
wqe->last_psn == pkt->psn)
state = COMPST_COMP_ACK;
else
state = COMPST_UPDATE_COMP;
break;
case COMPST_COMP_ACK:
state = complete_ack(qp, pkt, wqe);
break;
case COMPST_COMP_WQE:
state = complete_wqe(qp, pkt, wqe);
break;
case COMPST_UPDATE_COMP:
if (pkt->mask & RXE_END_MASK)
qp->comp.opcode = -1;
else
qp->comp.opcode = pkt->opcode;
if (psn_compare(pkt->psn, qp->comp.psn) >= 0)
qp->comp.psn = (pkt->psn + 1) & BTH_PSN_MASK;
if (qp->req.wait_psn) {
qp->req.wait_psn = 0;
rxe_run_task(&qp->req.task, 1);
}
state = COMPST_DONE;
break;
case COMPST_DONE:
if (pkt) {
rxe_drop_ref(pkt->qp);
kfree_skb(skb);
}
goto done;
case COMPST_EXIT:
if (qp->comp.timeout_retry && wqe) {
state = COMPST_ERROR_RETRY;
break;
}
/* re reset the timeout counter if
* (1) QP is type RC
* (2) the QP is alive
* (3) there is a packet sent by the requester that
* might be acked (we still might get spurious
* timeouts but try to keep them as few as possible)
* (4) the timeout parameter is set
*/
if ((qp_type(qp) == IB_QPT_RC) &&
(qp->req.state == QP_STATE_READY) &&
(psn_compare(qp->req.psn, qp->comp.psn) > 0) &&
qp->qp_timeout_jiffies)
mod_timer(&qp->retrans_timer,
jiffies + qp->qp_timeout_jiffies);
goto exit;
case COMPST_ERROR_RETRY:
/* we come here if the retry timer fired and we did
* not receive a response packet. try to retry the send
* queue if that makes sense and the limits have not
* been exceeded. remember that some timeouts are
* spurious since we do not reset the timer but kick
* it down the road or let it expire
*/
/* there is nothing to retry in this case */
if (!wqe || (wqe->state == wqe_state_posted))
goto exit;
if (qp->comp.retry_cnt > 0) {
if (qp->comp.retry_cnt != 7)
qp->comp.retry_cnt--;
/* no point in retrying if we have already
* seen the last ack that the requester could
* have caused
*/
if (psn_compare(qp->req.psn,
qp->comp.psn) > 0) {
/* tell the requester to retry the
* send send queue next time around
*/
qp->req.need_retry = 1;
rxe_run_task(&qp->req.task, 1);
}
goto exit;
} else {
wqe->status = IB_WC_RETRY_EXC_ERR;
state = COMPST_ERROR;
}
break;
case COMPST_RNR_RETRY:
if (qp->comp.rnr_retry > 0) {
if (qp->comp.rnr_retry != 7)
qp->comp.rnr_retry--;
qp->req.need_retry = 1;
pr_debug("set rnr nak timer\n");
mod_timer(&qp->rnr_nak_timer,
jiffies + rnrnak_jiffies(aeth_syn(pkt)
& ~AETH_TYPE_MASK));
goto exit;
} else {
wqe->status = IB_WC_RNR_RETRY_EXC_ERR;
state = COMPST_ERROR;
}
break;
case COMPST_ERROR:
do_complete(qp, wqe);
rxe_qp_error(qp);
goto exit;
}
}
exit:
/* we come here if we are done with processing and want the task to
* exit from the loop calling us
*/
return -EAGAIN;
done:
/* we come here if we have processed a packet we want the task to call
* us again to see if there is anything else to do
*/
return 0;
}

165
drivers/infiniband/sw/rxe/rxe_cq.c Normal file
View file

@ -0,0 +1,165 @@
/*
* Copyright (c) 2016 Mellanox Technologies Ltd. All rights reserved.
* Copyright (c) 2015 System Fabric Works, Inc. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenIB.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
#include "rxe.h"
#include "rxe_loc.h"
#include "rxe_queue.h"
int rxe_cq_chk_attr(struct rxe_dev *rxe, struct rxe_cq *cq,
int cqe, int comp_vector, struct ib_udata *udata)
{
int count;
if (cqe <= 0) {
pr_warn("cqe(%d) <= 0\n", cqe);
goto err1;
}
if (cqe > rxe->attr.max_cqe) {
pr_warn("cqe(%d) > max_cqe(%d)\n",
cqe, rxe->attr.max_cqe);
goto err1;
}
if (cq) {
count = queue_count(cq->queue);
if (cqe < count) {
pr_warn("cqe(%d) < current # elements in queue (%d)",
cqe, count);
goto err1;
}
}
return 0;
err1:
return -EINVAL;
}
static void rxe_send_complete(unsigned long data)
{
struct rxe_cq *cq = (struct rxe_cq *)data;
cq->ibcq.comp_handler(&cq->ibcq, cq->ibcq.cq_context);
}
int rxe_cq_from_init(struct rxe_dev *rxe, struct rxe_cq *cq, int cqe,
int comp_vector, struct ib_ucontext *context,
struct ib_udata *udata)
{
int err;
cq->queue = rxe_queue_init(rxe, &cqe,
sizeof(struct rxe_cqe));
if (!cq->queue) {
pr_warn("unable to create cq\n");
return -ENOMEM;
}
err = do_mmap_info(rxe, udata, false, context, cq->queue->buf,
cq->queue->buf_size, &cq->queue->ip);
if (err) {
kvfree(cq->queue->buf);
kfree(cq->queue);
return err;
}
if (udata)
cq->is_user = 1;
tasklet_init(&cq->comp_task, rxe_send_complete, (unsigned long)cq);
spin_lock_init(&cq->cq_lock);
cq->ibcq.cqe = cqe;
return 0;
}
int rxe_cq_resize_queue(struct rxe_cq *cq, int cqe, struct ib_udata *udata)
{
int err;
err = rxe_queue_resize(cq->queue, (unsigned int *)&cqe,
sizeof(struct rxe_cqe),
cq->queue->ip ? cq->queue->ip->context : NULL,
udata, NULL, &cq->cq_lock);
if (!err)
cq->ibcq.cqe = cqe;
return err;
}
int rxe_cq_post(struct rxe_cq *cq, struct rxe_cqe *cqe, int solicited)
{
struct ib_event ev;
unsigned long flags;
spin_lock_irqsave(&cq->cq_lock, flags);
if (unlikely(queue_full(cq->queue))) {
spin_unlock_irqrestore(&cq->cq_lock, flags);
if (cq->ibcq.event_handler) {
ev.device = cq->ibcq.device;
ev.element.cq = &cq->ibcq;
ev.event = IB_EVENT_CQ_ERR;
cq->ibcq.event_handler(&ev, cq->ibcq.cq_context);
}
return -EBUSY;
}
memcpy(producer_addr(cq->queue), cqe, sizeof(*cqe));
/* make sure all changes to the CQ are written before we update the
* producer pointer
*/
smp_wmb();
advance_producer(cq->queue);
spin_unlock_irqrestore(&cq->cq_lock, flags);
if ((cq->notify == IB_CQ_NEXT_COMP) ||
(cq->notify == IB_CQ_SOLICITED && solicited)) {
cq->notify = 0;
tasklet_schedule(&cq->comp_task);
}
return 0;
}
void rxe_cq_cleanup(void *arg)
{
struct rxe_cq *cq = arg;
if (cq->queue)
rxe_queue_cleanup(cq->queue);
}

166
drivers/infiniband/sw/rxe/rxe_dma.c Normal file
View file

@ -0,0 +1,166 @@
/*
* Copyright (c) 2016 Mellanox Technologies Ltd. All rights reserved.
* Copyright (c) 2015 System Fabric Works, Inc. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenIB.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
#include "rxe.h"
#include "rxe_loc.h"
#define DMA_BAD_ADDER ((u64)0)
static int rxe_mapping_error(struct ib_device *dev, u64 dma_addr)
{
return dma_addr == DMA_BAD_ADDER;
}
static u64 rxe_dma_map_single(struct ib_device *dev,
void *cpu_addr, size_t size,
enum dma_data_direction direction)
{
WARN_ON(!valid_dma_direction(direction));
return (uintptr_t)cpu_addr;
}
static void rxe_dma_unmap_single(struct ib_device *dev,
u64 addr, size_t size,
enum dma_data_direction direction)
{
WARN_ON(!valid_dma_direction(direction));
}
static u64 rxe_dma_map_page(struct ib_device *dev,
struct page *page,
unsigned long offset,
size_t size, enum dma_data_direction direction)
{
u64 addr;
WARN_ON(!valid_dma_direction(direction));
if (offset + size > PAGE_SIZE) {
addr = DMA_BAD_ADDER;
goto done;
}
addr = (uintptr_t)page_address(page);
if (addr)
addr += offset;
done:
return addr;
}
static void rxe_dma_unmap_page(struct ib_device *dev,
u64 addr, size_t size,
enum dma_data_direction direction)
{
WARN_ON(!valid_dma_direction(direction));
}
static int rxe_map_sg(struct ib_device *dev, struct scatterlist *sgl,
int nents, enum dma_data_direction direction)
{
struct scatterlist *sg;
u64 addr;
int i;
int ret = nents;
WARN_ON(!valid_dma_direction(direction));
for_each_sg(sgl, sg, nents, i) {
addr = (uintptr_t)page_address(sg_page(sg));
if (!addr) {
ret = 0;
break;
}
sg->dma_address = addr + sg->offset;
#ifdef CONFIG_NEED_SG_DMA_LENGTH
sg->dma_length = sg->length;
#endif
}
return ret;
}
static void rxe_unmap_sg(struct ib_device *dev,
struct scatterlist *sg, int nents,
enum dma_data_direction direction)
{
WARN_ON(!valid_dma_direction(direction));
}
static void rxe_sync_single_for_cpu(struct ib_device *dev,
u64 addr,
size_t size, enum dma_data_direction dir)
{
}
static void rxe_sync_single_for_device(struct ib_device *dev,
u64 addr,
size_t size, enum dma_data_direction dir)
{
}
static void *rxe_dma_alloc_coherent(struct ib_device *dev, size_t size,
u64 *dma_handle, gfp_t flag)
{
struct page *p;
void *addr = NULL;
p = alloc_pages(flag, get_order(size));
if (p)
addr = page_address(p);
if (dma_handle)
*dma_handle = (uintptr_t)addr;
return addr;
}
static void rxe_dma_free_coherent(struct ib_device *dev, size_t size,
void *cpu_addr, u64 dma_handle)
{
free_pages((unsigned long)cpu_addr, get_order(size));
}
struct ib_dma_mapping_ops rxe_dma_mapping_ops = {
.mapping_error = rxe_mapping_error,
.map_single = rxe_dma_map_single,
.unmap_single = rxe_dma_unmap_single,
.map_page = rxe_dma_map_page,
.unmap_page = rxe_dma_unmap_page,
.map_sg = rxe_map_sg,
.unmap_sg = rxe_unmap_sg,
.sync_single_for_cpu = rxe_sync_single_for_cpu,
.sync_single_for_device = rxe_sync_single_for_device,
.alloc_coherent = rxe_dma_alloc_coherent,
.free_coherent = rxe_dma_free_coherent
};

952
drivers/infiniband/sw/rxe/rxe_hdr.h Normal file
View file

@ -0,0 +1,952 @@
/*
* Copyright (c) 2016 Mellanox Technologies Ltd. All rights reserved.
* Copyright (c) 2015 System Fabric Works, Inc. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenIB.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
#ifndef RXE_HDR_H
#define RXE_HDR_H
/* extracted information about a packet carried in an sk_buff struct fits in
* the skbuff cb array. Must be at most 48 bytes. stored in control block of
* sk_buff for received packets.
*/
struct rxe_pkt_info {
struct rxe_dev *rxe; /* device that owns packet */
struct rxe_qp *qp; /* qp that owns packet */
struct rxe_send_wqe *wqe; /* send wqe */
u8 *hdr; /* points to bth */
u32 mask; /* useful info about pkt */
u32 psn; /* bth psn of packet */
u16 pkey_index; /* partition of pkt */
u16 paylen; /* length of bth - icrc */
u8 port_num; /* port pkt received on */
u8 opcode; /* bth opcode of packet */
u8 offset; /* bth offset from pkt->hdr */
};
/* Macros should be used only for received skb */
#define SKB_TO_PKT(skb) ((struct rxe_pkt_info *)(skb)->cb)
#define PKT_TO_SKB(pkt) container_of((void *)(pkt), struct sk_buff, cb)
/*
* IBA header types and methods
*
* Some of these are for reference and completeness only since
* rxe does not currently support RD transport
* most of this could be moved into IB core. ib_pack.h has
* part of this but is incomplete
*
* Header specific routines to insert/extract values to/from headers
* the routines that are named __hhh_(set_)fff() take a pointer to a
* hhh header and get(set) the fff field. The routines named
* hhh_(set_)fff take a packet info struct and find the
* header and field based on the opcode in the packet.
* Conversion to/from network byte order from cpu order is also done.
*/
#define RXE_ICRC_SIZE (4)
#define RXE_MAX_HDR_LENGTH (80)
/******************************************************************************
* Base Transport Header
******************************************************************************/
struct rxe_bth {
u8 opcode;
u8 flags;
__be16 pkey;
__be32 qpn;
__be32 apsn;
};
#define BTH_TVER (0)
#define BTH_DEF_PKEY (0xffff)
#define BTH_SE_MASK (0x80)
#define BTH_MIG_MASK (0x40)
#define BTH_PAD_MASK (0x30)
#define BTH_TVER_MASK (0x0f)
#define BTH_FECN_MASK (0x80000000)
#define BTH_BECN_MASK (0x40000000)
#define BTH_RESV6A_MASK (0x3f000000)
#define BTH_QPN_MASK (0x00ffffff)
#define BTH_ACK_MASK (0x80000000)
#define BTH_RESV7_MASK (0x7f000000)
#define BTH_PSN_MASK (0x00ffffff)
static inline u8 __bth_opcode(void *arg)
{
struct rxe_bth *bth = arg;
return bth->opcode;
}
static inline void __bth_set_opcode(void *arg, u8 opcode)
{
struct rxe_bth *bth = arg;
bth->opcode = opcode;
}
static inline u8 __bth_se(void *arg)
{
struct rxe_bth *bth = arg;
return 0 != (BTH_SE_MASK & bth->flags);
}
static inline void __bth_set_se(void *arg, int se)
{
struct rxe_bth *bth = arg;
if (se)
bth->flags |= BTH_SE_MASK;
else
bth->flags &= ~BTH_SE_MASK;
}
static inline u8 __bth_mig(void *arg)
{
struct rxe_bth *bth = arg;
return 0 != (BTH_MIG_MASK & bth->flags);
}
static inline void __bth_set_mig(void *arg, u8 mig)
{
struct rxe_bth *bth = arg;
if (mig)
bth->flags |= BTH_MIG_MASK;
else
bth->flags &= ~BTH_MIG_MASK;
}
static inline u8 __bth_pad(void *arg)
{
struct rxe_bth *bth = arg;
return (BTH_PAD_MASK & bth->flags) >> 4;
}
static inline void __bth_set_pad(void *arg, u8 pad)
{
struct rxe_bth *bth = arg;
bth->flags = (BTH_PAD_MASK & (pad << 4)) |
(~BTH_PAD_MASK & bth->flags);
}
static inline u8 __bth_tver(void *arg)
{
struct rxe_bth *bth = arg;
return BTH_TVER_MASK & bth->flags;
}
static inline void __bth_set_tver(void *arg, u8 tver)
{
struct rxe_bth *bth = arg;
bth->flags = (BTH_TVER_MASK & tver) |
(~BTH_TVER_MASK & bth->flags);
}
static inline u16 __bth_pkey(void *arg)
{
struct rxe_bth *bth = arg;
return be16_to_cpu(bth->pkey);
}
static inline void __bth_set_pkey(void *arg, u16 pkey)
{
struct rxe_bth *bth = arg;
bth->pkey = cpu_to_be16(pkey);
}
static inline u32 __bth_qpn(void *arg)
{
struct rxe_bth *bth = arg;
return BTH_QPN_MASK & be32_to_cpu(bth->qpn);
}
static inline void __bth_set_qpn(void *arg, u32 qpn)
{
struct rxe_bth *bth = arg;
u32 resvqpn = be32_to_cpu(bth->qpn);
bth->qpn = cpu_to_be32((BTH_QPN_MASK & qpn) |
(~BTH_QPN_MASK & resvqpn));
}
static inline int __bth_fecn(void *arg)
{
struct rxe_bth *bth = arg;
return 0 != (cpu_to_be32(BTH_FECN_MASK) & bth->qpn);
}
static inline void __bth_set_fecn(void *arg, int fecn)
{
struct rxe_bth *bth = arg;
if (fecn)
bth->qpn |= cpu_to_be32(BTH_FECN_MASK);
else
bth->qpn &= ~cpu_to_be32(BTH_FECN_MASK);
}
static inline int __bth_becn(void *arg)
{
struct rxe_bth *bth = arg;
return 0 != (cpu_to_be32(BTH_BECN_MASK) & bth->qpn);
}
static inline void __bth_set_becn(void *arg, int becn)
{
struct rxe_bth *bth = arg;
if (becn)
bth->qpn |= cpu_to_be32(BTH_BECN_MASK);
else
bth->qpn &= ~cpu_to_be32(BTH_BECN_MASK);
}
static inline u8 __bth_resv6a(void *arg)
{
struct rxe_bth *bth = arg;
return (BTH_RESV6A_MASK & be32_to_cpu(bth->qpn)) >> 24;
}
static inline void __bth_set_resv6a(void *arg)
{
struct rxe_bth *bth = arg;
bth->qpn = cpu_to_be32(~BTH_RESV6A_MASK);
}
static inline int __bth_ack(void *arg)
{
struct rxe_bth *bth = arg;
return 0 != (cpu_to_be32(BTH_ACK_MASK) & bth->apsn);
}
static inline void __bth_set_ack(void *arg, int ack)
{
struct rxe_bth *bth = arg;
if (ack)
bth->apsn |= cpu_to_be32(BTH_ACK_MASK);
else
bth->apsn &= ~cpu_to_be32(BTH_ACK_MASK);
}
static inline void __bth_set_resv7(void *arg)
{
struct rxe_bth *bth = arg;
bth->apsn &= ~cpu_to_be32(BTH_RESV7_MASK);
}
static inline u32 __bth_psn(void *arg)
{
struct rxe_bth *bth = arg;
return BTH_PSN_MASK & be32_to_cpu(bth->apsn);
}
static inline void __bth_set_psn(void *arg, u32 psn)
{
struct rxe_bth *bth = arg;
u32 apsn = be32_to_cpu(bth->apsn);
bth->apsn = cpu_to_be32((BTH_PSN_MASK & psn) |
(~BTH_PSN_MASK & apsn));
}
static inline u8 bth_opcode(struct rxe_pkt_info *pkt)
{
return __bth_opcode(pkt->hdr + pkt->offset);
}
static inline void bth_set_opcode(struct rxe_pkt_info *pkt, u8 opcode)
{
__bth_set_opcode(pkt->hdr + pkt->offset, opcode);
}
static inline u8 bth_se(struct rxe_pkt_info *pkt)
{
return __bth_se(pkt->hdr + pkt->offset);
}
static inline void bth_set_se(struct rxe_pkt_info *pkt, int se)
{
__bth_set_se(pkt->hdr + pkt->offset, se);
}
static inline u8 bth_mig(struct rxe_pkt_info *pkt)
{
return __bth_mig(pkt->hdr + pkt->offset);
}
static inline void bth_set_mig(struct rxe_pkt_info *pkt, u8 mig)
{
__bth_set_mig(pkt->hdr + pkt->offset, mig);
}
static inline u8 bth_pad(struct rxe_pkt_info *pkt)
{
return __bth_pad(pkt->hdr + pkt->offset);
}
static inline void bth_set_pad(struct rxe_pkt_info *pkt, u8 pad)
{
__bth_set_pad(pkt->hdr + pkt->offset, pad);
}
static inline u8 bth_tver(struct rxe_pkt_info *pkt)
{
return __bth_tver(pkt->hdr + pkt->offset);
}
static inline void bth_set_tver(struct rxe_pkt_info *pkt, u8 tver)
{
__bth_set_tver(pkt->hdr + pkt->offset, tver);
}
static inline u16 bth_pkey(struct rxe_pkt_info *pkt)
{
return __bth_pkey(pkt->hdr + pkt->offset);
}
static inline void bth_set_pkey(struct rxe_pkt_info *pkt, u16 pkey)
{
__bth_set_pkey(pkt->hdr + pkt->offset, pkey);
}
static inline u32 bth_qpn(struct rxe_pkt_info *pkt)
{
return __bth_qpn(pkt->hdr + pkt->offset);
}
static inline void bth_set_qpn(struct rxe_pkt_info *pkt, u32 qpn)
{
__bth_set_qpn(pkt->hdr + pkt->offset, qpn);
}
static inline int bth_fecn(struct rxe_pkt_info *pkt)
{
return __bth_fecn(pkt->hdr + pkt->offset);
}
static inline void bth_set_fecn(struct rxe_pkt_info *pkt, int fecn)
{
__bth_set_fecn(pkt->hdr + pkt->offset, fecn);
}
static inline int bth_becn(struct rxe_pkt_info *pkt)
{
return __bth_becn(pkt->hdr + pkt->offset);
}
static inline void bth_set_becn(struct rxe_pkt_info *pkt, int becn)
{
__bth_set_becn(pkt->hdr + pkt->offset, becn);
}
static inline u8 bth_resv6a(struct rxe_pkt_info *pkt)
{
return __bth_resv6a(pkt->hdr + pkt->offset);
}
static inline void bth_set_resv6a(struct rxe_pkt_info *pkt)
{
__bth_set_resv6a(pkt->hdr + pkt->offset);
}
static inline int bth_ack(struct rxe_pkt_info *pkt)
{
return __bth_ack(pkt->hdr + pkt->offset);
}
static inline void bth_set_ack(struct rxe_pkt_info *pkt, int ack)
{
__bth_set_ack(pkt->hdr + pkt->offset, ack);
}
static inline void bth_set_resv7(struct rxe_pkt_info *pkt)
{
__bth_set_resv7(pkt->hdr + pkt->offset);
}
static inline u32 bth_psn(struct rxe_pkt_info *pkt)
{
return __bth_psn(pkt->hdr + pkt->offset);
}
static inline void bth_set_psn(struct rxe_pkt_info *pkt, u32 psn)
{
__bth_set_psn(pkt->hdr + pkt->offset, psn);
}
static inline void bth_init(struct rxe_pkt_info *pkt, u8 opcode, int se,
int mig, int pad, u16 pkey, u32 qpn, int ack_req,
u32 psn)
{
struct rxe_bth *bth = (struct rxe_bth *)(pkt->hdr + pkt->offset);
bth->opcode = opcode;
bth->flags = (pad << 4) & BTH_PAD_MASK;
if (se)
bth->flags |= BTH_SE_MASK;
if (mig)
bth->flags |= BTH_MIG_MASK;
bth->pkey = cpu_to_be16(pkey);
bth->qpn = cpu_to_be32(qpn & BTH_QPN_MASK);
psn &= BTH_PSN_MASK;
if (ack_req)
psn |= BTH_ACK_MASK;
bth->apsn = cpu_to_be32(psn);
}
/******************************************************************************
* Reliable Datagram Extended Transport Header
******************************************************************************/
struct rxe_rdeth {
__be32 een;
};
#define RDETH_EEN_MASK (0x00ffffff)
static inline u8 __rdeth_een(void *arg)
{
struct rxe_rdeth *rdeth = arg;
return RDETH_EEN_MASK & be32_to_cpu(rdeth->een);
}
static inline void __rdeth_set_een(void *arg, u32 een)
{
struct rxe_rdeth *rdeth = arg;
rdeth->een = cpu_to_be32(RDETH_EEN_MASK & een);
}
static inline u8 rdeth_een(struct rxe_pkt_info *pkt)
{
return __rdeth_een(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_RDETH]);
}
static inline void rdeth_set_een(struct rxe_pkt_info *pkt, u32 een)
{
__rdeth_set_een(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_RDETH], een);
}
/******************************************************************************
* Datagram Extended Transport Header
******************************************************************************/
struct rxe_deth {
__be32 qkey;
__be32 sqp;
};
#define GSI_QKEY (0x80010000)
#define DETH_SQP_MASK (0x00ffffff)
static inline u32 __deth_qkey(void *arg)
{
struct rxe_deth *deth = arg;
return be32_to_cpu(deth->qkey);
}
static inline void __deth_set_qkey(void *arg, u32 qkey)
{
struct rxe_deth *deth = arg;
deth->qkey = cpu_to_be32(qkey);
}
static inline u32 __deth_sqp(void *arg)
{
struct rxe_deth *deth = arg;
return DETH_SQP_MASK & be32_to_cpu(deth->sqp);
}
static inline void __deth_set_sqp(void *arg, u32 sqp)
{
struct rxe_deth *deth = arg;
deth->sqp = cpu_to_be32(DETH_SQP_MASK & sqp);
}
static inline u32 deth_qkey(struct rxe_pkt_info *pkt)
{
return __deth_qkey(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_DETH]);
}
static inline void deth_set_qkey(struct rxe_pkt_info *pkt, u32 qkey)
{
__deth_set_qkey(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_DETH], qkey);
}
static inline u32 deth_sqp(struct rxe_pkt_info *pkt)
{
return __deth_sqp(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_DETH]);
}
static inline void deth_set_sqp(struct rxe_pkt_info *pkt, u32 sqp)
{
__deth_set_sqp(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_DETH], sqp);
}
/******************************************************************************
* RDMA Extended Transport Header
******************************************************************************/
struct rxe_reth {
__be64 va;
__be32 rkey;
__be32 len;
};
static inline u64 __reth_va(void *arg)
{
struct rxe_reth *reth = arg;
return be64_to_cpu(reth->va);
}
static inline void __reth_set_va(void *arg, u64 va)
{
struct rxe_reth *reth = arg;
reth->va = cpu_to_be64(va);
}
static inline u32 __reth_rkey(void *arg)
{
struct rxe_reth *reth = arg;
return be32_to_cpu(reth->rkey);
}
static inline void __reth_set_rkey(void *arg, u32 rkey)
{
struct rxe_reth *reth = arg;
reth->rkey = cpu_to_be32(rkey);
}
static inline u32 __reth_len(void *arg)
{
struct rxe_reth *reth = arg;
return be32_to_cpu(reth->len);
}
static inline void __reth_set_len(void *arg, u32 len)
{
struct rxe_reth *reth = arg;
reth->len = cpu_to_be32(len);
}
static inline u64 reth_va(struct rxe_pkt_info *pkt)
{
return __reth_va(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_RETH]);
}
static inline void reth_set_va(struct rxe_pkt_info *pkt, u64 va)
{
__reth_set_va(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_RETH], va);
}
static inline u32 reth_rkey(struct rxe_pkt_info *pkt)
{
return __reth_rkey(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_RETH]);
}
static inline void reth_set_rkey(struct rxe_pkt_info *pkt, u32 rkey)
{
__reth_set_rkey(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_RETH], rkey);
}
static inline u32 reth_len(struct rxe_pkt_info *pkt)
{
return __reth_len(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_RETH]);
}
static inline void reth_set_len(struct rxe_pkt_info *pkt, u32 len)
{
__reth_set_len(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_RETH], len);
}
/******************************************************************************
* Atomic Extended Transport Header
******************************************************************************/
struct rxe_atmeth {
__be64 va;
__be32 rkey;
__be64 swap_add;
__be64 comp;
} __attribute__((__packed__));
static inline u64 __atmeth_va(void *arg)
{
struct rxe_atmeth *atmeth = arg;
return be64_to_cpu(atmeth->va);
}
static inline void __atmeth_set_va(void *arg, u64 va)
{
struct rxe_atmeth *atmeth = arg;
atmeth->va = cpu_to_be64(va);
}
static inline u32 __atmeth_rkey(void *arg)
{
struct rxe_atmeth *atmeth = arg;
return be32_to_cpu(atmeth->rkey);
}
static inline void __atmeth_set_rkey(void *arg, u32 rkey)
{
struct rxe_atmeth *atmeth = arg;
atmeth->rkey = cpu_to_be32(rkey);
}
static inline u64 __atmeth_swap_add(void *arg)
{
struct rxe_atmeth *atmeth = arg;
return be64_to_cpu(atmeth->swap_add);
}
static inline void __atmeth_set_swap_add(void *arg, u64 swap_add)
{
struct rxe_atmeth *atmeth = arg;
atmeth->swap_add = cpu_to_be64(swap_add);
}
static inline u64 __atmeth_comp(void *arg)
{
struct rxe_atmeth *atmeth = arg;
return be64_to_cpu(atmeth->comp);
}
static inline void __atmeth_set_comp(void *arg, u64 comp)
{
struct rxe_atmeth *atmeth = arg;
atmeth->comp = cpu_to_be64(comp);
}
static inline u64 atmeth_va(struct rxe_pkt_info *pkt)
{
return __atmeth_va(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_ATMETH]);
}
static inline void atmeth_set_va(struct rxe_pkt_info *pkt, u64 va)
{
__atmeth_set_va(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_ATMETH], va);
}
static inline u32 atmeth_rkey(struct rxe_pkt_info *pkt)
{
return __atmeth_rkey(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_ATMETH]);
}
static inline void atmeth_set_rkey(struct rxe_pkt_info *pkt, u32 rkey)
{
__atmeth_set_rkey(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_ATMETH], rkey);
}
static inline u64 atmeth_swap_add(struct rxe_pkt_info *pkt)
{
return __atmeth_swap_add(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_ATMETH]);
}
static inline void atmeth_set_swap_add(struct rxe_pkt_info *pkt, u64 swap_add)
{
__atmeth_set_swap_add(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_ATMETH], swap_add);
}
static inline u64 atmeth_comp(struct rxe_pkt_info *pkt)
{
return __atmeth_comp(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_ATMETH]);
}
static inline void atmeth_set_comp(struct rxe_pkt_info *pkt, u64 comp)
{
__atmeth_set_comp(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_ATMETH], comp);
}
/******************************************************************************
* Ack Extended Transport Header
******************************************************************************/
struct rxe_aeth {
__be32 smsn;
};
#define AETH_SYN_MASK (0xff000000)
#define AETH_MSN_MASK (0x00ffffff)
enum aeth_syndrome {
AETH_TYPE_MASK = 0xe0,
AETH_ACK = 0x00,
AETH_RNR_NAK = 0x20,
AETH_RSVD = 0x40,
AETH_NAK = 0x60,
AETH_ACK_UNLIMITED = 0x1f,
AETH_NAK_PSN_SEQ_ERROR = 0x60,
AETH_NAK_INVALID_REQ = 0x61,
AETH_NAK_REM_ACC_ERR = 0x62,
AETH_NAK_REM_OP_ERR = 0x63,
AETH_NAK_INV_RD_REQ = 0x64,
};
static inline u8 __aeth_syn(void *arg)
{
struct rxe_aeth *aeth = arg;
return (AETH_SYN_MASK & be32_to_cpu(aeth->smsn)) >> 24;
}
static inline void __aeth_set_syn(void *arg, u8 syn)
{
struct rxe_aeth *aeth = arg;
u32 smsn = be32_to_cpu(aeth->smsn);
aeth->smsn = cpu_to_be32((AETH_SYN_MASK & (syn << 24)) |
(~AETH_SYN_MASK & smsn));
}
static inline u32 __aeth_msn(void *arg)
{
struct rxe_aeth *aeth = arg;
return AETH_MSN_MASK & be32_to_cpu(aeth->smsn);
}
static inline void __aeth_set_msn(void *arg, u32 msn)
{
struct rxe_aeth *aeth = arg;
u32 smsn = be32_to_cpu(aeth->smsn);
aeth->smsn = cpu_to_be32((AETH_MSN_MASK & msn) |
(~AETH_MSN_MASK & smsn));
}
static inline u8 aeth_syn(struct rxe_pkt_info *pkt)
{
return __aeth_syn(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_AETH]);
}
static inline void aeth_set_syn(struct rxe_pkt_info *pkt, u8 syn)
{
__aeth_set_syn(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_AETH], syn);
}
static inline u32 aeth_msn(struct rxe_pkt_info *pkt)
{
return __aeth_msn(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_AETH]);
}
static inline void aeth_set_msn(struct rxe_pkt_info *pkt, u32 msn)
{
__aeth_set_msn(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_AETH], msn);
}
/******************************************************************************
* Atomic Ack Extended Transport Header
******************************************************************************/
struct rxe_atmack {
__be64 orig;
};
static inline u64 __atmack_orig(void *arg)
{
struct rxe_atmack *atmack = arg;
return be64_to_cpu(atmack->orig);
}
static inline void __atmack_set_orig(void *arg, u64 orig)
{
struct rxe_atmack *atmack = arg;
atmack->orig = cpu_to_be64(orig);
}
static inline u64 atmack_orig(struct rxe_pkt_info *pkt)
{
return __atmack_orig(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_ATMACK]);
}
static inline void atmack_set_orig(struct rxe_pkt_info *pkt, u64 orig)
{
__atmack_set_orig(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_ATMACK], orig);
}
/******************************************************************************
* Immediate Extended Transport Header
******************************************************************************/
struct rxe_immdt {
__be32 imm;
};
static inline __be32 __immdt_imm(void *arg)
{
struct rxe_immdt *immdt = arg;
return immdt->imm;
}
static inline void __immdt_set_imm(void *arg, __be32 imm)
{
struct rxe_immdt *immdt = arg;
immdt->imm = imm;
}
static inline __be32 immdt_imm(struct rxe_pkt_info *pkt)
{
return __immdt_imm(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_IMMDT]);
}
static inline void immdt_set_imm(struct rxe_pkt_info *pkt, __be32 imm)
{
__immdt_set_imm(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_IMMDT], imm);
}
/******************************************************************************
* Invalidate Extended Transport Header
******************************************************************************/
struct rxe_ieth {
__be32 rkey;
};
static inline u32 __ieth_rkey(void *arg)
{
struct rxe_ieth *ieth = arg;
return be32_to_cpu(ieth->rkey);
}
static inline void __ieth_set_rkey(void *arg, u32 rkey)
{
struct rxe_ieth *ieth = arg;
ieth->rkey = cpu_to_be32(rkey);
}
static inline u32 ieth_rkey(struct rxe_pkt_info *pkt)
{
return __ieth_rkey(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_IETH]);
}
static inline void ieth_set_rkey(struct rxe_pkt_info *pkt, u32 rkey)
{
__ieth_set_rkey(pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_IETH], rkey);
}
enum rxe_hdr_length {
RXE_BTH_BYTES = sizeof(struct rxe_bth),
RXE_DETH_BYTES = sizeof(struct rxe_deth),
RXE_IMMDT_BYTES = sizeof(struct rxe_immdt),
RXE_RETH_BYTES = sizeof(struct rxe_reth),
RXE_AETH_BYTES = sizeof(struct rxe_aeth),
RXE_ATMACK_BYTES = sizeof(struct rxe_atmack),
RXE_ATMETH_BYTES = sizeof(struct rxe_atmeth),
RXE_IETH_BYTES = sizeof(struct rxe_ieth),
RXE_RDETH_BYTES = sizeof(struct rxe_rdeth),
};
static inline size_t header_size(struct rxe_pkt_info *pkt)
{
return pkt->offset + rxe_opcode[pkt->opcode].length;
}
static inline void *payload_addr(struct rxe_pkt_info *pkt)
{
return pkt->hdr + pkt->offset
+ rxe_opcode[pkt->opcode].offset[RXE_PAYLOAD];
}
static inline size_t payload_size(struct rxe_pkt_info *pkt)
{
return pkt->paylen - rxe_opcode[pkt->opcode].offset[RXE_PAYLOAD]
- bth_pad(pkt) - RXE_ICRC_SIZE;
}
#endif /* RXE_HDR_H */

96
drivers/infiniband/sw/rxe/rxe_icrc.c Normal file
View file

@ -0,0 +1,96 @@
/*
* Copyright (c) 2016 Mellanox Technologies Ltd. All rights reserved.
* Copyright (c) 2015 System Fabric Works, Inc. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenIB.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
#include "rxe.h"
#include "rxe_loc.h"
/* Compute a partial ICRC for all the IB transport headers. */
u32 rxe_icrc_hdr(struct rxe_pkt_info *pkt, struct sk_buff *skb)
{
unsigned int bth_offset = 0;
struct iphdr *ip4h = NULL;
struct ipv6hdr *ip6h = NULL;
struct udphdr *udph;
struct rxe_bth *bth;
int crc;
int length;
int hdr_size = sizeof(struct udphdr) +
(skb->protocol == htons(ETH_P_IP) ?
sizeof(struct iphdr) : sizeof(struct ipv6hdr));
/* pseudo header buffer size is calculate using ipv6 header size since
* it is bigger than ipv4
*/
u8 pshdr[sizeof(struct udphdr) +
sizeof(struct ipv6hdr) +
RXE_BTH_BYTES];
/* This seed is the result of computing a CRC with a seed of
* 0xfffffff and 8 bytes of 0xff representing a masked LRH.
*/
crc = 0xdebb20e3;
if (skb->protocol == htons(ETH_P_IP)) { /* IPv4 */
memcpy(pshdr, ip_hdr(skb), hdr_size);
ip4h = (struct iphdr *)pshdr;
udph = (struct udphdr *)(ip4h + 1);
ip4h->ttl = 0xff;
ip4h->check = CSUM_MANGLED_0;
ip4h->tos = 0xff;
} else { /* IPv6 */
memcpy(pshdr, ipv6_hdr(skb), hdr_size);
ip6h = (struct ipv6hdr *)pshdr;
udph = (struct udphdr *)(ip6h + 1);
memset(ip6h->flow_lbl, 0xff, sizeof(ip6h->flow_lbl));
ip6h->priority = 0xf;
ip6h->hop_limit = 0xff;
}
udph->check = CSUM_MANGLED_0;
bth_offset += hdr_size;
memcpy(&pshdr[bth_offset], pkt->hdr, RXE_BTH_BYTES);
bth = (struct rxe_bth *)&pshdr[bth_offset];
/* exclude bth.resv8a */
bth->qpn |= cpu_to_be32(~BTH_QPN_MASK);
length = hdr_size + RXE_BTH_BYTES;
crc = crc32_le(crc, pshdr, length);
/* And finish to compute the CRC on the remainder of the headers. */
crc = crc32_le(crc, pkt->hdr + RXE_BTH_BYTES,
rxe_opcode[pkt->opcode].length - RXE_BTH_BYTES);
return crc;
}

286
drivers/infiniband/sw/rxe/rxe_loc.h Normal file
View file

@ -0,0 +1,286 @@
/*
* Copyright (c) 2016 Mellanox Technologies Ltd. All rights reserved.
* Copyright (c) 2015 System Fabric Works, Inc. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenIB.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
#ifndef RXE_LOC_H
#define RXE_LOC_H
/* rxe_av.c */
int rxe_av_chk_attr(struct rxe_dev *rxe, struct ib_ah_attr *attr);
int rxe_av_from_attr(struct rxe_dev *rxe, u8 port_num,
struct rxe_av *av, struct ib_ah_attr *attr);
int rxe_av_to_attr(struct rxe_dev *rxe, struct rxe_av *av,
struct ib_ah_attr *attr);
int rxe_av_fill_ip_info(struct rxe_dev *rxe,
struct rxe_av *av,
struct ib_ah_attr *attr,
struct ib_gid_attr *sgid_attr,
union ib_gid *sgid);
struct rxe_av *rxe_get_av(struct rxe_pkt_info *pkt);
/* rxe_cq.c */
int rxe_cq_chk_attr(struct rxe_dev *rxe, struct rxe_cq *cq,
int cqe, int comp_vector, struct ib_udata *udata);
int rxe_cq_from_init(struct rxe_dev *rxe, struct rxe_cq *cq, int cqe,
int comp_vector, struct ib_ucontext *context,
struct ib_udata *udata);
int rxe_cq_resize_queue(struct rxe_cq *cq, int new_cqe, struct ib_udata *udata);
int rxe_cq_post(struct rxe_cq *cq, struct rxe_cqe *cqe, int solicited);
void rxe_cq_cleanup(void *arg);
/* rxe_mcast.c */
int rxe_mcast_get_grp(struct rxe_dev *rxe, union ib_gid *mgid,
struct rxe_mc_grp **grp_p);
int rxe_mcast_add_grp_elem(struct rxe_dev *rxe, struct rxe_qp *qp,
struct rxe_mc_grp *grp);
int rxe_mcast_drop_grp_elem(struct rxe_dev *rxe, struct rxe_qp *qp,
union ib_gid *mgid);
void rxe_drop_all_mcast_groups(struct rxe_qp *qp);
void rxe_mc_cleanup(void *arg);
/* rxe_mmap.c */
struct rxe_mmap_info {
struct list_head pending_mmaps;
struct ib_ucontext *context;
struct kref ref;
void *obj;
struct mminfo info;
};
void rxe_mmap_release(struct kref *ref);
struct rxe_mmap_info *rxe_create_mmap_info(struct rxe_dev *dev,
u32 size,
struct ib_ucontext *context,
void *obj);
int rxe_mmap(struct ib_ucontext *context, struct vm_area_struct *vma);
/* rxe_mr.c */
enum copy_direction {
to_mem_obj,
from_mem_obj,
};
int rxe_mem_init_dma(struct rxe_dev *rxe, struct rxe_pd *pd,
int access, struct rxe_mem *mem);
int rxe_mem_init_user(struct rxe_dev *rxe, struct rxe_pd *pd, u64 start,
u64 length, u64 iova, int access, struct ib_udata *udata,
struct rxe_mem *mr);
int rxe_mem_init_fast(struct rxe_dev *rxe, struct rxe_pd *pd,
int max_pages, struct rxe_mem *mem);
int rxe_mem_copy(struct rxe_mem *mem, u64 iova, void *addr,
int length, enum copy_direction dir, u32 *crcp);
int copy_data(struct rxe_dev *rxe, struct rxe_pd *pd, int access,
struct rxe_dma_info *dma, void *addr, int length,
enum copy_direction dir, u32 *crcp);
void *iova_to_vaddr(struct rxe_mem *mem, u64 iova, int length);
enum lookup_type {
lookup_local,
lookup_remote,
};
struct rxe_mem *lookup_mem(struct rxe_pd *pd, int access, u32 key,
enum lookup_type type);
int mem_check_range(struct rxe_mem *mem, u64 iova, size_t length);
int rxe_mem_map_pages(struct rxe_dev *rxe, struct rxe_mem *mem,
u64 *page, int num_pages, u64 iova);
void rxe_mem_cleanup(void *arg);
int advance_dma_data(struct rxe_dma_info *dma, unsigned int length);
/* rxe_qp.c */
int rxe_qp_chk_init(struct rxe_dev *rxe, struct ib_qp_init_attr *init);
int rxe_qp_from_init(struct rxe_dev *rxe, struct rxe_qp *qp, struct rxe_pd *pd,
struct ib_qp_init_attr *init, struct ib_udata *udata,
struct ib_pd *ibpd);
int rxe_qp_to_init(struct rxe_qp *qp, struct ib_qp_init_attr *init);
int rxe_qp_chk_attr(struct rxe_dev *rxe, struct rxe_qp *qp,
struct ib_qp_attr *attr, int mask);
int rxe_qp_from_attr(struct rxe_qp *qp, struct ib_qp_attr *attr,
int mask, struct ib_udata *udata);
int rxe_qp_to_attr(struct rxe_qp *qp, struct ib_qp_attr *attr, int mask);
void rxe_qp_error(struct rxe_qp *qp);
void rxe_qp_destroy(struct rxe_qp *qp);
void rxe_qp_cleanup(void *arg);
static inline int qp_num(struct rxe_qp *qp)
{
return qp->ibqp.qp_num;
}
static inline enum ib_qp_type qp_type(struct rxe_qp *qp)
{
return qp->ibqp.qp_type;
}
static inline enum ib_qp_state qp_state(struct rxe_qp *qp)
{
return qp->attr.qp_state;
}
static inline int qp_mtu(struct rxe_qp *qp)
{
if (qp->ibqp.qp_type == IB_QPT_RC || qp->ibqp.qp_type == IB_QPT_UC)
return qp->attr.path_mtu;
else
return RXE_PORT_MAX_MTU;
}
static inline int rcv_wqe_size(int max_sge)
{
return sizeof(struct rxe_recv_wqe) +
max_sge * sizeof(struct ib_sge);
}
void free_rd_atomic_resource(struct rxe_qp *qp, struct resp_res *res);
static inline void rxe_advance_resp_resource(struct rxe_qp *qp)
{
qp->resp.res_head++;
if (unlikely(qp->resp.res_head == qp->attr.max_rd_atomic))
qp->resp.res_head = 0;
}
void retransmit_timer(unsigned long data);
void rnr_nak_timer(unsigned long data);
void dump_qp(struct rxe_qp *qp);
/* rxe_srq.c */
#define IB_SRQ_INIT_MASK (~IB_SRQ_LIMIT)
int rxe_srq_chk_attr(struct rxe_dev *rxe, struct rxe_srq *srq,
struct ib_srq_attr *attr, enum ib_srq_attr_mask mask);
int rxe_srq_from_init(struct rxe_dev *rxe, struct rxe_srq *srq,
struct ib_srq_init_attr *init,
struct ib_ucontext *context, struct ib_udata *udata);
int rxe_srq_from_attr(struct rxe_dev *rxe, struct rxe_srq *srq,
struct ib_srq_attr *attr, enum ib_srq_attr_mask mask,
struct ib_udata *udata);
extern struct ib_dma_mapping_ops rxe_dma_mapping_ops;
void rxe_release(struct kref *kref);
int rxe_completer(void *arg);
int rxe_requester(void *arg);
int rxe_responder(void *arg);
u32 rxe_icrc_hdr(struct rxe_pkt_info *pkt, struct sk_buff *skb);
void rxe_resp_queue_pkt(struct rxe_dev *rxe,
struct rxe_qp *qp, struct sk_buff *skb);
void rxe_comp_queue_pkt(struct rxe_dev *rxe,
struct rxe_qp *qp, struct sk_buff *skb);
static inline unsigned wr_opcode_mask(int opcode, struct rxe_qp *qp)
{
return rxe_wr_opcode_info[opcode].mask[qp->ibqp.qp_type];
}
static inline int rxe_xmit_packet(struct rxe_dev *rxe, struct rxe_qp *qp,
struct rxe_pkt_info *pkt, struct sk_buff *skb)
{
int err;
int is_request = pkt->mask & RXE_REQ_MASK;
if ((is_request && (qp->req.state != QP_STATE_READY)) ||
(!is_request && (qp->resp.state != QP_STATE_READY))) {
pr_info("Packet dropped. QP is not in ready state\n");
goto drop;
}
if (pkt->mask & RXE_LOOPBACK_MASK) {
memcpy(SKB_TO_PKT(skb), pkt, sizeof(*pkt));
err = rxe->ifc_ops->loopback(skb);
} else {
err = rxe->ifc_ops->send(rxe, pkt, skb);
}
if (err) {
rxe->xmit_errors++;
return err;
}
atomic_inc(&qp->skb_out);
if ((qp_type(qp) != IB_QPT_RC) &&
(pkt->mask & RXE_END_MASK)) {
pkt->wqe->state = wqe_state_done;
rxe_run_task(&qp->comp.task, 1);
}
goto done;
drop:
kfree_skb(skb);
err = 0;
done:
return err;
}
#endif /* RXE_LOC_H */

190
drivers/infiniband/sw/rxe/rxe_mcast.c Normal file
View file

@ -0,0 +1,190 @@
/*
* Copyright (c) 2016 Mellanox Technologies Ltd. All rights reserved.
* Copyright (c) 2015 System Fabric Works, Inc. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenIB.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
#include "rxe.h"
#include "rxe_loc.h"
int rxe_mcast_get_grp(struct rxe_dev *rxe, union ib_gid *mgid,
struct rxe_mc_grp **grp_p)
{
int err;
struct rxe_mc_grp *grp;
if (rxe->attr.max_mcast_qp_attach == 0) {
err = -EINVAL;
goto err1;
}
grp = rxe_pool_get_key(&rxe->mc_grp_pool, mgid);
if (grp)
goto done;
grp = rxe_alloc(&rxe->mc_grp_pool);
if (!grp) {
err = -ENOMEM;
goto err1;
}
INIT_LIST_HEAD(&grp->qp_list);
spin_lock_init(&grp->mcg_lock);
grp->rxe = rxe;
rxe_add_key(grp, mgid);
err = rxe->ifc_ops->mcast_add(rxe, mgid);
if (err)
goto err2;
done:
*grp_p = grp;
return 0;
err2:
rxe_drop_ref(grp);
err1:
return err;
}
int rxe_mcast_add_grp_elem(struct rxe_dev *rxe, struct rxe_qp *qp,
struct rxe_mc_grp *grp)
{
int err;
struct rxe_mc_elem *elem;
/* check to see of the qp is already a member of the group */
spin_lock_bh(&qp->grp_lock);
spin_lock_bh(&grp->mcg_lock);
list_for_each_entry(elem, &grp->qp_list, qp_list) {
if (elem->qp == qp) {
err = 0;
goto out;
}
}
if (grp->num_qp >= rxe->attr.max_mcast_qp_attach) {
err = -ENOMEM;
goto out;
}
elem = rxe_alloc(&rxe->mc_elem_pool);
if (!elem) {
err = -ENOMEM;
goto out;
}
/* each qp holds a ref on the grp */
rxe_add_ref(grp);
grp->num_qp++;
elem->qp = qp;
elem->grp = grp;
list_add(&elem->qp_list, &grp->qp_list);
list_add(&elem->grp_list, &qp->grp_list);
err = 0;
out:
spin_unlock_bh(&grp->mcg_lock);
spin_unlock_bh(&qp->grp_lock);
return err;
}
int rxe_mcast_drop_grp_elem(struct rxe_dev *rxe, struct rxe_qp *qp,
union ib_gid *mgid)
{
struct rxe_mc_grp *grp;
struct rxe_mc_elem *elem, *tmp;
grp = rxe_pool_get_key(&rxe->mc_grp_pool, mgid);
if (!grp)
goto err1;
spin_lock_bh(&qp->grp_lock);
spin_lock_bh(&grp->mcg_lock);
list_for_each_entry_safe(elem, tmp, &grp->qp_list, qp_list) {
if (elem->qp == qp) {
list_del(&elem->qp_list);
list_del(&elem->grp_list);
grp->num_qp--;
spin_unlock_bh(&grp->mcg_lock);
spin_unlock_bh(&qp->grp_lock);
rxe_drop_ref(elem);
rxe_drop_ref(grp); /* ref held by QP */
rxe_drop_ref(grp); /* ref from get_key */
return 0;
}
}
spin_unlock_bh(&grp->mcg_lock);
spin_unlock_bh(&qp->grp_lock);
rxe_drop_ref(grp); /* ref from get_key */
err1:
return -EINVAL;
}
void rxe_drop_all_mcast_groups(struct rxe_qp *qp)
{
struct rxe_mc_grp *grp;
struct rxe_mc_elem *elem;
while (1) {
spin_lock_bh(&qp->grp_lock);
if (list_empty(&qp->grp_list)) {
spin_unlock_bh(&qp->grp_lock);
break;
}
elem = list_first_entry(&qp->grp_list, struct rxe_mc_elem,
grp_list);
list_del(&elem->grp_list);
spin_unlock_bh(&qp->grp_lock);
grp = elem->grp;
spin_lock_bh(&grp->mcg_lock);
list_del(&elem->qp_list);
grp->num_qp--;
spin_unlock_bh(&grp->mcg_lock);
rxe_drop_ref(grp);
rxe_drop_ref(elem);
}
}
void rxe_mc_cleanup(void *arg)
{
struct rxe_mc_grp *grp = arg;
struct rxe_dev *rxe = grp->rxe;
rxe_drop_key(grp);
rxe->ifc_ops->mcast_delete(rxe, &grp->mgid);
}

173
drivers/infiniband/sw/rxe/rxe_mmap.c Normal file
View file

@ -0,0 +1,173 @@
/*
* Copyright (c) 2016 Mellanox Technologies Ltd. All rights reserved.
* Copyright (c) 2015 System Fabric Works, Inc. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenIB.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
#include <linux/module.h>
#include <linux/vmalloc.h>
#include <linux/mm.h>
#include <linux/errno.h>
#include <asm/pgtable.h>
#include "rxe.h"
#include "rxe_loc.h"
#include "rxe_queue.h"
void rxe_mmap_release(struct kref *ref)
{
struct rxe_mmap_info *ip = container_of(ref,
struct rxe_mmap_info, ref);
struct rxe_dev *rxe = to_rdev(ip->context->device);
spin_lock_bh(&rxe->pending_lock);
if (!list_empty(&ip->pending_mmaps))
list_del(&ip->pending_mmaps);
spin_unlock_bh(&rxe->pending_lock);
vfree(ip->obj); /* buf */
kfree(ip);
}
/*
* open and close keep track of how many times the memory region is mapped,
* to avoid releasing it.
*/
static void rxe_vma_open(struct vm_area_struct *vma)
{
struct rxe_mmap_info *ip = vma->vm_private_data;
kref_get(&ip->ref);
}
static void rxe_vma_close(struct vm_area_struct *vma)
{
struct rxe_mmap_info *ip = vma->vm_private_data;
kref_put(&ip->ref, rxe_mmap_release);
}
static struct vm_operations_struct rxe_vm_ops = {
.open = rxe_vma_open,
.close = rxe_vma_close,
};
/**
* rxe_mmap - create a new mmap region
* @context: the IB user context of the process making the mmap() call
* @vma: the VMA to be initialized
* Return zero if the mmap is OK. Otherwise, return an errno.
*/
int rxe_mmap(struct ib_ucontext *context, struct vm_area_struct *vma)
{
struct rxe_dev *rxe = to_rdev(context->device);
unsigned long offset = vma->vm_pgoff << PAGE_SHIFT;
unsigned long size = vma->vm_end - vma->vm_start;
struct rxe_mmap_info *ip, *pp;
int ret;
/*
* Search the device's list of objects waiting for a mmap call.
* Normally, this list is very short since a call to create a
* CQ, QP, or SRQ is soon followed by a call to mmap().
*/
spin_lock_bh(&rxe->pending_lock);
list_for_each_entry_safe(ip, pp, &rxe->pending_mmaps, pending_mmaps) {
if (context != ip->context || (__u64)offset != ip->info.offset)
continue;
/* Don't allow a mmap larger than the object. */
if (size > ip->info.size) {
pr_err("mmap region is larger than the object!\n");
spin_unlock_bh(&rxe->pending_lock);
ret = -EINVAL;
goto done;
}
goto found_it;
}
pr_warn("unable to find pending mmap info\n");
spin_unlock_bh(&rxe->pending_lock);
ret = -EINVAL;
goto done;
found_it:
list_del_init(&ip->pending_mmaps);
spin_unlock_bh(&rxe->pending_lock);
ret = remap_vmalloc_range(vma, ip->obj, 0);
if (ret) {
pr_err("rxe: err %d from remap_vmalloc_range\n", ret);
goto done;
}
vma->vm_ops = &rxe_vm_ops;
vma->vm_private_data = ip;
rxe_vma_open(vma);
done:
return ret;
}
/*
* Allocate information for rxe_mmap
*/
struct rxe_mmap_info *rxe_create_mmap_info(struct rxe_dev *rxe,
u32 size,
struct ib_ucontext *context,
void *obj)
{
struct rxe_mmap_info *ip;
ip = kmalloc(sizeof(*ip), GFP_KERNEL);
if (!ip)
return NULL;
size = PAGE_ALIGN(size);
spin_lock_bh(&rxe->mmap_offset_lock);
if (rxe->mmap_offset == 0)
rxe->mmap_offset = PAGE_SIZE;
ip->info.offset = rxe->mmap_offset;
rxe->mmap_offset += size;
spin_unlock_bh(&rxe->mmap_offset_lock);
INIT_LIST_HEAD(&ip->pending_mmaps);
ip->info.size = size;
ip->context = context;
ip->obj = obj;
kref_init(&ip->ref);
return ip;
}

643
drivers/infiniband/sw/rxe/rxe_mr.c Normal file
View file

@ -0,0 +1,643 @@
/*
* Copyright (c) 2016 Mellanox Technologies Ltd. All rights reserved.
* Copyright (c) 2015 System Fabric Works, Inc. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenIB.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
#include "rxe.h"
#include "rxe_loc.h"
/*
* lfsr (linear feedback shift register) with period 255
*/
static u8 rxe_get_key(void)
{
static unsigned key = 1;
key = key << 1;
key |= (0 != (key & 0x100)) ^ (0 != (key & 0x10))
^ (0 != (key & 0x80)) ^ (0 != (key & 0x40));
key &= 0xff;
return key;
}
int mem_check_range(struct rxe_mem *mem, u64 iova, size_t length)
{
switch (mem->type) {
case RXE_MEM_TYPE_DMA:
return 0;
case RXE_MEM_TYPE_MR:
case RXE_MEM_TYPE_FMR:
return ((iova < mem->iova) ||
((iova + length) > (mem->iova + mem->length))) ?
-EFAULT : 0;
default:
return -EFAULT;
}
}
#define IB_ACCESS_REMOTE (IB_ACCESS_REMOTE_READ \
| IB_ACCESS_REMOTE_WRITE \
| IB_ACCESS_REMOTE_ATOMIC)
static void rxe_mem_init(int access, struct rxe_mem *mem)
{
u32 lkey = mem->pelem.index << 8 | rxe_get_key();
u32 rkey = (access & IB_ACCESS_REMOTE) ? lkey : 0;
if (mem->pelem.pool->type == RXE_TYPE_MR) {
mem->ibmr.lkey = lkey;
mem->ibmr.rkey = rkey;
}
mem->lkey = lkey;
mem->rkey = rkey;
mem->state = RXE_MEM_STATE_INVALID;
mem->type = RXE_MEM_TYPE_NONE;
mem->map_shift = ilog2(RXE_BUF_PER_MAP);
}
void rxe_mem_cleanup(void *arg)
{
struct rxe_mem *mem = arg;
int i;
if (mem->umem)
ib_umem_release(mem->umem);
if (mem->map) {
for (i = 0; i < mem->num_map; i++)
kfree(mem->map[i]);
kfree(mem->map);
}
}
static int rxe_mem_alloc(struct rxe_dev *rxe, struct rxe_mem *mem, int num_buf)
{
int i;
int num_map;
struct rxe_map **map = mem->map;
num_map = (num_buf + RXE_BUF_PER_MAP - 1) / RXE_BUF_PER_MAP;
mem->map = kmalloc_array(num_map, sizeof(*map), GFP_KERNEL);
if (!mem->map)
goto err1;
for (i = 0; i < num_map; i++) {
mem->map[i] = kmalloc(sizeof(**map), GFP_KERNEL);
if (!mem->map[i])
goto err2;
}
WARN_ON(!is_power_of_2(RXE_BUF_PER_MAP));
mem->map_shift = ilog2(RXE_BUF_PER_MAP);
mem->map_mask = RXE_BUF_PER_MAP - 1;
mem->num_buf = num_buf;
mem->num_map = num_map;
mem->max_buf = num_map * RXE_BUF_PER_MAP;
return 0;
err2:
for (i--; i >= 0; i--)
kfree(mem->map[i]);
kfree(mem->map);
err1:
return -ENOMEM;
}
int rxe_mem_init_dma(struct rxe_dev *rxe, struct rxe_pd *pd,
int access, struct rxe_mem *mem)
{
rxe_mem_init(access, mem);
mem->pd = pd;
mem->access = access;
mem->state = RXE_MEM_STATE_VALID;
mem->type = RXE_MEM_TYPE_DMA;
return 0;
}
int rxe_mem_init_user(struct rxe_dev *rxe, struct rxe_pd *pd, u64 start,
u64 length, u64 iova, int access, struct ib_udata *udata,
struct rxe_mem *mem)
{
int entry;
struct rxe_map **map;
struct rxe_phys_buf *buf = NULL;
struct ib_umem *umem;
struct scatterlist *sg;
int num_buf;
void *vaddr;
int err;
umem = ib_umem_get(pd->ibpd.uobject->context, start, length, access, 0);
if (IS_ERR(umem)) {
pr_warn("err %d from rxe_umem_get\n",
(int)PTR_ERR(umem));
err = -EINVAL;
goto err1;
}
mem->umem = umem;
num_buf = umem->nmap;
rxe_mem_init(access, mem);
err = rxe_mem_alloc(rxe, mem, num_buf);
if (err) {
pr_warn("err %d from rxe_mem_alloc\n", err);
ib_umem_release(umem);
goto err1;
}
WARN_ON(!is_power_of_2(umem->page_size));
mem->page_shift = ilog2(umem->page_size);
mem->page_mask = umem->page_size - 1;
num_buf = 0;
map = mem->map;
if (length > 0) {
buf = map[0]->buf;
for_each_sg(umem->sg_head.sgl, sg, umem->nmap, entry) {
vaddr = page_address(sg_page(sg));
if (!vaddr) {
pr_warn("null vaddr\n");
err = -ENOMEM;
goto err1;
}
buf->addr = (uintptr_t)vaddr;
buf->size = umem->page_size;
num_buf++;
buf++;
if (num_buf >= RXE_BUF_PER_MAP) {
map++;
buf = map[0]->buf;
num_buf = 0;
}
}
}
mem->pd = pd;
mem->umem = umem;
mem->access = access;
mem->length = length;
mem->iova = iova;
mem->va = start;
mem->offset = ib_umem_offset(umem);
mem->state = RXE_MEM_STATE_VALID;
mem->type = RXE_MEM_TYPE_MR;
return 0;
err1:
return err;
}
int rxe_mem_init_fast(struct rxe_dev *rxe, struct rxe_pd *pd,
int max_pages, struct rxe_mem *mem)
{
int err;
rxe_mem_init(0, mem);
/* In fastreg, we also set the rkey */
mem->ibmr.rkey = mem->ibmr.lkey;
err = rxe_mem_alloc(rxe, mem, max_pages);
if (err)
goto err1;
mem->pd = pd;
mem->max_buf = max_pages;
mem->state = RXE_MEM_STATE_FREE;
mem->type = RXE_MEM_TYPE_MR;
return 0;
err1:
return err;
}
static void lookup_iova(
struct rxe_mem *mem,
u64 iova,
int *m_out,
int *n_out,
size_t *offset_out)
{
size_t offset = iova - mem->iova + mem->offset;
int map_index;
int buf_index;
u64 length;
if (likely(mem->page_shift)) {
*offset_out = offset & mem->page_mask;
offset >>= mem->page_shift;
*n_out = offset & mem->map_mask;
*m_out = offset >> mem->map_shift;
} else {
map_index = 0;
buf_index = 0;
length = mem->map[map_index]->buf[buf_index].size;
while (offset >= length) {
offset -= length;
buf_index++;
if (buf_index == RXE_BUF_PER_MAP) {
map_index++;
buf_index = 0;
}
length = mem->map[map_index]->buf[buf_index].size;
}
*m_out = map_index;
*n_out = buf_index;
*offset_out = offset;
}
}
void *iova_to_vaddr(struct rxe_mem *mem, u64 iova, int length)
{
size_t offset;
int m, n;
void *addr;
if (mem->state != RXE_MEM_STATE_VALID) {
pr_warn("mem not in valid state\n");
addr = NULL;
goto out;
}
if (!mem->map) {
addr = (void *)(uintptr_t)iova;
goto out;
}
if (mem_check_range(mem, iova, length)) {
pr_warn("range violation\n");
addr = NULL;
goto out;
}
lookup_iova(mem, iova, &m, &n, &offset);
if (offset + length > mem->map[m]->buf[n].size) {
pr_warn("crosses page boundary\n");
addr = NULL;
goto out;
}
addr = (void *)(uintptr_t)mem->map[m]->buf[n].addr + offset;
out:
return addr;
}
/* copy data from a range (vaddr, vaddr+length-1) to or from
* a mem object starting at iova. Compute incremental value of
* crc32 if crcp is not zero. caller must hold a reference to mem
*/
int rxe_mem_copy(struct rxe_mem *mem, u64 iova, void *addr, int length,
enum copy_direction dir, u32 *crcp)
{
int err;
int bytes;
u8 *va;
struct rxe_map **map;
struct rxe_phys_buf *buf;
int m;
int i;
size_t offset;
u32 crc = crcp ? (*crcp) : 0;
if (mem->type == RXE_MEM_TYPE_DMA) {
u8 *src, *dest;
src = (dir == to_mem_obj) ?
addr : ((void *)(uintptr_t)iova);
dest = (dir == to_mem_obj) ?
((void *)(uintptr_t)iova) : addr;
if (crcp)
*crcp = crc32_le(*crcp, src, length);
memcpy(dest, src, length);
return 0;
}
WARN_ON(!mem->map);
err = mem_check_range(mem, iova, length);
if (err) {
err = -EFAULT;
goto err1;
}
lookup_iova(mem, iova, &m, &i, &offset);
map = mem->map + m;
buf = map[0]->buf + i;
while (length > 0) {
u8 *src, *dest;
va = (u8 *)(uintptr_t)buf->addr + offset;
src = (dir == to_mem_obj) ? addr : va;
dest = (dir == to_mem_obj) ? va : addr;
bytes = buf->size - offset;
if (bytes > length)
bytes = length;
if (crcp)
crc = crc32_le(crc, src, bytes);
memcpy(dest, src, bytes);
length -= bytes;
addr += bytes;
offset = 0;
buf++;
i++;
if (i == RXE_BUF_PER_MAP) {
i = 0;
map++;
buf = map[0]->buf;
}
}
if (crcp)
*crcp = crc;
return 0;
err1:
return err;
}
/* copy data in or out of a wqe, i.e. sg list
* under the control of a dma descriptor
*/
int copy_data(
struct rxe_dev *rxe,
struct rxe_pd *pd,
int access,
struct rxe_dma_info *dma,
void *addr,
int length,
enum copy_direction dir,
u32 *crcp)
{
int bytes;
struct rxe_sge *sge = &dma->sge[dma->cur_sge];
int offset = dma->sge_offset;
int resid = dma->resid;
struct rxe_mem *mem = NULL;
u64 iova;
int err;
if (length == 0)
return 0;
if (length > resid) {
err = -EINVAL;
goto err2;
}
if (sge->length && (offset < sge->length)) {
mem = lookup_mem(pd, access, sge->lkey, lookup_local);
if (!mem) {
err = -EINVAL;
goto err1;
}
}
while (length > 0) {
bytes = length;
if (offset >= sge->length) {
if (mem) {
rxe_drop_ref(mem);
mem = NULL;
}
sge++;
dma->cur_sge++;
offset = 0;
if (dma->cur_sge >= dma->num_sge) {
err = -ENOSPC;
goto err2;
}
if (sge->length) {
mem = lookup_mem(pd, access, sge->lkey,
lookup_local);
if (!mem) {
err = -EINVAL;
goto err1;
}
} else {
continue;
}
}
if (bytes > sge->length - offset)
bytes = sge->length - offset;
if (bytes > 0) {
iova = sge->addr + offset;
err = rxe_mem_copy(mem, iova, addr, bytes, dir, crcp);
if (err)
goto err2;
offset += bytes;
resid -= bytes;
length -= bytes;
addr += bytes;
}
}
dma->sge_offset = offset;
dma->resid = resid;
if (mem)
rxe_drop_ref(mem);
return 0;
err2:
if (mem)
rxe_drop_ref(mem);
err1:
return err;
}
int advance_dma_data(struct rxe_dma_info *dma, unsigned int length)
{
struct rxe_sge *sge = &dma->sge[dma->cur_sge];
int offset = dma->sge_offset;
int resid = dma->resid;
while (length) {
unsigned int bytes;
if (offset >= sge->length) {
sge++;
dma->cur_sge++;
offset = 0;
if (dma->cur_sge >= dma->num_sge)
return -ENOSPC;
}
bytes = length;
if (bytes > sge->length - offset)
bytes = sge->length - offset;
offset += bytes;
resid -= bytes;
length -= bytes;
}
dma->sge_offset = offset;
dma->resid = resid;
return 0;
}
/* (1) find the mem (mr or mw) corresponding to lkey/rkey
* depending on lookup_type
* (2) verify that the (qp) pd matches the mem pd
* (3) verify that the mem can support the requested access
* (4) verify that mem state is valid
*/
struct rxe_mem *lookup_mem(struct rxe_pd *pd, int access, u32 key,
enum lookup_type type)
{
struct rxe_mem *mem;
struct rxe_dev *rxe = to_rdev(pd->ibpd.device);
int index = key >> 8;
if (index >= RXE_MIN_MR_INDEX && index <= RXE_MAX_MR_INDEX) {
mem = rxe_pool_get_index(&rxe->mr_pool, index);
if (!mem)
goto err1;
} else {
goto err1;
}
if ((type == lookup_local && mem->lkey != key) ||
(type == lookup_remote && mem->rkey != key))
goto err2;
if (mem->pd != pd)
goto err2;
if (access && !(access & mem->access))
goto err2;
if (mem->state != RXE_MEM_STATE_VALID)
goto err2;
return mem;
err2:
rxe_drop_ref(mem);
err1:
return NULL;
}
int rxe_mem_map_pages(struct rxe_dev *rxe, struct rxe_mem *mem,
u64 *page, int num_pages, u64 iova)
{
int i;
int num_buf;
int err;
struct rxe_map **map;
struct rxe_phys_buf *buf;
int page_size;
if (num_pages > mem->max_buf) {
err = -EINVAL;
goto err1;
}
num_buf = 0;
page_size = 1 << mem->page_shift;
map = mem->map;
buf = map[0]->buf;
for (i = 0; i < num_pages; i++) {
buf->addr = *page++;
buf->size = page_size;
buf++;
num_buf++;
if (num_buf == RXE_BUF_PER_MAP) {
map++;
buf = map[0]->buf;
num_buf = 0;
}
}
mem->iova = iova;
mem->va = iova;
mem->length = num_pages << mem->page_shift;
mem->state = RXE_MEM_STATE_VALID;
return 0;
err1:
return err;
}

708
drivers/infiniband/sw/rxe/rxe_net.c Normal file
View file

@ -0,0 +1,708 @@
/*
* Copyright (c) 2016 Mellanox Technologies Ltd. All rights reserved.
* Copyright (c) 2015 System Fabric Works, Inc. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenIB.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
#include <linux/skbuff.h>
#include <linux/if_arp.h>
#include <linux/netdevice.h>
#include <linux/if.h>
#include <linux/if_vlan.h>
#include <net/udp_tunnel.h>
#include <net/sch_generic.h>
#include <linux/netfilter.h>
#include <rdma/ib_addr.h>
#include "rxe.h"
#include "rxe_net.h"
#include "rxe_loc.h"
static LIST_HEAD(rxe_dev_list);
static spinlock_t dev_list_lock; /* spinlock for device list */
struct rxe_dev *net_to_rxe(struct net_device *ndev)
{
struct rxe_dev *rxe;
struct rxe_dev *found = NULL;
spin_lock_bh(&dev_list_lock);
list_for_each_entry(rxe, &rxe_dev_list, list) {
if (rxe->ndev == ndev) {
found = rxe;
break;
}
}
spin_unlock_bh(&dev_list_lock);
return found;
}
struct rxe_dev *get_rxe_by_name(const char* name)
{
struct rxe_dev *rxe;
struct rxe_dev *found = NULL;
spin_lock_bh(&dev_list_lock);
list_for_each_entry(rxe, &rxe_dev_list, list) {
if (!strcmp(name, rxe->ib_dev.name)) {
found = rxe;
break;
}
}
spin_unlock_bh(&dev_list_lock);
return found;
}
struct rxe_recv_sockets recv_sockets;
static __be64 rxe_mac_to_eui64(struct net_device *ndev)
{
unsigned char *mac_addr = ndev->dev_addr;
__be64 eui64;
unsigned char *dst = (unsigned char *)&eui64;
dst[0] = mac_addr[0] ^ 2;
dst[1] = mac_addr[1];
dst[2] = mac_addr[2];
dst[3] = 0xff;
dst[4] = 0xfe;
dst[5] = mac_addr[3];
dst[6] = mac_addr[4];
dst[7] = mac_addr[5];
return eui64;
}
static __be64 node_guid(struct rxe_dev *rxe)
{
return rxe_mac_to_eui64(rxe->ndev);
}
static __be64 port_guid(struct rxe_dev *rxe)
{
return rxe_mac_to_eui64(rxe->ndev);
}
static struct device *dma_device(struct rxe_dev *rxe)
{
struct net_device *ndev;
ndev = rxe->ndev;
if (ndev->priv_flags & IFF_802_1Q_VLAN)
ndev = vlan_dev_real_dev(ndev);
return ndev->dev.parent;
}
static int mcast_add(struct rxe_dev *rxe, union ib_gid *mgid)
{
int err;
unsigned char ll_addr[ETH_ALEN];
ipv6_eth_mc_map((struct in6_addr *)mgid->raw, ll_addr);
err = dev_mc_add(rxe->ndev, ll_addr);
return err;
}
static int mcast_delete(struct rxe_dev *rxe, union ib_gid *mgid)
{
int err;
unsigned char ll_addr[ETH_ALEN];
ipv6_eth_mc_map((struct in6_addr *)mgid->raw, ll_addr);
err = dev_mc_del(rxe->ndev, ll_addr);
return err;
}
static struct dst_entry *rxe_find_route4(struct net_device *ndev,
struct in_addr *saddr,
struct in_addr *daddr)
{
struct rtable *rt;
struct flowi4 fl = { { 0 } };
memset(&fl, 0, sizeof(fl));
fl.flowi4_oif = ndev->ifindex;
memcpy(&fl.saddr, saddr, sizeof(*saddr));
memcpy(&fl.daddr, daddr, sizeof(*daddr));
fl.flowi4_proto = IPPROTO_UDP;
rt = ip_route_output_key(&init_net, &fl);
if (IS_ERR(rt)) {
pr_err_ratelimited("no route to %pI4\n", &daddr->s_addr);
return NULL;
}
return &rt->dst;
}
#if IS_ENABLED(CONFIG_IPV6)
static struct dst_entry *rxe_find_route6(struct net_device *ndev,
struct in6_addr *saddr,
struct in6_addr *daddr)
{
struct dst_entry *ndst;
struct flowi6 fl6 = { { 0 } };
memset(&fl6, 0, sizeof(fl6));
fl6.flowi6_oif = ndev->ifindex;
memcpy(&fl6.saddr, saddr, sizeof(*saddr));
memcpy(&fl6.daddr, daddr, sizeof(*daddr));
fl6.flowi6_proto = IPPROTO_UDP;
if (unlikely(ipv6_stub->ipv6_dst_lookup(sock_net(recv_sockets.sk6->sk),
recv_sockets.sk6->sk, &ndst, &fl6))) {
pr_err_ratelimited("no route to %pI6\n", daddr);
goto put;
}
if (unlikely(ndst->error)) {
pr_err("no route to %pI6\n", daddr);
goto put;
}
return ndst;
put:
dst_release(ndst);
return NULL;
}
#else
static struct dst_entry *rxe_find_route6(struct net_device *ndev,
struct in6_addr *saddr,
struct in6_addr *daddr)
{
return NULL;
}
#endif
static int rxe_udp_encap_recv(struct sock *sk, struct sk_buff *skb)
{
struct udphdr *udph;
struct net_device *ndev = skb->dev;
struct rxe_dev *rxe = net_to_rxe(ndev);
struct rxe_pkt_info *pkt = SKB_TO_PKT(skb);
if (!rxe)
goto drop;
if (skb_linearize(skb)) {
pr_err("skb_linearize failed\n");
goto drop;
}
udph = udp_hdr(skb);
pkt->rxe = rxe;
pkt->port_num = 1;
pkt->hdr = (u8 *)(udph + 1);
pkt->mask = RXE_GRH_MASK;
pkt->paylen = be16_to_cpu(udph->len) - sizeof(*udph);
return rxe_rcv(skb);
drop:
kfree_skb(skb);
return 0;
}
static struct socket *rxe_setup_udp_tunnel(struct net *net, __be16 port,
bool ipv6)
{
int err;
struct socket *sock;
struct udp_port_cfg udp_cfg;
struct udp_tunnel_sock_cfg tnl_cfg;
memset(&udp_cfg, 0, sizeof(udp_cfg));
if (ipv6) {
udp_cfg.family = AF_INET6;
udp_cfg.ipv6_v6only = 1;
} else {
udp_cfg.family = AF_INET;
}
udp_cfg.local_udp_port = port;
/* Create UDP socket */
err = udp_sock_create(net, &udp_cfg, &sock);
if (err < 0) {
pr_err("failed to create udp socket. err = %d\n", err);
return ERR_PTR(err);
}
tnl_cfg.sk_user_data = NULL;
tnl_cfg.encap_type = 1;
tnl_cfg.encap_rcv = rxe_udp_encap_recv;
tnl_cfg.encap_destroy = NULL;
/* Setup UDP tunnel */
setup_udp_tunnel_sock(net, sock, &tnl_cfg);
return sock;
}
static void rxe_release_udp_tunnel(struct socket *sk)
{
udp_tunnel_sock_release(sk);
}
static void prepare_udp_hdr(struct sk_buff *skb, __be16 src_port,
__be16 dst_port)
{
struct udphdr *udph;
__skb_push(skb, sizeof(*udph));
skb_reset_transport_header(skb);
udph = udp_hdr(skb);
udph->dest = dst_port;
udph->source = src_port;
udph->len = htons(skb->len);
udph->check = 0;
}
static void prepare_ipv4_hdr(struct dst_entry *dst, struct sk_buff *skb,
__be32 saddr, __be32 daddr, __u8 proto,
__u8 tos, __u8 ttl, __be16 df, bool xnet)
{
struct iphdr *iph;
skb_scrub_packet(skb, xnet);
skb_clear_hash(skb);
skb_dst_set(skb, dst);
memset(IPCB(skb), 0, sizeof(*IPCB(skb)));
skb_push(skb, sizeof(struct iphdr));
skb_reset_network_header(skb);
iph = ip_hdr(skb);
iph->version = IPVERSION;
iph->ihl = sizeof(struct iphdr) >> 2;
iph->frag_off = df;
iph->protocol = proto;
iph->tos = tos;
iph->daddr = daddr;
iph->saddr = saddr;
iph->ttl = ttl;
__ip_select_ident(dev_net(dst->dev), iph,
skb_shinfo(skb)->gso_segs ?: 1);
iph->tot_len = htons(skb->len);
ip_send_check(iph);
}
static void prepare_ipv6_hdr(struct dst_entry *dst, struct sk_buff *skb,
struct in6_addr *saddr, struct in6_addr *daddr,
__u8 proto, __u8 prio, __u8 ttl)
{
struct ipv6hdr *ip6h;
memset(&(IPCB(skb)->opt), 0, sizeof(IPCB(skb)->opt));
IPCB(skb)->flags &= ~(IPSKB_XFRM_TUNNEL_SIZE | IPSKB_XFRM_TRANSFORMED
| IPSKB_REROUTED);
skb_dst_set(skb, dst);
__skb_push(skb, sizeof(*ip6h));
skb_reset_network_header(skb);
ip6h = ipv6_hdr(skb);
ip6_flow_hdr(ip6h, prio, htonl(0));
ip6h->payload_len = htons(skb->len);
ip6h->nexthdr = proto;
ip6h->hop_limit = ttl;
ip6h->daddr = *daddr;
ip6h->saddr = *saddr;
ip6h->payload_len = htons(skb->len - sizeof(*ip6h));
}
static int prepare4(struct rxe_dev *rxe, struct sk_buff *skb, struct rxe_av *av)
{
struct dst_entry *dst;
bool xnet = false;
__be16 df = htons(IP_DF);
struct in_addr *saddr = &av->sgid_addr._sockaddr_in.sin_addr;
struct in_addr *daddr = &av->dgid_addr._sockaddr_in.sin_addr;
struct rxe_pkt_info *pkt = SKB_TO_PKT(skb);
dst = rxe_find_route4(rxe->ndev, saddr, daddr);
if (!dst) {
pr_err("Host not reachable\n");
return -EHOSTUNREACH;
}
if (!memcmp(saddr, daddr, sizeof(*daddr)))
pkt->mask |= RXE_LOOPBACK_MASK;
prepare_udp_hdr(skb, htons(RXE_ROCE_V2_SPORT),
htons(ROCE_V2_UDP_DPORT));
prepare_ipv4_hdr(dst, skb, saddr->s_addr, daddr->s_addr, IPPROTO_UDP,
av->grh.traffic_class, av->grh.hop_limit, df, xnet);
return 0;
}
static int prepare6(struct rxe_dev *rxe, struct sk_buff *skb, struct rxe_av *av)
{
struct dst_entry *dst;
struct in6_addr *saddr = &av->sgid_addr._sockaddr_in6.sin6_addr;
struct in6_addr *daddr = &av->dgid_addr._sockaddr_in6.sin6_addr;
struct rxe_pkt_info *pkt = SKB_TO_PKT(skb);
dst = rxe_find_route6(rxe->ndev, saddr, daddr);
if (!dst) {
pr_err("Host not reachable\n");
return -EHOSTUNREACH;
}
if (!memcmp(saddr, daddr, sizeof(*daddr)))
pkt->mask |= RXE_LOOPBACK_MASK;
prepare_udp_hdr(skb, htons(RXE_ROCE_V2_SPORT),
htons(ROCE_V2_UDP_DPORT));
prepare_ipv6_hdr(dst, skb, saddr, daddr, IPPROTO_UDP,
av->grh.traffic_class,
av->grh.hop_limit);
return 0;
}
static int prepare(struct rxe_dev *rxe, struct rxe_pkt_info *pkt,
struct sk_buff *skb, u32 *crc)
{
int err = 0;
struct rxe_av *av = rxe_get_av(pkt);
if (av->network_type == RDMA_NETWORK_IPV4)
err = prepare4(rxe, skb, av);
else if (av->network_type == RDMA_NETWORK_IPV6)
err = prepare6(rxe, skb, av);
*crc = rxe_icrc_hdr(pkt, skb);
return err;
}
static void rxe_skb_tx_dtor(struct sk_buff *skb)
{
struct sock *sk = skb->sk;
struct rxe_qp *qp = sk->sk_user_data;
int skb_out = atomic_dec_return(&qp->skb_out);
if (unlikely(qp->need_req_skb &&
skb_out < RXE_INFLIGHT_SKBS_PER_QP_LOW))
rxe_run_task(&qp->req.task, 1);
}
static int send(struct rxe_dev *rxe, struct rxe_pkt_info *pkt,
struct sk_buff *skb)
{
struct sk_buff *nskb;
struct rxe_av *av;
int err;
av = rxe_get_av(pkt);
nskb = skb_clone(skb, GFP_ATOMIC);
if (!nskb)
return -ENOMEM;
nskb->destructor = rxe_skb_tx_dtor;
nskb->sk = pkt->qp->sk->sk;
if (av->network_type == RDMA_NETWORK_IPV4) {
err = ip_local_out(dev_net(skb_dst(skb)->dev), nskb->sk, nskb);
} else if (av->network_type == RDMA_NETWORK_IPV6) {
err = ip6_local_out(dev_net(skb_dst(skb)->dev), nskb->sk, nskb);
} else {
pr_err("Unknown layer 3 protocol: %d\n", av->network_type);
kfree_skb(nskb);
return -EINVAL;
}
if (unlikely(net_xmit_eval(err))) {
pr_debug("error sending packet: %d\n", err);
return -EAGAIN;
}
kfree_skb(skb);
return 0;
}
static int loopback(struct sk_buff *skb)
{
return rxe_rcv(skb);
}
static inline int addr_same(struct rxe_dev *rxe, struct rxe_av *av)
{
return rxe->port.port_guid == av->grh.dgid.global.interface_id;
}
static struct sk_buff *init_packet(struct rxe_dev *rxe, struct rxe_av *av,
int paylen, struct rxe_pkt_info *pkt)
{
unsigned int hdr_len;
struct sk_buff *skb;
if (av->network_type == RDMA_NETWORK_IPV4)
hdr_len = ETH_HLEN + sizeof(struct udphdr) +
sizeof(struct iphdr);
else
hdr_len = ETH_HLEN + sizeof(struct udphdr) +
sizeof(struct ipv6hdr);
skb = alloc_skb(paylen + hdr_len + LL_RESERVED_SPACE(rxe->ndev),
GFP_ATOMIC);
if (unlikely(!skb))
return NULL;
skb_reserve(skb, hdr_len + LL_RESERVED_SPACE(rxe->ndev));
skb->dev = rxe->ndev;
if (av->network_type == RDMA_NETWORK_IPV4)
skb->protocol = htons(ETH_P_IP);
else
skb->protocol = htons(ETH_P_IPV6);
pkt->rxe = rxe;
pkt->port_num = 1;
pkt->hdr = skb_put(skb, paylen);
pkt->mask |= RXE_GRH_MASK;
memset(pkt->hdr, 0, paylen);
return skb;
}
/*
* this is required by rxe_cfg to match rxe devices in
* /sys/class/infiniband up with their underlying ethernet devices
*/
static char *parent_name(struct rxe_dev *rxe, unsigned int port_num)
{
return rxe->ndev->name;
}
static enum rdma_link_layer link_layer(struct rxe_dev *rxe,
unsigned int port_num)
{
return IB_LINK_LAYER_ETHERNET;
}
static struct rxe_ifc_ops ifc_ops = {
.node_guid = node_guid,
.port_guid = port_guid,
.dma_device = dma_device,
.mcast_add = mcast_add,
.mcast_delete = mcast_delete,
.prepare = prepare,
.send = send,
.loopback = loopback,
.init_packet = init_packet,
.parent_name = parent_name,
.link_layer = link_layer,
};
struct rxe_dev *rxe_net_add(struct net_device *ndev)
{
int err;
struct rxe_dev *rxe = NULL;
rxe = (struct rxe_dev *)ib_alloc_device(sizeof(*rxe));
if (!rxe)
return NULL;
rxe->ifc_ops = &ifc_ops;
rxe->ndev = ndev;
err = rxe_add(rxe, ndev->mtu);
if (err) {
ib_dealloc_device(&rxe->ib_dev);
return NULL;
}
spin_lock_bh(&dev_list_lock);
list_add_tail(&rxe_dev_list, &rxe->list);
spin_unlock_bh(&dev_list_lock);
return rxe;
}
void rxe_remove_all(void)
{
spin_lock_bh(&dev_list_lock);
while (!list_empty(&rxe_dev_list)) {
struct rxe_dev *rxe =
list_first_entry(&rxe_dev_list, struct rxe_dev, list);
list_del(&rxe->list);
spin_unlock_bh(&dev_list_lock);
rxe_remove(rxe);
spin_lock_bh(&dev_list_lock);
}
spin_unlock_bh(&dev_list_lock);
}
EXPORT_SYMBOL(rxe_remove_all);
static void rxe_port_event(struct rxe_dev *rxe,
enum ib_event_type event)
{
struct ib_event ev;
ev.device = &rxe->ib_dev;
ev.element.port_num = 1;
ev.event = event;
ib_dispatch_event(&ev);
}
/* Caller must hold net_info_lock */
void rxe_port_up(struct rxe_dev *rxe)
{
struct rxe_port *port;
port = &rxe->port;
port->attr.state = IB_PORT_ACTIVE;
port->attr.phys_state = IB_PHYS_STATE_LINK_UP;
rxe_port_event(rxe, IB_EVENT_PORT_ACTIVE);
pr_info("rxe: set %s active\n", rxe->ib_dev.name);
return;
}
/* Caller must hold net_info_lock */
void rxe_port_down(struct rxe_dev *rxe)
{
struct rxe_port *port;
port = &rxe->port;
port->attr.state = IB_PORT_DOWN;
port->attr.phys_state = IB_PHYS_STATE_LINK_DOWN;
rxe_port_event(rxe, IB_EVENT_PORT_ERR);
pr_info("rxe: set %s down\n", rxe->ib_dev.name);
return;
}
static int rxe_notify(struct notifier_block *not_blk,
unsigned long event,
void *arg)
{
struct net_device *ndev = netdev_notifier_info_to_dev(arg);
struct rxe_dev *rxe = net_to_rxe(ndev);
if (!rxe)
goto out;
switch (event) {
case NETDEV_UNREGISTER:
list_del(&rxe->list);
rxe_remove(rxe);
break;
case NETDEV_UP:
rxe_port_up(rxe);
break;
case NETDEV_DOWN:
rxe_port_down(rxe);
break;
case NETDEV_CHANGEMTU:
pr_info("rxe: %s changed mtu to %d\n", ndev->name, ndev->mtu);
rxe_set_mtu(rxe, ndev->mtu);
break;
case NETDEV_REBOOT:
case NETDEV_CHANGE:
case NETDEV_GOING_DOWN:
case NETDEV_CHANGEADDR:
case NETDEV_CHANGENAME:
case NETDEV_FEAT_CHANGE:
default:
pr_info("rxe: ignoring netdev event = %ld for %s\n",
event, ndev->name);
break;
}
out:
return NOTIFY_OK;
}
static struct notifier_block rxe_net_notifier = {
.notifier_call = rxe_notify,
};
int rxe_net_init(void)
{
int err;
spin_lock_init(&dev_list_lock);
recv_sockets.sk6 = rxe_setup_udp_tunnel(&init_net,
htons(ROCE_V2_UDP_DPORT), true);
if (IS_ERR(recv_sockets.sk6)) {
recv_sockets.sk6 = NULL;
pr_err("rxe: Failed to create IPv6 UDP tunnel\n");
return -1;
}
recv_sockets.sk4 = rxe_setup_udp_tunnel(&init_net,
htons(ROCE_V2_UDP_DPORT), false);
if (IS_ERR(recv_sockets.sk4)) {
rxe_release_udp_tunnel(recv_sockets.sk6);
recv_sockets.sk4 = NULL;
recv_sockets.sk6 = NULL;
pr_err("rxe: Failed to create IPv4 UDP tunnel\n");
return -1;
}
err = register_netdevice_notifier(&rxe_net_notifier);
if (err) {
rxe_release_udp_tunnel(recv_sockets.sk6);
rxe_release_udp_tunnel(recv_sockets.sk4);
pr_err("rxe: Failed to rigister netdev notifier\n");
}
return err;
}
void rxe_net_exit(void)
{
if (recv_sockets.sk6)
rxe_release_udp_tunnel(recv_sockets.sk6);
if (recv_sockets.sk4)
rxe_release_udp_tunnel(recv_sockets.sk4);
unregister_netdevice_notifier(&rxe_net_notifier);
}

53
drivers/infiniband/sw/rxe/rxe_net.h Normal file
View file

@ -0,0 +1,53 @@
/*
* Copyright (c) 2016 Mellanox Technologies Ltd. All rights reserved.
* Copyright (c) 2015 System Fabric Works, Inc. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenIB.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
#ifndef RXE_NET_H
#define RXE_NET_H
#include <net/sock.h>
#include <net/if_inet6.h>
#include <linux/module.h>
struct rxe_recv_sockets {
struct socket *sk4;
struct socket *sk6;
};
extern struct rxe_recv_sockets recv_sockets;
struct rxe_dev *rxe_net_add(struct net_device *ndev);
int rxe_net_init(void);
void rxe_net_exit(void);
#endif /* RXE_NET_H */

961
drivers/infiniband/sw/rxe/rxe_opcode.c Normal file
View file

@ -0,0 +1,961 @@
/*
* Copyright (c) 2016 Mellanox Technologies Ltd. All rights reserved.
* Copyright (c) 2015 System Fabric Works, Inc. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenIB.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
#include <rdma/ib_pack.h>
#include "rxe_opcode.h"
#include "rxe_hdr.h"
/* useful information about work request opcodes and pkt opcodes in
* table form
*/
struct rxe_wr_opcode_info rxe_wr_opcode_info[] = {
[IB_WR_RDMA_WRITE] = {
.name = "IB_WR_RDMA_WRITE",
.mask = {
[IB_QPT_RC] = WR_INLINE_MASK | WR_WRITE_MASK,
[IB_QPT_UC] = WR_INLINE_MASK | WR_WRITE_MASK,
},
},
[IB_WR_RDMA_WRITE_WITH_IMM] = {
.name = "IB_WR_RDMA_WRITE_WITH_IMM",
.mask = {
[IB_QPT_RC] = WR_INLINE_MASK | WR_WRITE_MASK,
[IB_QPT_UC] = WR_INLINE_MASK | WR_WRITE_MASK,
},
},
[IB_WR_SEND] = {
.name = "IB_WR_SEND",
.mask = {
[IB_QPT_SMI] = WR_INLINE_MASK | WR_SEND_MASK,
[IB_QPT_GSI] = WR_INLINE_MASK | WR_SEND_MASK,
[IB_QPT_RC] = WR_INLINE_MASK | WR_SEND_MASK,
[IB_QPT_UC] = WR_INLINE_MASK | WR_SEND_MASK,
[IB_QPT_UD] = WR_INLINE_MASK | WR_SEND_MASK,
},
},
[IB_WR_SEND_WITH_IMM] = {
.name = "IB_WR_SEND_WITH_IMM",
.mask = {
[IB_QPT_SMI] = WR_INLINE_MASK | WR_SEND_MASK,
[IB_QPT_GSI] = WR_INLINE_MASK | WR_SEND_MASK,
[IB_QPT_RC] = WR_INLINE_MASK | WR_SEND_MASK,
[IB_QPT_UC] = WR_INLINE_MASK | WR_SEND_MASK,
[IB_QPT_UD] = WR_INLINE_MASK | WR_SEND_MASK,
},
},
[IB_WR_RDMA_READ] = {
.name = "IB_WR_RDMA_READ",
.mask = {
[IB_QPT_RC] = WR_READ_MASK,
},
},
[IB_WR_ATOMIC_CMP_AND_SWP] = {
.name = "IB_WR_ATOMIC_CMP_AND_SWP",
.mask = {
[IB_QPT_RC] = WR_ATOMIC_MASK,
},
},
[IB_WR_ATOMIC_FETCH_AND_ADD] = {
.name = "IB_WR_ATOMIC_FETCH_AND_ADD",
.mask = {
[IB_QPT_RC] = WR_ATOMIC_MASK,
},
},
[IB_WR_LSO] = {
.name = "IB_WR_LSO",
.mask = {
/* not supported */
},
},
[IB_WR_SEND_WITH_INV] = {
.name = "IB_WR_SEND_WITH_INV",
.mask = {
[IB_QPT_RC] = WR_INLINE_MASK | WR_SEND_MASK,
[IB_QPT_UC] = WR_INLINE_MASK | WR_SEND_MASK,
[IB_QPT_UD] = WR_INLINE_MASK | WR_SEND_MASK,
},
},
[IB_WR_RDMA_READ_WITH_INV] = {
.name = "IB_WR_RDMA_READ_WITH_INV",
.mask = {
[IB_QPT_RC] = WR_READ_MASK,
},
},
[IB_WR_LOCAL_INV] = {
.name = "IB_WR_LOCAL_INV",
.mask = {
[IB_QPT_RC] = WR_REG_MASK,
},
},
[IB_WR_REG_MR] = {
.name = "IB_WR_REG_MR",
.mask = {
[IB_QPT_RC] = WR_REG_MASK,
},
},
};
struct rxe_opcode_info rxe_opcode[RXE_NUM_OPCODE] = {
[IB_OPCODE_RC_SEND_FIRST] = {
.name = "IB_OPCODE_RC_SEND_FIRST",
.mask = RXE_PAYLOAD_MASK | RXE_REQ_MASK | RXE_RWR_MASK
| RXE_SEND_MASK | RXE_START_MASK,
.length = RXE_BTH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_PAYLOAD] = RXE_BTH_BYTES,
}
},
[IB_OPCODE_RC_SEND_MIDDLE] = {
.name = "IB_OPCODE_RC_SEND_MIDDLE]",
.mask = RXE_PAYLOAD_MASK | RXE_REQ_MASK | RXE_SEND_MASK
| RXE_MIDDLE_MASK,
.length = RXE_BTH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_PAYLOAD] = RXE_BTH_BYTES,
}
},
[IB_OPCODE_RC_SEND_LAST] = {
.name = "IB_OPCODE_RC_SEND_LAST",
.mask = RXE_PAYLOAD_MASK | RXE_REQ_MASK | RXE_COMP_MASK
| RXE_SEND_MASK | RXE_END_MASK,
.length = RXE_BTH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_PAYLOAD] = RXE_BTH_BYTES,
}
},
[IB_OPCODE_RC_SEND_LAST_WITH_IMMEDIATE] = {
.name = "IB_OPCODE_RC_SEND_LAST_WITH_IMMEDIATE",
.mask = RXE_IMMDT_MASK | RXE_PAYLOAD_MASK | RXE_REQ_MASK
| RXE_COMP_MASK | RXE_SEND_MASK | RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_IMMDT_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_IMMDT] = RXE_BTH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_IMMDT_BYTES,
}
},
[IB_OPCODE_RC_SEND_ONLY] = {
.name = "IB_OPCODE_RC_SEND_ONLY",
.mask = RXE_PAYLOAD_MASK | RXE_REQ_MASK | RXE_COMP_MASK
| RXE_RWR_MASK | RXE_SEND_MASK
| RXE_START_MASK | RXE_END_MASK,
.length = RXE_BTH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_PAYLOAD] = RXE_BTH_BYTES,
}
},
[IB_OPCODE_RC_SEND_ONLY_WITH_IMMEDIATE] = {
.name = "IB_OPCODE_RC_SEND_ONLY_WITH_IMMEDIATE",
.mask = RXE_IMMDT_MASK | RXE_PAYLOAD_MASK | RXE_REQ_MASK
| RXE_COMP_MASK | RXE_RWR_MASK | RXE_SEND_MASK
| RXE_START_MASK | RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_IMMDT_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_IMMDT] = RXE_BTH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_IMMDT_BYTES,
}
},
[IB_OPCODE_RC_RDMA_WRITE_FIRST] = {
.name = "IB_OPCODE_RC_RDMA_WRITE_FIRST",
.mask = RXE_RETH_MASK | RXE_PAYLOAD_MASK | RXE_REQ_MASK
| RXE_WRITE_MASK | RXE_START_MASK,
.length = RXE_BTH_BYTES + RXE_RETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_RETH] = RXE_BTH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_RETH_BYTES,
}
},
[IB_OPCODE_RC_RDMA_WRITE_MIDDLE] = {
.name = "IB_OPCODE_RC_RDMA_WRITE_MIDDLE",
.mask = RXE_PAYLOAD_MASK | RXE_REQ_MASK | RXE_WRITE_MASK
| RXE_MIDDLE_MASK,
.length = RXE_BTH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_PAYLOAD] = RXE_BTH_BYTES,
}
},
[IB_OPCODE_RC_RDMA_WRITE_LAST] = {
.name = "IB_OPCODE_RC_RDMA_WRITE_LAST",
.mask = RXE_PAYLOAD_MASK | RXE_REQ_MASK | RXE_WRITE_MASK
| RXE_END_MASK,
.length = RXE_BTH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_PAYLOAD] = RXE_BTH_BYTES,
}
},
[IB_OPCODE_RC_RDMA_WRITE_LAST_WITH_IMMEDIATE] = {
.name = "IB_OPCODE_RC_RDMA_WRITE_LAST_WITH_IMMEDIATE",
.mask = RXE_IMMDT_MASK | RXE_PAYLOAD_MASK | RXE_REQ_MASK
| RXE_WRITE_MASK | RXE_COMP_MASK | RXE_RWR_MASK
| RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_IMMDT_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_IMMDT] = RXE_BTH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_IMMDT_BYTES,
}
},
[IB_OPCODE_RC_RDMA_WRITE_ONLY] = {
.name = "IB_OPCODE_RC_RDMA_WRITE_ONLY",
.mask = RXE_RETH_MASK | RXE_PAYLOAD_MASK | RXE_REQ_MASK
| RXE_WRITE_MASK | RXE_START_MASK
| RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_RETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_RETH] = RXE_BTH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_RETH_BYTES,
}
},
[IB_OPCODE_RC_RDMA_WRITE_ONLY_WITH_IMMEDIATE] = {
.name = "IB_OPCODE_RC_RDMA_WRITE_ONLY_WITH_IMMEDIATE",
.mask = RXE_RETH_MASK | RXE_IMMDT_MASK | RXE_PAYLOAD_MASK
| RXE_REQ_MASK | RXE_WRITE_MASK
| RXE_COMP_MASK | RXE_RWR_MASK
| RXE_START_MASK | RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_IMMDT_BYTES + RXE_RETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_RETH] = RXE_BTH_BYTES,
[RXE_IMMDT] = RXE_BTH_BYTES
+ RXE_RETH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_RETH_BYTES
+ RXE_IMMDT_BYTES,
}
},
[IB_OPCODE_RC_RDMA_READ_REQUEST] = {
.name = "IB_OPCODE_RC_RDMA_READ_REQUEST",
.mask = RXE_RETH_MASK | RXE_REQ_MASK | RXE_READ_MASK
| RXE_START_MASK | RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_RETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_RETH] = RXE_BTH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_RETH_BYTES,
}
},
[IB_OPCODE_RC_RDMA_READ_RESPONSE_FIRST] = {
.name = "IB_OPCODE_RC_RDMA_READ_RESPONSE_FIRST",
.mask = RXE_AETH_MASK | RXE_PAYLOAD_MASK | RXE_ACK_MASK
| RXE_START_MASK,
.length = RXE_BTH_BYTES + RXE_AETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_AETH] = RXE_BTH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_AETH_BYTES,
}
},
[IB_OPCODE_RC_RDMA_READ_RESPONSE_MIDDLE] = {
.name = "IB_OPCODE_RC_RDMA_READ_RESPONSE_MIDDLE",
.mask = RXE_PAYLOAD_MASK | RXE_ACK_MASK | RXE_MIDDLE_MASK,
.length = RXE_BTH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_PAYLOAD] = RXE_BTH_BYTES,
}
},
[IB_OPCODE_RC_RDMA_READ_RESPONSE_LAST] = {
.name = "IB_OPCODE_RC_RDMA_READ_RESPONSE_LAST",
.mask = RXE_AETH_MASK | RXE_PAYLOAD_MASK | RXE_ACK_MASK
| RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_AETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_AETH] = RXE_BTH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_AETH_BYTES,
}
},
[IB_OPCODE_RC_RDMA_READ_RESPONSE_ONLY] = {
.name = "IB_OPCODE_RC_RDMA_READ_RESPONSE_ONLY",
.mask = RXE_AETH_MASK | RXE_PAYLOAD_MASK | RXE_ACK_MASK
| RXE_START_MASK | RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_AETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_AETH] = RXE_BTH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_AETH_BYTES,
}
},
[IB_OPCODE_RC_ACKNOWLEDGE] = {
.name = "IB_OPCODE_RC_ACKNOWLEDGE",
.mask = RXE_AETH_MASK | RXE_ACK_MASK | RXE_START_MASK
| RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_AETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_AETH] = RXE_BTH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_AETH_BYTES,
}
},
[IB_OPCODE_RC_ATOMIC_ACKNOWLEDGE] = {
.name = "IB_OPCODE_RC_ATOMIC_ACKNOWLEDGE",
.mask = RXE_AETH_MASK | RXE_ATMACK_MASK | RXE_ACK_MASK
| RXE_START_MASK | RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_ATMACK_BYTES + RXE_AETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_AETH] = RXE_BTH_BYTES,
[RXE_ATMACK] = RXE_BTH_BYTES
+ RXE_AETH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_ATMACK_BYTES + RXE_AETH_BYTES,
}
},
[IB_OPCODE_RC_COMPARE_SWAP] = {
.name = "IB_OPCODE_RC_COMPARE_SWAP",
.mask = RXE_ATMETH_MASK | RXE_REQ_MASK | RXE_ATOMIC_MASK
| RXE_START_MASK | RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_ATMETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_ATMETH] = RXE_BTH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_ATMETH_BYTES,
}
},
[IB_OPCODE_RC_FETCH_ADD] = {
.name = "IB_OPCODE_RC_FETCH_ADD",
.mask = RXE_ATMETH_MASK | RXE_REQ_MASK | RXE_ATOMIC_MASK
| RXE_START_MASK | RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_ATMETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_ATMETH] = RXE_BTH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_ATMETH_BYTES,
}
},
[IB_OPCODE_RC_SEND_LAST_WITH_INVALIDATE] = {
.name = "IB_OPCODE_RC_SEND_LAST_WITH_INVALIDATE",
.mask = RXE_IETH_MASK | RXE_PAYLOAD_MASK | RXE_REQ_MASK
| RXE_COMP_MASK | RXE_SEND_MASK | RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_IETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_IETH] = RXE_BTH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_IETH_BYTES,
}
},
[IB_OPCODE_RC_SEND_ONLY_WITH_INVALIDATE] = {
.name = "IB_OPCODE_RC_SEND_ONLY_INV",
.mask = RXE_IETH_MASK | RXE_PAYLOAD_MASK | RXE_REQ_MASK
| RXE_COMP_MASK | RXE_RWR_MASK | RXE_SEND_MASK
| RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_IETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_IETH] = RXE_BTH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_IETH_BYTES,
}
},
/* UC */
[IB_OPCODE_UC_SEND_FIRST] = {
.name = "IB_OPCODE_UC_SEND_FIRST",
.mask = RXE_PAYLOAD_MASK | RXE_REQ_MASK | RXE_RWR_MASK
| RXE_SEND_MASK | RXE_START_MASK,
.length = RXE_BTH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_PAYLOAD] = RXE_BTH_BYTES,
}
},
[IB_OPCODE_UC_SEND_MIDDLE] = {
.name = "IB_OPCODE_UC_SEND_MIDDLE",
.mask = RXE_PAYLOAD_MASK | RXE_REQ_MASK | RXE_SEND_MASK
| RXE_MIDDLE_MASK,
.length = RXE_BTH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_PAYLOAD] = RXE_BTH_BYTES,
}
},
[IB_OPCODE_UC_SEND_LAST] = {
.name = "IB_OPCODE_UC_SEND_LAST",
.mask = RXE_PAYLOAD_MASK | RXE_REQ_MASK | RXE_COMP_MASK
| RXE_SEND_MASK | RXE_END_MASK,
.length = RXE_BTH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_PAYLOAD] = RXE_BTH_BYTES,
}
},
[IB_OPCODE_UC_SEND_LAST_WITH_IMMEDIATE] = {
.name = "IB_OPCODE_UC_SEND_LAST_WITH_IMMEDIATE",
.mask = RXE_IMMDT_MASK | RXE_PAYLOAD_MASK | RXE_REQ_MASK
| RXE_COMP_MASK | RXE_SEND_MASK | RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_IMMDT_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_IMMDT] = RXE_BTH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_IMMDT_BYTES,
}
},
[IB_OPCODE_UC_SEND_ONLY] = {
.name = "IB_OPCODE_UC_SEND_ONLY",
.mask = RXE_PAYLOAD_MASK | RXE_REQ_MASK | RXE_COMP_MASK
| RXE_RWR_MASK | RXE_SEND_MASK
| RXE_START_MASK | RXE_END_MASK,
.length = RXE_BTH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_PAYLOAD] = RXE_BTH_BYTES,
}
},
[IB_OPCODE_UC_SEND_ONLY_WITH_IMMEDIATE] = {
.name = "IB_OPCODE_UC_SEND_ONLY_WITH_IMMEDIATE",
.mask = RXE_IMMDT_MASK | RXE_PAYLOAD_MASK | RXE_REQ_MASK
| RXE_COMP_MASK | RXE_RWR_MASK | RXE_SEND_MASK
| RXE_START_MASK | RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_IMMDT_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_IMMDT] = RXE_BTH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_IMMDT_BYTES,
}
},
[IB_OPCODE_UC_RDMA_WRITE_FIRST] = {
.name = "IB_OPCODE_UC_RDMA_WRITE_FIRST",
.mask = RXE_RETH_MASK | RXE_PAYLOAD_MASK | RXE_REQ_MASK
| RXE_WRITE_MASK | RXE_START_MASK,
.length = RXE_BTH_BYTES + RXE_RETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_RETH] = RXE_BTH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_RETH_BYTES,
}
},
[IB_OPCODE_UC_RDMA_WRITE_MIDDLE] = {
.name = "IB_OPCODE_UC_RDMA_WRITE_MIDDLE",
.mask = RXE_PAYLOAD_MASK | RXE_REQ_MASK | RXE_WRITE_MASK
| RXE_MIDDLE_MASK,
.length = RXE_BTH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_PAYLOAD] = RXE_BTH_BYTES,
}
},
[IB_OPCODE_UC_RDMA_WRITE_LAST] = {
.name = "IB_OPCODE_UC_RDMA_WRITE_LAST",
.mask = RXE_PAYLOAD_MASK | RXE_REQ_MASK | RXE_WRITE_MASK
| RXE_END_MASK,
.length = RXE_BTH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_PAYLOAD] = RXE_BTH_BYTES,
}
},
[IB_OPCODE_UC_RDMA_WRITE_LAST_WITH_IMMEDIATE] = {
.name = "IB_OPCODE_UC_RDMA_WRITE_LAST_WITH_IMMEDIATE",
.mask = RXE_IMMDT_MASK | RXE_PAYLOAD_MASK | RXE_REQ_MASK
| RXE_WRITE_MASK | RXE_COMP_MASK | RXE_RWR_MASK
| RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_IMMDT_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_IMMDT] = RXE_BTH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_IMMDT_BYTES,
}
},
[IB_OPCODE_UC_RDMA_WRITE_ONLY] = {
.name = "IB_OPCODE_UC_RDMA_WRITE_ONLY",
.mask = RXE_RETH_MASK | RXE_PAYLOAD_MASK | RXE_REQ_MASK
| RXE_WRITE_MASK | RXE_START_MASK
| RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_RETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_RETH] = RXE_BTH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_RETH_BYTES,
}
},
[IB_OPCODE_UC_RDMA_WRITE_ONLY_WITH_IMMEDIATE] = {
.name = "IB_OPCODE_UC_RDMA_WRITE_ONLY_WITH_IMMEDIATE",
.mask = RXE_RETH_MASK | RXE_IMMDT_MASK | RXE_PAYLOAD_MASK
| RXE_REQ_MASK | RXE_WRITE_MASK
| RXE_COMP_MASK | RXE_RWR_MASK
| RXE_START_MASK | RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_IMMDT_BYTES + RXE_RETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_RETH] = RXE_BTH_BYTES,
[RXE_IMMDT] = RXE_BTH_BYTES
+ RXE_RETH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_RETH_BYTES
+ RXE_IMMDT_BYTES,
}
},
/* RD */
[IB_OPCODE_RD_SEND_FIRST] = {
.name = "IB_OPCODE_RD_SEND_FIRST",
.mask = RXE_RDETH_MASK | RXE_DETH_MASK | RXE_PAYLOAD_MASK
| RXE_REQ_MASK | RXE_RWR_MASK | RXE_SEND_MASK
| RXE_START_MASK,
.length = RXE_BTH_BYTES + RXE_DETH_BYTES + RXE_RDETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_RDETH] = RXE_BTH_BYTES,
[RXE_DETH] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES
+ RXE_DETH_BYTES,
}
},
[IB_OPCODE_RD_SEND_MIDDLE] = {
.name = "IB_OPCODE_RD_SEND_MIDDLE",
.mask = RXE_RDETH_MASK | RXE_DETH_MASK | RXE_PAYLOAD_MASK
| RXE_REQ_MASK | RXE_SEND_MASK
| RXE_MIDDLE_MASK,
.length = RXE_BTH_BYTES + RXE_DETH_BYTES + RXE_RDETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_RDETH] = RXE_BTH_BYTES,
[RXE_DETH] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES
+ RXE_DETH_BYTES,
}
},
[IB_OPCODE_RD_SEND_LAST] = {
.name = "IB_OPCODE_RD_SEND_LAST",
.mask = RXE_RDETH_MASK | RXE_DETH_MASK | RXE_PAYLOAD_MASK
| RXE_REQ_MASK | RXE_COMP_MASK | RXE_SEND_MASK
| RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_DETH_BYTES + RXE_RDETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_RDETH] = RXE_BTH_BYTES,
[RXE_DETH] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES
+ RXE_DETH_BYTES,
}
},
[IB_OPCODE_RD_SEND_LAST_WITH_IMMEDIATE] = {
.name = "IB_OPCODE_RD_SEND_LAST_WITH_IMMEDIATE",
.mask = RXE_RDETH_MASK | RXE_DETH_MASK | RXE_IMMDT_MASK
| RXE_PAYLOAD_MASK | RXE_REQ_MASK
| RXE_COMP_MASK | RXE_SEND_MASK
| RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_IMMDT_BYTES + RXE_DETH_BYTES
+ RXE_RDETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_RDETH] = RXE_BTH_BYTES,
[RXE_DETH] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES,
[RXE_IMMDT] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES
+ RXE_DETH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES
+ RXE_DETH_BYTES
+ RXE_IMMDT_BYTES,
}
},
[IB_OPCODE_RD_SEND_ONLY] = {
.name = "IB_OPCODE_RD_SEND_ONLY",
.mask = RXE_RDETH_MASK | RXE_DETH_MASK | RXE_PAYLOAD_MASK
| RXE_REQ_MASK | RXE_COMP_MASK | RXE_RWR_MASK
| RXE_SEND_MASK | RXE_START_MASK | RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_DETH_BYTES + RXE_RDETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_RDETH] = RXE_BTH_BYTES,
[RXE_DETH] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES
+ RXE_DETH_BYTES,
}
},
[IB_OPCODE_RD_SEND_ONLY_WITH_IMMEDIATE] = {
.name = "IB_OPCODE_RD_SEND_ONLY_WITH_IMMEDIATE",
.mask = RXE_RDETH_MASK | RXE_DETH_MASK | RXE_IMMDT_MASK
| RXE_PAYLOAD_MASK | RXE_REQ_MASK
| RXE_COMP_MASK | RXE_RWR_MASK | RXE_SEND_MASK
| RXE_START_MASK | RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_IMMDT_BYTES + RXE_DETH_BYTES
+ RXE_RDETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_RDETH] = RXE_BTH_BYTES,
[RXE_DETH] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES,
[RXE_IMMDT] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES
+ RXE_DETH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES
+ RXE_DETH_BYTES
+ RXE_IMMDT_BYTES,
}
},
[IB_OPCODE_RD_RDMA_WRITE_FIRST] = {
.name = "IB_OPCODE_RD_RDMA_WRITE_FIRST",
.mask = RXE_RDETH_MASK | RXE_DETH_MASK | RXE_RETH_MASK
| RXE_PAYLOAD_MASK | RXE_REQ_MASK
| RXE_WRITE_MASK | RXE_START_MASK,
.length = RXE_BTH_BYTES + RXE_RETH_BYTES + RXE_DETH_BYTES
+ RXE_RDETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_RDETH] = RXE_BTH_BYTES,
[RXE_DETH] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES,
[RXE_RETH] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES
+ RXE_DETH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES
+ RXE_DETH_BYTES
+ RXE_RETH_BYTES,
}
},
[IB_OPCODE_RD_RDMA_WRITE_MIDDLE] = {
.name = "IB_OPCODE_RD_RDMA_WRITE_MIDDLE",
.mask = RXE_RDETH_MASK | RXE_DETH_MASK | RXE_PAYLOAD_MASK
| RXE_REQ_MASK | RXE_WRITE_MASK
| RXE_MIDDLE_MASK,
.length = RXE_BTH_BYTES + RXE_DETH_BYTES + RXE_RDETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_RDETH] = RXE_BTH_BYTES,
[RXE_DETH] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES
+ RXE_DETH_BYTES,
}
},
[IB_OPCODE_RD_RDMA_WRITE_LAST] = {
.name = "IB_OPCODE_RD_RDMA_WRITE_LAST",
.mask = RXE_RDETH_MASK | RXE_DETH_MASK | RXE_PAYLOAD_MASK
| RXE_REQ_MASK | RXE_WRITE_MASK
| RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_DETH_BYTES + RXE_RDETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_RDETH] = RXE_BTH_BYTES,
[RXE_DETH] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES
+ RXE_DETH_BYTES,
}
},
[IB_OPCODE_RD_RDMA_WRITE_LAST_WITH_IMMEDIATE] = {
.name = "IB_OPCODE_RD_RDMA_WRITE_LAST_WITH_IMMEDIATE",
.mask = RXE_RDETH_MASK | RXE_DETH_MASK | RXE_IMMDT_MASK
| RXE_PAYLOAD_MASK | RXE_REQ_MASK
| RXE_WRITE_MASK | RXE_COMP_MASK | RXE_RWR_MASK
| RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_IMMDT_BYTES + RXE_DETH_BYTES
+ RXE_RDETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_RDETH] = RXE_BTH_BYTES,
[RXE_DETH] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES,
[RXE_IMMDT] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES
+ RXE_DETH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES
+ RXE_DETH_BYTES
+ RXE_IMMDT_BYTES,
}
},
[IB_OPCODE_RD_RDMA_WRITE_ONLY] = {
.name = "IB_OPCODE_RD_RDMA_WRITE_ONLY",
.mask = RXE_RDETH_MASK | RXE_DETH_MASK | RXE_RETH_MASK
| RXE_PAYLOAD_MASK | RXE_REQ_MASK
| RXE_WRITE_MASK | RXE_START_MASK
| RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_RETH_BYTES + RXE_DETH_BYTES
+ RXE_RDETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_RDETH] = RXE_BTH_BYTES,
[RXE_DETH] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES,
[RXE_RETH] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES
+ RXE_DETH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES
+ RXE_DETH_BYTES
+ RXE_RETH_BYTES,
}
},
[IB_OPCODE_RD_RDMA_WRITE_ONLY_WITH_IMMEDIATE] = {
.name = "IB_OPCODE_RD_RDMA_WRITE_ONLY_WITH_IMMEDIATE",
.mask = RXE_RDETH_MASK | RXE_DETH_MASK | RXE_RETH_MASK
| RXE_IMMDT_MASK | RXE_PAYLOAD_MASK
| RXE_REQ_MASK | RXE_WRITE_MASK
| RXE_COMP_MASK | RXE_RWR_MASK
| RXE_START_MASK | RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_IMMDT_BYTES + RXE_RETH_BYTES
+ RXE_DETH_BYTES + RXE_RDETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_RDETH] = RXE_BTH_BYTES,
[RXE_DETH] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES,
[RXE_RETH] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES
+ RXE_DETH_BYTES,
[RXE_IMMDT] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES
+ RXE_DETH_BYTES
+ RXE_RETH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES
+ RXE_DETH_BYTES
+ RXE_RETH_BYTES
+ RXE_IMMDT_BYTES,
}
},
[IB_OPCODE_RD_RDMA_READ_REQUEST] = {
.name = "IB_OPCODE_RD_RDMA_READ_REQUEST",
.mask = RXE_RDETH_MASK | RXE_DETH_MASK | RXE_RETH_MASK
| RXE_REQ_MASK | RXE_READ_MASK
| RXE_START_MASK | RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_RETH_BYTES + RXE_DETH_BYTES
+ RXE_RDETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_RDETH] = RXE_BTH_BYTES,
[RXE_DETH] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES,
[RXE_RETH] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES
+ RXE_DETH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_RETH_BYTES
+ RXE_DETH_BYTES
+ RXE_RDETH_BYTES,
}
},
[IB_OPCODE_RD_RDMA_READ_RESPONSE_FIRST] = {
.name = "IB_OPCODE_RD_RDMA_READ_RESPONSE_FIRST",
.mask = RXE_RDETH_MASK | RXE_AETH_MASK
| RXE_PAYLOAD_MASK | RXE_ACK_MASK
| RXE_START_MASK,
.length = RXE_BTH_BYTES + RXE_AETH_BYTES + RXE_RDETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_RDETH] = RXE_BTH_BYTES,
[RXE_AETH] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES
+ RXE_AETH_BYTES,
}
},
[IB_OPCODE_RD_RDMA_READ_RESPONSE_MIDDLE] = {
.name = "IB_OPCODE_RD_RDMA_READ_RESPONSE_MIDDLE",
.mask = RXE_RDETH_MASK | RXE_PAYLOAD_MASK | RXE_ACK_MASK
| RXE_MIDDLE_MASK,
.length = RXE_BTH_BYTES + RXE_RDETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_RDETH] = RXE_BTH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES,
}
},
[IB_OPCODE_RD_RDMA_READ_RESPONSE_LAST] = {
.name = "IB_OPCODE_RD_RDMA_READ_RESPONSE_LAST",
.mask = RXE_RDETH_MASK | RXE_AETH_MASK | RXE_PAYLOAD_MASK
| RXE_ACK_MASK | RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_AETH_BYTES + RXE_RDETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_RDETH] = RXE_BTH_BYTES,
[RXE_AETH] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES
+ RXE_AETH_BYTES,
}
},
[IB_OPCODE_RD_RDMA_READ_RESPONSE_ONLY] = {
.name = "IB_OPCODE_RD_RDMA_READ_RESPONSE_ONLY",
.mask = RXE_RDETH_MASK | RXE_AETH_MASK | RXE_PAYLOAD_MASK
| RXE_ACK_MASK | RXE_START_MASK | RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_AETH_BYTES + RXE_RDETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_RDETH] = RXE_BTH_BYTES,
[RXE_AETH] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES
+ RXE_AETH_BYTES,
}
},
[IB_OPCODE_RD_ACKNOWLEDGE] = {
.name = "IB_OPCODE_RD_ACKNOWLEDGE",
.mask = RXE_RDETH_MASK | RXE_AETH_MASK | RXE_ACK_MASK
| RXE_START_MASK | RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_AETH_BYTES + RXE_RDETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_RDETH] = RXE_BTH_BYTES,
[RXE_AETH] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES,
}
},
[IB_OPCODE_RD_ATOMIC_ACKNOWLEDGE] = {
.name = "IB_OPCODE_RD_ATOMIC_ACKNOWLEDGE",
.mask = RXE_RDETH_MASK | RXE_AETH_MASK | RXE_ATMACK_MASK
| RXE_ACK_MASK | RXE_START_MASK | RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_ATMACK_BYTES + RXE_AETH_BYTES
+ RXE_RDETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_RDETH] = RXE_BTH_BYTES,
[RXE_AETH] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES,
[RXE_ATMACK] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES
+ RXE_AETH_BYTES,
}
},
[IB_OPCODE_RD_COMPARE_SWAP] = {
.name = "RD_COMPARE_SWAP",
.mask = RXE_RDETH_MASK | RXE_DETH_MASK | RXE_ATMETH_MASK
| RXE_REQ_MASK | RXE_ATOMIC_MASK
| RXE_START_MASK | RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_ATMETH_BYTES + RXE_DETH_BYTES
+ RXE_RDETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_RDETH] = RXE_BTH_BYTES,
[RXE_DETH] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES,
[RXE_ATMETH] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES
+ RXE_DETH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES +
+ RXE_ATMETH_BYTES
+ RXE_DETH_BYTES +
+ RXE_RDETH_BYTES,
}
},
[IB_OPCODE_RD_FETCH_ADD] = {
.name = "IB_OPCODE_RD_FETCH_ADD",
.mask = RXE_RDETH_MASK | RXE_DETH_MASK | RXE_ATMETH_MASK
| RXE_REQ_MASK | RXE_ATOMIC_MASK
| RXE_START_MASK | RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_ATMETH_BYTES + RXE_DETH_BYTES
+ RXE_RDETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_RDETH] = RXE_BTH_BYTES,
[RXE_DETH] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES,
[RXE_ATMETH] = RXE_BTH_BYTES
+ RXE_RDETH_BYTES
+ RXE_DETH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES +
+ RXE_ATMETH_BYTES
+ RXE_DETH_BYTES +
+ RXE_RDETH_BYTES,
}
},
/* UD */
[IB_OPCODE_UD_SEND_ONLY] = {
.name = "IB_OPCODE_UD_SEND_ONLY",
.mask = RXE_DETH_MASK | RXE_PAYLOAD_MASK | RXE_REQ_MASK
| RXE_COMP_MASK | RXE_RWR_MASK | RXE_SEND_MASK
| RXE_START_MASK | RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_DETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_DETH] = RXE_BTH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_DETH_BYTES,
}
},
[IB_OPCODE_UD_SEND_ONLY_WITH_IMMEDIATE] = {
.name = "IB_OPCODE_UD_SEND_ONLY_WITH_IMMEDIATE",
.mask = RXE_DETH_MASK | RXE_IMMDT_MASK | RXE_PAYLOAD_MASK
| RXE_REQ_MASK | RXE_COMP_MASK | RXE_RWR_MASK
| RXE_SEND_MASK | RXE_START_MASK | RXE_END_MASK,
.length = RXE_BTH_BYTES + RXE_IMMDT_BYTES + RXE_DETH_BYTES,
.offset = {
[RXE_BTH] = 0,
[RXE_DETH] = RXE_BTH_BYTES,
[RXE_IMMDT] = RXE_BTH_BYTES
+ RXE_DETH_BYTES,
[RXE_PAYLOAD] = RXE_BTH_BYTES
+ RXE_DETH_BYTES
+ RXE_IMMDT_BYTES,
}
},
};

Some files were not shown because too many files have changed in this diff Show more