Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-18 Thread Alexander Duyck
On Fri, Dec 18, 2020 at 12:18 PM Jason Gunthorpe wrote: > > On Fri, Dec 18, 2020 at 11:22:12AM -0800, Alexander Duyck wrote: > > > Also as far as the patch count complaints I have seen in a few threads > > I would be fine with splitting things up so that the devlink and aux > > device creation get

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-18 Thread Jason Gunthorpe
On Fri, Dec 18, 2020 at 11:22:12AM -0800, Alexander Duyck wrote: > Also as far as the patch count complaints I have seen in a few threads > I would be fine with splitting things up so that the devlink and aux > device creation get handled in one set, and then we work out the > details of mlx5 atta

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-18 Thread Alexander Duyck
On Fri, Dec 18, 2020 at 10:01 AM Parav Pandit wrote: > > > > From: Alexander Duyck > > Sent: Friday, December 18, 2020 9:31 PM > > > > On Thu, Dec 17, 2020 at 9:20 PM Parav Pandit wrote: > > > > > > > > > > From: Alexander Duyck > > > > Sent: Friday, December 18, 2020 8:41 AM > > > > > > > > On

RE: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-18 Thread Parav Pandit
> From: Alexander Duyck > Sent: Friday, December 18, 2020 9:31 PM > > On Thu, Dec 17, 2020 at 9:20 PM Parav Pandit wrote: > > > > > > > From: Alexander Duyck > > > Sent: Friday, December 18, 2020 8:41 AM > > > > > > On Thu, Dec 17, 2020 at 5:30 PM David Ahern > wrote: > > > > > > > > On 12/16

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-18 Thread Alexander Duyck
On Thu, Dec 17, 2020 at 9:20 PM Parav Pandit wrote: > > > > From: Alexander Duyck > > Sent: Friday, December 18, 2020 8:41 AM > > > > On Thu, Dec 17, 2020 at 5:30 PM David Ahern wrote: > > > > > > On 12/16/20 3:53 PM, Alexander Duyck wrote: > > The problem is PCIe DMA wasn't designed to function

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-18 Thread Alexander Duyck
On Thu, Dec 17, 2020 at 7:55 PM David Ahern wrote: > > On 12/17/20 8:11 PM, Alexander Duyck wrote: > > On Thu, Dec 17, 2020 at 5:30 PM David Ahern wrote: > >> > >> On 12/16/20 3:53 PM, Alexander Duyck wrote: > >>> The problem in my case was based on a past experience where east-west > >>> traffic

RE: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-17 Thread Parav Pandit
> From: Parav Pandit > Sent: Friday, December 18, 2020 10:51 AM > > > From: Alexander Duyck > > Sent: Friday, December 18, 2020 8:41 AM > > > > On Thu, Dec 17, 2020 at 5:30 PM David Ahern wrote: > > > > > > On 12/16/20 3:53 PM, Alexander Duyck wrote: > > The problem is PCIe DMA wasn't designe

RE: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-17 Thread Parav Pandit
> From: Alexander Duyck > Sent: Friday, December 18, 2020 8:41 AM > > On Thu, Dec 17, 2020 at 5:30 PM David Ahern wrote: > > > > On 12/16/20 3:53 PM, Alexander Duyck wrote: > The problem is PCIe DMA wasn't designed to function as a network switch > fabric and when we start talking about a 400Gb

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-17 Thread David Ahern
On 12/17/20 8:11 PM, Alexander Duyck wrote: > On Thu, Dec 17, 2020 at 5:30 PM David Ahern wrote: >> >> On 12/16/20 3:53 PM, Alexander Duyck wrote: >>> The problem in my case was based on a past experience where east-west >>> traffic became a problem and it was easily shown that bypassing the >>> N

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-17 Thread Alexander Duyck
On Thu, Dec 17, 2020 at 5:30 PM David Ahern wrote: > > On 12/16/20 3:53 PM, Alexander Duyck wrote: > > The problem in my case was based on a past experience where east-west > > traffic became a problem and it was easily shown that bypassing the > > NIC for traffic was significantly faster. > > If

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-17 Thread David Ahern
On 12/16/20 3:53 PM, Alexander Duyck wrote: > The problem in my case was based on a past experience where east-west > traffic became a problem and it was easily shown that bypassing the > NIC for traffic was significantly faster. If a deployment expects a lot of east-west traffic *within a host* w

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-17 Thread Jason Gunthorpe
On Thu, Dec 17, 2020 at 01:05:03PM -0800, Alexander Duyck wrote: > > I view the SW bypass path you are talking about similarly to > > GSO/etc. It should be accessed by the HW driver as an optional service > > provided by the core netdev, not implemented as some wrapper netdev > > around a HW imple

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-17 Thread Alexander Duyck
On Thu, Dec 17, 2020 at 11:40 AM Jason Gunthorpe wrote: > > On Thu, Dec 17, 2020 at 10:48:48AM -0800, Alexander Duyck wrote: > > > Just to clarify I am not with Intel, nor do I plan to work on any > > Intel drivers related to this. > > Sure > > > I disagree here. In my mind a design where two inte

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-17 Thread Jason Gunthorpe
On Thu, Dec 17, 2020 at 10:48:48AM -0800, Alexander Duyck wrote: > Just to clarify I am not with Intel, nor do I plan to work on any > Intel drivers related to this. Sure > I disagree here. In my mind a design where two interfaces, which both > exist in the kernel, have to go to hardware in ord

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-17 Thread Alexander Duyck
On Wed, Dec 16, 2020 at 4:38 PM Jason Gunthorpe wrote: > > On Wed, Dec 16, 2020 at 02:53:07PM -0800, Alexander Duyck wrote: > > > It isn't about the association, it is about who is handling the > > traffic. Going back to the macvlan model what we did is we had a group > > of rings on the device th

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-16 Thread Jason Gunthorpe
On Wed, Dec 16, 2020 at 02:53:07PM -0800, Alexander Duyck wrote: > It isn't about the association, it is about who is handling the > traffic. Going back to the macvlan model what we did is we had a group > of rings on the device that would automatically forward unicast > packets to the macvlan in

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-16 Thread Alexander Duyck
On Wed, Dec 16, 2020 at 12:35 PM Jason Gunthorpe wrote: > > On Wed, Dec 16, 2020 at 11:27:32AM -0800, Alexander Duyck wrote: > > > That has been the case for a long time. However it had been my > > experience that SR-IOV never scaled well to meet those needs and so it > > hadn't been used in such

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-16 Thread Jason Gunthorpe
On Wed, Dec 16, 2020 at 11:27:32AM -0800, Alexander Duyck wrote: > That has been the case for a long time. However it had been my > experience that SR-IOV never scaled well to meet those needs and so it > hadn't been used in such deployments. Seems to be going quite well here, perhaps the applica

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-16 Thread Alexander Duyck
On Wed, Dec 16, 2020 at 9:51 AM Jason Gunthorpe wrote: > > On Wed, Dec 16, 2020 at 08:31:44AM -0800, Alexander Duyck wrote: > > > You say this will scale better but I am not even sure about that. The > > fact is SR-IOV could scale to 256 VFs, but for networking I kind of > > doubt the limitation w

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-16 Thread Saeed Mahameed
On Wed, 2020-12-16 at 08:50 +0200, Leon Romanovsky wrote: > On Tue, Dec 15, 2020 at 01:28:05PM -0800, Jakub Kicinski wrote: > > On Tue, 15 Dec 2020 12:35:20 -0800 Saeed Mahameed wrote: > > > > I think the big thing we really should do if we are going to go > > > > this > > > > route is to look at s

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-16 Thread Jason Gunthorpe
On Wed, Dec 16, 2020 at 08:31:44AM -0800, Alexander Duyck wrote: > You say this will scale better but I am not even sure about that. The > fact is SR-IOV could scale to 256 VFs, but for networking I kind of > doubt the limitation would have been the bus number and would more > likely be issues wit

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-16 Thread Alexander Duyck
On Wed, Dec 16, 2020 at 5:33 AM Jason Gunthorpe wrote: > > On Tue, Dec 15, 2020 at 08:13:21PM -0800, Alexander Duyck wrote: > > > > > Ugh, don't get me started on switchdev. The biggest issue as I see it > > > > with switchev is that you have to have a true switch in order to > > > > really be abl

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-16 Thread Jason Gunthorpe
On Tue, Dec 15, 2020 at 08:13:21PM -0800, Alexander Duyck wrote: > > > Ugh, don't get me started on switchdev. The biggest issue as I see it > > > with switchev is that you have to have a true switch in order to > > > really be able to use it. > > > > That cuts both ways, suggesting HW with a true

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Leon Romanovsky
On Tue, Dec 15, 2020 at 01:28:05PM -0800, Jakub Kicinski wrote: > On Tue, 15 Dec 2020 12:35:20 -0800 Saeed Mahameed wrote: > > > I think the big thing we really should do if we are going to go this > > > route is to look at standardizing what the flavours are that get > > > created by the parent ne

RE: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Parav Pandit
> From: Alexander Duyck > Sent: Wednesday, December 16, 2020 9:43 AM > > > > > That is goal here. This is not about creating just a netdev, this is > > about the whole kit: rdma, netdev, vdpa virtio-net, virtio-mdev. > > One issue is right now we are only seeing the rdma and netdev. It is kind

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Alexander Duyck
On Tue, Dec 15, 2020 at 7:04 PM Jason Gunthorpe wrote: > > On Tue, Dec 15, 2020 at 06:19:18PM -0800, Alexander Duyck wrote: > > > > > I would really like to see is a solid standardization of what this is. > > > > Otherwise the comparison is going to be made. Especially since a year > > > > ago Mel

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Alexander Duyck
On Tue, Dec 15, 2020 at 5:13 PM Edwin Peer wrote: > > On Tue, Dec 15, 2020 at 10:49 AM Alexander Duyck > wrote: > > > It isn't "SR-IOV done right" it seems more like "VMDq done better". > > I don't think I agree with that assertion. The fact that VMDq can talk > to a common driver still makes VMD

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Jason Gunthorpe
On Tue, Dec 15, 2020 at 06:19:18PM -0800, Alexander Duyck wrote: > > > I would really like to see is a solid standardization of what this is. > > > Otherwise the comparison is going to be made. Especially since a year > > > ago Mellanox was pushing this as an mdev type interface. > > > > mdev was

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Jason Gunthorpe
On Tue, Dec 15, 2020 at 05:12:33PM -0800, Edwin Peer wrote: > 1) More than 256 SFs are possible: Maybe it's about time PCI-SIG > addresses this limit for VFs? They can't, the Bus/Device/Function is limited by protocol and changing that would upend the entire PCI world. Instead PCI-SIG said PASI

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Alexander Duyck
On Tue, Dec 15, 2020 at 4:20 PM Jason Gunthorpe wrote: > > On Tue, Dec 15, 2020 at 01:41:04PM -0800, Alexander Duyck wrote: > > > > not just devlink and switchdev, auxbus was also introduced to > > > standardize some of the interfaces. > > > > The auxbus is just there to make up for the fact that

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Edwin Peer
On Tue, Dec 15, 2020 at 10:49 AM Alexander Duyck wrote: > It isn't "SR-IOV done right" it seems more like "VMDq done better". I don't think I agree with that assertion. The fact that VMDq can talk to a common driver still makes VMDq preferable in some respects. Thus, subfunctions do appear to be

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Jason Gunthorpe
On Tue, Dec 15, 2020 at 01:41:04PM -0800, Alexander Duyck wrote: > > not just devlink and switchdev, auxbus was also introduced to > > standardize some of the interfaces. > > The auxbus is just there to make up for the fact that there isn't > another bus type for this though. I would imagine othe

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Alexander Duyck
On Tue, Dec 15, 2020 at 12:35 PM Saeed Mahameed wrote: > > On Tue, 2020-12-15 at 11:12 -0800, Alexander Duyck wrote: > > On Mon, Dec 14, 2020 at 10:15 PM Saeed Mahameed > > wrote: > > > On Mon, 2020-12-14 at 17:53 -0800, Alexander Duyck wrote: > > > > On Mon, Dec 14, 2020 at 1:49 PM Saeed Mahamee

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Jakub Kicinski
On Tue, 15 Dec 2020 12:35:20 -0800 Saeed Mahameed wrote: > > I think the big thing we really should do if we are going to go this > > route is to look at standardizing what the flavours are that get > > created by the parent netdevice. Otherwise we are just creating the > > same mess we had with SR

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Jason Gunthorpe
On Tue, Dec 15, 2020 at 10:47:36AM -0800, Alexander Duyck wrote: > > Jason and Saeed explained this in great detail few weeks back in v0 version > > of the patchset at [1], [2] and [3]. > > I better not repeat all of it here again. Please go through it. > > If you may want to read precursor to it

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread David Ahern
On 12/14/20 10:48 PM, Parav Pandit wrote: > >> From: Alexander Duyck >> Sent: Tuesday, December 15, 2020 7:24 AM >> >> On Mon, Dec 14, 2020 at 1:49 PM Saeed Mahameed >> wrote: >>> >>> Hi Dave, Jakub, Jason, >>> >> >> Just to clarify a few things for myself. You mention virtualization and >> SR-

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Saeed Mahameed
On Tue, 2020-12-15 at 11:12 -0800, Alexander Duyck wrote: > On Mon, Dec 14, 2020 at 10:15 PM Saeed Mahameed > wrote: > > On Mon, 2020-12-14 at 17:53 -0800, Alexander Duyck wrote: > > > On Mon, Dec 14, 2020 at 1:49 PM Saeed Mahameed > > > wrote: > > > > Hi Dave, Jakub, Jason, > > > > > > > > This

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Saeed Mahameed
On Tue, 2020-12-15 at 10:47 -0800, Alexander Duyck wrote: > On Mon, Dec 14, 2020 at 9:48 PM Parav Pandit > wrote: > > > > > From: Alexander Duyck > > > Sent: Tuesday, December 15, 2020 7:24 AM > > > > > > On Mon, Dec 14, 2020 at 1:49 PM Saeed Mahameed > > > wrote: > > > > Hi Dave, Jakub, Jason

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Alexander Duyck
On Mon, Dec 14, 2020 at 10:15 PM Saeed Mahameed wrote: > > On Mon, 2020-12-14 at 17:53 -0800, Alexander Duyck wrote: > > On Mon, Dec 14, 2020 at 1:49 PM Saeed Mahameed > > wrote: > > > Hi Dave, Jakub, Jason, > > > > > > This series form Parav was the theme of this mlx5 release cycle, > > > we've

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Alexander Duyck
On Mon, Dec 14, 2020 at 9:48 PM Parav Pandit wrote: > > > > From: Alexander Duyck > > Sent: Tuesday, December 15, 2020 7:24 AM > > > > On Mon, Dec 14, 2020 at 1:49 PM Saeed Mahameed > > wrote: > > > > > > Hi Dave, Jakub, Jason, > > > > > > > Just to clarify a few things for myself. You mention v

RE: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Parav Pandit
> From: Alexander Duyck > Sent: Tuesday, December 15, 2020 9:47 PM > > On Mon, Dec 14, 2020 at 6:44 PM David Ahern wrote: > > > > On 12/14/20 6:53 PM, Alexander Duyck wrote: > > >> example subfunction usage sequence: > > >> --- > > >> Change device to switchdev

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Alexander Duyck
On Mon, Dec 14, 2020 at 6:44 PM David Ahern wrote: > > On 12/14/20 6:53 PM, Alexander Duyck wrote: > >> example subfunction usage sequence: > >> --- > >> Change device to switchdev mode: > >> $ devlink dev eswitch set pci/:06:00.0 mode switchdev > >> > >> Add a

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-14 Thread Saeed Mahameed
On Mon, 2020-12-14 at 17:53 -0800, Alexander Duyck wrote: > On Mon, Dec 14, 2020 at 1:49 PM Saeed Mahameed > wrote: > > Hi Dave, Jakub, Jason, > > > > This series form Parav was the theme of this mlx5 release cycle, > > we've been waiting anxiously for the auxbus infrastructure to make > > it int

RE: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-14 Thread Parav Pandit
> From: Alexander Duyck > Sent: Tuesday, December 15, 2020 7:24 AM > > On Mon, Dec 14, 2020 at 1:49 PM Saeed Mahameed > wrote: > > > > Hi Dave, Jakub, Jason, > > > > Just to clarify a few things for myself. You mention virtualization and SR-IOV > in your patch description but you cannot suppor

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-14 Thread David Ahern
On 12/14/20 6:53 PM, Alexander Duyck wrote: >> example subfunction usage sequence: >> --- >> Change device to switchdev mode: >> $ devlink dev eswitch set pci/:06:00.0 mode switchdev >> >> Add a devlink port of subfunction flaovur: >> $ devlink port add pci/:

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-14 Thread Alexander Duyck
On Mon, Dec 14, 2020 at 1:49 PM Saeed Mahameed wrote: > > Hi Dave, Jakub, Jason, > > This series form Parav was the theme of this mlx5 release cycle, > we've been waiting anxiously for the auxbus infrastructure to make it into > the kernel, and now as the auxbus is in and all the stars are aligned

[net-next v4 00/15] Add mlx5 subfunction support

2020-12-14 Thread Saeed Mahameed
Hi Dave, Jakub, Jason, This series form Parav was the theme of this mlx5 release cycle, we've been waiting anxiously for the auxbus infrastructure to make it into the kernel, and now as the auxbus is in and all the stars are aligned, I can finally submit this V2 of the devlink and mlx5 subfunction