Re: Re: [PATCH v6 09/10] vduse: Introduce VDUSE - vDPA Device in Userspace

2021-04-16 Thread Yongji Xie
On Fri, Apr 16, 2021 at 11:24 AM Jason Wang wrote: > > > 在 2021/3/31 下午4:05, Xie Yongji 写道: > > + } > > + case VDUSE_INJECT_VQ_IRQ: > > + ret = -EINVAL; > > + if (arg >= dev->vq_num) > > + break; > > + > > + ret = 0; > > +

Re: Re: [PATCH v6 10/10] Documentation: Add documentation for VDUSE

2021-04-15 Thread Yongji Xie
On Fri, Apr 16, 2021 at 10:24 AM Jason Wang wrote: > > > 在 2021/4/15 下午10:38, Stefan Hajnoczi 写道: > > On Thu, Apr 15, 2021 at 04:36:35PM +0800, Jason Wang wrote: > >> 在 2021/4/15 下午3:19, Stefan Hajnoczi 写道: > >>> On Thu, Apr 15, 2021 at 01:38:37PM +0800, Yon

Re: Re: [PATCH v6 10/10] Documentation: Add documentation for VDUSE

2021-04-15 Thread Yongji Xie
On Fri, Apr 16, 2021 at 11:03 AM Jason Wang wrote: > > > 在 2021/4/16 上午10:58, Yongji Xie 写道: > > On Fri, Apr 16, 2021 at 10:20 AM Jason Wang wrote: > >> > >> 在 2021/4/15 下午7:17, Yongji Xie 写道: > >>> On Thu, Apr 15, 2021 at 5:05 PM Jason Wang wr

Re: Re: [PATCH v6 10/10] Documentation: Add documentation for VDUSE

2021-04-15 Thread Yongji Xie
On Thu, Apr 15, 2021 at 10:38 PM Stefan Hajnoczi wrote: > > On Thu, Apr 15, 2021 at 04:36:35PM +0800, Jason Wang wrote: > > > > 在 2021/4/15 下午3:19, Stefan Hajnoczi 写道: > > > On Thu, Apr 15, 2021 at 01:38:37PM +0800, Yongji Xie wrote: > > > > On Wed, A

Re: Re: [PATCH v6 10/10] Documentation: Add documentation for VDUSE

2021-04-15 Thread Yongji Xie
On Fri, Apr 16, 2021 at 10:20 AM Jason Wang wrote: > > > 在 2021/4/15 下午7:17, Yongji Xie 写道: > > On Thu, Apr 15, 2021 at 5:05 PM Jason Wang wrote: > >> > >> 在 2021/4/15 下午4:36, Jason Wang 写道: > >>>> Please state this explicitly at the start of th

Re: Re: [PATCH v6 10/10] Documentation: Add documentation for VDUSE

2021-04-15 Thread Yongji Xie
On Thu, Apr 15, 2021 at 5:05 PM Jason Wang wrote: > > > 在 2021/4/15 下午4:36, Jason Wang 写道: > >>> > >> Please state this explicitly at the start of the document. Existing > >> interfaces like FUSE are designed to avoid trusting userspace. > > > > > > There're some subtle difference here. VDUSE pres

Re: Re: Re: [PATCH v6 10/10] Documentation: Add documentation for VDUSE

2021-04-15 Thread Yongji Xie
On Thu, Apr 15, 2021 at 3:19 PM Stefan Hajnoczi wrote: > > On Thu, Apr 15, 2021 at 01:38:37PM +0800, Yongji Xie wrote: > > On Wed, Apr 14, 2021 at 10:15 PM Stefan Hajnoczi > > wrote: > > > > > > On Wed, Mar 31, 2021 at 04:05:19PM +0800, Xie Yongji wrote: >

Re: Re: [PATCH v6 10/10] Documentation: Add documentation for VDUSE

2021-04-14 Thread Yongji Xie
On Wed, Apr 14, 2021 at 10:15 PM Stefan Hajnoczi wrote: > > On Wed, Mar 31, 2021 at 04:05:19PM +0800, Xie Yongji wrote: > > VDUSE (vDPA Device in Userspace) is a framework to support > > implementing software-emulated vDPA devices in userspace. This > > document is intended to clarify the VDUSE de

Re: Re: [PATCH v6 00/10] Introduce VDUSE - vDPA Device in Userspace

2021-04-14 Thread Yongji Xie
On Wed, Apr 14, 2021 at 3:35 PM Michael S. Tsirkin wrote: > > On Wed, Mar 31, 2021 at 04:05:09PM +0800, Xie Yongji wrote: > > This series introduces a framework, which can be used to implement > > vDPA Devices in a userspace program. The work consist of two parts: > > control path forwarding and d

Re: Re: [PATCH v6 09/10] vduse: Introduce VDUSE - vDPA Device in Userspace

2021-04-12 Thread Yongji Xie
On Tue, Apr 13, 2021 at 11:35 AM Jason Wang wrote: > > > 在 2021/4/12 下午5:59, Yongji Xie 写道: > > On Mon, Apr 12, 2021 at 5:37 PM Jason Wang wrote: > >> > >> 在 2021/4/12 下午4:02, Yongji Xie 写道: > >>> On Mon, Apr 12, 2021 at 3:16 PM Jason Wang w

Re: Re: [PATCH v6 09/10] vduse: Introduce VDUSE - vDPA Device in Userspace

2021-04-12 Thread Yongji Xie
On Mon, Apr 12, 2021 at 5:37 PM Jason Wang wrote: > > > 在 2021/4/12 下午4:02, Yongji Xie 写道: > > On Mon, Apr 12, 2021 at 3:16 PM Jason Wang wrote: > >> > >> 在 2021/4/9 下午4:02, Yongji Xie 写道: > >>>>>>> +}; > >>>>>>> +

Re: Re: [PATCH v6 09/10] vduse: Introduce VDUSE - vDPA Device in Userspace

2021-04-12 Thread Yongji Xie
On Mon, Apr 12, 2021 at 3:16 PM Jason Wang wrote: > > > 在 2021/4/9 下午4:02, Yongji Xie 写道: > >>>>> +}; > >>>>> + > >>>>> +struct vduse_dev_config_data { > >>>>> + __u32 offset; /* offset from the

Re: Re: Re: [PATCH v6 03/10] vhost-vdpa: protect concurrent access to vhost device iotlb

2021-04-11 Thread Yongji Xie
On Mon, Apr 12, 2021 at 4:49 AM Michael S. Tsirkin wrote: > > On Sun, Apr 11, 2021 at 01:36:18PM +0800, Yongji Xie wrote: > > On Sat, Apr 10, 2021 at 12:16 AM Michael S. Tsirkin wrote: > > > > > > On Wed, Mar 31, 2021 at 04:05:12PM +0800, Xie Yongji wrote: > >

Re: Re: [PATCH v6 03/10] vhost-vdpa: protect concurrent access to vhost device iotlb

2021-04-10 Thread Yongji Xie
On Sat, Apr 10, 2021 at 12:16 AM Michael S. Tsirkin wrote: > > On Wed, Mar 31, 2021 at 04:05:12PM +0800, Xie Yongji wrote: > > Use vhost_dev->mutex to protect vhost device iotlb from > > concurrent access. > > > > Fixes: 4c8cf318("vhost: introduce vDPA-based backend") > > Cc: sta...@vger.kernel.or

Re: Re: [PATCH v6 09/10] vduse: Introduce VDUSE - vDPA Device in Userspace

2021-04-09 Thread Yongji Xie
On Fri, Apr 9, 2021 at 1:36 PM Jason Wang wrote: > > > 在 2021/4/8 下午5:36, Yongji Xie 写道: > > On Thu, Apr 8, 2021 at 2:57 PM Jason Wang wrote: > >> > >> 在 2021/3/31 下午4:05, Xie Yongji 写道: > >>> This VDUSE driver enables implementing vDPA devices in use

Re: Re: [PATCH v6 09/10] vduse: Introduce VDUSE - vDPA Device in Userspace

2021-04-08 Thread Yongji Xie
dpa_user/vduse_dev.c > > b/drivers/vdpa/vdpa_user/vduse_dev.c > > new file mode 100644 > > index ..51ca73464d0d > > --- /dev/null > > +++ b/drivers/vdpa/vdpa_user/vduse_dev.c > > @@ -0,0 +1,1362 @@ > > +// SPDX-License-Identifier: GPL-2.0-only > >

Re: Re: [PATCH v6 10/10] Documentation: Add documentation for VDUSE

2021-04-08 Thread Yongji Xie
On Thu, Apr 8, 2021 at 3:18 PM Jason Wang wrote: > > > 在 2021/3/31 下午4:05, Xie Yongji 写道: > > VDUSE (vDPA Device in Userspace) is a framework to support > > implementing software-emulated vDPA devices in userspace. This > > document is intended to clarify the VDUSE design and usage. > > > > Signed

Re: Re: [PATCH v6 08/10] vduse: Implement an MMU-based IOMMU driver

2021-04-07 Thread Yongji Xie
On Thu, Apr 8, 2021 at 11:26 AM Jason Wang wrote: > > > 在 2021/3/31 下午4:05, Xie Yongji 写道: > > This implements an MMU-based IOMMU driver to support mapping > > kernel dma buffer into userspace. The basic idea behind it is > > treating MMU (VA->PA) as IOMMU (IOVA->PA). The driver will set > > up MM

Re: Re: [PATCH v6 01/10] file: Export receive_fd() to modules

2021-03-31 Thread Yongji Xie
On Wed, Mar 31, 2021 at 10:08 PM Christian Brauner wrote: > > On Wed, Mar 31, 2021 at 09:59:07PM +0800, Yongji Xie wrote: > > On Wed, Mar 31, 2021 at 8:23 PM Christian Brauner > > wrote: > > > > > > On Wed, Mar 31, 2021 at 07:32:33PM +0800, Yongji Xie wrote:

Re: Re: [PATCH v6 01/10] file: Export receive_fd() to modules

2021-03-31 Thread Yongji Xie
On Wed, Mar 31, 2021 at 8:23 PM Christian Brauner wrote: > > On Wed, Mar 31, 2021 at 07:32:33PM +0800, Yongji Xie wrote: > > On Wed, Mar 31, 2021 at 5:15 PM Christian Brauner > > wrote: > > > > > > On Wed, Mar 31, 2021 at 04:05:10PM +0800, Xie Yongji wrote:

Re: Re: [PATCH v6 01/10] file: Export receive_fd() to modules

2021-03-31 Thread Yongji Xie
On Wed, Mar 31, 2021 at 5:15 PM Christian Brauner wrote: > > On Wed, Mar 31, 2021 at 04:05:10PM +0800, Xie Yongji wrote: > > Export receive_fd() so that some modules can use > > it to pass file descriptor between processes without > > missing any security stuffs. > > > > Signed-off-by: Xie Yongji

Re: Re: [PATCH v5 08/11] vduse: Implement an MMU-based IOMMU driver

2021-03-25 Thread Yongji Xie
On Fri, Mar 26, 2021 at 2:16 PM Jason Wang wrote: > > > 在 2021/3/26 下午1:14, Yongji Xie 写道: > > + } > + map->bounce_page = page; > + > + /* paired with vduse_domain_map_page() */ > + smp_mb(); > > So this is suspicious. It's better to ex

Re: Re: [PATCH v5 08/11] vduse: Implement an MMU-based IOMMU driver

2021-03-25 Thread Yongji Xie
On Fri, Mar 26, 2021 at 12:27 PM Jason Wang wrote: > > > 在 2021/3/25 下午3:38, Yongji Xie 写道: > > On Thu, Mar 25, 2021 at 12:53 PM Jason Wang wrote: > >> > >> 在 2021/3/24 下午3:39, Yongji Xie 写道: > >>> On Wed, Mar 24, 2021 at 11:54 AM Jason Wang wr

Re: Re: [PATCH v5 01/11] file: Export __receive_fd() to modules

2021-03-25 Thread Yongji Xie
On Thu, Mar 25, 2021 at 4:25 PM Christoph Hellwig wrote: > > On Mon, Mar 15, 2021 at 05:46:43PM +0800, Yongji Xie wrote: > > On Mon, Mar 15, 2021 at 5:08 PM Christoph Hellwig > > wrote: > > > > > > On Mon, Mar 15, 2021 at 01:37:11PM +0800, Xie Yongji wrote: &

Re: Re: [PATCH v5 08/11] vduse: Implement an MMU-based IOMMU driver

2021-03-25 Thread Yongji Xie
On Thu, Mar 25, 2021 at 12:53 PM Jason Wang wrote: > > > 在 2021/3/24 下午3:39, Yongji Xie 写道: > > On Wed, Mar 24, 2021 at 11:54 AM Jason Wang wrote: > >> > >> 在 2021/3/15 下午1:37, Xie Yongji 写道: > >>> This implements an MMU-based IOMMU driver to

Re: Re: [PATCH v5 09/11] vduse: Introduce VDUSE - vDPA Device in Userspace

2021-03-25 Thread Yongji Xie
On Thu, Mar 25, 2021 at 2:31 PM Jason Wang wrote: > > > 在 2021/3/24 下午4:55, Yongji Xie 写道: > > On Wed, Mar 24, 2021 at 12:43 PM Jason Wang wrote: > >> > >> 在 2021/3/15 下午1:37, Xie Yongji 写道: > >>> This VDUSE driver enables implementing vDPA devices i

Re: Re: [PATCH v5 10/11] vduse: Add config interrupt support

2021-03-24 Thread Yongji Xie
On Wed, Mar 24, 2021 at 12:45 PM Jason Wang wrote: > > > 在 2021/3/15 下午1:37, Xie Yongji 写道: > > This patch introduces a new ioctl VDUSE_INJECT_CONFIG_IRQ > > to support injecting config interrupt. > > > > Signed-off-by: Xie Yongji > > > I suggest to squash this into path 9. > Will do it in v6.

Re: Re: [PATCH v5 09/11] vduse: Introduce VDUSE - vDPA Device in Userspace

2021-03-24 Thread Yongji Xie
t; new file mode 100644 > > index ..07d0ae92d470 > > --- /dev/null > > +++ b/drivers/vdpa/vdpa_user/vduse_dev.c > > @@ -0,0 +1,1281 @@ > > +// SPDX-License-Identifier: GPL-2.0-only > > +/* > > + * VDUSE: vDPA Device in Userspace

Re: Re: [PATCH v5 08/11] vduse: Implement an MMU-based IOMMU driver

2021-03-24 Thread Yongji Xie
On Wed, Mar 24, 2021 at 11:54 AM Jason Wang wrote: > > > 在 2021/3/15 下午1:37, Xie Yongji 写道: > > This implements an MMU-based IOMMU driver to support mapping > > kernel dma buffer into userspace. The basic idea behind it is > > treating MMU (VA->PA) as IOMMU (IOVA->PA). The driver will set > > up M

Re: Re: [PATCH v5 07/11] vdpa: Support transferring virtual addressing during DMA mapping

2021-03-23 Thread Yongji Xie
On Tue, Mar 23, 2021 at 11:13 AM Jason Wang wrote: > > > 在 2021/3/15 下午1:37, Xie Yongji 写道: > > This patch introduces an attribute for vDPA device to indicate > > whether virtual address can be used. If vDPA device driver set > > it, vhost-vdpa bus driver will not pin user page and transfer > > us

Re: Re: [PATCH v5 03/11] vhost-vdpa: protect concurrent access to vhost device iotlb

2021-03-23 Thread Yongji Xie
On Tue, Mar 23, 2021 at 11:02 AM Jason Wang wrote: > > > 在 2021/3/15 下午1:37, Xie Yongji 写道: > > Use vhost_dev->mutex to protect vhost device iotlb from > > concurrent access. > > > > Fixes: 4c8cf318("vhost: introduce vDPA-based backend") > > Signed-off-by: Xie Yongji > > > Acked-by: Jason Wang >

Re: Re: [PATCH v5 01/11] file: Export __receive_fd() to modules

2021-03-15 Thread Yongji Xie
On Mon, Mar 15, 2021 at 10:44 PM Christian Brauner wrote: > > On Mon, Mar 15, 2021 at 05:46:43PM +0800, Yongji Xie wrote: > > On Mon, Mar 15, 2021 at 5:08 PM Christoph Hellwig > > wrote: > > > > > > On Mon, Mar 15, 2021 at 01:37:11PM +0800, Xie Yongji wro

Re: Re: [PATCH v5 01/11] file: Export __receive_fd() to modules

2021-03-15 Thread Yongji Xie
On Mon, Mar 15, 2021 at 5:08 PM Christoph Hellwig wrote: > > On Mon, Mar 15, 2021 at 01:37:11PM +0800, Xie Yongji wrote: > > Export __receive_fd() so that some modules can use > > it to pass file descriptor between processes. > > I really don't think any non-core code should do that, especilly not

Re: Re: [RFC v4 07/11] vduse: Introduce VDUSE - vDPA Device in Userspace

2021-03-10 Thread Yongji Xie
On Wed, Mar 10, 2021 at 8:58 PM Jason Wang wrote: > > > On 2021/2/23 7:50 下午, Xie Yongji wrote: > > + > > + switch (cmd) { > > + case VDUSE_IOTLB_GET_FD: { > > + struct vduse_iotlb_entry entry; > > + struct vhost_iotlb_map *map; > > + struct vdpa_map_fil

Re: Re: [RFC v4 10/11] vduse: Introduce a workqueue for irq injection

2021-03-07 Thread Yongji Xie
On Mon, Mar 8, 2021 at 3:02 PM Jason Wang wrote: > > > On 2021/3/8 12:50 下午, Yongji Xie wrote: > > On Mon, Mar 8, 2021 at 11:04 AM Jason Wang wrote: > >> > >> On 2021/3/5 4:12 下午, Yongji Xie wrote: > >>> On Fri, Mar 5, 2021 at 3:37 PM Jason Wang

Re: Re: [RFC v4 06/11] vduse: Implement an MMU-based IOMMU driver

2021-03-07 Thread Yongji Xie
On Mon, Mar 8, 2021 at 3:04 PM Jason Wang wrote: > > > On 2021/3/8 1:05 下午, Yongji Xie wrote: > > On Mon, Mar 8, 2021 at 11:52 AM Jason Wang wrote: > >> > >> On 2021/3/8 11:45 上午, Yongji Xie wrote: > >>> On Mon, Mar 8, 2021 at 11:17 AM Jason Wang

Re: Re: [RFC v4 06/11] vduse: Implement an MMU-based IOMMU driver

2021-03-07 Thread Yongji Xie
On Mon, Mar 8, 2021 at 11:52 AM Jason Wang wrote: > > > On 2021/3/8 11:45 上午, Yongji Xie wrote: > > On Mon, Mar 8, 2021 at 11:17 AM Jason Wang wrote: > >> > >> On 2021/3/5 3:59 下午, Yongji Xie wrote: > >>> On Fri, Mar 5, 2021 at 3:27 PM Jason Wang

Re: Re: [RFC v4 10/11] vduse: Introduce a workqueue for irq injection

2021-03-07 Thread Yongji Xie
On Mon, Mar 8, 2021 at 11:04 AM Jason Wang wrote: > > > On 2021/3/5 4:12 下午, Yongji Xie wrote: > > On Fri, Mar 5, 2021 at 3:37 PM Jason Wang wrote: > >> > >> On 2021/3/5 3:27 下午, Yongji Xie wrote: > >>> On Fri, Mar 5, 2021 at 3:01 PM Jason Wang

Re: Re: [RFC v4 06/11] vduse: Implement an MMU-based IOMMU driver

2021-03-07 Thread Yongji Xie
On Mon, Mar 8, 2021 at 11:17 AM Jason Wang wrote: > > > On 2021/3/5 3:59 下午, Yongji Xie wrote: > > On Fri, Mar 5, 2021 at 3:27 PM Jason Wang wrote: > >> > >> On 2021/3/5 3:13 下午, Yongji Xie wrote: > >>> On Fri, Mar 5, 2021 at 2:52 PM Jason Wang

Re: Re: [RFC v4 10/11] vduse: Introduce a workqueue for irq injection

2021-03-05 Thread Yongji Xie
On Fri, Mar 5, 2021 at 3:37 PM Jason Wang wrote: > > > On 2021/3/5 3:27 下午, Yongji Xie wrote: > > On Fri, Mar 5, 2021 at 3:01 PM Jason Wang wrote: > >> > >> On 2021/3/5 2:36 下午, Yongji Xie wrote: > >>> On Fri, Mar 5, 2021 at 11:42 AM Jason Wang w

Re: Re: [RFC v4 06/11] vduse: Implement an MMU-based IOMMU driver

2021-03-04 Thread Yongji Xie
On Fri, Mar 5, 2021 at 3:27 PM Jason Wang wrote: > > > On 2021/3/5 3:13 下午, Yongji Xie wrote: > > On Fri, Mar 5, 2021 at 2:52 PM Jason Wang wrote: > >> > >> On 2021/3/5 2:15 下午, Yongji Xie wrote: > >> > >> Sorry if I've asked this before. &

Re: Re: [RFC v4 10/11] vduse: Introduce a workqueue for irq injection

2021-03-04 Thread Yongji Xie
On Fri, Mar 5, 2021 at 3:01 PM Jason Wang wrote: > > > On 2021/3/5 2:36 下午, Yongji Xie wrote: > > On Fri, Mar 5, 2021 at 11:42 AM Jason Wang wrote: > >> > >> On 2021/3/5 11:30 上午, Yongji Xie wrote: > >>> On Fri, Mar 5, 2021 at 11:05 AM Jason Wang

Re: Re: [RFC v4 06/11] vduse: Implement an MMU-based IOMMU driver

2021-03-04 Thread Yongji Xie
On Fri, Mar 5, 2021 at 2:52 PM Jason Wang wrote: > > > On 2021/3/5 2:15 下午, Yongji Xie wrote: > > Sorry if I've asked this before. > > But what's the reason for maintaing a dedicated IOTLB here? I think we > could reuse vduse_dev->iommu since the device can n

Re: Re: [RFC v4 11/11] vduse: Support binding irq to the specified cpu

2021-03-04 Thread Yongji Xie
On Fri, Mar 5, 2021 at 11:44 AM Jason Wang wrote: > > > On 2021/3/5 11:37 上午, Yongji Xie wrote: > > On Fri, Mar 5, 2021 at 11:11 AM Jason Wang wrote: > >> > >> On 2021/3/4 4:19 下午, Yongji Xie wrote: > >>> On Thu, Mar 4, 2021 at 3:30 PM Jason Wang w

Re: Re: [RFC v4 10/11] vduse: Introduce a workqueue for irq injection

2021-03-04 Thread Yongji Xie
On Fri, Mar 5, 2021 at 11:42 AM Jason Wang wrote: > > > On 2021/3/5 11:30 上午, Yongji Xie wrote: > > On Fri, Mar 5, 2021 at 11:05 AM Jason Wang wrote: > >> > >> On 2021/3/4 4:58 下午, Yongji Xie wrote: > >>> On Thu, Mar 4, 2021 at 2:59 PM Jason Wang w

Re: Re: [RFC v4 06/11] vduse: Implement an MMU-based IOMMU driver

2021-03-04 Thread Yongji Xie
On Fri, Mar 5, 2021 at 11:36 AM Jason Wang wrote: > > > On 2021/3/4 1:12 下午, Yongji Xie wrote: > > On Thu, Mar 4, 2021 at 12:21 PM Jason Wang wrote: > >> > >> On 2021/2/23 7:50 下午, Xie Yongji wrote: > >>> This implements a MMU-based IOMMU driver to

Re: Re: [RFC v4 07/11] vduse: Introduce VDUSE - vDPA Device in Userspace

2021-03-04 Thread Yongji Xie
On Fri, Mar 5, 2021 at 11:20 AM Jason Wang wrote: > > > On 2021/3/4 4:05 下午, Yongji Xie wrote: > > On Thu, Mar 4, 2021 at 2:27 PM Jason Wang wrote: > >> > >> On 2021/2/23 7:50 下午, Xie Yongji wrote: > >>> This VDUSE driver enables implementing vDPA

Re: Re: [RFC v4 11/11] vduse: Support binding irq to the specified cpu

2021-03-04 Thread Yongji Xie
On Fri, Mar 5, 2021 at 11:11 AM Jason Wang wrote: > > > On 2021/3/4 4:19 下午, Yongji Xie wrote: > > On Thu, Mar 4, 2021 at 3:30 PM Jason Wang wrote: > >> > >> On 2021/2/23 7:50 下午, Xie Yongji wrote: > >>> Add a parameter for the ioctl VDUSE_INJECT_

Re: Re: [RFC v4 10/11] vduse: Introduce a workqueue for irq injection

2021-03-04 Thread Yongji Xie
On Fri, Mar 5, 2021 at 11:05 AM Jason Wang wrote: > > > On 2021/3/4 4:58 下午, Yongji Xie wrote: > > On Thu, Mar 4, 2021 at 2:59 PM Jason Wang wrote: > >> > >> On 2021/2/23 7:50 下午, Xie Yongji wrote: > >>> This patch introduces a workqueue to

Re: Re: [RFC v4 09/11] Documentation: Add documentation for VDUSE

2021-03-04 Thread Yongji Xie
On Thu, Mar 4, 2021 at 2:40 PM Jason Wang wrote: > > > On 2021/2/23 7:50 下午, Xie Yongji wrote: > > VDUSE (vDPA Device in Userspace) is a framework to support > > implementing software-emulated vDPA devices in userspace. This > > document is intended to clarify the VDUSE design and usage. > > > > S

Re: Re: [RFC v4 10/11] vduse: Introduce a workqueue for irq injection

2021-03-04 Thread Yongji Xie
On Thu, Mar 4, 2021 at 2:59 PM Jason Wang wrote: > > > On 2021/2/23 7:50 下午, Xie Yongji wrote: > > This patch introduces a workqueue to support injecting > > virtqueue's interrupt asynchronously. This is mainly > > for performance considerations which makes sure the push() > > and pop() for used v

Re: Re: [RFC v4 11/11] vduse: Support binding irq to the specified cpu

2021-03-04 Thread Yongji Xie
On Thu, Mar 4, 2021 at 3:30 PM Jason Wang wrote: > > > On 2021/2/23 7:50 下午, Xie Yongji wrote: > > Add a parameter for the ioctl VDUSE_INJECT_VQ_IRQ to support > > injecting virtqueue's interrupt to the specified cpu. > > > How userspace know which CPU is this irq for? It looks to me we need to >

Re: Re: [RFC v4 07/11] vduse: Introduce VDUSE - vDPA Device in Userspace

2021-03-04 Thread Yongji Xie
/vdpa_user/vduse_dev.c > > new file mode 100644 > > index ..393bf99c48be > > --- /dev/null > > +++ b/drivers/vdpa/vdpa_user/vduse_dev.c > > @@ -0,0 +1,1348 @@ > > +// SPDX-License-Identifier: GPL-2.0-only > > +/* > > + * VDUSE: vDPA Device

Re: Re: [RFC v4 05/11] vdpa: Support transferring virtual addressing during DMA mapping

2021-03-03 Thread Yongji Xie
On Thu, Mar 4, 2021 at 11:07 AM Jason Wang wrote: > > > On 2021/2/23 7:50 下午, Xie Yongji wrote: > > This patch introduces an attribute for vDPA device to indicate > > whether virtual address can be used. If vDPA device driver set > > it, vhost-vdpa bus driver will not pin user page and transfer >

Re: Re: [RFC v4 06/11] vduse: Implement an MMU-based IOMMU driver

2021-03-03 Thread Yongji Xie
On Thu, Mar 4, 2021 at 12:21 PM Jason Wang wrote: > > > On 2021/2/23 7:50 下午, Xie Yongji wrote: > > This implements a MMU-based IOMMU driver to support mapping > > kernel dma buffer into userspace. The basic idea behind it is > > treating MMU (VA->PA) as IOMMU (IOVA->PA). The driver will set > > u

Re: Re: [RFC v4 05/11] vdpa: Support transferring virtual addressing during DMA mapping

2021-03-03 Thread Yongji Xie
On Wed, Mar 3, 2021 at 6:52 PM Mika Penttilä wrote: > > > > On 23.2.2021 13.50, Xie Yongji wrote: > > This patch introduces an attribute for vDPA device to indicate > > whether virtual address can be used. If vDPA device driver set > > it, vhost-vdpa bus driver will not pin user page and transfer

Re: Re: [RFC v4 01/11] eventfd: Increase the recursion depth of eventfd_signal()

2021-03-02 Thread Yongji Xie
On Tue, Mar 2, 2021 at 2:44 PM Jason Wang wrote: > > > On 2021/2/23 7:50 下午, Xie Yongji wrote: > > Increase the recursion depth of eventfd_signal() to 1. This > > is the maximum recursion depth we have found so far. > > > > Signed-off-by: Xie Yongji > > > Acked-by: Jason Wang > > It might be use

Re: Re: [RFC v4 02/11] vhost-vdpa: protect concurrent access to vhost device iotlb

2021-03-02 Thread Yongji Xie
On Tue, Mar 2, 2021 at 2:47 PM Jason Wang wrote: > > > On 2021/2/23 7:50 下午, Xie Yongji wrote: > > Use vhost_dev->mutex to protect vhost device iotlb from > > concurrent access. > > > > Fixes: 4c8cf318("vhost: introduce vDPA-based backend") > > Signed-off-by: Xie Yongji > > --- > > drivers/vhos

Re: Re: [RFC v3 03/11] vdpa: Remove the restriction that only supports virtio-net devices

2021-01-30 Thread Yongji Xie
On Fri, Jan 29, 2021 at 11:04 PM Stefano Garzarella wrote: > > On Thu, Jan 28, 2021 at 11:11:49AM +0800, Jason Wang wrote: > > > >On 2021/1/27 下午4:57, Stefano Garzarella wrote: > >>On Wed, Jan 27, 2021 at 11:33:03AM +0800, Jason Wang wrote: > >>> > >>>On 2021/1/20 下午7:08, Stefano Garzarella wrote:

Re: Re: [RFC v3 08/11] vduse: Introduce VDUSE - vDPA Device in Userspace

2021-01-27 Thread Yongji Xie
On Thu, Jan 28, 2021 at 2:14 PM Jason Wang wrote: > > > On 2021/1/28 下午2:03, Yongji Xie wrote: > >>>>> + > >>>>> +static const struct file_operations vduse_domain_fops = { > >>>>> + .mmap = vduse_domain_mmap, > >>>

Re: Re: [RFC v3 01/11] eventfd: track eventfd_signal() recursion depth separately in different cases

2021-01-27 Thread Yongji Xie
On Thu, Jan 28, 2021 at 12:31 PM Jason Wang wrote: > > > On 2021/1/28 上午11:52, Yongji Xie wrote: > > On Thu, Jan 28, 2021 at 11:05 AM Jason Wang wrote: > >> > >> On 2021/1/27 下午5:11, Yongji Xie wrote: > >>> On Wed, Jan 27, 2021 at 11:38 AM Jason Wa

Re: Re: [RFC v3 08/11] vduse: Introduce VDUSE - vDPA Device in Userspace

2021-01-27 Thread Yongji Xie
On Thu, Jan 28, 2021 at 12:27 PM Jason Wang wrote: > > > On 2021/1/27 下午4:50, Yongji Xie wrote: > > On Tue, Jan 26, 2021 at 4:09 PM Jason Wang wrote: > >> > >> On 2021/1/19 下午1:07, Xie Yongji wrote: > >>> This VDUSE driver enables implementing vDPA

Re: Re: [RFC v3 01/11] eventfd: track eventfd_signal() recursion depth separately in different cases

2021-01-27 Thread Yongji Xie
On Thu, Jan 28, 2021 at 11:08 AM Jens Axboe wrote: > > On 1/27/21 8:04 PM, Jason Wang wrote: > > > > On 2021/1/27 下午5:11, Yongji Xie wrote: > >> On Wed, Jan 27, 2021 at 11:38 AM Jason Wang wrote: > >>> > >>> On 2021/1/20 下午2:52, Yongji Xie w

Re: Re: [RFC v3 01/11] eventfd: track eventfd_signal() recursion depth separately in different cases

2021-01-27 Thread Yongji Xie
On Thu, Jan 28, 2021 at 11:05 AM Jason Wang wrote: > > > On 2021/1/27 下午5:11, Yongji Xie wrote: > > On Wed, Jan 27, 2021 at 11:38 AM Jason Wang wrote: > >> > >> On 2021/1/20 下午2:52, Yongji Xie wrote: > >>> On Wed, Jan 20, 2021 at 12:24 PM Jason

Re: Re: [RFC v3 08/11] vduse: Introduce VDUSE - vDPA Device in Userspace

2021-01-27 Thread Yongji Xie
; + struct file *file; > > + unsigned long bounce_pfns = PAGE_ALIGN(bounce_size) >> PAGE_SHIFT; > > + > > + domain = kzalloc(sizeof(*domain), GFP_KERNEL); > > + if (!domain) > > + return NULL; > > + > > + domain

Re: Re: [RFC v3 06/11] vhost-vdpa: Add an opaque pointer for vhost IOTLB

2021-01-27 Thread Yongji Xie
On Wed, Jan 27, 2021 at 11:51 AM Jason Wang wrote: > > > On 2021/1/20 下午3:52, Yongji Xie wrote: > > On Wed, Jan 20, 2021 at 2:24 PM Jason Wang wrote: > >> > >> On 2021/1/19 下午12:59, Xie Yongji wrote: > >>> Add an opaque pointer for vhost IOTLB to s

Re: Re: [RFC v3 01/11] eventfd: track eventfd_signal() recursion depth separately in different cases

2021-01-27 Thread Yongji Xie
On Wed, Jan 27, 2021 at 11:38 AM Jason Wang wrote: > > > On 2021/1/20 下午2:52, Yongji Xie wrote: > > On Wed, Jan 20, 2021 at 12:24 PM Jason Wang wrote: > >> > >> On 2021/1/19 下午12:59, Xie Yongji wrote: > >>> Now we have a global percpu counter to limi

Re: Re: [RFC v3 03/11] vdpa: Remove the restriction that only supports virtio-net devices

2021-01-27 Thread Yongji Xie
On Wed, Jan 27, 2021 at 4:59 PM Stefano Garzarella wrote: > > On Tue, Jan 19, 2021 at 12:59:12PM +0800, Xie Yongji wrote: > >With VDUSE, we should be able to support all kinds of virtio devices. > > > >Signed-off-by: Xie Yongji > >--- > > drivers/vhost/vdpa.c | 29 +++-- >

Re: Re: [RFC v3 11/11] vduse: Introduce a workqueue for irq injection

2021-01-27 Thread Yongji Xie
On Tue, Jan 26, 2021 at 4:17 PM Jason Wang wrote: > > > On 2021/1/19 下午1:07, Xie Yongji wrote: > > This patch introduces a dedicated workqueue for irq injection > > so that we are able to do some performance tuning for it. > > > > Signed-off-by: Xie Yongji > > > If we want the split like this. >

Re: Re: [RFC v3 08/11] vduse: Introduce VDUSE - vDPA Device in Userspace

2021-01-27 Thread Yongji Xie
On Tue, Jan 26, 2021 at 4:19 PM Jason Wang wrote: > > > On 2021/1/19 下午1:07, Xie Yongji wrote: > > This VDUSE driver enables implementing vDPA devices in userspace. > > Both control path and data path of vDPA devices will be able to > > be handled in userspace. > > > > In the control path, the VDU

Re: Re: [RFC v3 10/11] vduse: grab the module's references until there is no vduse device

2021-01-27 Thread Yongji Xie
On Tue, Jan 26, 2021 at 4:10 PM Jason Wang wrote: > > > On 2021/1/19 下午1:07, Xie Yongji wrote: > > The module should not be unloaded if any vduse device exists. > > So increase the module's reference count when creating vduse > > device. And the reference count is kept until the device is > > dest

Re: Re: [RFC v3 06/11] vhost-vdpa: Add an opaque pointer for vhost IOTLB

2021-01-19 Thread Yongji Xie
On Wed, Jan 20, 2021 at 2:24 PM Jason Wang wrote: > > > On 2021/1/19 下午12:59, Xie Yongji wrote: > > Add an opaque pointer for vhost IOTLB to store the > > corresponding vma->vm_file and offset on the DMA mapping. > > > Let's split the patch into two. > > 1) opaque pointer > 2) vma stuffs > OK. >

Re: Re: [RFC v3 05/11] vdpa: shared virtual addressing support

2021-01-19 Thread Yongji Xie
On Wed, Jan 20, 2021 at 1:55 PM Jason Wang wrote: > > > On 2021/1/19 下午12:59, Xie Yongji wrote: > > This patches introduces SVA (Shared Virtual Addressing) > > support for vDPA device. During vDPA device allocation, > > vDPA device driver needs to indicate whether SVA is > > supported by the devic

Re: Re: [RFC v3 01/11] eventfd: track eventfd_signal() recursion depth separately in different cases

2021-01-19 Thread Yongji Xie
On Wed, Jan 20, 2021 at 12:24 PM Jason Wang wrote: > > > On 2021/1/19 下午12:59, Xie Yongji wrote: > > Now we have a global percpu counter to limit the recursion depth > > of eventfd_signal(). This can avoid deadlock or stack overflow. > > But in stack overflow case, it should be OK to increase the

Re: Re: [RFC v3 03/11] vdpa: Remove the restriction that only supports virtio-net devices

2021-01-19 Thread Yongji Xie
On Wed, Jan 20, 2021 at 11:47 AM Jason Wang wrote: > > > On 2021/1/19 下午12:59, Xie Yongji wrote: > > With VDUSE, we should be able to support all kinds of virtio devices. > > > > Signed-off-by: Xie Yongji > > --- > > drivers/vhost/vdpa.c | 29 +++-- > > 1 file changed,

Re: Re: [RFC v3 04/11] vhost-vdpa: protect concurrent access to vhost device iotlb

2021-01-19 Thread Yongji Xie
On Wed, Jan 20, 2021 at 11:44 AM Jason Wang wrote: > > > On 2021/1/19 下午12:59, Xie Yongji wrote: > > Introduce a mutex to protect vhost device iotlb from > > concurrent access. > > > > Fixes: 4c8cf318("vhost: introduce vDPA-based backend") > > Signed-off-by: Xie Yongji > > --- > > drivers/vhost

Re: Re: [RFC v3 08/11] vduse: Introduce VDUSE - vDPA Device in Userspace

2021-01-19 Thread Yongji Xie
On Wed, Jan 20, 2021 at 1:54 AM Randy Dunlap wrote: > > Hi, > > Documentation comments only: > Will fix it. Thanks, Yongji > On 1/18/21 9:07 PM, Xie Yongji wrote: > > > > Signed-off-by: Xie Yongji > > --- > > Documentation/driver-api/vduse.rst | 85 ++ > > > > diff --git a/D

Re: Re: [RFC v3 08/11] vduse: Introduce VDUSE - vDPA Device in Userspace

2021-01-19 Thread Yongji Xie
On Tue, Jan 19, 2021 at 10:54 PM Jonathan Corbet wrote: > > X-Gm-Spam: 0 > X-Gm-Phishy: 0 > > On Tue, 19 Jan 2021 13:07:53 +0800 > Xie Yongji wrote: > > > diff --git a/Documentation/driver-api/vduse.rst > > b/Documentation/driver-api/vduse.rst > > new file mode 100644 > > index ..941

Re: Re: [RFC v2 06/13] vduse: Introduce VDUSE - vDPA Device in Userspace

2021-01-10 Thread Yongji Xie
On Fri, Jan 8, 2021 at 9:32 PM Bob Liu wrote: > > On 12/22/20 10:52 PM, Xie Yongji wrote: > > This VDUSE driver enables implementing vDPA devices in userspace. > > Both control path and data path of vDPA devices will be able to > > be handled in userspace. > > > > In the control path, the VDUSE dr

Re: Re: [RFC v2 09/13] vduse: Add support for processing vhost iotlb message

2020-12-31 Thread Yongji Xie
On Thu, Dec 31, 2020 at 3:12 PM Jason Wang wrote: > > > On 2020/12/31 下午2:52, Yongji Xie wrote: > > On Thu, Dec 31, 2020 at 1:50 PM Jason Wang wrote: > >> > >> On 2020/12/31 下午1:15, Yongji Xie wrote: > >>> On Thu, Dec 31, 2020 at 10:49 AM Jason Wan

Re: Re: [RFC v2 09/13] vduse: Add support for processing vhost iotlb message

2020-12-30 Thread Yongji Xie
On Thu, Dec 31, 2020 at 1:50 PM Jason Wang wrote: > > > On 2020/12/31 下午1:15, Yongji Xie wrote: > > On Thu, Dec 31, 2020 at 10:49 AM Jason Wang wrote: > >> > >> On 2020/12/30 下午6:12, Yongji Xie wrote: > >>> On Wed, Dec 30, 2020 at 4:41 PM Jason Wan

Re: Re: [RFC v2 09/13] vduse: Add support for processing vhost iotlb message

2020-12-30 Thread Yongji Xie
On Thu, Dec 31, 2020 at 10:49 AM Jason Wang wrote: > > > On 2020/12/30 下午6:12, Yongji Xie wrote: > > On Wed, Dec 30, 2020 at 4:41 PM Jason Wang wrote: > >> > >> On 2020/12/30 下午3:09, Yongji Xie wrote: > >>> On Wed, Dec 30, 2020 at 2:11 PM Jason Wan

Re: Re: [RFC v2 09/13] vduse: Add support for processing vhost iotlb message

2020-12-30 Thread Yongji Xie
On Wed, Dec 30, 2020 at 4:41 PM Jason Wang wrote: > > > On 2020/12/30 下午3:09, Yongji Xie wrote: > > On Wed, Dec 30, 2020 at 2:11 PM Jason Wang wrote: > >> > >> On 2020/12/29 下午6:26, Yongji Xie wrote: > >>> On Tue, Dec 29, 2020 at 5:11 PM Jason Wang

Re: Re: [RFC v2 09/13] vduse: Add support for processing vhost iotlb message

2020-12-29 Thread Yongji Xie
On Wed, Dec 30, 2020 at 2:11 PM Jason Wang wrote: > > > On 2020/12/29 下午6:26, Yongji Xie wrote: > > On Tue, Dec 29, 2020 at 5:11 PM Jason Wang wrote: > >> > >> > >> - Original Message - > >>> On Mon, Dec 28, 2020 at 4:43 PM Jason W

Re: Re: [RFC v2 09/13] vduse: Add support for processing vhost iotlb message

2020-12-29 Thread Yongji Xie
On Tue, Dec 29, 2020 at 5:11 PM Jason Wang wrote: > > > > - Original Message - > > On Mon, Dec 28, 2020 at 4:43 PM Jason Wang wrote: > > > > > > > > > On 2020/12/28 下午4:14, Yongji Xie wrote: > > > >> I see. So all the abov

Re: Re: [RFC v2 09/13] vduse: Add support for processing vhost iotlb message

2020-12-28 Thread Yongji Xie
On Mon, Dec 28, 2020 at 4:43 PM Jason Wang wrote: > > > On 2020/12/28 下午4:14, Yongji Xie wrote: > >> I see. So all the above two questions are because VHOST_IOTLB_INVALIDATE > >> is expected to be synchronous. This need to be solved by tweaking the > >> curren

Re: Re: [RFC v2 09/13] vduse: Add support for processing vhost iotlb message

2020-12-28 Thread Yongji Xie
On Mon, Dec 28, 2020 at 3:43 PM Jason Wang wrote: > > > On 2020/12/25 下午6:31, Yongji Xie wrote: > > On Fri, Dec 25, 2020 at 2:58 PM Jason Wang wrote: > >> > >> On 2020/12/24 下午3:37, Yongji Xie wrote: > >>> On Thu, Dec 24, 2020 at 10:41 AM Jason Wan

Re: Re: [RFC v2 09/13] vduse: Add support for processing vhost iotlb message

2020-12-25 Thread Yongji Xie
On Fri, Dec 25, 2020 at 3:02 PM Jason Wang wrote: > > > On 2020/12/25 上午10:37, Yongji Xie wrote: > > On Thu, Dec 24, 2020 at 3:37 PM Yongji Xie wrote: > >> On Thu, Dec 24, 2020 at 10:41 AM Jason Wang wrote: > >>> > >>> On 2020/12/23 下午8:14, Yongj

Re: Re: [RFC v2 09/13] vduse: Add support for processing vhost iotlb message

2020-12-25 Thread Yongji Xie
On Fri, Dec 25, 2020 at 2:58 PM Jason Wang wrote: > > > On 2020/12/24 下午3:37, Yongji Xie wrote: > > On Thu, Dec 24, 2020 at 10:41 AM Jason Wang wrote: > >> > >> On 2020/12/23 下午8:14, Yongji Xie wrote: > >>> On Wed, Dec 23, 2020 at 5:05 PM Jason W

Re: Re: [RFC v2 09/13] vduse: Add support for processing vhost iotlb message

2020-12-24 Thread Yongji Xie
On Thu, Dec 24, 2020 at 3:37 PM Yongji Xie wrote: > > On Thu, Dec 24, 2020 at 10:41 AM Jason Wang wrote: > > > > > > On 2020/12/23 下午8:14, Yongji Xie wrote: > > > On Wed, Dec 23, 2020 at 5:05 PM Jason Wang wrote: > > >> > > >> On 2020/

Re: Re: [RFC v2 06/13] vduse: Introduce VDUSE - vDPA Device in Userspace

2020-12-24 Thread Yongji Xie
On Thu, Dec 24, 2020 at 11:01 AM Jason Wang wrote: > > > On 2020/12/23 下午10:17, Yongji Xie wrote: > > On Wed, Dec 23, 2020 at 4:08 PM Jason Wang wrote: > >> > >> On 2020/12/22 下午10:52, Xie Yongji wrote: > >>> This VDUSE driver enables implementing

Re: Re: [RFC v2 09/13] vduse: Add support for processing vhost iotlb message

2020-12-23 Thread Yongji Xie
On Thu, Dec 24, 2020 at 10:41 AM Jason Wang wrote: > > > On 2020/12/23 下午8:14, Yongji Xie wrote: > > On Wed, Dec 23, 2020 at 5:05 PM Jason Wang wrote: > >> > >> On 2020/12/22 下午10:52, Xie Yongji wrote: > >>> To support vhost-vdpa bus driver, we nee

Re: Re: [RFC v2 08/13] vdpa: Introduce process_iotlb_msg() in vdpa_config_ops

2020-12-23 Thread Yongji Xie
On Thu, Dec 24, 2020 at 10:37 AM Jason Wang wrote: > > > On 2020/12/23 下午7:06, Yongji Xie wrote: > > On Wed, Dec 23, 2020 at 4:37 PM Jason Wang wrote: > >> > >> On 2020/12/22 下午10:52, Xie Yongji wrote: > >>> This patch introduces a new method in th

Re: [External] Re: [RFC v2 01/13] mm: export zap_page_range() for driver use

2020-12-23 Thread Yongji Xie
On Wed, Dec 23, 2020 at 4:13 PM Christoph Hellwig wrote: > > On Wed, Dec 23, 2020 at 02:32:07PM +0800, Yongji Xie wrote: > > Now I want to map/unmap some pages in an userland vma dynamically. The > > vm_insert_page() is being used for mapping. In the unmapping case, it

Re: [RFC v2 06/13] vduse: Introduce VDUSE - vDPA Device in Userspace

2020-12-23 Thread Yongji Xie
> + domain->size = size; > > + INIT_LIST_HEAD(&domain->vma_list); > > + mutex_init(&domain->vma_lock); > > + mutex_init(&domain->map_lock); > > + > > + return domain; > > +err: > > + for (j

Re: [External] Re: [RFC v2 09/13] vduse: Add support for processing vhost iotlb message

2020-12-23 Thread Yongji Xie
On Wed, Dec 23, 2020 at 5:05 PM Jason Wang wrote: > > > On 2020/12/22 下午10:52, Xie Yongji wrote: > > To support vhost-vdpa bus driver, we need a way to share the > > vhost-vdpa backend process's memory with the userspace VDUSE process. > > > > This patch tries to make use of the vhost iotlb messag

Re: [RFC v2 08/13] vdpa: Introduce process_iotlb_msg() in vdpa_config_ops

2020-12-23 Thread Yongji Xie
On Wed, Dec 23, 2020 at 4:37 PM Jason Wang wrote: > > > On 2020/12/22 下午10:52, Xie Yongji wrote: > > This patch introduces a new method in the vdpa_config_ops to > > support processing the raw vhost memory mapping message in the > > vDPA device driver. > > > > Signed-off-by: Xie Yongji > > --- >

Re: [RFC v2 00/13] Introduce VDUSE - vDPA Device in Userspace

2020-12-23 Thread Yongji Xie
On Wed, Dec 23, 2020 at 2:38 PM Jason Wang wrote: > > > On 2020/12/22 下午10:52, Xie Yongji wrote: > > This series introduces a framework, which can be used to implement > > vDPA Devices in a userspace program. The work consist of two parts: > > control path forwarding and data path offloading. > >

Re: [RFC v2 01/13] mm: export zap_page_range() for driver use

2020-12-22 Thread Yongji Xie
On Tue, Dec 22, 2020 at 11:44 PM Christoph Hellwig wrote: > > On Tue, Dec 22, 2020 at 10:52:09PM +0800, Xie Yongji wrote: > > Export zap_page_range() for use in VDUSE. > > Err, no. This has absolutely no business being used by drivers. Now I want to map/unmap some pages in an userland vma dynami

Re: [External] Re: [PATCH 0/7] Introduce vdpa management tool

2020-12-02 Thread Yongji Xie
On Wed, Dec 2, 2020 at 7:13 PM Parav Pandit wrote: > > > > > From: Yongji Xie > > Sent: Wednesday, December 2, 2020 2:52 PM > > > > On Wed, Dec 2, 2020 at 12:53 PM Parav Pandit wrote: > > > > > > > > > > > >

  1   2   >