On 2016年11月09日 15:14, Jason Wang wrote: >On 2016年11月08日 19:58, Zhangming (James, Euler) wrote: >> On 2016年11月08日 19:17, Jason Wang wrote: >> >>> On 2016年11月08日 19:13, Jason Wang wrote: >>>> Cc Michael >>>> >>>> On 2016年11月08日 16:34, Zhangming (James, Euler) wrote: >>>>> In container scenario, OVS is installed in the Virtual machine, and >>>>> all the containers connected to the OVS will communicated through >>>>> VXLAN encapsulation. >>>>> >>>>> By now, virtio_net does not support TSO offload for VXLAN >>>>> encapsulated TSO package. In this condition, the performance is not >>>>> good, sender is bottleneck >>>>> >>>>> I googled this scenario, but I didn’t find any information. Will >>>>> virtio_net support VXLAN encapsulation package TSO offload later? >>>>> >>>> Yes and for both sender and receiver. >>>> >>>>> My idea is virtio_net open encapsulated TSO offload, and transport >>>>> encapsulation info to TUN, TUN will parse the info and build skb >>>>> with encapsulation info. >>>>> >>>>> OVS or kernel on the host should be modified to support this. Using >>>>> this method, the TCP performance aremore than 2x as before. >>>>> >>>>> Any advice and suggestions for this idea or new idea will be >>>>> greatly appreciated! >>>>> >>>>> Best regards, >>>>> >>>>> James zhang >>>>> >>>> Sounds very good. And we may also need features bits >>>> (VIRTIO_NET_F_GUEST|HOST_GSO_X) for this. >>>> >>>> This is in fact one of items in networking todo list. (See >>>> http://www.linux-kvm.org/page/NetworkingTodo). While at it, we'd >>>> better support not only VXLAN but also other tunnels. >>> Cc Vlad who is working on extending virtio-net headers. >>> >>>> We can start with the spec work, or if you've already had some bits >>>> you can post them as RFC for early review. >>>> >>>> Thanks >> Below is my demo code >> Virtio_net.c >> static int virtnet_probe(struct virtio_device *vdev), add belows codes: >> if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF) || >> // avoid gso segment, it should be negotiation later, >> because in the demo I reuse num_buffers. >> virtio_has_feature(vdev, VIRTIO_F_VERSION_1)) { >> dev->hw_enc_features |= NETIF_F_TSO; >> dev->hw_enc_features |= NETIF_F_ALL_CSUM; >> dev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL; >> dev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL_CSUM; >> dev->hw_enc_features |= NETIF_F_GSO_TUNNEL_REMCSUM; >> >> dev->features |= NETIF_F_GSO_UDP_TUNNEL; >> dev->features |= NETIF_F_GSO_UDP_TUNNEL_CSUM; >> dev->features |= NETIF_F_GSO_TUNNEL_REMCSUM; >> } >> >> static int xmit_skb(struct send_queue *sq, struct sk_buff *skb), add >> below to pieces of codes >> >> if (skb_shinfo(skb)->gso_type & SKB_GSO_UDP_TUNNEL) >> hdr->hdr.gso_type |= VIRTIO_NET_HDR_GSO_TUNNEL; >> if (skb_shinfo(skb)->gso_type & SKB_GSO_UDP_TUNNEL_CSUM) >> hdr->hdr.gso_type |= VIRTIO_NET_HDR_GSO_TUNNEL_CSUM; >> if (skb_shinfo(skb)->gso_type & SKB_GSO_TUNNEL_REMCSUM) >> hdr->hdr.gso_type |= >> VIRTIO_NET_HDR_GSO_TUNNEL_REMCSUM; >> >> if (skb->encapsulation && skb_is_gso(skb)) { >> inner_mac_len = skb_inner_network_header(skb) - >> skb_inner_mac_header(skb); >> tnl_len = skb_inner_mac_header(skb) - skb_mac_header(skb); >> if ( !(inner_mac_len >> DATA_LEN_SHIFT) && !(tnl_len >> >> DATA_LEN_SHIFT) ) { >> hdr->hdr.flags |= VIRTIO_NET_HDR_F_ENCAPSULATION; >> hdr->num_buffers = (__virtio16)((inner_mac_len << >> DATA_LEN_SHIFT) | tnl_len); //we reuse num_buffers for simple >> , we should add extend member for later. >> } else >> hdr->num_buffers = 0; >> } >> >> Tun.c >> if (memcpy_fromiovecend((void *)&hdr, iv, offset, >> tun->vnet_hdr_sz)) //read header with negotiation length >> return -EFAULT; >> >> if (hdr.gso_type & VIRTIO_NET_HDR_GSO_TUNNEL) >> //set tunnel gso info >> skb_shinfo(skb)->gso_type |= SKB_GSO_UDP_TUNNEL; >> if (hdr.gso_type & VIRTIO_NET_HDR_GSO_TUNNEL_CSUM) >> skb_shinfo(skb)->gso_type |= >> SKB_GSO_UDP_TUNNEL_CSUM; >> if (hdr.gso_type & VIRTIO_NET_HDR_GSO_TUNNEL_REMCSUM) >> skb_shinfo(skb)->gso_type |= >> SKB_GSO_TUNNEL_REMCSUM; >> >> if (hdr.flags & VIRTIO_NET_HDR_F_ENCAPSULATION) { >> //read tunnel info from header and set to built skb. >> tnl_len = tun16_to_cpu(tun, hdr.num_buffers) & >> TUN_TNL_LEN_MASK; >> payload_mac_len = tun16_to_cpu(tun, hdr.num_buffers) >> >> TUN_DATA_LEN_SHIFT; >> mac_len = skb_network_header(skb) - skb_mac_header(skb); >> skb_set_inner_mac_header(skb, tnl_len - mac_len); >> skb_set_inner_network_header(skb, tnl_len + payload_mac_len >> - mac_len); >> skb->encapsulation = 1; >> } >> >> > >Something like this, and you probably need do something more: > >- use net-next.git to generate the patch (for the latest code) >- add feature negotiation >- tun/macvtap/qemu patches for this, you can start with tun/macvtap patches >- support for all other SKB_GSO_* types which is not supported >- use a new field instead of num_buffers >- a virtio spec patch to describe the support for encapsulation offload > >Thanks
Thank you for your advice, I will start it right now. Thanks
