Fantastic! I really appreciate you working with the community on this one. ~ David Smiley Apache Lucene/Solr Search Developer http://www.linkedin.com/in/davidwsmiley
On Mon, Mar 6, 2023 at 6:32 PM Fikavec F <fika...@yandex.ru> wrote: > Thank you, you are very kind. I took measurements on two physical servers > with a 10 Gigabit link, the speed and time of full fetching 10 Gb > collection (one shard; empty "Accept-Encoding: " header; collection with > only id and string stored fields) are as follows: > original wt=json - 419 Mb/s fetching time: 3m 26s > original wt=csv - 463 Mb/s fetching time: 3m 6s > my wt=myfastjson - 2.33 Gb/s fetching time: 37s > original wt=smile - 2.38 Gb/s fetching time: 36s > my wt=myfastcbor - 2.55 Gb/s fetching time: 34s > original wt=javabin - 2.76 Gb/s fetching time: 31s > > With "Accept-Encoding: gzip" header: > original wt=json - 81 Mb/s fetching time: 12m 3s > my wt=myfastjson - 114 Mb/s fetching time: 8m 32s > > When I placed the test collection on 4 shards, wt=json worked at speeds > of about 430 Mb/s, and javabin, myfastcbor, myfastjson worked at speeds of > 5.2-6.7 Gb/s, however, before the start of data transfer, there was a > significant delay (possibly for data collection from individual shards and > sorting) and the final full fetching time did not exceed the above. > Downloading a 10 GB file from Jetty took place at a speed of 9.87 Gb/s. > The time to move the cursor through the collection documents and get > the field values in my python application has been reduced from 4m 24s to > 1m 4s, moreover, half of this minute was spent to deserialization in python > from json and cbor. No failures were noticed, everything seemed to work as > before, only very quickly (4х+ faster). > I dreamed of seeing Solr work at 5Gigabit+ speeds and with the help of > your support, everything worked out, thank you all. > > Best Regards, > > >
