I tested "streaming expressions" ('expr=search(test_collection,q="*:*",fl="id, text_sn",sort="id asc",rows=1600)') on collection with one shard with small documents - a long preparation of the server response before the data transfer begins (it looks like when the collection consisted of 8 shar
I was able to create a collection with "solr.SimpleTextCodecFactory" codecFactory and solr can proces (return) only 2x more documents per second from it (214 410 documents per second vs 115 000 "solr.SchemaCodecFactory" with compression). I expected much much more, because this is a simple itera
Thank you for working on the Solr performance issues raised here. LZ4 is a great solution, but let's look at how things are today. As far as I understand, uncompressed fields have been abandoned since version 4.1 (early 2013). At that time, 15,000 RPM SAS disks produced 350 MB/s or 150-180 iops a
Continuing my research on the performance of data fetching from Solr, I noticed a significant drop in the transfer rate when the size of stored fields decreased. Below are the results of measuring the data transfer rate (wt=javabin) from a collection of 10 gigabytes in size, but consisting of a
I checked how an increase in the size of the outputAggregationSize and outputBufferSize in Jetty.xml affects the data transfer speed. - outputBufferSize - set the size of the buffer into which response content is aggregated before being sent to the client. A larger buffer can improve performance
Thank you for your work Noble Paul, it's very interesting. You were so attentive that you noticed and replaced "else if (val instanceof Double) { gen.writeNumber(val.floatValue()); }" with "else if (val instanceof Double) { gen.writeNumber(val.doubleValue()); }". But I just copied the code (Line
Thank you, you are very kind. I took measurements on two physical servers with a 10 Gigabit link, the speed and time of full fetching 10 Gb collection (one shard; empty "Accept-Encoding: " header; collection with only id and string stored fields) are as follows:original wt=json - 419 Mb/s fet
Thanks. In the coming days I will conduct testing and measurements on real hardware. Unfortunately my code is not ready to become part of the project directly, since this is a very serious place for changes and I am not a Java developer, I am not deeply familiar with the work of internal Solr mec
Thanks to everyone for your help (especially Mikhail Khludnev and Michael Gibney)! I rewrote JSONResponseWriter using fasterxml jackson library (com.fasterxml.jackson.core.JsonGenerator, I took the SmileResponseWriter code as a basis) and added it to the collection as my own queryResponseWriter
Thank you for helping me figure out the performance of "Response Writers". If you wish, you can close the topic or move it to issues under the title performance of non-binary types of "Response Writers" (if in your opinion the difference of 4-8 times between wt=javabin and wt=json is not normal). M
David Smiley, sorry for my terminology, I’m used to calling a full data fetching by small parts from DB table (collection) as "scrolling". Of course, in Solr cursors (cursorMark) are designed for this and I use them. Large "rows" values in my examples (measurements) are needed to show the speed at
Thank you for your help with slow single threaded data receiving from Solr. Today I was able to reach a speed of 3Gigabit+ and got results that may be useful in the future. I turned out to be wrong in assuming that the main problem is in the FastWriter output buffer, but this was the most obvious t
Thanks for the patch for testing. I could not see significant improvements on virtual machines, I will try again this week on servers.I tried the following values for buffers: 65536 - 64Kb, 262144 - 256Kb, 524288 - 512Kb, 1048576 - 1MB, 4194304 - 4MB, 16777216 - 16MB, 33554432 - 32Mb, 67108864 - 6
Today I created a 1GB collection with RAMDirectoryFactory and only id, stored unindexed without docValues text_sn fields in the schema. In solrconfig.xml:... and in section:${solr.lock.type:single} In managed-schema:... ... Receiving all data from it is also carried out at slow speeds of ap
jetty.xml used in default solr configuration, file downloaded by http and nginx maybe faster because in /etc/nginx/sites-available/default - location / { aio threads; sendfile on; }I'm reproduce this in Ubuntu on virtual machines in home PC:# 0 set /etc/sysctl.conf settings from https://darksideclo
I'm installed Solr 8.11.1 (SOLR_JAVA_MEM="-Xms31g -Xmx31g") into ram disk in hi-performance server with 10-Gigabit network adapters. Jumbo Frames (MTU) enabled and sets to 9000, linux core tcp buffers tunned for 10-Gigabit network (/etc/sysctl.conf):net.ipv4.tcp_rmem = 8192 87380 134217728net.ipv4.
16 matches
Mail list logo
