The tricky part of setData as compared to getData is that in getData we can tell the data is compressed based on the initial bytes read. For setData the only way to know if we should compress the provided data is if we read the solr.xml to know if compression is enabled or by adding an argument to classes like ZkCpTool.
Putting an uncompressed state.json into a cluster where compression is used should still work fine so at this point I've erred on people using these tools outside of the Solr cluster having an understanding of how they have set up compression or not for when adding data into the cluster. Not opposed to adding more arguments for some classes, but we don't have a way to just handle compression in setData in the same way we do with getData. On Sat, Mar 2, 2024 at 11:23 AM Eric Pugh <ep...@opensourceconnections.com> wrote: > Looking at this, when we use the ZkCpTool to upload content, I’m not sure > it goes through the compression step? > > Looks like eventually we get to SolrZkClient.setData() and I don’t see > anything about compression.. Unlike the SolrZkClient.getData()… > > Am I reading this right? Shouldn’t setData mimic getData in handling > compression? > > Thanks for looking at this! > > > > > On Feb 29, 2024, at 12:29 PM, Justin Sweeney <justin.sweene...@gmail.com> > wrote: > > > > I actually think that use case should just work since the SolrZkClient > can > > already handle compressed state.json, assuming you are just using the > > default ZLib implementation of compression. When getting data it looks > like > > the ZkCpTool calls SolrZkClient.getData() which is able to check if the > > data is compressed. > > > > On Thu, Feb 29, 2024 at 8:58 AM Eric Pugh < > ep...@opensourceconnections.com <mailto:ep...@opensourceconnections.com>> > > wrote: > > > >> Hi all, > >> > >> I am poking around ZkCLI.java, and noticed that the compression for a > >> “state.json” file logic is in this file. I’m realizing that the > existing > >> bin/solr zk cp command knows nothing about a “state.json” file being > >> compressed or not, and so if you do > >> > >> bin/solr zk cp my_local_state.json zk:/state.json -z > localhost:9983 > >> > >> Then I think you don’t get the compression aspect kicking in. > >> > >> I could copy that logic into the ZkCpTool.java, but wondering if there > is > >> a better refactoring? Could this logic live in either > SolrZkClient.java > >> or ZkMaintenanceUtils.java ? > >> > >> Thoughts? > >> > >> Eric > >> _______________________ > >> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | > >> http://www.opensourceconnections.com < > >> http://www.opensourceconnections.com/> | My Free/Busy < > >> http://tinyurl.com/eric-cal> > >> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed < > >> > https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw > > > >> > >> This e-mail and all contents, including attachments, is considered to be > >> Company Confidential unless explicitly stated otherwise, regardless of > >> whether attachments are marked as such. > > _______________________ > Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | > http://www.opensourceconnections.com < > http://www.opensourceconnections.com/> | My Free/Busy < > http://tinyurl.com/eric-cal> > Co-Author: Apache Solr Enterprise Search Server, 3rd Ed < > https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw> > > This e-mail and all contents, including attachments, is considered to be > Company Confidential unless explicitly stated otherwise, regardless of > whether attachments are marked as such. > >
