Hi Sounds like something that could be useful, although it would be a bit uncommon to have a bunch of data in Solr already without already knowing your fields?
The proposal reminds me of OpenRefine https://openrefine.org which is a great tool to use *before* you start indexing a dataset to Solr. It will eat a CSV, XMl or whatever, and let you discover what's in the data, cardinality, data quality etc. A different use case, but just wanted to x-ref it. Jan > 13. nov. 2026 kl. 19:38 skrev David Smiley via dev <[email protected]>: > > (disclaimer: I worked with Srikanth on this proposal) > > I've seen a number of situations where we fumble in the absence of this or > things are built similar outside of Solr. The experts here may not fully > appreciate this as we are experts; Solr newbies and sometimes agents, are > not. I think there is growing utility for this with LLM-based > interrogation of what's in Solr, and I could definitely see Solr's MCP > using this. > > Safety: This handler is only intended for use by a single person/LLM at a > time to protect the "production" load assumed to be happening concurrently > on the cluster. Of course this aspect could be toggled off if desired > (e.g. a "dev" cluster). Assume a timeAllowed/cpuAllowed. Maybe even rate > limit to no more than one or two requests at a time (Solr has code for > this). > > There will be a slippery slope & tension on what /discover should do versus > leave to existing/standard Solr methods. In my view, /discover shouldn't > return entire documents -- just use /select if that is needed. > > I think it'd be helpful for the response structure to mirror /select where > it fits. Certainly for facets & stats, which is mostly what /discover > exposes. Much of the logic of /discover is just applying safe defaults on > what facet/stats to return (possibly blocking "unsafe" requests), plus a > rather new field coverage component. That's powered by > "facet.query=field:*" ultimately, but still. In order to determine if it's > "safe" to get the facet values for a field, an approximate cardinality > check may be necessary first. > > On Wed, Nov 11, 2026 at 4:40 PM Srikanth Tumati (BLOOMBERG/ 120 PARK) via > dev <[email protected]> wrote: > >> Hi all, >> >> I'd like your thoughts on adding a new data discovery handler to Solr. The >> design would revolve around two principles, simplicity and safety. It >> should be easy for anyone to use, and it should have protections built in >> so that exploring your data never puts the cluster at risk. In some cases, >> that may mean returning approximate results instead of exact ones. >> Questions, comments and concerns are all welcome. >> >> Problem >> People often come to Solr with a question but no idea how the data is >> structured. Which fields exist, which ones are populated, and what do they >> contain? Developers, analysts and (increasingly) LLM agents all run into >> this, and Solr makes these questions hard to answer. Additionally, dynamic >> fields mean the schema doesn't show which fields actually exist, and >> documents in the same collection can often have different fields. >> Performing a very naive data discovery has the potential to execute very >> expensive Solr queries (e.g. faceting on a high cardinality field, >> returning a significant number of rows, etc…). >> >> What exists today >> *Schema XML/Schema API shows the fields that are defined, but not the >> dynamic fields that actually exist in the index. It also has no statistics, >> so clients are unable to tell which fields have data. >> *Luke (/admin/luke) has per-field stats, but you can't narrow them down >> with filter queries. Newer versions of Solr can run it across shards, but >> the detailed term stats are still per-shard. Finally, it's also an advanced >> tool and hard for beginners to use. >> *Facets can answer the question, but only if you already know how to write >> a good Solr query. Every client has to build a facet for each field and >> batch them, which isn't obvious to someone new to Solr. The concern of >> expensive queries is directly related to utilizing facets. >> >> Proposal >> A new DiscoveryHandler that takes filter queries (fq) and supports: >> *Field coverage: how many matching documents have each field >> *Field statistics/facets: distinct count, top values, and min/max for >> numbers and dates. For low cardinality fields, the handler would return the >> top facets. Where an exact answer would be too expensive, these may be >> provided as approximate values. >> *Pivot field: optionally compute stats and coverage by the values of a >> field, such as the document type. Each value-group then gets its own >> numbers. >> Future Plans >> Once the basic stats are available, there's a lot more the handler could >> return: >> *Field relationships: field coverage inter-relationships, such as fields >> that only appear when another field is present or has a certain value. >> *Visualization: the Admin UI could eventually be updated to display this >> information. How exactly is out of scope for this proposal. >> The aim is for discovery to eventually grow from "what fields are here" >> into a real picture of how a user’s data is structured. >> >> Thank you, >> Srikanth >> >> >> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
