Thank you for the response Brian,
with both config a and config b, i was unable to see the source label.
suggested config a)
metric_relabel_configs:
- source_labels: [ source, __address__ ]
regex: ";(.*)(:.*)?"
target_label: source
replacement: '${1}'
output:
*query: custom_metrics_from_server*
response:
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
source="server1.example.com", status="pend"} 3215
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
source="server1.example.com", status="run"} 3213
*query: up*
response:
up{instance="localhost:9104", job="nodeexporter"} 1
*query: node_cooling_device_max_state{instance="localhost:9104",
job="nodeexporter", name="0", type="Processor"}*
response:
node_cooling_device_max_state{instance="localhost:9104", job="nodeexporter"
, name="0", type="Processor"} 0
source= metrics was not available.
now to debug, i changed the query slightly , and it seems that the
__address__ is not available with metric_relabel_configs?
modified config a)
- source_labels: [ source, __address__ ]
regex: "(.*)"
target_label: source
replacement: '${1}'
*query: custom_metrics_from_server*
response:
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
source="server1.example.com;", status="pend"} 3215
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
source="server1.example.com;", status="run"} 3213
*query: up*
up{instance="localhost:9104", job="nodeexporter"}
*query: node_cooling_device_max_state{instance="localhost:9104",
job="nodeexporter", name="0", type="Processor"}*
node_cooling_device_max_state{instance="localhost:9104", job="nodeexporter"
, name="0", source=";", type="Processor"} 0
so from the last query, it seems that __address__ is blank, and since
source= is also blank, we see source=";" which appears to be concatenation
of 2 blank values with ; as delimiter.
Q1: here i am unable to understand that how ; was appended after source
value in *custom_metrics_from_server* .
config b) :
metric_relabel_configs:
- source_labels: [ source ]
regex: ""
target_label: __tmp_source_empty
replacement: Y
- source_labels: [ __tmp_source_empty, __address__ ]
regex: "Y;(.*)(:.*)?"
target_label: source
replacement: "${1}"
- target_label: __tmp_source_empty
replacement: ""
output:
*query: custom_metrics_from_server*
response:
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
source="server1.example.com", status="pend"} 3215
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
source="server1.example.com", status="run"} 3213
*query: up*
response:
up{instance="localhost:9104", job="nodeexporter"} 1
*query: node_cooling_device_max_state{instance="localhost:9104",
job="nodeexporter", name="0", type="Processor"}*
response:
node_cooling_device_max_state{instance="localhost:9104", job="nodeexporter"
, name="0", type="Processor"} 0
source= metrics was not available.
now to debug, i changed the query slightly , and it seems that the
__address__ is not available with metric_relabel_configs with config b?
modified config b)
*query: custom_metrics_from_server*
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
source=";", status="pend"} 3215
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
source=";", status="run"} 3213
*query: up*
up{instance="localhost:9104", job="nodeexporter"}
*query: node_cooling_device_max_state{instance="localhost:9104",
job="nodeexporter", name="0", type="Processor"}*
node_cooling_device_max_state{instance="localhost:9104", job="nodeexporter"
, name="0", source="Y;", type="Processor"}
so from the last query, it seems that __address__ is blank, and since
source= is also blank, we see source=";" which appears to be concatenation
of 2 blank values with ; as delimiter.
so far , it appears that the __address__ field is not available with
metrics_relabel_config?
Please advice.
On Wednesday, March 12, 2025 at 12:24:11 AM UTC+5:30 Brian Candler wrote:
> It's better to avoid honor_labels unless you really need it. It allows the
> exporter to do bad things like overriding the "job" and "instance" labels.
>
> Better to use metric_relabel_configs to relabel, at which point you have
> the combined set of labels from service discovery and the scrape. A missing
> label is the same as a label whose value is the empty string. So you can do
> (untested):
>
> metric_relabel_configs:
> - source_labels: [ source, __address__ ]
> regex: ";(.*)(:.*)?"
>
> target_label: source
> replacement: "${1}"
>
> This works as long as the source label's value cannot start with a
> semicolon - if it can, define a different "separator" character. If you
> don't like that, another approach is to set a temporary label as a flag:
>
> metric_relabel_configs:
> - source_labels: [ source ]
> regex: ""
> target_label: __tmp_source_empty
> replacement: Y
> - source_labels: [ __tmp_source_empty, __address__ ]
> regex: "Y;(.*)(:.*)?"
> target_label: source
> replacement: "${1}"
> - target_label: __tmp_source_empty
> replacement: ""
>
> On Tuesday, 11 March 2025 at 17:39:58 UTC mohan garden wrote:
>
>> Hi Everyone,
>>
>> I need to handle a scenario where I add a label to ingested metrics, but
>> only if the label does not already exist in the collected data. If the
>> label is present in a metric, it should remain unchanged.
>>
>> For example, given a set of metrics:
>>
>> custom_metrics_from_server{ source="server1.example.com",status="run" }
>> 3213
>> custom_metrics_from_server{ source="server1.example.com",status="pend" }
>> 3215
>>
>> When additional metrics are collected from a node exporter, some will
>> already have a "source=" label, while others will not:
>>
>> custom_metrics_from_server{ source="server1.example.com",status="run" }
>> 3213
>> custom_metrics_from_server{ source="server1.example.com",status="pend" }
>> 3215
>> node_cooling_device_max_state{name="7",type="Processor"} 0
>>
>> To ensure that all metrics contain the "source=" label without overriding
>> existing ones, I figured the following approach using honor_labels and
>> relabel_configs:
>>
>> honor_labels: true
>> relabel_configs:
>> - source_labels: [ _address_ ]
>> regex: "(.*):(.*)"
>> target_label: "source"
>> replacement: '${1}'
>>
>> Is this the best way to handle this, or is there a more optimal approach?
>> Please advice.
>>
>> Thanks!
>>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion visit
https://groups.google.com/d/msgid/prometheus-users/ff6ef6e6-3431-4ee7-93c9-82333de86537n%40googlegroups.com.
