Restrictinf xml file size

[girisubhra]

I want to read the xml file whose size is above 600 bytes. so i modifed code as below, but it is not giving expected result. can you please suggest on this.
File name : datasource.go

func (d *Datasource) ReadAll(ctx context.Context) ([]byte, error) {
var reader io.Reader
rc, err := d.GetStream(ctx)
if err != nil {
return nil, err
}
defer rc.Close()
reader = io.LimitReader(rc, *d.MaxContentLength)
if reader > 600 {
return ioutil.ReadAll(reader)
}
}

[MicroOps-cn]

Are you sure it is 600 bytes?
The XML file in the example "examples/xml_3gpp_ts /" is 2.86 KB. 600 bytes must be OK.

[MicroOps-cn]

data_exporter/collector/datasource.go

Line 191 in b0e93f9

const DefaultMaxContent = 102400000

data_exporter/collector/datasource.go

Lines 268 to 277 in b0e93f9

	if d.LineMaxContentLength == nil {
	d.LineMaxContentLength = new(int)
	*d.LineMaxContentLength = DefaultMaxContent
	}
	if d.MaxContentLength == nil {
	d.MaxContentLength = new(int64)
	if d.ReadMode != Stream {
	*d.MaxContentLength = DefaultMaxContent
	}
	}

The size limit for file reading is set by max_content_length is controlled by the configuration parameter. If it is not specified, the default value is 102400000bytes (about 100MB)

It is also described in the document:
https://github.com/MicroOps-cn/data_exporter/blob/master/README-en.md#file

datasource:
  - type: "file"
    name: <string> # datasource name 
    max_content_length: <int> # The maximum read length, in bytes, When the value of "read_mode" is stream, it defaults to 0 (unlimited),otherwise the default value is 102400000  
    line_max_content_length: <int> # The maximum read length per line,in bytes,0 means unlimited, defaults: 102400000
    
    relabel_configs: [ <relabel_config>, ... ] # reference: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
    timeout: <duration>  # The default is "30s", which cannot be less than "1ms", reference: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#duration
    read_mode: <string> # read mode, The value can be: "stream","line" or "full", defaults: "full"
    url: "../examples/weather.xml"
    line_separator: [<string>,...] # The value can be string、[string,...], Line separator, default: "\n"
    end_of: # Message end flag. When this flag is read, it will stop reading and close the connection. The message is line buffered, so The end_of value of cannot be multiple rows.

[girisubhra]

Are you sure it is 600 bytes? The XML file in the example "examples/xml_3gpp_ts /" is 2.86 KB. 600 bytes must be OK.
It is not about maximum limit of file.

I want if a file size is greater than 600 bytes or 1Kb or any size (for example take 600bytes), then we should read the file otherwise we should not read by printing a message"Insufficient data"

[MicroOps-cn]

Can I understand it this way: you want to control the minimum size of the file. If the length of the file content is below the threshold, it will not be parsed.

[girisubhra]

Can I understand it this way: you want to control the minimum size of the file. If the length of the file content is below the threshold, it will not be parsed.

Yes you have understood it right.

[MicroOps-cn]

On the issue-10 branch, the features you need have been implemented. You can try to switch the code to the issue-10 branch, and then try to compile and run it yourself.

It may merge into the main branch after a while.

[girisubhra]

in issue-10 branch code, I added min_content_length to 4000 to examples/xml_3gpp_ts)/xml_exporter.yaml, but it is reading the file which is less than 4000 bytes

collects:

• name: "meas_info"
  data_format: "xml"
  datasource:
  • type: "file"
    min_content_length: 4000
    url: "/etc/data_exporter/pmdata.xml"
    metrics:
  • name: "meas_info"
    match:
    datapoint: "//measData/measInfo/measValue/measResults"
    labels:

[MicroOps-cn]

My test results are all right:

$ head -10 xml_exporter.yaml 
collects:
  - name: "meas_info"
    data_format: "xml"
    datasource:
      - type: "file"
        url: "./pmdata.xml"
        min_content_length: 4000
    metrics:
      - name: "meas_info"
        match:
$ go run ../../ debug --config.path=./
logfmt
level=info ts=2022-09-12T07:17:22.694Z caller=main.go:90 msg="Loaded config file"
debug
================================================================================================= [Process] =================================================================================================
[collect]           meas_info
[msg]               Failed to get datasource.
[err]               data is too short
============================================================================================== [Output Metrics] ==============================================================================================
# HELP data_exporter_collect_error_count datasource or metric collect error count
# TYPE data_exporter_collect_error_count counter
data_exporter_collect_error_count{name="",type="datasource"} 1

[girisubhra]

I am also getting same result by running go run command, but i cannot see same result while running by docker

[MicroOps-cn]

Are you using the docker I provided or the docker you compiled yourself? The docker image I provided is compiled from the master branch, this feature is not currently supported. If you want to run with docker, you must compile it yourself.

Compiling an image is very simple. You can compile it by executing this command directly in the code root directory:
make common-build && docker build -t data_exporter .

[MicroOps-cn]

Or you can use this command to run:
docker run --rm -it -v ${PWD}:/etc/data_exporter/ microops/data_exporter:0.4.3-alpha debug --config.path=/etc/data_exporter/

[girisubhra]

I executed below command which you have giver before, but it is reading the file which is less than threshold value.
docker run -itd --privileged -p 9116:9116 -v /home/subhra/new_data_exporter/data_exporter/examples/xml_3gpp_ts:/etc/data_exporter/ --name data_exporter microops/data_exporter --config.path=/etc/data_exporter/xml_exporter.yaml

Output
[root@MYPC data_exporter]# docker run -itd --privileged -p 9116:9116 -v /home/subhra/new_data_exporter/data_exporter/examples/xml_3gpp_ts:/etc/data_exporter/ --name data_exporter microops/data_exporter --config.path=/etc/data_exporter/xml_exporter.yaml
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
7b8b6ba597a47db55d8733e188b4da34bf02c626a14f819b7ed803b965f8ef61

[MicroOps-cn]

Try changing the image to microops/data_exporter:0.4.3-alpha.

[girisubhra]

Try changing the image to microops/data_exporter:0.4.3-alpha.

I changed as you suggested, but still it is reading the file which is less than threshold value.