fix dataset reading and add support for full dataset by wojtuss · Pull Request #16559 · PaddlePaddle/Paddle

wojtuss · 2019-03-29T12:08:19Z

In this patch, we fix a bug in reading dataset and add support for reading a whole imagenet dataset preprocessed using the tool from #16529.

Most changes come from the diff between #16532 and merged #16399.

Additional methods are added to tester_helper.h. Could be refactored to reuse existing methods but would require more changes in other tests - left to further refactoring.

test=develop

… wojtuss/fix-int8-test

test=develop

luotao1 · 2019-03-29T12:15:15Z

paddle/fluid/inference/tests/api/analyzer_int8_image_classification_tester.cc

+       "depthwise_conv_mkldnn_pass", "conv_bn_fuse_pass",
+       "conv_eltwiseadd_bn_fuse_pass", "conv_bias_mkldnn_fuse_pass",
+       "conv_elementwise_add_mkldnn_fuse_pass", "conv_relu_mkldnn_fuse_pass",
+       "fc_fuse_pass", "is_test_pass"});


Fix SetPass will in another PR?

@luotao1 , it turned out that no special fix is required for this and the call to SetPasses() can be simply removed now. Some other modifications must have fixed the accuracy problem. Of course, repeated passes problem would still benefit from a cleanup, but it is not critical here. We will prepare a PR with a cleanup later.

We will prepare a PR with a cleanup later.

Got it. Does new PR fix #16559 (comment) as well?

I will adjust the fps calculation and remove redundant PredictionRun method in this PR.
WIP

luotao1 · 2019-03-29T12:16:21Z

paddle/fluid/inference/tests/api/CMakeLists.txt

@@ -148,7 +151,7 @@ inference_analysis_api_test_with_fake_data(test_analyzer_mobilenet_depthwise_con
 if(WITH_MKLDNN)
  set(INT8_DATA_DIR "${INFERENCE_DEMO_INSTALL_DIR}/int8")


please change the INT8_DATA_DIR if you change the data. Otherwise, since INT8_DATA_DIR is already existed ON CI, it will be error.

With this patch, we keep the data.bin file name but change the archive name into imagenet_val_100_tail.tar.gz, so there should be no conflict of archive names, or might there be a conflict of data.bin between several builds?

if INT8_DATA_DIR is already exised ON CI, it will not download imagenet_val_100_tail.tar.gz

do you mean that several CI builds share the same directory with the dataset?

Our CI cache the INFERENCE_DEMO_INSTALL_DIR for speedup.

set(INT8_DATA_DIR "${INFERENCE_DEMO_INSTALL_DIR}/int8v2") or other name

luotao1 · 2019-03-29T12:17:11Z

paddle/fluid/inference/tests/api/tester_helper.h

+DEFINE_int32(iterations, 0, "number of batches to process");  // setting to 0,
+                                                              // means process
+                                                              // the whole
+                                                              // dataset


comment from 46-49 should be in one line.

luotao1 · 2019-03-29T12:20:09Z

paddle/fluid/inference/tests/api/tester_helper.h

  }
 }

+// With support for multiple batches (multiple outputs)


I wonder why do you rewrite this again, could you enhance line331predictor->Run(inputs[i], outputs, batch_size); to predictor->Run(inputs[i], &(*outputs)[i], FLAGS_batch_size);

The original PredictionRun method discards outputs of all but the last iteration and outputs is of type std::vector<PaddleTensor> *. To calculate average accuracy over all iterations we have to keep all the output data and have outputs of type std::vector<std::vector<PaddleTensor>> *. I could modify just the original function, but it would require updates also in several other test files. I thought it would be refactored later, but I will do it here if you wish.

It's better to refactor here, thanks very much!

@luotao1 , I realized that refactoring would require modification of the latency calculation formula. This could influence some latency statistics of other tests. I am not sure whether these statistics are being gathered after running the tests and whether changing them could break anything beyond the tests. I cannot verify that quickly enough so I left the refactoring for later.

I wonder why refactoring requires modification of the latency calculation formula?

Yours:

auto latency = elapsed_time / (iterations * num_times * FLAGS_batch_size); PrintTime(FLAGS_batch_size, num_times, FLAGS_paddle_num_threads, 0, latency, 1);

Ours:

PrintTime(batch_size, num_times, num_threads, tid, elapsed_time / num_times, inputs.size());

Why you don't use ours?

With the current implementation fps value would be incorrect. It is 1/latency, so latency is related to a single frame, not a batch. The current formula does not handle that correctly.

If current implementation fps value would be incorrect, you can correct it directly.

test=develop

luotao1 · 2019-04-01T12:40:53Z

paddle/fluid/inference/api/helper.h

-              << "ms ======";
-  }
+  LOG(INFO) << "====== batch_size: " << batch_size << ", iterations: " << epoch
+            << ", repetitons: " << repeat << " ======";


repetitions

luotao1 · 2019-04-01T13:06:50Z

paddle/fluid/inference/tests/api/tester_helper.h

+  LOG(INFO) << "FP32 & INT8 prediction run: batch_size " << FLAGS_batch_size
+            << ", warmup batch size " << FLAGS_warmup_batch_size << ".";
+  PrintConfig(reinterpret_cast<const PaddlePredictor::Config *>(qconfig), true);
+  PrintConfig(reinterpret_cast<const PaddlePredictor::Config *>(config), true);


Since line502 is the same as line503, how about move line503 after line507?

luotao1 · 2019-04-01T13:11:36Z

I test this PR on 6271 CPU with 50000 images generated by danqing.
command

./test_analyzer_int8_resnet50 --infer_model=/dev/shm/.cache/paddle/dataset/int8/download/mobilenetv1_fp32/model/ --infer_data=/dev/shm/.cache/paddle/dataset/int8/download/int8_full_val.bin --test_all_data --gtest_filter=Analyzer_int8_resnet50.quantization 2> mobilenet_v2.log

log:
mobilenet_v2.log


I0401 20:46:35.058284 164022 analysis_predictor.cc:429] == optimize end ==
I0401 20:46:35.059271 164022 tester_helper.h:323] Thread 0, number of threads 1, run 1 times...
I0401 20:57:37.384917 164022 helper.h:273] ====== threads: 1, thread id: 0 ======
I0401 20:57:37.384954 164022 helper.h:275] ====== batch_size: 1, iterations: 50000, repetitons: 1 ======
I0401 20:57:37.384959 164022 helper.h:277] ====== batch latency: 13.2465ms, number of samples: 50000, sample latency: 13.2465ms, fps: 75.4916 ======
I0401 20:57:37.388720 164022 tester_helper.h:508] --- INT8 prediction start ---
I0401 20:57:45.525290 164022 mkldnn_quantizer.cc:393] == optimize 2 end ==
I0401 20:57:45.526273 164022 tester_helper.h:323] Thread 0, number of threads 1, run 1 times...
I0401 21:01:20.693503 164022 helper.h:273] ====== threads: 1, thread id: 0 ======
I0401 21:01:20.693539 164022 helper.h:275] ====== batch_size: 1, iterations: 50000, repetitons: 1 ======
I0401 21:01:20.693543 164022 helper.h:277] ====== batch latency: 4.30334ms, number of samples: 50000, sample latency: 4.30334ms, fps: 232.377 ======
I0401 21:01:20.696801 164022 tester_helper.h:510] --- comparing outputs --- 
I0401 21:01:20.705694 164022 tester_helper.h:456] Avg top1 INT8 accuracy: 0.7036
I0401 21:01:20.705705 164022 tester_helper.h:458] Avg top1 FP32 accuracy: 0.0010
I0401 21:01:20.705708 164022 tester_helper.h:460] Accepted accuracy drop threshold: 0.01
F0401 21:01:20.705729 164022 tester_helper.h:461] Check failed: std::abs(avg_acc1_quant - avg_acc1_ref) <= FLAGS_quantized_accuracy (0.70258 vs. 0.01)

It seems the thoughput of FP32 is 75 and INT8 is 232? And the accuracy larger than 0.7?

luotao1 · 2019-04-01T13:15:14Z

I test this PR in May 29th, the accuracy is correct:

You use SetPasses() in the PR.

luotao1 · 2019-04-01T13:18:19Z

Besides, could you fix the compiler warning

/Paddle/paddle/fluid/inference/tests/api/tester_helper.h: In function ‘void paddle::inference::PredictionRun(paddle::PaddlePredictor*, const std::vector<std::vector<paddle::PaddleTensor> >&, std::vector<std::vector<paddle::PaddleTensor> >*, int, int)’:
/Paddle/paddle/fluid/inference/tests/api/tester_helper.h:319:62: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   if (FLAGS_iterations > 0 && FLAGS_iterations < inputs.size())
                                                              ^
/Paddle/paddle/fluid/inference/tests/api/tester_helper.h:332:28: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     for (size_t i = 0; i < iterations; i++) {
                            ^
/Paddle/paddle/fluid/inference/tests/api/tester_helper.h:339:28: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     for (size_t i = 0; i < iterations; i++) {
                            ^
In file included from /Paddle/paddle/fluid/inference/tests/api/analyzer_int8_image_classification_tester.cc:18:0:
/Paddle/paddle/fluid/inference/tests/api/tester_helper.h: In function ‘void paddle::inference::PredictionRun(paddle::PaddlePredictor*, const std::vector<std::vector<paddle::PaddleTensor> >&, std::vector<std::vector<paddle::PaddleTensor> >*, int, int)’:
/Paddle/paddle/fluid/inference/tests/api/tester_helper.h:319:62: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   if (FLAGS_iterations > 0 && FLAGS_iterations < inputs.size())
                                                              ^
/Paddle/paddle/fluid/inference/tests/api/tester_helper.h:332:28: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     for (size_t i = 0; i < iterations; i++) {
                            ^
/Paddle/paddle/fluid/inference/tests/api/tester_helper.h:339:28: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     for (size_t i = 0; i < iterations; i++) {

wojtuss · 2019-04-01T13:21:24Z

I test this PR on 6271 CPU with 50000 images generated by danqing.
command

./test_analyzer_int8_resnet50 --infer_model=/dev/shm/.cache/paddle/dataset/int8/download/mobilenetv1_fp32/model/ --infer_data=/dev/shm/.cache/paddle/dataset/int8/download/int8_full_val.bin --test_all_data --gtest_filter=Analyzer_int8_resnet50.quantization 2> mobilenet_v2.log

log:
mobilenet_v2.log


I0401 20:46:35.058284 164022 analysis_predictor.cc:429] == optimize end ==
I0401 20:46:35.059271 164022 tester_helper.h:323] Thread 0, number of threads 1, run 1 times...
I0401 20:57:37.384917 164022 helper.h:273] ====== threads: 1, thread id: 0 ======
I0401 20:57:37.384954 164022 helper.h:275] ====== batch_size: 1, iterations: 50000, repetitons: 1 ======
I0401 20:57:37.384959 164022 helper.h:277] ====== batch latency: 13.2465ms, number of samples: 50000, sample latency: 13.2465ms, fps: 75.4916 ======
I0401 20:57:37.388720 164022 tester_helper.h:508] --- INT8 prediction start ---
I0401 20:57:45.525290 164022 mkldnn_quantizer.cc:393] == optimize 2 end ==
I0401 20:57:45.526273 164022 tester_helper.h:323] Thread 0, number of threads 1, run 1 times...
I0401 21:01:20.693503 164022 helper.h:273] ====== threads: 1, thread id: 0 ======
I0401 21:01:20.693539 164022 helper.h:275] ====== batch_size: 1, iterations: 50000, repetitons: 1 ======
I0401 21:01:20.693543 164022 helper.h:277] ====== batch latency: 4.30334ms, number of samples: 50000, sample latency: 4.30334ms, fps: 232.377 ======
I0401 21:01:20.696801 164022 tester_helper.h:510] --- comparing outputs --- 
I0401 21:01:20.705694 164022 tester_helper.h:456] Avg top1 INT8 accuracy: 0.7036
I0401 21:01:20.705705 164022 tester_helper.h:458] Avg top1 FP32 accuracy: 0.0010
I0401 21:01:20.705708 164022 tester_helper.h:460] Accepted accuracy drop threshold: 0.01
F0401 21:01:20.705729 164022 tester_helper.h:461] Check failed: std::abs(avg_acc1_quant - avg_acc1_ref) <= FLAGS_quantized_accuracy (0.70258 vs. 0.01)

It seems the thoughput of FP32 is 75 and INT8 is 232? And the accuracy larger than 0.7?

As for accuracy, it is for 100 images only, so it may be larger than 0.7.

By default there is only one iteration of batch size 100, without warmup phase. The latency is not precise then. We assumed the precise latency and accuracy will be measured on the whole dataset, then the warmup is negligible.

As for the accuracy for FP32, that was the problem with the passes I had on Friday. Today I have built it up from scratch and it worked fine.
I have the fix for repeated passes ready and could add the commit with the fix to this PR, or submit as a separate PR. What do you think?

luotao1 · 2019-04-01T13:28:15Z

Today I have built it up from scratch and it worked fine.

What is scratch?

luotao1 · 2019-04-01T13:30:30Z

I have the fix for repeated passes ready and could add the commit with the fix to this PR, or submit as a separate PR. What do you think?

A separate PR is better.

Besides, the thoughput of FP32 is 75 and INT8 is 232, which is different with V1 in python/paddle/fluid/contrib/int8_inference/README.md

wojtuss · 2019-04-01T13:30:57Z

"From scratch" means starting from clean directory.

Oh, I have just noticed you have run this on the full dataset.

Yes, with this test we get the following accuracy on the whole dataset:
ResNet50:
Avg top1 INT8 accuracy: 0.7640
Avg top1 FP32 accuracy: 0.7663
MobileNet-v1:
Avg top1 INT8 accuracy: 0.7039
Avg top1 FP32 accuracy: 0.7078

For throughput, you have to set larger batch_size.

luotao1 · 2019-04-01T13:35:55Z

"From scratch" means starting from clean directory.

I clean the directory, and must add SetPasses to get the right accuracy.

wojtuss · 2019-04-01T13:37:44Z

Then I am sending a new PR with the fix for passes. With it, the SetPasses problem should be fixed.

luotao1 · 2019-04-01T13:50:10Z

paddle/fluid/inference/tests/api/tester_helper.h

+  std::vector<std::vector<PaddleTensor>> analysis_outputs;
+  std::vector<std::vector<PaddleTensor>> quantized_outputs;
+  LOG(INFO) << "--- FP32 prediction start ---";
+  TestAnalysisPrediction(config, inputs, &analysis_outputs, true);


Why add TestAnalysisPrediction? Could you use TestOneThreadPrediction directly? Then you can remove TestAnalysisPrediction and line502-503

To me, it seems cleaner this way for both AnalysisConfigs, less pointer casting. I can change it if you like.

please change it. Thanks very much!

wojtuss

I have the fix for repeated passes ready and could add the commit with the fix to this PR, or submit as a separate PR. What do you think?

A separate PR is better.

Submitted #16606
Could you please verify that this fixes the issue in the test on your machine?

wojtuss · 2019-04-01T14:00:22Z

paddle/fluid/inference/tests/api/tester_helper.h

+  std::vector<std::vector<PaddleTensor>> analysis_outputs;
+  std::vector<std::vector<PaddleTensor>> quantized_outputs;
+  LOG(INFO) << "--- FP32 prediction start ---";
+  TestAnalysisPrediction(config, inputs, &analysis_outputs, true);


To me, it seems cleaner this way for both AnalysisConfigs, less pointer casting. I can change it if you like.

test=develop

wojtuss · 2019-04-01T19:18:08Z

@luotao1 , when comparing latency, keep in mind that here the latency does not include loading data as the inference starts when the whole dataset is loaded into memory

wojtuss · 2019-04-01T23:17:21Z

@luotao1 , I have added a comment regarding accuracy drop for FP32 inference: #16606 (comment)

… wojtuss/fix-int8-test

test=develop

luotao1 · 2019-04-02T02:49:57Z

The resnet50 result:

I0402 00:12:28.451478 150483 helper.h:277] ====== batch latency: 75.9028ms, number of samples: 50000, sample latency: 75.9028ms, fps: 13.1747 ======
I0402 00:29:30.439829 150483 helper.h:277] ====== batch latency: 20.0641ms, number of samples: 50000, sample latency: 20.0641ms, fps: 49.8403 ======
I0402 00:29:30.444981 150483 tester_helper.h:510] --- comparing outputs ---
I0402 00:29:30.462234 150483 tester_helper.h:456] Avg top1 INT8 accuracy: 0.7648
I0402 00:29:30.462246 150483 tester_helper.h:458] Avg top1 FP32 accuracy: 0.7663
I0402 00:29:30.462249 150483 tester_helper.h:460] Accepted accuracy drop threshold: 0.01

luotao1 · 2019-04-02T04:12:24Z

I have added a comment regarding accuracy drop for FP32 inference

I will investigate it. Please solve the conflict and use SetPasses in this PR for merge at first.

wojtuss · 2019-04-02T06:54:22Z

I have added a comment regarding accuracy drop for FP32 inference

I will investigate it. Please solve the conflict and use SetPasses in this PR for merge at first.

Done.

luotao1 · 2019-04-02T07:59:42Z

PR_CI fails. Maybe an error caused by #16584 . You can change like

--- a/paddle/fluid/inference/tests/api/CMakeLists.txt
+++ b/paddle/fluid/inference/tests/api/CMakeLists.txt
@@ -152,20 +152,20 @@ inference_analysis_api_test_with_fake_data(test_analyzer_mobilenet_depthwise_con
 if(WITH_MKLDNN)
   set(INT8_DATA_DIR "${INFERENCE_DEMO_INSTALL_DIR}/int8v2")
   if (NOT EXISTS ${INT8_DATA_DIR})
-    inference_download_and_uncompress(${INT8_DATA_DIR} ${INFERENCE_URL}"/int8" "imagenet_val_100_tail.tar.gz")
+    inference_download_and_uncompress(${INT8_DATA_DIR} "${INFERENCE_URL}/int8" "imagenet_val_100_tail.tar.gz")
   endif()

   #resnet50 int8
   set(INT8_RESNET50_MODEL_DIR "${INT8_DATA_DIR}/resnet50")
   if (NOT EXISTS ${INT8_RESNET50_MODEL_DIR})
-    inference_download_and_uncompress(${INT8_RESNET50_MODEL_DIR} ${INFERENCE_URL}"/int8" "resnet50_int8_model.tar.gz" )
+    inference_download_and_uncompress(${INT8_RESNET50_MODEL_DIR} "${INFERENCE_URL}/int8" "resnet50_int8_model.tar.gz" )
   endif()
   inference_analysis_api_int8_test(test_analyzer_int8_resnet50 ${INT8_RESNET50_MODEL_DIR} ${INT8_DATA_DIR} analyzer_int8_image_classification_tester.cc SERIAL)

   #mobilenet int8
   set(INT8_MOBILENET_MODEL_DIR "${INT8_DATA_DIR}/mobilenet")
   if (NOT EXISTS ${INT8_MOBILENET_MODEL_DIR})
-    inference_download_and_uncompress(${INT8_MOBILENET_MODEL_DIR} ${INFERENCE_URL}"/int8" "mobilenetv1_int8_model.tar.gz" )
+    inference_download_and_uncompress(${INT8_MOBILENET_MODEL_DIR} "${INFERENCE_URL}/int8" "mobilenetv1_int8_model.tar.gz" )
   endif()

wojtuss · 2019-04-02T08:35:22Z

Yes, that helped. Thank you.
Done.

luotao1

LGTM

Cherry-pick of #16559, #16606 and #16608

Sylwester Fraczek and others added 5 commits March 29, 2019 06:07

add reading full preprocessed imagenet val dataset

85ff63c

test=develop

refine comment

ce6e94e

fix info printing

87cb50e

test=develop

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

183bd2f

… wojtuss/fix-int8-test

renamed directory for dataset and models

e650d39

test=develop

wojtuss added Intel int8 labels Mar 29, 2019

wojtuss requested a review from luotao1 March 29, 2019 12:08

luotao1 reviewed Mar 29, 2019

View reviewed changes

Wojciech Uss added 2 commits April 1, 2019 02:05

remove call to SetPasses()

fef0238

test=develop

fix test output handling and fps calculation

8c36e58

test=develop

luotao1 reviewed Apr 1, 2019

View reviewed changes

wojtuss commented Apr 1, 2019

View reviewed changes

Wojciech Uss added 3 commits April 1, 2019 18:03

fixes after review

ea9e948

test=develop

reuse TestOneThreadPrediction function

9b8140b

test=develop

detect error in runtime_context_cache_pass

46aaa36

test=develop

This was referenced Apr 1, 2019

runtime_context_cache_pass breaks accuracy of FP32 C-API inference of ResNet50 and MobileNet-v1 #16609

Closed

fix repeating passes #16606

Merged

wojtuss force-pushed the wojtuss/fix-int8-test branch from 69cdebb to 46aaa36 Compare April 1, 2019 23:15

Wojciech Uss added 3 commits April 2, 2019 01:53

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

842fb78

… wojtuss/fix-int8-test

use SetPasses

3549930

test=develop

fix download url

e569296

test=develop

luotao1 approved these changes Apr 2, 2019

View reviewed changes

luotao1 merged commit 9b6a029 into PaddlePaddle:develop Apr 2, 2019

luotao1 added a commit that referenced this pull request Apr 3, 2019

Merge pull request #16643 from sfraczek/fix-repeating-passes

d3b6291

Cherry-pick of #16559, #16606 and #16608

luotao1 mentioned this pull request Apr 3, 2019

test_analyzer_int8 tests use default pass order #16678

Merged

		@@ -148,7 +151,7 @@ inference_analysis_api_test_with_fake_data(test_analyzer_mobilenet_depthwise_con
		if(WITH_MKLDNN)
		set(INT8_DATA_DIR "${INFERENCE_DEMO_INSTALL_DIR}/int8")

Conversation

wojtuss commented Mar 29, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wojtuss Apr 1, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wojtuss Apr 1, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wojtuss Mar 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

luotao1 commented Apr 1, 2019

Uh oh!

luotao1 commented Apr 1, 2019

Uh oh!

luotao1 commented Apr 1, 2019

Uh oh!

wojtuss commented Apr 1, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

luotao1 commented Apr 1, 2019

Uh oh!

luotao1 commented Apr 1, 2019

Uh oh!

wojtuss commented Apr 1, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

luotao1 commented Apr 1, 2019

wojtuss Apr 1, 2019 •

edited

Loading

wojtuss Apr 1, 2019 •

edited

Loading

wojtuss Mar 29, 2019 •

edited

Loading

wojtuss commented Apr 1, 2019 •

edited

Loading

wojtuss commented Apr 1, 2019 •

edited

Loading

wojtuss left a comment •

edited

Loading

wojtuss commented Apr 1, 2019 •

edited

Loading

wojtuss commented Apr 2, 2019 •

edited

Loading