Adds initial scripts to use rsyslog for logging in workstation by kushaldas · Pull Request #8 · freedomofpress/securedrop-log

kushaldas · 2020-01-09T13:51:07Z

closes #10
sd-rsyslog is the output plugin of rsyslog to be installed in
/usr/sbin

sdlog.conf is the configuration of rsyslog in /etc/rsyslog.d/

securedrop-redis-log is part of the Qrexec service inside of
sd-log vm. This will receive the messages from any other vm,
and add them into a queue in Redis.

securedrop-log-saver will be the service inside sd-log VM,
this will monitor the queue, and save any incoming message to
the syslog.log file of the respective directory for each VM.

freedomofpress/securedrop-builder#132 needs to be merged first as
we need redis to build the package. You can still test this PR with the steps below.

How to test?

The following are the manual steps.

Make a source tarball from this branch python3 setup.py sdist
Build the package using this PR Adds changes for updated securedrop-log package securedrop-builder#133

Then install the fresh debian package in sd-log vm (it will require network connection to get a dependency), then also install redis-server and redis

sudo apt install redis-server redis

After installing the debian package of securedrop-log, execute the following commands.

sudo systemctl daemon-reload
sudo systemctl start redis
sudo systemctl start securedrop-log

Next, in sd-app vm

Put sd-rsyslog-example.conf file to /etc/sd-rsyslog.conf, make sure update
it so that is shows the right localvm name (sd-app).
Install the fresh debian package (will require network access for dependency installation)
Next, restart the rsyslog service.

systemctl restart rsyslog

logger hello, this is sd-app run this command in sd-app, this should up in the /home/user/QubesIncomingLogs/sd-app/syslog.log file
Now, if start the client application, it should create logs to /home/user/QubesIncomingLogs/sd-app/syslog.log file.

rmol · 2020-01-10T20:30:36Z

Why introduce Redis? Why not just use syslog there as well?

kushaldas · 2020-01-13T17:07:07Z

Why introduce Redis? Why not just use syslog there as well?

Because we want to smoothly split the logs in different files based on which vm they are coming from, and redis helps as it automagically makes them proper even if more than one thread is pushing them into the queue.

Tested this approach before while processing jobs across multiple processes over many machines.

rmol · 2020-01-13T17:15:36Z

rsyslog can route to different destinations based on hostname, application name, and other criteria, and it's pretty good at handling concurrent sources.

kushaldas · 2020-01-13T17:18:18Z

rsyslog can route to different destinations based on hostname, application name, and other criteria, and it's pretty good at handling concurrent sources.

Yup, but, I was unable to see how to use it here. Can you please help in the PR on that front? in the sd-log vm, how should we push the log lines so that rsyslog can put them to the right files?

rmol · 2020-01-13T17:20:55Z

Of course. I hope to have time later today, will see what's required or what I'm missing. 😉

rmol

I looked at just using rsyslog instead of redis within sd-log, and I think getting syslog-formatted messages across the qrexec link in the chain and then having rsyslog in sd-log route them is problematic. We could probably figure out rsyslog templates and parsing, maybe using the mmnormalize plugin, and get the messages routed properly, but frankly it looks more painful than what you've got now. I've spent a few hours experimenting and digging through the rsyslog docs, and I'm ready to be done.

This might be easier if we had rsyslog in the various VMs talking directly to rsyslog in sd-log, but setting up encrypted transport and configuring the firewall or qvm-connect-tcp is a lot of work on top of figuring out the rsyslog configuration, and ultimately less safe than qrexec.

Any other multiprocess-safe Python logging solution looks to be about the same amount of work, still involves a daemon process to collect and write the logs (I haven't found one that tries file locking), and will overall likely be less solid than using Redis.

I think this just needs some more documentation and some operational polishing -- getting Redis installed in sd-log-buster-template and added to the pip requirements here; figuring out how securedrop-log-saver should be run (systemd service as user?) -- and then we should ship it. It works, and if we want to fine-tune it later it should be much easier for any of us to work with securedrop-log-saver than rsyslog configuration.

sd-rsyslog

rmol · 2020-01-17T21:36:19Z

securedrop-log-saver

+                    f"{vmname}",
+                    "syslog.log",
+                )
+                fh = open(filepath, "a")


This bit needs the directory creation logic from securedrop-log.

securedrop-redis-log

emkll · 2020-01-21T17:40:06Z

securedrop-log-saver

+
+import os
+import sys
+import redis


will this require a pip install?

Yeah, which of course will need to be done in the template.

kushaldas · 2020-01-22T08:49:18Z

@rmol thanks for the feedback. Now, I can work to make it more production ready from that draft state.

kushaldas · 2020-01-22T08:59:25Z

Q: If we want to control the python redis package, then we can follow like any other our packages, and install via dh-virtualenv, this will also require to add the redis wheel. Or, we can use the system provided package and use the debian package dependency to pull it down.

@redshiftzero @rmol @emkll Just asking this as it was in mind about this particular code/application packaging.

For now I am working to make it a dh-virtualenv based package.

rmol · 2020-01-22T12:07:38Z

I prefer installing the redis Python module via dh-virtualenv, particularly if we're using a version that's already been reviewed. 😉 Better to have all the Python dependencies specified in one place and installed the same way, and to not build the virtualenv with system packages.

kushaldas · 2020-01-22T18:40:00Z

I will be submitting the PR tomorrow, I will have to add a wheel for redis for this.

emkll · 2020-01-24T21:07:59Z

sd-rsyslog

+        confirmMessages="on"
+        ...)
+
+Licensed under the Apache License, Version 2.0 (the "License");


should we license (a)GPLv3 here for consistency with both other repos and your setup.py: https://github.com/freedomofpress/securedrop-log/pull/8/files#diff-2eeaed663bd0d25b7e608891384b7298R27

We can not as this is copied from the official example.

kushaldas · 2020-01-27T09:08:41Z

Updated with the latest wheel hashes from debian packaging repo for the production beta release.

emkll

Thanks @kushaldas I have gone through the test plan and tried to adapt it to a test plan where, I have some questions, and some files appear to be missing from the packaging, and ran into issues with redis. Perhaps a PR to securedrop-workstation is required to populate the systemd service files? Would it be possible to update the test plan as well, it seems like your test plan is out-of-date, now that you've done packaging work.

In dom0, add the following file /etc/qubes-rpc/policy/securedrop.Log and then the text below.

@tag:sd-workstation sd-log allow
@anyvm @anyvm deny

Is this step necessary? We already add those in master: https://github.com/freedomofpress/securedrop-workstation/blob/master/dom0/sd-log.sls#L44-L49

in sd-log vm, Add the following content to /etc/qubes-rpc/securedrop.Log
This appears to be already handled by https://github.com/freedomofpress/securedrop-debian-packaging/pull/133/files#diff-c3454cdfb26226700f264ea620b88cc2R1

/usr/sbin/securedrop-log

Create a virtualenv at /opt/venvs/securedrop-log, you may have to apt install python3-venv for this.
and then place securedrop-redis-log and securedrop-log-saver scripts to the
virtualenv at /opt/venvs/securedrop-log and create links to /usr/sbin/
directory and make sure that they are executable. This step will be automated via
the Debian package.
Same as above, this seems to be handled by the packaging logic
❓ : Copy securedrop-log.service file to /usr/systemd/system and then
Is there a reason this isn't dropped into place by https://github.com/freedomofpress/securedrop-debian-packaging/pull/133/files ?
❗ the target folder is /etc/systemd/system

sudo systemctl daemon-reload
sudo systemctl start redis
sudo systemctl start securedrop-log

❗ when running sudo systemctl start redis, i get the following error:

Failed to start redis.service: Unit redis.service not found

I stopped the test plan at this point, please let me know if I missed something. An updated test plan (using the packaging logic) should make things easier and more reproducible for a reviewer, and also allow to review the packaging logic as well.

kushaldas · 2020-02-04T05:49:29Z

❗ the target folder is /etc/systemd/system
Correct, I had a typing mistake :(

Also, you are correct that I missed to put the systemd file on place, I was waiting for this review to finish (verifying that the idea is correct).

You will have to manually install redis-server on the log vm/template. I don't want to put that as a direct dependency of the Debian package, as it is only required on the sd-log vm.

I have updated the test steps.

kushaldas · 2020-02-04T06:33:22Z

Is this step necessary? We already add those in master: https://github.com/freedomofpress/securedrop-workstation/blob/master/dom0/sd-log.sls#L44-L49

No, but, I just wanted to have clear steps.

emkll

Thanks @kushaldas for your patience, i have gotten much further this time. A few comments:

I have opened a ticket to track the provisioning/configuration for all VMs in freedomofpress/securedrop-workstation#440, it might be good to tackle this prior to merge of this PR to ensure the configuration is reliable and reproducible. What do you think?
python3-distutils package was required to get the scripts to work on the sd-log vm see stacktrace in [1]. Do we need to include the system package or a wheel?
I would recommend stating in the Readme that this package is dual-purpose (provides scripts to both send and receive). I can see some advantages with this approach, but I am curious if you issues in the future? specifically, if we need to automate some configuration changes (in postinst or other), the logic to distinguish a system that is sending logs from one receiving logs is complex.
Having some issues sending logs over: invoking sd-rsyslog directly allows me to send amessage (and i see in qvm-rpc logs), but I don't see the message being logged/processed on sd-log . The redis server logs don't indicate a connection, but maybe I am doing something wrong. securedrop-log.service seems properly configured [2]. I think this highlights the issues described in 1), where it would be helpful to have a more reliable and reproducible configuration

[1]

Traceback (most recent call last):
  File "/opt/venvs/securedrop-log/sbin/securedrop-log-saver", line 5, in <module>
    import redis
  File "/opt/venvs/securedrop-log/lib/python3.7/site-packages/redis/__init__.py", line 1, in <module>
    from redis.client import Redis, StrictRedis
  File "/opt/venvs/securedrop-log/lib/python3.7/site-packages/redis/client.py", line 13, in <module>
    from redis.connection import (ConnectionPool, UnixDomainSocketConnection,
  File "/opt/venvs/securedrop-log/lib/python3.7/site-packages/redis/connection.py", line 2, in <module>
    from distutils.version import StrictVersion
  File "/opt/venvs/securedrop-log/lib/python3.7/distutils/__init__.py", line 25, in <module>
    from distutils import dist, sysconfig
ImportError: cannot import name 'dist' from 'distutils' (/opt/venvs/securedrop-log/lib/python3.7/distutils/__init__.py)

[2]

● securedrop-log.service - securedrop logging Service
   Loaded: loaded (/etc/systemd/system/securedrop-log.service; disabled; vendor preset: enabled)
   Active: active (running) since Wed 2020-02-05 14:27:48 EST; 48min ago
 Main PID: 3828 (securedrop-log-)
    Tasks: 1 (limit: 597)
   Memory: 8.1M
   CGroup: /system.slice/securedrop-log.service
           └─3828 /opt/venvs/securedrop-log/bin/python3 /opt/venvs/securedrop-log/sbin/securedrop-log-saver

`sd-rsyslog` is the output plugin of rsyslog to be installed in /usr/sbin `sdlog.conf` is the configuration of rsyslog in /etc/rsyslog.d/ `securedrop-redis-log` is part of the Qrexec service inside of sd-log vm. This will receive the messages from any other vm, and add them into a queue in Redis. `securedrop-log-saver` will be the service inside `sd-log` VM, this will monitor the queue, and save any incoming message to the `syslog.log` file of the respective directory for each VM. It also has the Makefile for the project.

kushaldas · 2020-02-06T09:03:01Z

@emkll Updated the test steps using Debian package only.

emkll

Thanks for your patience, @kushaldas, this now works as expected 🎉

I am approving this pr but we should not merge it right away. Let's ensure CI passes since nightlies will be built once we merge. I propose we do as follows:

push a 0.0.4 tag
restart CI and observe ci passing in freedomofpress/securedrop-builder#133
merge freedomofpress/securedrop-builder#133
Open follow up issue to track configuration freedomofpress/securedrop-workstation#440
merge this PR ❗ @rmol has requested changes, however the 0.0.4 tag and nightlies should unblock work in freedomofpress/securedrop-workstation#440 and we can wait a little bit longer to merge this one (we can discuss in standup)

emkll · 2020-02-06T16:19:47Z

I've appended 9fbcae4 to add a build job to CI for this repository

emkll · 2020-02-07T17:32:55Z

I have tracked the split packaging in freedomofpress/securedrop-client#1721
Because I have merged the packaging changes yesterday in freedomofpress/securedrop-builder#133, nightlies are currently broken because we didn't merge this PR, and those PRs were interdependent (initially reported by @redshiftzero , see https://circleci.com/gh/freedomofpress/securedrop-debian-packaging/2749)

It seems like most issues @rmol requested changes on were address, with the exception of a follow up tracked in freedomofpress/securedrop-client#1721., in order to unblock nightlies. As i am not an admin in this repo, and because I have added a commit for CI, I cannot dismiss @rmol 's review and should not merge this pr.

Could someone else @redshiftzero perhaps? review my commit and dismiss the review?

redshiftzero

confirming that I've reviewed the last commit added by @emkll, lgtm, relying on @emkll's review for the previous commits

@rmol

dismissing as @rmol is not available and we want to merge this to resolve nightlies and address further comments in follow ups which Mickael has filed

rmol previously requested changes Jan 17, 2020

View reviewed changes

emkll reviewed Jan 21, 2020

View reviewed changes

kushaldas force-pushed the oh_my_syslog branch 3 times, most recently from afe7803 to 6b8e8ca Compare January 22, 2020 12:00

eloquence mentioned this pull request Jan 22, 2020

decide on approach for integrating into each project #7

Closed

kushaldas force-pushed the oh_my_syslog branch 2 times, most recently from ad05c89 to 7c5ab29 Compare January 23, 2020 12:10

kushaldas marked this pull request as ready for review January 23, 2020 12:13

kushaldas mentioned this pull request Jan 23, 2020

Adds changes for updated securedrop-log package freedomofpress/securedrop-builder#133

Merged

emkll reviewed Jan 24, 2020

View reviewed changes

kushaldas force-pushed the oh_my_syslog branch from 7c5ab29 to 2ee7c61 Compare January 27, 2020 09:07

kushaldas force-pushed the oh_my_syslog branch from 2ee7c61 to 10aae1e Compare January 27, 2020 16:47

emkll suggested changes Jan 30, 2020

View reviewed changes

emkll reviewed Feb 5, 2020

View reviewed changes

kushaldas force-pushed the oh_my_syslog branch from 10aae1e to 29c1cf7 Compare February 6, 2020 06:43

emkll approved these changes Feb 6, 2020

View reviewed changes

ci: Add build-buster job

9fbcae4

emkll requested a review from rmol February 7, 2020 17:28

redshiftzero approved these changes Feb 7, 2020

View reviewed changes

redshiftzero merged commit 58e7523 into master Feb 7, 2020

redshiftzero deleted the oh_my_syslog branch February 7, 2020 17:41

redshiftzero mentioned this pull request Feb 7, 2020

nightly builds partially broken freedomofpress/securedrop-builder#139

Closed

Conversation

kushaldas commented Jan 9, 2020 • edited by emkll Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to test?

Uh oh!

rmol commented Jan 10, 2020

Uh oh!

kushaldas commented Jan 13, 2020

Uh oh!

rmol commented Jan 13, 2020

Uh oh!

kushaldas commented Jan 13, 2020

Uh oh!

rmol commented Jan 13, 2020

Uh oh!

rmol left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rmol Jan 17, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

emkll Jan 21, 2020

Choose a reason for hiding this comment

Uh oh!

rmol Jan 21, 2020

Choose a reason for hiding this comment

Uh oh!

kushaldas commented Jan 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kushaldas commented Jan 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rmol commented Jan 22, 2020

Uh oh!

kushaldas commented Jan 22, 2020

Uh oh!

emkll Jan 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kushaldas Jan 27, 2020

Choose a reason for hiding this comment

Uh oh!

kushaldas commented Jan 27, 2020

Uh oh!

emkll left a comment

Choose a reason for hiding this comment

Uh oh!

kushaldas commented Feb 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kushaldas commented Feb 4, 2020

Uh oh!

emkll left a comment

Choose a reason for hiding this comment

Uh oh!

kushaldas commented Feb 6, 2020

Uh oh!

emkll left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emkll commented Feb 6, 2020

Uh oh!

emkll commented Feb 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

redshiftzero left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

kushaldas commented Jan 9, 2020 •

edited by emkll

Loading

kushaldas commented Jan 22, 2020 •

edited

Loading

kushaldas commented Jan 22, 2020 •

edited

Loading

emkll Jan 24, 2020 •

edited

Loading

kushaldas commented Feb 4, 2020 •

edited

Loading

emkll left a comment •

edited

Loading

emkll commented Feb 7, 2020 •

edited

Loading