Dev/Qubes Remote Support

From Whonix
Jump to navigation Jump to search

QubesOS to be remotely manageable from on-demand, ephemeral, hidden onion service to dom0/AdminVM.

challenges

  • There are no Open Source remote support tools which are usable by laymen.
    • There is teamviewer but it's proprietary and therefore the teamviewer company could take over any users who use it.
  • The existing Open Source tools do not account for that most users nowadays are behind common NAT routers which make it hard to receive unsolicited incoming connections, i.e. hard to run servers. Would require port forwardings which are difficult for users. The internet is full of users seeking help how and failing to create port forwardings. Users are also often on networks (3/4G) that do not even support incoming (server) connections.
  • Even if some remote support instructions use Open Source software and work on Linux, these won't work easily in Qubes dom0 since non-networked by default.
  • Most if not all existing remote support Open Source software does not use encryption by default.
    • Therefore any man-in-the-middle could take over the connection and compromise the remote support receiver.
      • It's possible to run VNC over encrypted/authenticated SSH but this is not something which laymen users that need remote support are capable to use.

goals

  • functional for users with reasonable slow internet connections
  • reasonable security
  • Open Source
  • work behind NAT

desired security properties

  • encrypted
  • authenticated
  • clear and reliable signaling to the user whether remote admin is enabled and whether it is currently in use
  • Even after sys-whonix compromise the attacker should not be able to access dom0 through VNC.

implementation

  • Command line tools (which can be called by any GUI).
  • Whonix-Workstation script. (GUI and command line tools would run here.)
  • Whonix-Gateway script. (Tor onion service)
  • dom0 package (Qubes dom0 configuration, SSH/VNC server)
  • Packaging of a (python) GUI [1] (the GUI itself to be written by Marek)

sys-whonix GUI

able to see already existing hidden services per sys-whonix in a widget.

Adding remote support thought requirements:

  • List configured onion services.
  • Delete them.
  • Create new remote support related hidden onion service. When done, output onion address, port and shared secret for the user to copy paste to chosen secured communication channel.
  • create time based or persistent remote support
  • multiple remote support onions at the same time remove?
  • optionally a feature to send access info to the support agent from the GUI

Missing parts

Avoidable?

  • Qubes dom0 salt
  • Qubes dom0 GUI

Idea

general

  • x2go supports SSH.
    • (Needless to say that SSH or any extra encryption complicates things.)
    • TODO: test if x2go can work in acceptable speed over a Tor onion. FreeNX (maybe outdated), SPICE, x2go... x2go [in packages.debian.org: yes] seems most promising.
  • Run an SSH server in dom0.
  • magic-wormhole?
    • [in packages.debian.org: yes]
    • (File_Sharing#Magic-Wormhole)
    • These code words are quite usable. Example: 7-crossover-clockwork We'd need two of these.

questions:

  • SSH server running in dom0 is OK?
  • I guess running VNC (and perhaps SSH) in dom0 is OK?
  • Things can always be more compartmentalized, secure but also getting more and more complicated, fragile.

I am considering to script this entirely from dom0 using `qvm-run`. Only expectation: installed (use salt?) and non-broken sys-whonix

Draft Concept

remote support provider

On remote support provider machine.

  • dom0 qvm-start sys-whonix
  • dom0 qvm-run sys-whonix create .onion service config file in /usr/local/etc/torrc.d/40_remote_support.conf
  • dom0 qvm-run sys-whonix restart Tor [1]
  • dom0 qvm-start remote-support-provider-server-vm
  • dom0 qvm-run remote-support-provider-server-vm install openssh-server
  • make openssh-server reachable through .onion running in sys-whonix
  • dom0 wormhole send connection.zip
    • write wormhole code into script variable
  • connection.zip contains
    • .onion hostname of remote support provider
    • ~/.ssh/known_hosts
    • .auth_private file (private key) for authenticated Tor onion service
    • ssh key
  • tell remote support receiver out of band (telephone, chat, e-mail) the wormhole code
  • wait for remote support receiver to start the remote support receiver tool and to enter the wormhole code
  • in remote-support-provider-server-vm: x2go connect to remote support receiver localhost:port [reverse ssh]

remote support receiver

On remote support receiver machine.

  • dom0 sudo dnf install x2go-server
  • dom0 qvm-start sys-whonix
  • dom0 wormhole receive connection.zip
    • Must be done in dom0. Not sys-whonix. Because goal is sys-whonix to be untrusted.
  • dom0 install ~/.ssh/known_hosts
  • dom0 qvm-run sys-whonix install .auth_private file (private key)
    • sudo sourcefile=~/user.auth_private anon-server-to-client-install (adjust path)
  • dom0 qvm-run sys-whonix test connection to .onion
  • dom0 test connection to .onion
  • dom0 reverse ssh connect to remote support provider .onion
  • There is now an open port on remote support receiver machine over ssh and over .onion available to remote support provider.

qubes-remote-support-provider-gui

Not needed. Remote support provider can use CLI tools for connection setup. Would be encapsulated into a simple CLI call and telling the remote support receiver the wormhole code. Once the remote support receiver connected to the remote support provider (reverse SSH), the remote support provider can use a $VNC viewer (which is graphical) to connect.)

notes

  • Can use two wormhole codes if not avoidable.
  • Automatically re-connect if reverse SSH is down.

ssh keys

challenges:

  • Have to manage file /root/.ssh/authorized_keys using scripts.
    • Use secure, long, script generated passwords instead?

send access into to gpg agent

> optionally a feature to send access info to the support agent from the GUI (for example GPG encrypted email)

Patrick said: I would discourage that. Challanges:

  • calling e-mail / gpg from command line
  • calling e-mail / gpg from dom0 or adminVM?

Preliminary Work Done

See also

x2go

pros

  • Easy to use, built-in support for using over SSH.
  • Connection settings for slow/high latency connections (i.e. hopefully useful over Tor).

x2go bugs

Happening with hardened SSH config file.

https://bugs.x2go.org/cgi-bin/bugreport.cgi?bug=1349archive.org iconarchive.today icon 

NXPROXY - Version 3.5.99.19

Copyright (c) 2001, 2011 NoMachine (http://www.nomachine.com)
Copyright (c) 2008-2014 Oleksandr Shneyder <o.shneyder@phoca-gmbh.de>
Copyright (c) 2014-2016 Ulrich Sibiller <uli42@gmx.de>
Copyright (c) 2014-2016 Mihai Moldovan <ionic@ionic.de>
Copyright (c) 2011-2016 Mike Gabriel <mike.gabriel@das-netzwerkteam.de>
Copyright (c) 2015-2016 Qindel Group (http://www.qindel.com)

NXCOMP, NX protocol compression and NX extensions to this software
are copyright of the aforementioned persons and companies.

Redistribution and use of the present software is allowed according
to terms specified in the file LICENSE.nxcomp which comes in the
source distribution.

All rights reserved.

NOTE: This software has received contributions from various other
contributors, only the core maintainers and supporters are listed as
copyright holders. Please contact us, if you feel you should be listed
as copyright holder, as well.

NX protocol compression is derived from DXPC project.

Copyright (c) 1995,1996 Brian Pane
Copyright (c) 1996,1997 Zachary Vonler and Brian Pane
Copyright (c) 1999 Kevin Vigor and Brian Pane
Copyright (c) 2000,2003 Gian Filippo Pinzari and Brian Pane

All rights reserved.

See https://github.com/ArcticaProject/nx-libs for more information.

Info: Proxy running in server mode with pid '5072'.
Session: Starting session at 'Mon Jul  6 18:19:31 2020'.
Info: Using errors file '/home/user/.x2go/S-user-50-1594059610_stDXFCE_dp24/sessions'.
Info: Using stats file '/home/user/.x2go/S-50/stats'.
Loop: WARNING! Overriding auxiliary X11 port with new value '1'.
Warning: Overriding auxiliary X11 port with new value '1'.
Info: Using abstract X11 socket in kernel namespace for accessing DISPLAY=:0.
Info: Connecting to remote host 'localhost:51683'.
Info: Connected to remote proxy on FD#5.
Loop: WARNING! Non printable character decimal '10' received in remote data from FD#5.
Warning: Non printable character decimal '10' received in remote data from FD#5.
Connection timeout, abortingInfo: Aborting the procedure due to signal '15'.
Session: Session terminated at 'Mon Jul  6 18:20:01 2020'.

Unable to execute: rdesktop  -g 800x600

Proof of Concept

ssh into Qubes dom0

sys-whonix setup

x) sys-whonix software installation

In sys-whonix VM. Install ssh and x2go. 

sudo apt --no-install-recommends install openssh-client x2goclient

x) sys-whonix socat listener setup

In sys-whonix VM. Start socat listener.

TODO: use Qubes TcpConnect instead 

sudo socat TCP-LISTEN:22,fork EXEC:"qrexec-client-vm dom0 local.ssh"

x) setup onion v3

In sys-whonix VM.

Add onion v3 service configuration to Tor config /usr/local/etc/torrc.d/50_user.conf 

HiddenServiceDir /var/lib/tor/remote_service/
HiddenServicePort 22 127.0.0.1:22
HiddenServiceVersion 3

x) restart Tor

In sys-whonix VM.

[1] 

sudo systemctl restart tor@default

x) get onion v3 domain name

In sys-whonix VM. 

sudo cat /var/lib/tor/remote_service/hostname

remote-support-provider setup

x) remote-support-provider VM creation.

Create a VM named remote-support-provider (or any other name) based on whonix-ws-15 TemplateVM with networing set to sys-whonix. Usual instructions for Multiple Whonix-Workstation. Could even skip VM creation and use an already existing VM anon-whonix instead.

x) Create SSH key.

In remote-support-provider VM. 

ssh-keygen -t ed25519

x) View id_ed25519 public key.

In remote-support-provider VM. 

cat /home/user/.ssh/id_ed25519.pub

x) ssh client configuration hardening

In remote-support-provider VM.

Optional. [2] [3]

dom0 setup

x) In dom0. Install packages.

Optional. Some tool to simplify copying files from a VM to dom0 will later greatly simplify the setup. 

mkdir -p ~/bin

nano ~/bin/qubes-copy-from-vm 

#!/bin/bash
destination="/home/user/$(basename $2)"
qvm-run --pass-io "$1" "cat $2" > "$destination"

chmod +x ~/bin/qubes-copy-from-vm

x) In dom0. Install packages. 

sudo qubes-dom0-update socat openssh-server x2goserver x2goserver-xsession

[4]

x) dom0 SSH server configuration hardening

In dom0.

Optional.

[5] [3]

x) ssh PasswordAuthentication No

At least the following should be set.

/etc/ssh/sshd_config 

PasswordAuthentication No

x) enable sshd

In dom0. 

sudo systemctl enable sshd

x) start sshd

In dom0. 

sudo systemctl start sshd

x) reload sshd

In dom0.

[6] 

sudo systemctl reload sshd

x) check sshd status

In dom0. 

sudo systemctl status sshd

x) dom0 x2go setup 

sudo x2godbadmin --createdb

sudo systemctl enable x2gocleansessions.service

sudo systemctl start x2gocleansessions.service

x) dom0 qrexec rpc setup

In dom0. Setup qrexec rpc.

Open file /etc/qubes-rpc/local.ssh in an editor with administrative ("root") rights.

1 Select your platform.

Non-Qubes-Whonix

2 Notes.

  • Sudoedit guidance: See Kicksecure logo Open File with Root RightsOnion network Logo for details on why using sudoedit improves security and how to use it.
  • Editor requirement: Close Featherpad (or the chosen text editor) before running the sudoedit command.

3 Open the file with root rights.

sudoedit /etc/qubes-rpc/local.ssh

Qubes-Whonix

2 Notes.

  • Sudoedit guidance: See Kicksecure logo Open File with Root RightsOnion network Logo for details on why using sudoedit improves security and how to use it.
  • Editor requirement: Close Featherpad (or the chosen text editor) before running the sudoedit command.
  • Template requirement: When using Qubes-Whonix, this must be done inside the Template.

3 Open the file with root rights.

sudoedit /etc/qubes-rpc/local.ssh

4 Notes.

  • Shut down Template: After applying this change, shut down the Template.
  • Restart App Qubes: All App Qubes based on the Template need to be restarted if they were already running.
  • Qubes persistence: See also Kicksecure logo Qubes PersistenceOnion network Logo
  • General procedure: This is a general procedure required for Qubes and is unspecific to Qubes-Whonix.

Others and Alternatives

2 Notes.

  • Example only: This is just an example. Other tools could achieve the same goal.
  • Troubleshooting and alternatives: If this example does not work for you, or if you are not using Whonix, please refer to Open File with Root Rights.

3 Open the file with root rights.

sudoedit /etc/qubes-rpc/local.ssh

Add. 

#!/bin/sh
exec socat STDIO TCP-CONNECT:localhost:22

Save.

Make executable. [7] 

sudo chmod +x /etc/qubes-rpc/local.ssh

x) dom0 qrexec policy setup

In dom0.

Open file /etc/qubes-rpc/policy/local.ssh in an editor with administrative ("root") rights.

1 Select your platform.

Non-Qubes-Whonix

2 Notes.

  • Sudoedit guidance: See Kicksecure logo Open File with Root RightsOnion network Logo for details on why using sudoedit improves security and how to use it.
  • Editor requirement: Close Featherpad (or the chosen text editor) before running the sudoedit command.

3 Open the file with root rights.

sudoedit /etc/qubes-rpc/policy/local.ssh

Qubes-Whonix

2 Notes.

  • Sudoedit guidance: See Kicksecure logo Open File with Root RightsOnion network Logo for details on why using sudoedit improves security and how to use it.
  • Editor requirement: Close Featherpad (or the chosen text editor) before running the sudoedit command.
  • Template requirement: When using Qubes-Whonix, this must be done inside the Template.

3 Open the file with root rights.

sudoedit /etc/qubes-rpc/policy/local.ssh

4 Notes.

  • Shut down Template: After applying this change, shut down the Template.
  • Restart App Qubes: All App Qubes based on the Template need to be restarted if they were already running.
  • Qubes persistence: See also Kicksecure logo Qubes PersistenceOnion network Logo
  • General procedure: This is a general procedure required for Qubes and is unspecific to Qubes-Whonix.

Others and Alternatives

2 Notes.

  • Example only: This is just an example. Other tools could achieve the same goal.
  • Troubleshooting and alternatives: If this example does not work for you, or if you are not using Whonix, please refer to Open File with Root Rights.

3 Open the file with root rights.

sudoedit /etc/qubes-rpc/policy/local.ssh

Add. 

sys-whonix dom0 allow

Save.

x) dom0 ssh authorized_keys setup

Contents of remote-support-provider file /home/user/.ssh/id_ed25519.pub need to be transferred to dom0 /home/user/.ssh/id_ed25519.pub.

In remote-support-provider VM.

In dom0. As user. Not root! 

qubes-copy-from-vm sys-whonix /home/user/.ssh/id_ed25519.pub

In dom0. As user. Not root! 

mkdir -p ~/.ssh

In dom0. As user. Not root!

WARNING: If you used ssh previously, you might want to append instead. The following command would overwrite your ~/.ssh/authorized_keys. 

mv id_ed25519.pub ~/.ssh/authorized_keys

In dom0. As user. Not root! [8] 

sudo chmod --recursive og-rwx ~/.ssh

Usage

Introduction

After making an SSH server available in Qubes dom0 as per above instructions it will be accessible through an anonymous onion service.

ssh

Test this first.

In remote-support-provider VM.

Qubes dom0 requires the SSH key being added to Qubes dom0 ~/.ssh/authorized_keys as documented earlier on this page. 

ssh xxx.onion

x2goclient

Start x2goclient in a remote-support-provider (or another VM name). Should be based on Whonix-Workstation TemplateVM and Networking set to sys-whonix. Could alternatively use an already existing anon-whonix VM. [9]

As user. Run x2goclient.

  • host: the onion v3 domain
  • login: the user name such as user
  • password: the user login password such as changeme
  • session type: connect to local desktop
  • media -> disable client side printing support
  • media -> disable sound support
  • settings for better performance
    • connection -> connection speed -> WAN
    • connection -> image quality 0
    • connection -> method -> 8-jpeg
    • feel free to experiment with these settings
  • OK
  • Click on the newly setup connection
  • OK

Alternatively session type terminal could be used.

If you can only see a black screen, the the desktop might be locked (xscreensaver). Any mouse movement or any keypress should prompt for password.

Test Results

All times are created with an external stopwatch and should have +/- 2 seconds human caused inaccuracy.

  • Qubes R4.0
    • ssh:
      • ssh connection initialization takes 12 seconds.
      • There is a 2 second delay after each simple command such as ls.
      • Running a simple command such as ls -la in a folder with 60 files takes 2 seconds until results are drawn.
      • Comparable to a SSH connection over clearnet.
    • x2go session type terminal:
      • connection speed setting "modem":
        • Establishing a connection takes 90 seconds.
        • There is an two seconds delay between a keypress on in remote-support-provider VM until it gets visible in the remote terminal.
        • Running a simple command such as ls -la in a folder with 60 files takes 18 seconds until results are drawn.
        • from opening the x2go viewer to having the full desktop contents drawn it takes 180 seconds
      • connection speed setting "WAN":
        • time to open x2go viewer: 100 seconds
        • delay from clicking menu bar to actually seeing menu bar: 6 seconds
        • Running a simple command such as ls -la in a folder with 60 files takes 18 seconds until results are drawn.
        • Redraw screen after pressing enter: 18 seconds
    • This is much faster with Non-Qubes-Whonix when following instructions for SSH into Whonix with x2go. This might be because Qubes R4.0 dom0 is based Fedora 25 which ships an older version of x2go. It could also be because the x2go version of Qubes dom0 Fedora 25 is older than the x2go version in Debian buster based Whonix 15. The speed issues might be fixed with the release of Qubes R4.1 which will come with a more newer dom0 version of Fedora and thereby with a newer version of x2go.
  • Qubes R4.1
    • Running a simple command such as ls -la takes 1 second until results are drawn.
    • There is an 1 seconds delay between a keypress on in remote-support-provider VM until it gets visible in the remote terminal.
    • Click Xfce start menu: 2 seconds
    • clock in systray (shows seconds) update: every 1 or 2 seconds
    • There is a bug where I have no idea if caused by Qubes R4.1 testing version or x2go or a combination of it. When pressing any key it sometimes happen that this key is auto-repeat "jammed". I.e. "a" becomes "aaaaaaaaaaaaaaa" (no exact counting).

Debugging

  • Perhaps Exercise setting up a simple onion service web server first.
  • SSH into Whonix-Gateway (conflicts with above Tor configuration)
  • Could copy remote-support-provider folder ~/.ssh key to sys-whonix and then use ssh 127.0.0.1 directly inside sys-whonix to test if the SSH connection is functional. This was successfully tested during development. Useful to exclude any potential Tor connectivity issues.
    • Same as above but can also be done using x2goclient.
  • dom0: sudo journalctl -f -u sshd
  • dom0: journalctl to watch qrexec logs

Footnotes

  1. ↑ 1.0 1.1 reload should be sufficient but probably some bug in Tor required restart instead.
  2. ↑ Currently broken. Harden /etc/ssh/ssh_config as per here.
  3. ↑ 3.0 3.1 Currently broken due to x2go bugs.
  4. ↑ sftp-server required. 
    Unable to find the sftp-server binary. Neither bundled, nor found in $PATH nor additional directories.

If you are using a Linux-based operating system, please ask your system administrator to install the package containing the sftp-server binary. Common names are openssh, openssh-server or openssh-sftp-server depending upon distribution. 

If the sftp-server binary is installed on your system, please report a bug mentioning its path on: 

https://wiki.x2go.org/doku.php/wiki:bugs
  5. ↑ Currently broken. Use this file as /etc/ssh/sshd_config.
  6. ↑ This should not be required only in cases where sshd was previously running.
  7. ↑ Symptom if qubes-rpc is not executable. 
    2020/07/10 14:00:35 socat[14462] E waitpid(): child 14463 exited with status 127
  8. ↑ chmod og-rwx ~/.ssh/authorized_keys was insufficient, sshd would refuse with insecure permissions error message. chmod --recursive og-rwx ~/.ssh was insufficient probably because file was owned by root, not user.
  9. ↑ Or any other system with transparent torification. It cannot be easily made to work using socksifier torsocks.
Retrieved from "https://www.whonix.org/w/index.php?title=Dev/Qubes_Remote_Support&oldid=58559"