= Sell-Your-Saas installation and operation document This document describes the technical and functional implementation of Sell-Your-Saas - automated deployment and sales system in SaaS of a WAMP application (like Dolibarr ERP CRM, GLPI, ...) - Laurent Destailleur - www.sellyoursaas.org :source-highlighter: red :title: Document installation and operation of SellYourSaas :subject: This document describes the technical and functional implementation of SellYourSaas (automated deployment and sale system in SaaS of a WAMP application (like Dolibarr ERP CRM, GLPI, ...). :keywords: sellyoursaas, saas, dolibarr, wamp, glpi :imagesdir: ./img :toc: manual :toclevels: 3 :toc-title: Table of contents :toc-placement: preamble <<<< == Introduction == This document present how to install a remote server for Backups in a SellYourSaas environement. <<<< === Choice of machine and OS and create it * Obtain a server with SSH access that can pass root (We will use Ubuntu LTS minimum *18.04* to maximu *24.04*) * Add DNS entries of the server(s) (Entry A for IP4 and entry AAAA for IP6) [[adding_disk]] === Adding the harddisk of data (home of user instances and home of backups) We will add on the *RemoteBackupServer*, an independent disk for user instances and backups. With OVH Public Cloud or ScaleWay: * Create the disk of data from backoffice. You can imagine to reserve 250MB for each customer instance so choose a size in consideration. * Associate the disk with the server (each additional disk is added in /dev/vdb, /dev/vdc, /dev/vdd, ...). Note, the disk becomes visible with *fdisk -l* and *lsblk* * If it is a disk never partitioned, add the partition on the disk (Linux type) and format it by doing: [source, bash] --------------- fdisk -l fdisk /dev/vdx option n then p (then choose the partition number, first and last sector) then w fdisk -l fsck -N /dev/vdxY mkfs.btrfs /dev/vdxY --------------- Whether the disk has just been formatted or whether it is an added disk already formatted, the rest of the procedure is identical: * Recover the value of the UUID at the end of the formatting which is displayed, otherwise, recover it with the command [source, bash] --------------- blkid --------------- * Declare the assembly for an automatic assembly at each reboot by adding a line in */etc/fstab* [source, bash] --------------- UUID=94817f83-a2ad-46c4-81e0-06e6dd0e95f1 /mnt/diskX btrfs noatime,nofail 0 0 (does not block the server from starting) --------------- * Mount disk [source, bash] --------------- mkdir /mnt/diskbackup; chown admin:admin /mnt/diskbackup mount /dev/vdxZ /mnt/diskbackup blkid --------------- Note: A reboot may be required if disk or mount is not visible. * Optimize the filesystem by removing the update of the "atime" read access To see options for optimizing filesystems: [source, bash] --------------- tune2fs -l /dev/vdxY | grep features --------------- return Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize To add -noatime to the filesystem in the */etc/fstab* file: [source, bash] --------------- UUID=94817f83-a2ad-46c4-81e0-06e6dd0e95f1 /mnt/diskX btrfs noatime,nofail 0 0 --------------- To take the change into account: [source, bash] --------------- mount -oremount /dev/diskX/ --------------- To check: [source, bash] --------------- cat /proc/mounts | grep diskX --------------- Rem: If you need to recover data files from another disk, use: [source, bash] --------------- rsync --info=progress2 -au serveursource:/mnt/diskSource/* /mnt/diskTarget --------------- === SSH and sudo === Unix admin account Create the user account *admin*. It will be used to install and administer the system when root is not required. [source, bash] --------------- groupadd admin; useradd -m -g admin admin; usermod -a -G adm admin; mkdir /home/admin/logs; chown root:adm /home/admin/logs; chmod 770 /home/admin/logs; mkdir /mnt/diskbackup/.snapshots; chown admin:admin /mnt/diskbackup; chown admin:admin /mnt/diskbackup/.snapshots; --------------- Check that the id of this user *admin* is greater than or equal to 1000. Create a user account for yourself (or other administrators), for example: *myunixlogin*. It will be used to log in. [source, bash] --------------- adduser myunixlogin --------------- ==== ssh setup Fix permission on */etc/ssh/sshd_config* so only root has read and write access: [source,conf] --------------- chmod go-rw /etc/ssh/sshd_config --------------- Create a file */etc/ssh/sshd_config.d/sellyoursaas.conf* to change login permissions with the following content: [source, conf] --------------- # Privilege Separation is turned on for security UsePrivilegeSeparation yes # Permissions on files must be correct to allow login StrictModes yes # MaxSessions 10 MaxSessions 25 # Disallow login to root PermitRootLogin no # Disallow empty passwords PermitEmptyPasswords no # Do not support the "keyboard-interactive" authentication scheme defined in RFC-4256. ChallengeResponseAuthentication no # Define list of allowed method to authenticate PasswordAuthentication yes PubkeyAuthentication yes DenyUsers guest AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys_support AllowUsers admin AllowUsers myunixlogin --------------- Please note: replace *myunixlogin* with the correct value before taking changes into account with: [source, conf] --------------- /etc/init.d/ssh reload --------------- Create a file */etc/sudoers.d/myunixlogin* with the owner *root*.*root* and the permissions *r-r-----* and the content [source, conf] --------------- myunixlogin ALL=(ALL) NOPASSWD:ALL --------------- Test that you can connect using *myunixlogin* and make a sudo with [source,bash] --------------- ssh -v myunixlogin@x.y.z.a sudo -s --------------- Add your public key to your unix account. [source, bash] --------------- ssh-copy-id myunixlogin@x.y.z.a --------------- Warning: Sometime ssh-copy-id copy the dss key instead of rsa key and ssh fails with dss. Define or redefine the password for *root*, *admin* with a secure password. [source,bash] --------------- passwd root passwd admin --------------- Launch *ssh-keygen* on each of thee 3 accounts *root*, *admin* and *myunixlogin* ==== Default shell Modify the default shell to use bash (instead of dh sh or dash) [source, bash] --------------- ln -fs /bin/bash /usr/bin/sh --------------- === Deletion of information files at login In order not to give information to users doing SSH, on the deployment servers: [source, bash] --------------- rm /etc/update-motd.d/10-help-text /etc/update-motd.d/20-runabove rm /etc/update-motd.d/50-landscape-sysinfo /etc/update-motd.d/50-landscape-sysinfo rm /etc/update-motd.d/9*-update*-available /etc/update-motd.d/92-unattended-upgrades --------------- === Add alias Add at the end of */etc/bash.bashrc*: [source, bash] --------------- alias psld='ps -fax -eo user:12,pid,ppid,pcpu,pmem,vsz:12,size:12,tty,start_time:6,utime,time,cmd' --------------- === Hostname and IP configuration Add an entry from the new server to the DNS provided by the domain provider. Go to the OVH IP management interface, to add the reverse on the server IP. Go to the management interface of OVH servers, to modify their short name. This will modify the */etc/hostname* file automatically (if not manually modify) with the short name. The file will then have as sole content: [source, bash] --------------- nameofserver --------------- Connect and modify the file */etc/hosts* with the entry of the new server [source, bash] --------------- main.ip.of.server nameofserver.mysaasdomainname.com --------------- === Added support for IP v6 (optional, if ipv6 wanted but not yet enabled) ==== With netplan (Ubuntu 18.04 +) Add a conf file */etc/netplan/51-ipv6-ovh.yaml*. Note: OVH provides a /128 for ipv6 but netplan wants /64 Example for an IPv6 1234:41d0:1234:1000::1234 with as gateway 1234:41d0:1234:1000::1 [source, conf] --------------- network: version: 2 ethernets: eth0: match: name: eth0 addresses: - "1234:41d0:1234:1000::1234/64" gateway6: "1234:41d0:1234:1000::1" --------------- Note: Use 4 spaces for tabulation. [source, bash] --------------- netplan try netplan apply --------------- Rem: *eth0* can be something else, for example *ens3*. === Add virtual IP (optional) - Add the virtual IP via the OVH manager. - Add and remove the virtual network interface on the server dynamically (for test). Addition: [source, bash] --------------- ifconfig eth0: 0 a.b.c.d --------------- Deletion: [source, bash] --------------- ifconfig eth0: 0 down --------------- - For a persistent reboot definition, declare the interface in */etc/network/interfaces* or in a file in */etc/network/interfaces.d* (Ubuntu <17.10) Example for 2 virtual IPs: [source, conf] --------------- auto eth0: 0 iface eth0: 0 inet static address a.b.c.d netmask 255.255.255.255 broadcast a.b.c.d # To declare a persistent virtual IP auto eth0: 1 iface eth0: 1 inet static address e.f.g.h netmask 255.255.255.255 broadcast e.f.g.h --------------- Rem: *eth0* can be something else, for example *ens3*. To take this into account, try this, otherwise, reboot. [source, bash] --------------- /etc/init.d/networking restart --------------- - Associate the virtual IP with the server from the OVH manager. === Creation of working directories Only */mnt/diskbackup* is required, so no creation of directory has to be done. <<<< == Installation of system and application components === Installation of packages There are two scenario depending on your version of Ubuntu. Follow the instruction *18.04-* OR the *20.04+* one * Installation of the 18.04- Ubuntu packages [source,bash] --------------- sudo apt update sudo apt install ntp git gzip zip zstd ncdu duc sudo apt install rkhunter chkrootkit sudo apt install spamc spamassassin clamav clamav-daemon sudo apt install fail2ban --------------- * Installation of the 20.04+ Ubuntu packages [source,bash] --------------- sudo apt update sudo apt install systemd-timesyncd git gzip zip zstd ncdu sudo apt install rkhunter chkrootkit sudo apt install spamc spamassassin clamav clamav-daemon sudo apt install fail2ban --------------- Delete all snap packages (we don't need snap for backup servers): [source,bash] --------------- sudo apt remove --purge --assume-yes snapd systemctl daemon-reload --------------- === Disabling automatic update Uninstall the package *unattended-upgrades* if it was installed. [source, bash] --------------- apt remove unattended-upgrades --------------- === Installation of the firewall === * Create a firewall to accept input of SSH only and allow output for NTP and DNS and HTTPS [source, bash] --------------- ufw allow 22 ufw allow out ntp ufw allow out 53 ufw allow out 443 ufw status ufw enable --------------- Port 22 is for SSH input, NTP is 123/upd, 53 if for DNS, 443 is to allow monitoring agents to push reporting data === Installation of fail2ban === * Installation of fail2ban and activation of the following fail2ban rules: *pam-generic*, *xinetd-fail* To do this, first create a */etc/fail2ban/jail.local* file with this content: NOTE: The rules available may vary depending on the version of the OS installed. NOTE: Remember to also modify "mybusinessips" by your ip(s) separated by spaces as well as the parameter *destemail* by the supervision email of your company. [source, bash] --------------- # Fail2Ban configuration file. # # This file was composed for Debian systems from the original one # provided now under /usr/share/doc/fail2ban/examples/jail.conf # for additional examples. # # Comments: use '#' for comment lines and ';' for inline comments # # To avoid merges during upgrades DO NOT MODIFY THIS FILE # and rather provide your changes in /etc/fail2ban/jail.local # # The DEFAULT allows a global definition of the options. They can be overridden # in each jail afterwards. [DEFAULT] # "ignoreip" can be an IP address, a CIDR mask or a DNS host. Fail2ban will not # ban a host which matches an address in this list. Several addresses can be # defined using space separator. ignoreip = 127.0.0.1/8 mybusinessips # "bantime" is the number of seconds that a host is banned. bantime = 3600 # A host is banned if it has generated "maxretry" during the last "findtime" # seconds. findtime = 600 maxretry = 3 # "backend" specifies the backend used to get files modification. # Available options are "pyinotify", "gamin", "polling" and "auto". # This option can be overridden in each jail as well. # # pyinotify: requires pyinotify (a file alteration monitor) to be installed. # If pyinotify is not installed, Fail2ban will use auto. # gamin: requires Gamin (a file alteration monitor) to be installed. # If Gamin is not installed, Fail2ban will use auto. # polling: uses a polling algorithm which does not require external libraries. # auto: will try to use the following backends, in order: # pyinotify, gamin, polling. backend = auto # "usedns" specifies if jails should trust hostnames in logs, # warn when reverse DNS lookups are performed, or ignore all hostnames in logs # # yes: if a hostname is encountered, a reverse DNS lookup will be performed. # warn: if a hostname is encountered, a reverse DNS lookup will be performed, # but it will be logged as a warning. # no: if a hostname is encountered, will not be used for banning, # but it will be logged as info. usedns = warn # # Destination email address used solely for the interpolations in # jail.{conf,local} configuration files. destemail = supervision@mydomain.com # # Name of the sender for mta actions sendername = Fail2Ban # # ACTIONS # # Default banning action (e.g. iptables, iptables-new, # iptables-multiport, shorewall, etc) It is used to define # action_* variables. Can be overridden globally or per # section within jail.local file banaction = iptables-multiport # email action. Since 0.8.1 upstream fail2ban uses sendmail # MTA for the mailing. Change mta configuration parameter to mail # if you want to revert to conventional 'mail'. mta = sendmail [pam-generic] enabled = true [sshd] enabled = true [xinetd-fail] enabled = true --------------- Then place the filter files supplied with the project in *etc/fail2ban/filter.d* in the directory of the same name */etc/fail2ban/filter.d* === Test spamassassin === The process *spamd* must be running. Start it manually if it is not the case the first time. To test that spamassassin client is working, create a file */tmp/testspam* with content Subject: Test spam mail (GTUBE) Message-ID: Date: Wed, 23 Jul 2003 23:30:00 +0200 From: Sender To: Recipient Precedence: junk MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit This is the GTUBE, the Generic Test for Unsolicited Bulk Email If your spam filter supports it, the GTUBE provides a test by which you can verify that the filter is installed correctly and is detecting incoming spam. You can send yourself a test mail containing the following string of characters (in upper case and with no white spaces and line breaks): XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X You should send this test mail from an account outside of your network. Then test with: [source,bash] --------------- spamc < /tmp/testspam spamc -c < /tmp/testspam echo $? --------------- === Installation of ClamAV The process *freshclam* and *clamd* must be running. Start them manually the first time. To test clamav tool, create a file */tmp/testvirus* with content [source,bash] --------------- X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H* --------------- And to test *clamav* command line and daemon: [source,bash] --------------- clamscan /tmp/testvirus clamdscan /tmp/testvirus --fdpass --------------- [source,bash] --------------- aa-status --------------- You should see into the status a line saying that Profile *usr/sbin/clamd* is disabled. === Setup of logrotate * Add a line if not already present into file */etc/logrotate.conf* [source,bash] --------------- # use the syslog group by default, since this is the owning group of /var/log. su root syslog --------------- === Setup of journalctl Journals are stored into */var/log/journal/* (or into memory */run/log/journal/*) * Edit the file */etc/systemd/journald.conf* to define the max size for systemd journals [source,conf] --------------- ... SystemMaxUse=1G ... --------------- Take into account the change with [source,bash] --------------- systemctl stop systemd-journald systemctl start systemd-journald --------------- To force clear of journal: [source,bash] --------------- journalctl --flush --rotate journalctl --vacuum-size=1G journalctl --vacuum-time=1d --------------- To read journal: [source,bash] --------------- journalctl --disk-usage journalctl --header --------------- === Désactivation ou activation de apport (optionnel, "on" recommandé) Pour activer: [source,bash] --------------- sudo systemctl enable apport.service sudo systemctl start apport.service sudo systemctl status apport.service --------------- Pour désactiver: [source,bash] --------------- sudo systemctl disable apport.service sudo systemctl stop apport.service sudo systemctl status apport.service --------------- Note: Reports are into */var/crash* === Installation of cron tasks Delete very old files daily by adding the crontab task: [source,bash] --------------- #0 18 * * 0 find /mnt/diskbackup/*/backupold_* -type f -mtime +50 -ls -delete > /var/log/find_delete_old_files.log 2>&1 #0 18 * * 1 find /mnt/diskbackup/*/backupold_* -type d -empty -ls -delete > /var/log/find_delete_empty_dir.log 2>&1 55 23 * * * btrfs subvolume delete /mnt/diskbackup/.snapshots/diskbackup-`date +\%j` >/var/log/sellyoursaas_btrfs.log; btrfs subvolume snapshot /mnt/diskbackup /mnt/diskbackup/.snapshots/diskbackup-`date +\%j` >>/var/log/sellyoursaas_btrfs.log; rm -fr /mnt/diskbackup/.snapshots/diskbackup-`date +\%j`/.snapshots >>/var/log/sellyoursaas_btrfs.log; btrfs property set -ts /mnt/diskbackup/.snapshots/diskbackup-`date +\%j` ro true >>/var/log/sellyoursaas_btrfs.log; 0 0 * * * find /mnt/diskbackup/.snapshots -maxdepth 1 -type d -mtime +40 -exec btrfs subvolume delete {} \; >>/var/log/sellyoursaas_btrfs.log; 0 1 * * * find /mnt/diskbackup/backup_*/osu*/ -maxdepth 0 -type d -mtime +730 -exec rm -fr {} \; >>/var/log/sellyoursaas_btrfs.log --------------- <<<< == Installation d'outils externes === Installation of DataDog (optionnel pour supervision) * Create an account on DataDog. * Install the agent on serveur with: [source,bash] --------------- DD_AGENT_MAJOR_VERSION=7 DD_API_KEY=YOURDATADOGAPIKEY bash -c "$(curl -L https://raw.githubusercontent.com/DataDog/datadog-agent/master/cmd/agent/install_script.sh)" --------------- Relancer datadog [source,bash] --------------- sudo service datadog-agent stop sudo service datadog-agent start --------------- <<<< == Exploitation - Supervision === Backup / Restauration ==== Backup system La sauvegarde du serveur+bases peut se faire par un snapshot d'image de la VM. Il est aussi possible de ne faire un snapshot que des disques complémentaires. Voir chapitre <> ==== Restauration system Depuis l'espace "Snapshots" d'OVH, on peut demander à le restaurer sur un serveur (pour une image VM) ou sur un aute disque (pour une image disque complémentaire), à condition que la cible (serveur ou disque) soit supérieure ou égale en terme de capacité de stockage. Voir chapitre <> === Increase size of disk * Faire le snapshot du disque à redimensionner pour sauvegarde. Créer un nouveau disque depuis ce snapshot et le rattacher à un autre serveur (voir chapitre <>) pour s'assurer qu'il est lisible et ainsi avoir les fichiers de la sauvegarde sous la main. * Unmount the filesystem: [source,bash] --------------- umount /mnt/disk/ --------------- Rem: Pour voir les fichiers ouverts sur un disque si le démontage échoue: [source,bash] --------------- lsof | grep "/mnt/disk" --------------- * Détacher le disque du serveur. S'assurer que son nom ne contient pas d'espaces ou caractères spéciaux. Changer la taille du disque depuis le manager du Public Cloud et le réattacher au serveur. * Agrandir la partition en lançant: [source,bash] --------------- fdisk -l parted /dev/vdX (X=a, b, !!! SANS le chiffre, on veut le disque complet) print all resizepart Y 999GB (Ne pas saisir la valeur proposé par défaut mais la valeur max du disque qui a été affiché par le "print all") q --------------- * Remonter le disque pour prise en compte et augmenter le formatage du filesystem sans effacement. [source,bash] --------------- mount /mnt/disk/ resize2fs /dev/vdX9 --------------- === Upgrade OS Pour mettre à jour Ubuntu x.y vers z.w sur un serveur SellYourSaas: [source,sql] --------------- apt dist-upgrade --------------- === Passage en mode rescue d'un serveur Aller sur l'interface du service Cloud pour passer en mode rescue. Le serveur sera rebooté et un lien pour se logué sera fourni. Trouver les disques attachées et montez le disque système. [source,bash] --------------- lsblk mount /dev/sdXY /mnt --------------- Il est alors possible d'agir sur le disque en écriture accessible dans /mnt === Migrate a ext4 partition into btrfs Run command to see partition types: [source,bash] --------------- df -Th lsblk -f --------------- If the data disk is ext4, you can convert it into BTRFS with: [source,bash] --------------- btrfs-convert -p /dev/vdX lsblk -f --------------- Then edit he */etc/fstab* to the type *ext4* to *btrfs* (change the UUID if it was changed) Then you create a subvolume for each directory you want to be able to make snapshots. === BTRFS admin To list existing subvolumes: [source,bash] --------------- btrfs subvolume list /mnt/diskbackup --------------- To create a subvolume for a server [source,bash] --------------- btrfs subvolume create /mnt/diskbackup/xxx # with xxx = home_serverX or backup_serverX chown admin:admin /mnt/diskbackup/xxx chmod -R o-rw /mnt/diskbackup/xxx cp -ax --reflink=always src/. dest --------------- To convert an existing directory into a subvolume or convert a subvolume into a common directory [source,bash] --------------- export dirtoconvert=xxx # with xxx = home_serverX or backup_serverX # To convert directory into subvolume mv /mnt/diskbackup/$dirtoconvert /mnt/diskbackup/$dirtoconvert-old; btrfs subvolume create /mnt/diskbackup/$dirtoconvert; cp -ax --reflink=always /mnt/diskbackup/$dirtoconvert-old/. /mnt/diskbackup/$dirtoconvert; rm -fr /mnt/diskbackup/$dirtoconvert-old; # To convert subvolume into directory mv /mnt/diskbackup/$dirtoconvert /mnt/diskbackup/$dirtoconvert-old; cp -ax --reflink=always /mnt/diskbackup/$dirtoconvert-old/. /mnt/diskbackup/$dirtoconvert; btrfs subvolume delete /mnt/diskbackup/$dirtoconvert-old; --------------- -- Example of CLI to convert directoris not btrfs into btrfs subvolume -- btrfs subvolume show /mnt/diskbackup/not 2>&1 | grep -c "Not a Btrfs subvolume" && mv /mnt/diskbackup/not /mnt/diskbackup/not-old && btrfs subvolume create /mnt/diskbackup/not && cp -ax --reflink=always /mnt/diskbackup/not-old/. /mnt/diskbackup/not && rm -fr /mnt/diskbackup/not-old To create a readonly snapshot [source,bash] --------------- btrfs subvolume snapshot /mnt/diskbackup /mnt/diskbackup/.snapshots/diskbackup-`date +%j`; rm -fr /mnt/diskbackup/.snapshots/diskbackup-`date +%j`/.snapshots; btrfs property set -ts /mnt/diskbackup/.snapshots/diskbackup-`date +%j` ro true; --------------- To delete a readonly snapshot of a day. Note: space is not freed immediatly. [source,bash] --------------- btrfs subvolume delete /mnt/diskbackup/.snapshots/diskbackup-`date +%j`; --------------- To delete all old snapshot [source,bash] --------------- find /mnt/diskbackup/.snapshots -maxdepth 1 -type d -mtime +60 -exec btrfs subvolume delete {} \; >>/var/log/sellyoursaas_btrfs.log; --------------- == TroubleShooting See the chapter available into the *Documentation SellYourSaas - Master and Deployment servers*