Go away, the cloud is full

Using ffmpeg to convert audio cassettes to videos

Spiros Evangelatos — Wed, 10 Jun 2020 13:04:20 GMT

The excellent ffmpeg is the swiss army knife of all video processing. Just as a reminder to myself, here's a command line to convert an image and audio file to a video.

ffmpeg -y -loop 1 -framerate 1  -i image.jpg  -i Unknown.mp3 \
    -c:v libx264 -preset medium -tune stillimage -crf 23 \
    -vf scale=-1:720 -c:a copy -shortest -pix_fmt yuv420p \
    -movflags +faststart output.mp4

The options are as follows:

-y Yes, go ahead and overwrite output file. Useful when experimenting.
-loop 1 Loop input stream 1. Makes the single image last forever as a stream.
-framerate 1 Set video frame rate to 1FPS. This saves space. But going below 1 FPS causes compatibility issues with some players.
-c:v libx264 Encode video as h.264 using the excellent x264 library. Good quality, great compatibility.
-preset medium Use medium quality settings for video encoder. Other options: ultrafast, superfast, veryfast, faster, fast, medium, slow, slower, veryslow. No need to go overboard for still images.
-tune stillimage Tune the video encoder for a still image.
-crf 23 Video quality/bitrate. Values of 18 - 25 are reasonable. Lower values give higher quality and bitrates.
-vf scale=-1:720 Scale the video image to 720 pixels of height, preserving aspect ration. Adjust to your needs.
-c:a copy Copy the audio stream without re-encoding it.
-shortest Finish encoding when the shortest input stream ends. Since we made the image stream infinite with -loop 1, the audio stream is the shortest.
-pix_fmt yuv420p Use yuv pixel format for better compatibility.
-movflags +faststart Arrange all the mp4 headers and stuff at the beginning of the file so that it can be streamed efficiently.

Optionally you may use -c:a aac -b:a 192k instead of -c:a copy to re-encode audio as AAC, which is more common in the mp4 container.

So there you go...

Yet another rsync based backup script

Spiros Evangelatos — Tue, 30 Apr 2019 13:36:15 GMT

Just putting this here for easy reference, or to help any random passers by. I know there are tools that are supposed to automate this like rsnapshot but I was disappointed to see that rsnapshot is probably unmaintained

On top of that, this looked like something easy enough to roll my own and adapt to my needs. So there you go...

#!/bin/bash
# A script to perform daily/weekly/monthly backups using rsync

SOURCE="user@remote:/path/some_files"
DESTINATION="/backups/my_backups"

# How many backups of each type shall we keep?
DAILY_COUNT=7
WEEKLY_COUNT=4
MONTHLY_COUNT=18

DAILY=$(date  "+daily_%Y_%m_%d")
WEEKLY=$(date  "+weekly_%Y_%W")
MONTHLY=$(date  "+monthly_%Y_%m")
ARCHIVE="archive"
LATEST="latest"

set -e

# Make sure destination dirs exist
mkdir -p "${DESTINATION}/${ARCHIVE}"
mkdir -p "${DESTINATION}/${LATEST}"

cd "${DESTINATION}"

ABS_LATEST="$(pwd)/${LATEST}"
# Do the rsync 
rsync -a --quiet --delete --link-dest="${ABS_LATEST}" -- "${SOURCE}" "${ARCHIVE}/${DAILY}"

# Replace the last latest
/bin/rm -rf -- "${ABS_LATEST}"
/bin/cp -al -- "${ARCHIVE}/${DAILY}" "${ABS_LATEST}"

# Add new weekly if needed
if [ ! -d "${ARCHIVE}/${WEEKLY}" ]; then
    /bin/cp -al -- "${ABS_LATEST}" "${ARCHIVE}/${WEEKLY}"
fi

# Add new monthly if needed
if [ ! -d "${ARCHIVE}/${MONTHLY}" ]; then
    /bin/cp -al -- "${ABS_LATEST}" "${ARCHIVE}/${MONTHLY}"
fi

# Cull archives
cd "${ARCHIVE}"

# Daily culling
/usr/bin/find . -maxdepth 1 -type d -regextype posix-egrep \
    -regex ".*\/daily_[0-9]{4}_[0-9]{2}_[0-9]{2}" -printf "%f\n" |
    sort -r |
    tail -n +$((${DAILY_COUNT} + 1)) |
    while read EXPIRED; do
        #echo "Removing ${EXPIRED}"
        /bin/rm -rf -- "$EXPIRED"
    done

# Weekly culling
/usr/bin/find . -maxdepth 1 -type d -regextype posix-egrep \
    -regex ".*\/weekly_[0-9]{4}_[0-9]{2}" -printf "%f\n" |
    sort -r |
    tail -n +$((${WEEKLY_COUNT} + 1)) |
    while read EXPIRED; do
        #echo "Removing ${EXPIRED}"
        /bin/rm -rf -- "$EXPIRED"
    done

# Monthly culling
/usr/bin/find . -maxdepth 1 -type d -regextype posix-egrep \
    -regex ".*\/monthly_[0-9]{4}_[0-9]{2}" -printf "%f\n" |
    sort -r |
    tail -n +$((${MONTHLY_COUNT} + 1)) |
    while read EXPIRED; do
        #echo "Removing ${EXPIRED}"
        /bin/rm -rf -- "$EXPIRED"
    done

Setting up owncloud on ubuntu 18.04

Spiros Evangelatos — Sun, 14 Apr 2019 17:53:43 GMT

In this article I will walk us throught installing a bare metal owncloud server based on ubuntu server 18.04. Throughout this guide we will assue that our server is named myowncloud.mydomain.com.

Base system installation

First steps as usual, we download the ubuntu server install image. If you wish to use full disk encryption, please use the "alternate installer" ubuntu-18.04.x-server-amd64.iso images instead of the "standard" ubuntu-18.04.x-live-server-amd64.iso images that do not currently support advanced LVM options. Next, "burn" the iso on a USB stick and boot it.

Now, if you wish to use full disk encryption, in the "Partition disks" step select: "Guided - user entire disk and setup encrypted LVM".

During installation I also chose the "Install security updates automatically" option. As for the software selection (tasksel), I only picked the "OpenSSH server" option as I wished to install everything else manually.

System preparation

After installation is complete, boot the new system and log on to it.
Let's first bring everything up to date and install a firewall.

apt update
apt upgrade -y
apt dist-upgrade -y

#firewall 
ufw allow OpenSSH
ufw enable

If you did not setup unattended package upgrades during installation you may do so now.

apt install unattended-upgrades
dpkg-reconfigure -p low unattended-upgrades

OwnCloud prerequisites

Next we will install the owncloud prerequisites in addition to some nice-to have packages like vim and tmux. We will also update our firewall to allow ports 80 and 443 to pass through to apache.

apt-get install \
apache2 \
curl \
mysql-server \
php libapache2-mod-php php-mysql \
certbot python-certbot-apache \
vim tmux wget unattended-upgrades \
php-bz2 php-curl php-gd php-imagick php-intl php-mbstring \
php-xml php-zip php-redis redis

ufw app list
ufw app info "Apache Full"
ufw allow "Apache Full"

Install dyndns update script

In my case, I needed to setup dynamic DNS for my server. To do that I chose the excelent FreeDNS service that allows you to update your IP using a simple http request done through wget. Go here to get the command for your address.

So we now edit the crontab to add the updates:

crontab -e
# Add the following lines:
#@hourly sleep 12 ; wget -O - http://freedns.afraid.org/dynamic/update.php?SOME_HASH_HERE >> /tmp/freedns_owncloud.log 2>&1 &
#@reboot sleep 12 ; wget -O - http://freedns.afraid.org/dynamic/update.php?SOME_HASH_HERE >> /tmp/freedns_owncloud.log 2>&1 &

Run certbot to get SSL certificates

Now that we have our DNS up and running, we need to use certbot to get SSL certificates for our server. This is crucial since using owncloud over the internet without SSL encryption would allow anyone to read our login information and exchanged files. Not a good idea obviously. This is more or less what you should see during this process. At this point, to be able to run certbot, you must configure your router to allow acessing ports 80 and 443 of your server from the internet.

certbot
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Enter email address (used for urgent renewal and security notices) (Enter 'c' to
cancel): ****REMOVED****@mailserver.com

-------------------------------------------------------------------------------
Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf. You must
agree in order to register with the ACME server at
https://acme-v01.api.letsencrypt.org/directory
-------------------------------------------------------------------------------
(A)gree/(C)ancel: A
No names were found in your configuration files. Please enter in your domain
name(s) (comma and/or space separated)  (Enter 'c' to cancel): myowncloud.mydomain.com
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for myowncloud.mydomain.com
Enabled Apache rewrite module
Waiting for verification...
Cleaning up challenges
Created an SSL vhost at /etc/apache2/sites-available/000-default-le-ssl.conf
Enabled Apache socache_shmcb module
Enabled Apache ssl module
Deploying Certificate to VirtualHost /etc/apache2/sites-available/000-default-le-ssl.conf
Enabling available site: /etc/apache2/sites-available/000-default-le-ssl.conf

Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
-------------------------------------------------------------------------------
1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Enabled Apache rewrite module
Redirecting vhost in /etc/apache2/sites-enabled/000-default.conf to ssl vhost in /etc/apache2/sites-available/000-default-le-ssl.conf
...

Now that we got our SSL certificates, we need to configure cron to renew them periodically, as these certificates expire every 3 months. We also need to restart apache after the certificates have been removed.

crontab -e
@daily /usr/bin/certbot renew -n --post-hook "systemctl reload apache2"

Secure mysql server

Run:

mysql_secure_installation

Follow the prompts to setup a secure mysql password and remove test databases/users and disable remote root logins.

Get latest version of owncloud

Go to the owncloud downloads page and get the URL of the latest owncloud tarball.

cd /var/www 
wget  https://download.owncloud.org/community/owncloud-10.1.1.tar.bz2 
tar xjf owncloud-10.1.1.tar.bz2
# Change ownership of owncloud files
find /var/www/owncloud \( \! -user www-data -o \! -group root \) -print0 | xargs -r -0 chown www-data:root && \
  chmod g+w /var/www/owncloud

# Make data directory
mkdir /mnt/owncloud_data
chown www-data:root /mnt/owncloud_data &&   chmod g+w /mnt/owncloud_data

Configure apache

First configure the default servername for apache.

# Add this line near the end of the file: 
# ServerName myowncloud.mydomain.com
vim /etc/apache2/apache2.conf

Now edit /etc/apache2/mods-enabled/dir.conf to give precedence to php files by bringing index.php in front of the rest of the file types.

vim /etc/apache2/mods-enabled/dir.conf

Next, we need to edit our apache configuration. First find the configuration file for the owncloud website:

apache2ctl -t -D DUMP_VHOSTS | grep myowncloud.mydomain.com

In my case that is /etc/apache2/sites-enabled/000-default-le-ssl.conf.
Now enable the mod_headers apache plugin and then edit the website config file.

a2enmod headers # Enable mod_headers
vim /etc/apache2/sites-enabled/000-default-le-ssl.conf

Within the config file add the following under the tag:


  Header always set Strict-Transport-Security "max-age=15552000; includeSubDomains"

and set:

DocumentRoot /var/www/owncloud

Finally, restart apache:

systemctl restart apache2

Configure php for production

Make sure the following options in /etc/php/7.2/apache2/php.ini are set as below to reduce unintended information exposure through debugging messages:

display_errors = Off
display_startup_errors = Off

Also you may want to edit /var/www/owncloud/.user.ini to increase some
limits. Mine currently reads:

upload_max_filesize=1025M
post_max_size=1025M
memory_limit=1024M
mbstring.func_overload=0
always_populate_raw_post_data=-1
default_charset='UTF-8'
output_buffering=0

Configure mysql owncloud user and privileges

Use your own uniqueue_password for the owncloud user here

mysql -u root -p <