This has grown into a very useful list of things that make my life easier. Divided into sections for clarity. Mostly applicable to Ubuntu as well.

Nifty shortcuts, one-liners and useful programs

• netcat for file transfer receiver: nc -l -p 1234 > somefile.png
  sender: nc -w 3 192.168.178.20 1234 < somefile.png
• ssh port forwarding ssh -R 80:localhost:8080 user@server.com -i key.pem
• multiple x sessions exec startx -- :1 vt8
• get random SALT openssl rand -base64 32
• use imagemagick to resize pictures convert <infile> -resize 100x100 [<outfile>]
• force aspect ratio convert <infile> -thumbnail '100x100>' -background white -alpha remove -gravity center -extent 100x100 [<outfile>]
• use imagemagick to create quick thumbnails POSIX, only works with .jpeg, height=200px: for f in *; do; echo $f; convert -thumbnail x200 -background white -alpha remove $f ${f:%.jpeg}_thumb.png; done
  with zsh: for f in *; do; echo $f; convert -thumbnail x200 -background white -alpha remove $f ${f:r}_thumb.png; done
  where ${f:r} means “remove file extension from $f”, see zsh archive
• fast video thumbnails with ffmpegthumbnailer for f in *; do; echo $f; ffmpegthumbnailer -i $f -o ${f:r}_thumb.png -s 400; done
• imagemagick: crop and then append images Images are 1080x1920, we want to crop them to the middle, but full height. First, create the needed folders folders, with files named identically: mkdir before now cropped_{before,now}, so that you have e.g. before/1.png, now/1.png, before/2.png, now/2.png and so forth.
  Then, figure out the crop dimensions, e.g. 800x1080+580+0 where we want an image 800x1080 in dimension, and we carve it out with the x-offset of 580 and y-offset of 0.
  Then, run the following imagemagick commands:

  # Crop all before images
  for f in before/*; do
    i=${f#before/} && echo $i && \
    convert -crop 800x1080+580+0 before/$i cropped_before/cropped_$i
  done`
  # Crop all after images
  for f in now/*; do
    i=${f#now/} && echo $i && \
    convert -crop 800x1080+580+0 now/$i cropped_now/cropped_$i
  done`
  # Merge the cropped before and after pictures side-by-side
  for f in before/*; do
    i=${f#before/} && echo $i && \
    convert +append cropped_before/cropped_$i cropped_now/cropped_$i $i
  done`
  

• ffmpeg gif optimizer
  ffmpeg -i input.mp4 -r 10 -f image2pipe -vcodec ppm - | convert -loop 0 - output.gif
• find and remove “~” files from emacs etc. find . -name*\~ | xargs rm
• use quvi to download videos etc quvi --exec "wget -O <name.webm> %u" <link>
  or just use youtube-dl
• count words (or lines) wc [-l]
• print all file types in a directory find . -type f -name "*.*" | awk -F. '{print $NF}' | sort -u
  or just print number of files of one filetpye: find . -name "*.mp3" -type f | wc -l
• count number of files find <dir> -type f | wc -l
• generate random number shuf -i 2000-65000 -n 1
• recover data from btrfs partitions from superuser
  run strings /dev/sda1 | less and then use / to search for the strings that could be in that file. delimit the start of it with “m” then move to the end and use |cat >file to save it to a file
  (note: worked on text for me, binary could be difficult)
• tre-agrep
  cat file.txt | tre-agrep -i(ignorecase) -s(show match cost) -3(max cost of 3) 'Fuzzy match'
  • -D: cost of missing characters
  • -I: cost of extra characters
  • -S: cost of substitutions(should be x2 since deletion and insertion are 2 operations)
• load testing use siege --concurrent=100

Pdf handling

• use pdftk to split a document into multiple pages pdftk A=input.pdf cat A1-100 output output-1-100.pdf

• use pdftk to join multiple pdfs
  pdftk input1.pdf input2.pdf cat output output.pdf

• use pdfjam to make presentations printable
  pdf90 inputs.pdf pdfjam Inputs_rotated.pdf --nup 2x1 --landscape --outfile Outname.pdf pdf270 Outname.pdf

• convenient fish function

  function pp -d 'Make presentations printable'
      pdf90 $argv
      set -x pdf_file (echo $argv | cut -f 1 -d '.')
      pdfjam (echo $pdf_file)-rotated90.pdf --nup 2x1 \
      --landscape --outfile print-(echo $pdf_file).pdf
      pdf270 print-(echo $pdf_file).pdf
      rm print-(echo $pdf_file).pdf
      mv print-(echo $pdf_file)-rotated270.pdf print-(echo $pdf_file).pdf
      rm (echo $pdf_file)-rotated90.pdf
  end
  

• k2pdfopt
  -om 0.3 for margins, -n for disabling ocr and bitmap generation

• removing margins
  use pdfcrop to remove whitespace automatically
  use --margins '5 10 20 30' to remove margins in pts(left, top, right, bottom), use negative values (-50 -50 -50 -50) for actually cropping away

• generate blank page
  echo "" | ps2pdf -sPAPERSIZE=a4 - blankpage.pdf

Misc

(Difficult-to-remember commands or usage of programs, accomplishing very specific tasks)

• tar tar xvzf file to extract file.tar.gz, tar cvzf file.tar.gz files_to_include to create

• find find . -iname "*thing*", case insensitive

• multiple keyboard locales setxkbmap -option grp:switch,grp:alt_shift_toggle,grp_led:scroll us,de

• encrypt harddrive (from hermann-uwe.de)

  badblocks -c 10240 -s -w -t random -v /dev/sdX
  # create partition using fdisk
  cryptsetup --verbose --verify-passphrase luksFormat /dev/sdX1
  cryptsetup luksOpen /dev/sdX1 name
  mkfs.ext3 -j -m 1 -O dir_index,filetype,sparse_super /dev/mapper/name
  # mount it
  # write to it
  # unmount it
  cryptsetup luksClose /dev/mapper/name
  

• invert bash wildcards
  shopt (-s)(-u) extglob
ls !(*.zip)

• the sudo tee duo sh unprivileged-command | sudo tee output-file >/dev/null

• sed instead of head sed 11q file is equal to head file
sed '3q;d' file to get 3rd line of file

• bridge terminal and clipboard
  echo test | xclip -selection c - put into clipboard
  xclip -selection clipboard -o - print out clipboard contents

• use notify-send sanely
  use notify-send 'Heading' 'Body text' --icon=dialog-information with --hint int:transient:1 for ephemeral notifications that do not clutter the message view

• qemu
  qemu-img create disk.img 10G
qemu -hda disk.img -boot d -cdrom distro.iso -m 1024

• download a whole webpage
  wget --recursive --no-clobber --page-requisites --html-extension --convert-links --domains domain.com domain.com

• reinstall all python packages
  sudo dpkg --get-selections | sed 's:install$::' | sed -e 's/^[ \t]*//' -e 's/[ \t]*$//' | grep -v 'python3' | grep python sudo dpkg --get-selections | sed 's:install$::' | sed -e 's/^[ \t]*//' -e 's/[ \t]*$//' | grep python3 | xargs sudo apt-get --reinstall install

• thunderbird: disable annoyances thunderbird for enterprise

• upgrade all pip packages
  add --local for virtualenvs

    pip install -U `pip list --outdated | awk '{ print $1}'`
  

• Clear systemd journal
  sudo journalctl --rotate && sudo journalctl --vacuum-time=1s

• use tar to back up a server
  crude, but effective

  tar -zcvpf /archive/full-backup-`date '+%d-%B-%Y'`.tar.gz --directory / \
      --exclude=mnt --exclude=proc --exclude=var/spool/squid --exclude=sys \
      --exclude=cdrom --exclude=sys --exclude=tmp
  

  or from serverpronto.com

  tar cvpzf /backups/backup.tgz --exclude=/proc --exclude=/lost+found \
      --exclude=/backups --exclude=/dev --exclude=/sys --exclude=/boot/grub \
      --exclude=/etc/fstab --exclude=/etc/network/interfaces \
      --exclude=/etc/udev/rules.d/70-persistent-net.rules /
  

  or

  tar cvpzf /root/backups.tgz --exclude=/root --exclude=/proc \
      --exclude=/lost+found --exclude=/backups --exclude=/dev \
      --exclude=/sys --exclude=/boot/grub --exclude=/etc/fstab \
      --exclude=/etc/network/interfaces
  

rsync

• basic rsync -avs --delete -h . user@domain.tld:/home/user/server
• copy directory structure only rsync -av -f"+ */" -f"- *" /path/to/src /path/to/dest/
• ignore version control files(warning: ignores folders named core) rsync --cvs-exclude

git

• set up new repo

  git init
  git add .
  git commit -m "commit message"
  git remote add remotename user@domain.tld:/home/user/proj.git
  git push --set-upstream remotename master
  # on server
  git --bare init
  

• update submodules
  git pull --recurse-submodules
  with proper commits: git submodule update --recursive

• change author info
  from github:
  git clone --bare https://github.com/user/repo.git
cd repo.git
  run git-author-rewrite.sh, check logs
  git push --force --tags origin 'refs/heads/*'

Media

• convert ogg to mp3
  for name in *.ogg; do ffmpeg -i "$name" -ab 128k -map_metadata 0:s:0 "${name/.ogg/.mp3}"; done;

• convert flac to mp3(fish shell)

  for i in (ls *.flac)
      ffmpeg -i $i -ab 320k -map_metadata 0 -id3v2_version 3 $i.flac
  end
  

• ffmpeg: reduce video size
  ffmpeg -i input.mp4 -preset medium -c:v libx265 -crf 28 -c:a copy output.mp4
  (crf: low is better quality, high is worse. 20-28 are sane values, 23 is default, see stackoverflow)

• ffmpeg: convert to x264 with good quality
  ffmpeg -i input.mp4 -c:a copy -vcodec libx264 -crf 20 output.mp4

• ffmpeg: use VAAPI acceleration for converting

  ffmpeg -init_hw_device vaapi=default_vaapi:/dev/dri/renderD128 -hwaccel vaapi \
    -hwaccel_output_format vaapi -hwaccel_device default_vaapi \
    -i input.mkv -filter_hw_device default_vaapi -vf 'format=nv12|vaapi,hwupload' \
    -preset medium -c:v hevc_vaapi -crf 25 -c:a copy output.mkv
  

  See also ffmpeg: VAAPI

  Codec                        | VAAPI name  
  -----------------------------|------------
  H.262 / MPEG-2 part 2        | mpeg2_vaapi 
  H.264 / MPEG-4 part 10 (AVC) | h264_vaapi  
  H.265 / MPEG-H part 2 (HEVC) | hevc_vaapi  
  MJPEG / JPEG                 | mjpeg_vaapi 
  VP8                          | vp8_vaapi   
  VP9                          | vp9_vaapi   
  

• change volume ffmpeg -i input.mp3 -af "volume=0.8" output.mp3

• save all mp3s from an m3u file
  wget -r 'url.m3u' -O - | wget -nd -N -i - && ls *.mp3 *.ogg *.flac *.wav > play.m3u

• split audio files without reencoding
  install mp3splt
mp3splt -S 5 input.mp3 (-D directory): split into 5 equal parts
  mp3splt -t 10.30 input.mp3: split into files of minutes.seconds

• remove audio tracks from an mkv file
  mkvmerge --identify input.mkv (get track info)
  mkvmerge -o output.mkv --audio-tracks 2 input.mkv
  other usage: mkvtoolnix documentation

• merge video and subtitles into container file mkvmerge -o output.mkv --default-track 0 --language 0:eng subtitles.srt input.mp4

• audio players
  from ubuntuusers
  place a .is_audio_player file in root directory of audio player

• splitting flac files (from ubuntugenius)
  install cuetools, shntool and flac packages
  run cuebreakpoints album.cue | shnsplit -o flac album.flac and then(after converting to mp3) cuetag album.cue *.mp3
  note: enclose filesnames containing special characters with double quotes

Typography

• create specific unicode characters CTRL-SHIFT-U <NUMBER> ENTER
• via compose key Set Right control(makes most sense) as compose key using gnome-tweak-tool, then either look up the corresponding sequence here or try to guess it, the mappings make sense most of the time
  (No need to keep pressing the compose modifier while entering)
  Examples: - . . for … (ellipsis) - ~ n for ñ(eñe) - ? ? for ¿(inverted ?) - C C C P(uppercase!) for ☭ (hammer & sickle)

Display management

• start display manager (xmonad, …)
  in .xinitrc

  xrandr -s 0
  xmonad &
  wait
  

• set brightness xrandr --output DVI0 --brightness 0.5
  also use arandr

• turn off display xset dpms force off, combine with lock: xset dpms force off; slock

System deployment

• manipulate installed packages
  sudo dpkg --get-selections > packages.txt to back up
  then sudo dpkg --clear-selections to set all to deinstall
sudo dpkg --set-selections < packages.txt to import the saved package list
  sudo aptitude install or sudo apt-get -u dselect-upgrade(much faster)

fish shell

• ‘clear cache’ and speed up: remove ~/.config/fish/fishd*
• save alias: alias la 'ls -la' and then funcsave <func>
• clipboard: kill with CTRL-w or ALT-BCKSP -> kill ring.
  Paste:CTRL-y, cycle: ALT-y
• man dmenu has a good explanation of movement
• use ALT and left/right to move through directory history
• use z with tab completion for a fuzzy ncurses-like file manager

Shortcuts from emacs-mode

Fix specific application problems or configure for desired mode of operation

• apt-get without diffs diff files are great for low bandwidth connections but they are not needed most of the time
  either run apt-get update with -o Acquire::Pdiffs=false
  or edit /etc/apt/apt.conf to include Acquire::PDiffs "false";
• apt: disable automatic updates edit /etc/apt/apt.conf.d/10periodic and set APT::Periodic::Update-Package-Lists "0";
• fix alsa problems (sudo) alsactl init
• pandoc generate proper html with utf-8 encoding pandoc -f markdown infile.mkd -t html -s -o outfile.html
• pandoc: website to epub pandoc -s -r html http://url.tld/page -o epub
• try to explicitly log into an ssh server using password instead of an ssh key ssh -o PreferredAuthentications=keyboard-interactive -o \ PubkeyAuthentication=no user@host
• gnome, systemd dependency madness on ubuntu from shallowsky.com
  install equivs package
  run equivs-control, edit Package to package name, Version to 2:42 or something high, Maintainer to some name and <email@address.com>
  run equivs-build and install with dpkg -i <packagename>
  Use this to remove e.g. evolution-data-server
• package that a file belongs to dpkg -S file
• disable bootup of services on ubuntu update-rc.d cups remove (note: at least for now, systemd might change this)
• let apt ignore dependencies edit /var/lib/dpkg/status and remove Depends: entries (note: only short-term, this will get overwritten)
• flask install directly from github tarball to use flask * commands
• use sudo without entering your password all the time sudo visudo
user ALL=NOPASSWD: ALL
  however, this is very insecure, so it is better to just whitelist specific commands:
  user ALL=(root) NOPASSWD: /usr/bin/command [args]
%group ALL=(root) NOPASSWD: /path/to/comand [args]
• sudo password asterisks Defaults env_reset,pwfeedback in visudo
• pip problems cryptography need libssl-dev on ubuntu
  cffi needs python-cffi
Pillow needs libjpeg-dev
• stop mongodb from allocating 3GB of space (shutdown mongodb), edit /etc/mongodb.conf and insert smallfiles=true
  remove /var/lib/mongodb/journal/\*, (startup mongodb)
• partition woes
  • partprobe /dev/sdX or kpartx /dev/sdX
  • try hdparm -Z /dev/sdX or sfdisk -R /dev/sdX
• QT5
  • set ENVVAR with export QT_AUTO_SCREEN_SCALE_FACTOR=0.8 in /etc/profile.d/qt.sh or try with .bash_profile, .profile, .bashrc etc.
  • install libqt5libqgtk2 qt5-image-formats-plugins qtwayland5 qt5-qmltooling-plugins
  • run qt5ct

Power management and laptop handling

• power saving
  • install laptop-mode-tools or, preferably tlp
  • install powertop and set toggles
  • utilize your desktop environment’s built-in solution, or run pm-suspend, pm-hibernate or pm-suspend-hybrid like an animal
  • edit /etc/systemd/logind.conf and set HandleLidSwitch=ignore, then run systemctl restart systemd-logind.service clean up boot with update-rc.d or disable systemd units, use the Startup applications program to manage autostarts

gsettings

• Reset app grid
  gsettings reset org.gnome.shell app-picker-layout
• List tracker settings
  gsettings list-recursively | grep -i org.freedesktop.Tracker | sort | uniq

systemctl –user

• Get DBus access as su-d normal user
  export DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/$(id -u)/bus See Start systemd units at boot time

Fix problems pertaining to specific hardware configurations

• wifi on Asus EEE PC 1000H

  install firmware-ralink

  edit /etc/network/interfaces

    auto wlan0
    iface wlan0 inet dhcp
        wpa-ssid YOUR-SSID-HERE
        wpa-psk YOUR-PASSWORD-HERE
    /etc/init.d/networking restart
    iwlist
    iwconfig -a
    iwconfig wlan0 down
    ifconfig wlan0 hw ether 01:23:45:67:89:00
    ifconfig wlan0 up
  

  I don’t use my old netbook any more, I hear OpenBSD has good drivers for wireless networking. Go have fun installing that one.

• mitigate problems with intel ac 7260

  edit /etc/modprobe.d/iwlwifi-11n.conf and add options iwlwifi 11n_disable=8 (or disable=1)
  (at runtime) run sudo rmmod {iwlmvm,iwlwifi}, then sudo modprobe iwlwifi 11n_disable=1, sudo modprobe iwlmvm
  edit /etc/default/crda and change REGDOMAIN= to your country code
  note: this still does not fix all problems, especially eduroam is still wonky

• bluetooth
  purge gnome-bluetooth
  install bluez-tools bluetooth blueman bluez-hcidump python-bluez and pulseaudio-module-blueooth (for audio)
  run sudo hciconfig hci0 reset and sudo pactl load-module module-blueotooth-discover
  if applicable, edit /etc/bluetooth/main.conf