ix5 tech

Pandas: Apply function per row with context

Wed, 12 Apr 2023 17:44:36 +0200

def get_affected_days(start, finish):
    pass

pairs = df.apply(
    lambda row: get_affected_days(row['started_at'], row['finished_at']),
    axis=1
)

Plotly

Fri, 31 Mar 2023 22:06:28 +0200

Sane plotly defaults, margins, with legends inside chart

import pandas as pd
import numpy as np
import plotly.express as px
import plotly.graph_objects as go
plotly_config = dict(
    autosize=True,
    margin=dict(l=0, r=0, b=0, t=0, pad=0),
)
fig.update_layout(**plotly_config)
fig.update_layout(legend=dict(
    yanchor="top",
    y=0.99,
    xanchor="center",
    x=0.1,
))
_filename = 'filename'
fig.write_image(f'{_filename}.svg')
fig.show()

Setting plotly labels

fig = px.bar(df['url'].resample('1w').count(), labels={'value': 'Disruptions'}).update_layout(
    autosize=True,
    margin=dict(l=0, r=0, b=0, t=0, pad=0),
).update_layout(showlegend=False)
fig.write_image("image.svg")
fig.show()

Pivot operations and MultiIndex to flat columns

important = russia_bans
important.loc[~russia_bans['product'].str.startswith(_cols), 'product'] = 'Other'

sanctions_with_other = (important
    .groupby(
        by=[important['date.implemented'], important['product'].str[:2]],
        group_keys=True,
    )
    .count()
    .rename(columns={'product': 'product.count'})
    .reset_index('product')
    .loc[:, ['product', 'product.count']]
)
sanctions_with_other = pd.DataFrame(
    sanctions_with_other.pivot(columns=['product'])
)
sanctions_with_other.columns = sanctions_with_other.columns.get_level_values(1)
# Only for including "Other"
sanctions_with_other = sanctions_with_other.loc[:, ('Ot',) + _cols].rename(columns={'Ot': 'Other'})
sanctions_with_other.index = pd.to_datetime(sanctions_with_other.index)

Helper: Embedding plotly charts as static images for (PDF) export

EXPORT_TO_PDF = False

import os
from IPython.display import Image
def plotly_img_save_and_inline(fig, filename):
    fig.write_image(f'{filename}.svg')
    fig.write_image(f'{filename}.png', width=1000)
    if EXPORT_TO_PDF:
        print('To pdf, saving')
        img = Image(data=f'{filename}.png', width=1000)
        # Declutter by removing the png file from disk again,
        # we only need vector graphics
        os.remove(f'{filename}.png')
        return img
    print('Inline')
    fig.show()

fig = px.line(
    strikes.resample('d').count()
)
_filename = 'image'
plotly_img_save_and_inline(fig, _filename)

Changed Zoho IMAP Access for Custom Domain

Fri, 24 Mar 2023 23:38:10 +0100

Zoho, a suite of business applications allowing, among others, having an email account using your own domain name, silently changed their access procedure, leading to Login failed messages in e.g. Thunderbird.

The gist of it is that you need to use imappro.zoho.eu for your IMAP server and smtppro.zoho.eu for outgoing SMTP server to continue using your own domain name.

The updated guide is at the Help pages for Zoho mail.

Incoming server hostname:
If you are an organization user with a domain-based account, provide imappro.zoho.eu, select 993 for Port from the drop-down, and Normal Password for authentication.

For Outgoing server hostname:
If you are an organization user with a domain-based account, provide smtppro.zoho.eu, select 465 for Port from the drop-down, and Normal Password for authentication.

GNOME: Clear lock timer

Tue, 21 Feb 2023 22:27:40 +0100

Run faillock --user <username> --reset.

Sources: reddit, reddit

Zotero: Eliminate imported duplicates

Tue, 14 Feb 2023 07:43:36 +0100

After exporting a “collection” in Zotero to send to an acquaintance, I imported the exported file to check that the archive file was valid, choosing to “link” the imported files rather than importing them (i.e., storing them inside the Zotero data folder).

The import had gone smoothly and I deleted the collection again, but foolishly did not chose to delete the collection contents. This left me with a large number of duplicate items scattered around, as “orphan” items with no parents.

At this point, I should have used the Unfiled Items pane to find those duplicates and chosen to permanently delete them. But, as I had not noticed the duplicate items, I forgot to do this and over time, more unfiled items filled up with unrelated items (e.g. through saving snapshots from the web extension without selecting a collection).

I then noticed the duplicated items and first tried the “merge” function via the Duplicate Items pane. This left me with no duplicate items any more, but meant that inside the items previously affected by duplication, the notes, PDF attachments and website snapshots were duplicated, with duplicates pointing to the data export folder.

After a little digging, it turns out Zotero marks imported “linked” attachments and especially internally with a “link mode” of 2:

LINK_MODE_IMPORTED_FILE = 0;
LINK_MODE_IMPORTED_URL = 1;
LINK_MODE_LINKED_FILE = 2;
LINK_MODE_LINKED_URL = 3;
LINK_MODE_EMBEDDED_IMAGE = 4;

Using the Zotero Javascript API, I cleaned up the duplicates.

Go to Tools -> Developer -> Run JavaScript and enter this code, uncheck Run as async function, click Run. Adjust FS_LOC to the start of the full path of the linked imported Zotero collection, e.g. C:/ or C:\ for Windows.

const ZoteroPane = Zotero.getActiveZoteroPane();
const selectedItems = [ZoteroPane.getSelectedItems()[0]];
const attachmentsToDelete = [];
const itemsWithDuplicateNotes = [];
const itemsWithDuplicateLinks = [];

// Change this to 'C:/' for Windows or '/User' for Mac
const FS_LOC = '/home'

for (let item of selectedItems) {
  if (item.isRegularItem()) {
    let attachmentIDs = item.getAttachments();
    let links = [];
    for (let id of attachmentIDs) {
      let attachment = Zotero.Items.get(id);
      let url = attachment.getField('url');
      // Remove "linked" attachments
      if (attachment['attachmentLinkMode'] == 2 &&
        attachment['attachmentPath'].startsWith(FS_LOC)) {
        // Could also check if attachment with same name already exists
        attachmentsToDelete.push(attachment);
        attachment.erase();
        attachment.saveTx();
        Zotero.Fulltext.indexItems([attachment.id])
      }
      // Remove duplicate linked url attachments
      if (attachment.getAttachmentLinkMode() == 3 && links.includes(url)) {
        itemsWithDuplicateLinks.push({
          f: attachment,
          url: url,
        });
        attachment.erase();
        attachment.saveTx();
        Zotero.Fulltext.indexItems([attachment.id])
      }
      links.push(url);
    }
    let notes = item.getNotes();
    let noteContents = []
    for (let id of notes) {
      let note = Zotero.Items.get(id);
      let noteHTML = note.getNote();
      // Remove duplicates notes
      if (noteContents.includes(noteHTML)) {
        itemsWithDuplicateNotes.push({
          text: noteHTML,
          id: item.getID(),
          title: item.getDisplayTitle(),
        });
        note.erase();
        note.saveTx();
        Zotero.Fulltext.indexItems([note.id])
      }
      noteContents.push(noteHTML);
    }
  }
}

//attachmentsToDelete;
//itemsWithDuplicateNotes;
//itemsWithDuplicateLinks;

For all methods of the Zotero.Item object, see Zotero’s source code for item.js.

spaCy Terms Cheat Sheet

Thu, 22 Dec 2022 01:35:24 +0100

I am currently learning to deal with Natural Language Processing (NLP) using spaCy. These are the terms that I often end up having to look up.

I wrote a small bash helper function for myself:

# spaCy explain term
function explain() {
    _query=$1
    # activate virtualenv here, if necessary
    python -c "from spacy import explain; print(explain('"$_query"'))"
}

Glossary

Extracted from spaCy: glossary.py and polished up a bit. spaCy license: MIT.

POS tags: Universal POS Tags

See universaldependencies.org/u/pos/

Term	Explanation
`ADJ`	adjective
`ADP`	adposition
`ADV`	adverb
`AUX`	auxiliary
`CONJ`	conjunction
`CCONJ`	coordinating conjunction
`DET`	determiner
`INTJ`	interjection
`NOUN`	noun
`NUM`	numeral
`PART`	particle
`PRON`	pronoun
`PROPN`	proper noun
`PUNCT`	punctuation
`SCONJ`	subordinating conjunction
`SYM`	symbol
`VERB`	verb
`X`	other
`EOL`	end of line
`SPACE`	space

POS tags (English)

From OntoNotes 5 / Penn Treebank, see ling.upenn.edu

Term	Explanation
`.`	punctuation mark, sentence closer
`,`	punctuation mark, comma
`-LRB-`	left round bracket
`-RRB-`	right round bracket
`\```	opening quotation mark
`"`	: closing quotation mark
`''`	closing quotation mark
`:`	punctuation mark, colon or ellipsis
`$`	symbol, currency
`#`	symbol, number sign
`AFX`	affix
`CC`	conjunction, coordinating
`CD`	cardinal number
`DT`	determiner
`EX`	existential there
`FW`	foreign word
`HYPH`	punctuation mark, hyphen
`IN`	conjunction, subordinating or preposition
`JJ`	adjective (English), other noun-modifier (Chinese)
`JJR`	adjective, comparative
`JJS`	adjective, superlative
`LS`	list item marker
`MD`	verb, modal auxiliary
`NIL`	missing tag
`NN`	noun, singular or mass
`NNP`	noun, proper singular
`NNPS`	noun, proper plural
`NNS`	noun, plural
`PDT`	predeterminer
`POS`	possessive ending
`PRP`	pronoun, personal
`PRP$`	pronoun, possessive
`RB`	adverb
`RBR`	adverb, comparative
`RBS`	adverb, superlative
`RP`	adverb, particle
`TO`	‘infinitival to
`UH`	interjection
`VB`	verb, base form
`VBD`	verb, past tense
`VBG`	verb, gerund or present participle
`VBN`	verb, past participle
`VBP`	verb, non-3rd person singular present
`VBZ`	verb, 3rd person singular present
`WDT`	wh-determiner
`WP`	wh-pronoun, personal
`WP$`	wh-pronoun, possessive
`WRB`	wh-adverb
`SP`	space (English), sentence-final particle (Chinese)
`ADD`	email
`NFP`	superfluous punctuation
`GW`	additional word in multi-word expression
`XX`	unknown
`BES`	auxiliary ‘be’
`HVS`	forms of ‘have’
`_SP`	whitespace

POS Tags (German)

From TIGER Treebank, see ims.uni-stuttgart.de

Term	Explanation
`$(`	other sentence-internal punctuation mark
`$,`	comma
`$.`	sentence-final punctuation mark
`ADJA`	adjective, attributive
`ADJD`	adjective, adverbial or predicative
`APPO`	postposition
`APPR`	preposition; circumposition left
`APPRART`	preposition with article
`APZR`	circumposition right
`ART`	definite or indefinite article
`CARD`	cardinal number
`FM`	foreign language material
`ITJ`	interjection
`KOKOM`	comparative conjunction
`KON`	coordinate conjunction
`KOUI`	‘subordinate conjunction with zu
`KOUS`	subordinate conjunction with sentence
`NE`	proper noun
`NNE`	proper noun
`PAV`	pronominal adverb
`PROAV`	pronominal adverb
`PDAT`	attributive demonstrative pronoun
`PDS`	substituting demonstrative pronoun
`PIAT`	attributive indefinite pronoun without determiner
`PIDAT`	attributive indefinite pronoun with determiner
`PIS`	substituting indefinite pronoun
`PPER`	non-reflexive personal pronoun
`PPOSAT`	attributive possessive pronoun
`PPOSS`	substituting possessive pronoun
`PRELAT`	attributive relative pronoun
`PRELS`	substituting relative pronoun
`PRF`	reflexive personal pronoun
`PTKA`	particle with adjective or adverb
`PTKANT`	answer particle
`PTKNEG`	negative particle
`PTKVZ`	separable verbal particle
`PTKZU`	‘zu
`PWAT`	attributive interrogative pronoun
`PWAV`	adverbial interrogative or relative pronoun
`PWS`	substituting interrogative pronoun
`TRUNC`	word remnant
`VAFIN`	finite verb, auxiliary
`VAIMP`	imperative, auxiliary
`VAINF`	infinitive, auxiliary
`VAPP`	perfect participle, auxiliary
`VMFIN`	finite verb, modal
`VMINF`	infinitive, modal
`VMPP`	perfect participle, modal
`VVFIN`	finite verb, full
`VVIMP`	imperative, full
`VVINF`	infinitive, full
`VVIZU`	‘infinitive with zu
`VVPP`	perfect participle, full
`XY`	non-word containing non-letter

POS Tags (Chinese)

From OntoNotes / Chinese Penn Treebank, see repository.upenn.edu

Term	Explanation
`AD`	adverb
`AS`	aspect marker
`BA`	把 in ba-construction
`CS`	subordinating conjunction
`DEC`	的 in a relative clause
`DEG`	associative 的
`DER`	得 in V-de const. and V-de-R
`DEV`	地 before VP
`ETC`	for words 等, 等等
`IJ`	interjection
`LB`	被 in long bei-const
`LC`	localizer
`M`	measure word
`MSP`	other particle
`NR`	proper noun
`NT`	temporal noun
`OD`	ordinal number
`ON`	onomatopoeia
`P`	preposition excluding 把 and 被
`PN`	pronoun
`PU`	punctuation
`SB`	被 in short bei-const
`VA`	predicative adjective
`VC`	是 (copula)
`VE`	有 as the main verb
`VV`	other verb

Noun chunks

Term	Explanation
`NP`	noun phrase
`PP`	prepositional phrase
`VP`	verb phrase
`ADVP`	adverb phrase
`ADJP`	adjective phrase
`SBAR`	subordinating conjunction
`PRT`	particle
`PNP`	prepositional noun phrase

Dependency Labels (English)

From ClearNLP / Universal Dependencies, see github.com/clir/clearnlp-guidelines

Term	Explanation
`acl`	clausal modifier of noun (adjectival clause)
`acomp`	adjectival complement
`advcl`	adverbial clause modifier
`advmod`	adverbial modifier
`agent`	agent
`amod`	adjectival modifier
`appos`	appositional modifier
`attr`	attribute
`aux`	auxiliary
`auxpass`	auxiliary (passive)
`case`	case marking
`cc`	coordinating conjunction
`ccomp`	clausal complement
`clf`	classifier
`complm`	complementizer
`compound`	compound
`conj`	conjunct
`cop`	copula
`csubj`	clausal subject
`csubjpass`	clausal subject (passive)
`dative`	dative
`dep`	unclassified dependent
`det`	determiner
`discourse`	discourse element
`dislocated`	dislocated elements
`dobj`	direct object
`expl`	expletive
`fixed`	fixed multiword expression
`flat`	flat multiword expression
`goeswith`	goes with
`hmod`	modifier in hyphenation
`hyph`	hyphen
`infmod`	infinitival modifier
`intj`	interjection
`iobj`	indirect object
`list`	list
`mark`	marker
`meta`	meta modifier
`neg`	negation modifier
`nmod`	modifier of nominal
`nn`	noun compound modifier
`npadvmod`	noun phrase as adverbial modifier
`nsubj`	nominal subject
`nsubjpass`	nominal subject (passive)
`nounmod`	modifier of nominal
`npmod`	noun phrase as adverbial modifier
`num`	number modifier
`number`	number compound modifier
`nummod`	numeric modifier
`oprd`	object predicate
`obj`	object
`obl`	oblique nominal
`orphan`	orphan
`parataxis`	parataxis
`partmod`	participal modifier
`pcomp`	complement of preposition
`pobj`	object of preposition
`poss`	possession modifier
`possessive`	possessive modifier
`preconj`	pre-correlative conjunction
`prep`	prepositional modifier
`prt`	particle
`punct`	punctuation
`quantmod`	modifier of quantifier
`rcmod`	relative clause modifier
`relcl`	relative clause modifier
`reparandum`	overridden disfluency
`root`	root
`ROOT`	root
`vocative`	vocative
`xcomp`	open clausal complement

Dependency labels (German)

From TIGER Treebank, see ims.uni-stuttgart.de

currently missing: ‘cc’ (comparative complement) because of conflict with English labels

Term	Explanation
`ac`	adpositional case marker
`adc`	adjective component
`ag`	genitive attribute
`ams`	measure argument of adjective
`app`	apposition
`avc`	adverbial phrase component
`cd`	coordinating conjunction
`cj`	conjunct
`cm`	comparative conjunction
`cp`	complementizer
`cvc`	collocational verb construction
`da`	dative
`dh`	discourse-level head
`dm`	discourse marker
`ep`	expletive es
`hd`	head
`ju`	junctor
`mnr`	postnominal modifier
`mo`	modifier
`ng`	negation
`nk`	noun kernel element
`nmc`	numerical component
`oa`	accusative object
`oc`	clausal object
`og`	genitive object
`op`	prepositional object
`par`	parenthetical element
`pd`	predicate
`pg`	phrasal genitive
`ph`	placeholder
`pm`	morphological particle
`pnc`	proper noun component
`rc`	relative clause
`re`	repeated element
`rs`	reported speech
`sb`	subject
`sbp`	passivized subject (PP)
`sp`	subject or predicate
`svp`	separable verb prefix
`uc`	unit component
`vo`	vocative

Named Entity Recognition

From OntoNotes 5, see catalog.ldc.upenn.edu

Term	Explanation
`PERSON`	People, including fictional
`NORP`	Nationalities or religious or political groups
`FACILITY`	Buildings, airports, highways, bridges, etc.
`FAC`	Buildings, airports, highways, bridges, etc.
`ORG`	Companies, agencies, institutions, etc.
`GPE`	Countries, cities, states
`LOC`	Non-GPE locations, mountain ranges, bodies of water
`PRODUCT`	Objects, vehicles, foods, etc. (not services)
`EVENT`	Named hurricanes, battles, wars, sports events, etc.
`WORK_OF_ART`	Titles of books, songs, etc.
`LAW`	Named documents made into laws.
`LANGUAGE`	Any named language
`DATE`	Absolute or relative dates or periods
`TIME`	Times smaller than a day
`PERCENT`	‘Percentage, including %
`MONEY`	Monetary values, including unit
`QUANTITY`	Measurements, as of weight or distance
`ORDINAL`	‘first
`CARDINAL`	Numerals that do not fall under another type

Named Entity Recognition (Wikipedia)

See Article 1 and Article 2

Term	Explanation
`PER`	Named person or family.
`MISC`	Miscellaneous entities, e.g. events, nationalities, products or works of art

NER Miscellaneous

From github.com/ltgoslo/norne:

Term	Explanation
`EVT`	Festivals, cultural events, sports events, weather phenomena, wars, etc.
`PROD`	Product, i.e. artificially produced entities including speeches, radio shows, programming languages, contracts, laws and ideas
`DRV`	Words (and phrases?) that are dervied from a name, but not a name in themselves, e.g. ‘Oslo-mannen’ (’the man from Oslo’)
`GPE_LOC`	Geo-political entity, with a locative sense, e.g. ‘John lives in Spain’
`GPE_ORG`	Geo-political entity, with an organisation sense, e.g. ‘Spain declined to meet with Belgium’

Launching scripts from GNOME shortcuts

Sat, 24 Sep 2022 11:01:24 +0200

As of at least 2022-09-24, launching small scripts that spawn a rofi interface via the GNOME shell shortcuts interface would fail for me with the following message:

app-gnome-\x7e\x2fcontrol.sh-3565.scope: No PIDs left to attach to the scope's control group, refusing.
app-gnome-\x7e\x2fcontrol.sh-3565.scope: Failed with result 'resources'.
Failed to start Application launched by gsd-media-keys.

My control script would be executable and located at ~/control.sh and launch something like rofi -dmenu -i <choices>. I had the association (e.g. for <Super>-i) entered through the GNOME settings keyboard shortcuts as command ~/control.sh and this used to work just fine for years.

Researching the error message, it now seems you are supposed to set your own “scope” (purportedly somehow related to cgroups) and use systemd-run.

The fix was to change the command ~/control.sh to an absolute path, prefixed by systemd-run:

systemd-run --scope --user /home/<user>/control.sh

Anki Webview Issues

Fri, 15 Apr 2022 15:27:10 +0200

As of 2022-04-15, Anki will open on my up-to-date Arch Linux install, but show a blank page instead of my decks.

$ pacman -Q anki qt5-base qt6-base
anki-official-binary-bundle 2.1.49-1
qt5-base 5.15.3+kde+r137-1
qt6-base 6.3.0-1

I suspected an issue with the webview implementation, since menus, preferences and the like still work.

A topic on the [Ankiweb Forums][forums] surfaces the suggestion to disable the Qt webengine seccomp filter: QTWEBENGINE_CHROMIUM_FLAGS="--disable-seccomp-filter-sandbox"

Add the environment variable to your .bashrc or wherever or just prepend it to any launches of Anki:

QTWEBENGINE_CHROMIUM_FLAGS="--disable-seccomp-filter-sandbox" anki

Note: Now simply using anki-bin AUR package instead of anki-official-binary-bundle

[forums]: https://forums.ankiweb.net/t/webviews-blank-on-linux/1433/6)

Netgear R6120 OpenWRT

Tue, 16 Feb 2021 12:29:02 +0100

Get nmrpflash

$ git clone https://github.com/jclehner/nmrpflash.git
$ cd nmrpflash/
$ make

Check interfaces

./nmrpflash -L
enp0s31f6        192.168.1.8      8c:16:45:22:f5:71
wlp2s0           192.168.1.6      74:e5:f9:e7:79:26

For me, enp0s31f6 showed 0.0.0.0 as IP:

enp0s31f6        0.0.0.0.0        8c:16:45:22:f5:71

Flash

Plug LAN cable from computer into LAN port 1 of the router. NOT into the WAN port.

Turn off router.

$ sudo ./nmrpflash -i enp0s31f6 -f ~/downloads/netgear-openwrt/factory/openwrt-ramips-mt76x8-netgear_r6120-squashfs-factory.img
Waiting for physical connection.

Turn on router

Advertising NMRP server on enp0s31f6 ...

Wait for flash to finish, restart router, you should be able to ssh into it like the guide says.

Caveats and bugs

There are some things that somehow fail for me from time to time.

It seems that updating to a newer OpenWRT version by flashing a new sysupgrade file completely breaks any access, both via web and via ssh.

While using nmrpflash, this might happen:

Advertising NMRP server on enp0s31f6 ...
Received configuration request from 00:00:00:00:00:00.
Sending configuration: 10.164.183.252/24.
Received upload request without filename.
Uploading openwrt-ramips-mt76x8-netgear_r6120-squashfs-factory.img ... errno=99
bind: Cannot assign requested address

Also, make sure to really plug into LAN port 1, not any other!

N.b.: The following is wrong imo, better to let the tool use automatic IP:

sudo ./nmrpflash -i enp0s31f6 -f ~/downloads/netgear-openwrt/factory/openwrt-ramips-mt76x8-netgear_r6120-squashfs-factory.img -a 192.168.1.200

Languages

Sun, 07 Feb 2021 20:57:42 +0100

For reasons that (I swear!) have nothing to do with my nationality, I have always avoided acquiring any modicum of French language skills. I took Latin in high school and while at university, Spanish sounded like the clearly superior investment, especially for my social life. Hearing people speak French, I was having trouble holding it together, because, obviously this was a fake language designed to make you emit as many silly noncoherent noises as possible.

Similarly, I had put off learning Javascript until very recently. The language of “sprinkle some ads on it”, the language of ADHD-riddled kids who import a package to left-align a number, the language that makes the web an increasingly unpleasant experience. Javascript did not appeal to me.

l’ordinateur - the computer

$ 'wtf' instanceof String
> false
$ typeof 'wtf'
> 'string'

As happens from time to time, it is not by choice but rather by necessity that you have to pick up a skill. So while I am now forced to learn French for future foreign assignments, I also had to delve into the murky waters that are the Javascript “ecosystem”.

Ubuntu Touch & just a small bug

Flashing custom firmware on mobile phones has historically been a manual and painful process. All the more was I happy to hear that my buddy Konrad had worked hard to submit my favourite device, the Sony Xperia XZ, for inclusion in Ubuntu Touch’s fabulous interactive GUI installer.

There was just a small issue with the (Electron/Javascript-based) installer failing to verify a specific file.

error: Error: core:manual_download: Error: checksum mismatch

I know about Android, Linux administration and hardware debugging. I know where to look and I can solve issues rather quickly. From investing quite a bit of time, I’ve seen a lot of errors and know how to look for the part in the whole source tree that might be responsible.

With Javascript however - or any new and unfamiliar environment - hunting down bugs takes me ages. It’s incredibly frustrating, like you’re a preschooler being made to navigate complex algebra.

le magazine - the magazine; le magasin - the store

There are so many convenitions, syntax oddities and build/ecosystem tooling to learn, none of which I could give a rat’s ass about. I am not interested in any of the individual tools involved, I just want to solve a problem that has been caused by them.

Not to be deterred though, I dug in. This, treasured reader, is the story of that problem.

Promises and Lies

The code I am about to reference makes heavy use of asynchronous Promise functionality.

A basic usage pattern would look like this:

const prom = new Promise( (resolutionFunc, rejectionFunc) => {
    console.log("Look, ma!");
    return true;
});

With this block, a new Promise object would be created (and immediately executed). It executes the log statement and calls back to one of the two callback functions, one for success (fulfilled) and one for failure (rejected).

More commonly, developers use .then to handle promise resolution:

const myPromise =
  (new Promise(resolutionFunc))
  .then(() =>
    // handle first fulfilled, return second fulfilled
    dostuff;
  )
  .then(() =>
    // handle second fulfilled, return something
    domorestuff;
  )
  .then(resolved => {
    // handle domorestuff success
  }, rejected {
    // handle domorestuff failure
  })

This is called a “chained promise” and is basically a try/catch, but for Javascript and asynchrooonouuuuus.

in, an, en, eau, eux - all pronounced pretty much the same

Having learned the semantics of promises, this block of code looked perfectly innocent to me:

return new Promise(resolve => {
  // get user input for path - simplified here
  return "/home/user/path/file.zip";
})
.then(downloadedFilePath => {
  fs.ensureDir(
      path.join(this.cachePath, this.props.config.codename, group)
    )
    .then(() =>
      fs.copyFile(
        downloadedFilePath,
        path.join(
          this.cachePath,
          this.props.config.codename,
          group,
          file.name
        )
      )
    );
})
.then(() =>
  checkFile(
    {
      checksum: file.checksum,
      path: path.join(
        this.cachePath,
        this.props.config.codename,
        group,
        file.name
      )
    },
    true
  )
)
.then(ok => {
  if (ok) {
    return ok;
  } else {
    throw new Error("checksum mismatch");
  }
});

Simple, right? The first promise passes the downloadedFilePath to the ensureDir func, which then triggers copyFile. Once the block is resolved, checkFile is run on the copied file. All functions politely wait for the previous func to resolve via the .then() statement.

Or do they? Has this all been a lie?

Les Nombres

Similarly, lulled into a false sense of security, I thought I might approach learning French numbers above the single-digit range. Une, deux, trois… that’s practically Spanish, right? So how hard can those until 100 be?

4, 5, 6, 7, 8, 9, 10 - quatre, cinq, six, sept, huit, neuf, dix

So far, so good.

11, 12, 13, 14, 15, 16 - onze, douze, treize, quatorze, quinze, seize

Great!

17, 18, 19, 20, 21 - dix-sept, dix-huit, dix-neuf, vingt, vingt-et-un

Okay, small inconsistency, but I can get used to that…

30, 40, 50 - trente, quarante, cinquante

So from 20 onwards, there’s a nice pattern again. Sweet.

Continuing on with Javascript.

The intended flow of the specific part of the program I was looking at was:

If file already present, checksum and compare with predefined checksum, on success skip all the next steps
If checksum mismatch or file not present:
Let user pick file with system file picker
Return path of picked file as promise
Ensure target dir inside cache is present
Copy user-picked file to cache location (~/.cache/ubports/<device>/firmware/)
Checksum newly copied file
Unpack newly copied file (extract .img from zip file)

To reiterate, the bug which I was encountering looked like this:

error: Error: core:manual_download: Error: checksum mismatch

I quickly narrowed down the issue to a call of checkFile() in src/core/plugins/core/plugin.js.

At first, I thought maybe the crypto library was failing to call the right hash functions, but when a file was already in the right place, the hashes were verified as correct.

I then quickly disabled the checksum verification after the file had been copied and was met with a failure of the unpack step, which complained about a missing input file. The file was being copied alright, but it seems the checksum and unpack steps were winning a race against the copyFile() operation even though I thought they were running in sequence.

After some further digging, maybe what I needed was the synchronous variant of copyFile? Swapped it for copyFileSync, but no change in results.

I added an artificial delay of 100ms after the whole ensuredir/copy block and before the checksum, and everything went smoothly. So, why was the copy operation not being finished in time?

Having spent way too much time on this issue already, there must’ve been something wrong with all this asynchronous promising going on…

Shadows looming

Helpfully, Mozilla’s excellent MDN includes an admonition about the subtle ways you can shoot yourself in the foot:

An action can be assigned to an already “settled” promise. In that case, the action (if appropriate) will be performed at the first asynchronous opportunity. Note that promises are guaranteed to be asynchronous. Therefore, an action for an already “settled” promise will occur only after the stack has cleared and a clock-tick has passed. The effect is much like that of setTimeout(action,10).

and this little snippet:

const promiseA = new Promise( (resolutionFunc,rejectionFunc) => {
    resolutionFunc(777);
});
// At this point, "promiseA" is already settled.
promiseA.then( (val) => console.log("asynchronous logging has val:",val) );
console.log("immediate logging");

// produces output in this order:
// immediate logging
// asynchronous logging has val: 777

Not quite the problem I was facing, but a harbinger of things to come.

Now for the final nail in the coffin: Implied return statements.

This:

() => 1 // returns 1

is the same as this:

() => { return 1; } // returns 1

but not the same as this:

() => { 1; } // returns nothing

With all the pieces of the puzzle collected, we are ready to solve this riddle.

But not before going back again to the French numbers. We left off at 50 and I’m excited for what lies ahead:

60 - soixante

Excellent. This is beginning to make a lot of sense to me.

70 - soixante-dix (60 + 10)

What. The. Fuck.

71 - soixante et onze (60 + 11)

So from 70 on, they just use 60 as a base? Alright, very confusing, but I’m in it for the long haul, I can deal with that. How do the numbers continue after 70?

80 - quatre-vingts (4 x 20)

At this point I’m scrambling to cross-reference, because clear as day my study material has been written by a prankster. That can’t possibly be right, no?

90 - quatre-vingt-dix ((4 x 20) + 10)

By now I have consulted several sources and confirmed this is indeed how the French count. We might have been premature in mocking the Anglos for their imperial units. This is worse.

But I’ll leave les nombres be for now.

Deadly void

Aaaand back into the joyous world of machines! Let’s look at the mutinous code block again.

return new Promise(resolve => {
  // Return downloadedFilePath
  return "/home/user/path/file.zip";
})
// block start
.then(downloadedFilePath => {
  fs.ensureDir("/home/user/path")
    .then(() =>
      fs.copyFile(
        downloadedFilePath,
        "/home/user/.cache/path/file.zip"
      )
    );
}) // block end
.then(() =>
  checkFile();
)
.then(ok => {
  if (ok) { // handle stuff
}

I’ve simplified it for better readibility. Astute readers will have spotted the error by now:

The whole “block” was never going to resolve its promise with an actual return value (inside curly braces but no return statement)
checkFile() was being called before any of the promise functions inside the previous block had the chance to resolve
The copied file was not yet available
checkFile tried to verify a non-existent path
Checksum mismatch error.

This is the sort of bug that makes you want to strangle a kitten. To shout obscenities à la Captain Haddock. To permanently disable the power grid and return to farming sheep.

parle, parles, parle - I speak, you speak, he speaks - all pronounced the same

Solution & Reflections

The first solution I landed on was to make the whole block explicitly return something, in this case the value of the last chained promise:

.then(downloadedFilePath => {
  // Added return statement below
  return fs.ensureDir("/home/user/path")
    .then(() =>
      fs.copyFile(
        downloadedFilePath,
        "/home/user/.cache/path/file.zip"
      )
    );
})

I finally settled (after consultation with Marijn) on removing the curly braces to convert the block into an implied return, as that would fit better with the syntax of the surrounding code:

// Removed curly braces:
.then(downloadedFilePath =>
  fs.ensureDir("/home/user/path")
    .then(() =>
      fs.copyFile(
        downloadedFilePath,
        "/home/user/.cache/path/file.zip"
      )
    )
)

I submitted a Pull Request, as you do nowadays, and waited for the fix to propagate to the users.

The saddest part about all this is that none of these functions need to be asynchronous. They are executed in sequence and the UI blocks until they return anyway.

The developer who wrote all this, Jan Sprinz, is a perfectly nice guy, and if it wasn’t for him, people would probably still have to fumble around with arcane command line tools to install Ubuntu Touch. I can’t fault him for seeing common patterns in Javascript development and, thinking they’re good practices, writing the whole function using promises.

In conclusion, programming is terrible, hug your loved ones, go outside and enjoy the sun.

P.S.

Huge thanks to my buddy Marijn for proofreading and giving corrections!

Customizing Isso for my needs

Tue, 19 Jan 2021 00:00:00 +0000

Some functionality that is not (yet) available with upstream Isso.

All features mentioned in this post are available on this frozen branch.

I run a local development server using the following command:

ISSO_SETTINGS=share/isso-dev.conf \
    venv-isso/bin/gunicorn \
    -b 0.0.0.0:8080 \
    -w 1 \
    --preload isso.run \
    --worker-tmp-dir /dev/shm

Thanks button - A small button with a counter to simply say “Thanks” instead of posting a comment. Has been suggested upstream but so far no one attempted the work.

Telegram integration - Notify me via Telegram when a new comment is posted. Gotta find a way to lose the requests dependency and POST directly. Also integrated with the custom Thanks button functionality via signals.

Use with these settings in your isso.cfg

[telegram]
token = <your-token>
chat_id = <chat-id> # Can be obtained from @getidsbot

Absolute URIs for threads helps with hosting multiple sites on the same commenting instance.

Use pre for comments in admin area (PR opened) means that comments will be rendered as source in admin and be properly contenteditable. Used with this fix for PR #470.

Restore comment after cancelling edit in admin, from PR #470.

js: Remove unused languages results in a reduction of embed.min.js from ~65kB to ~43kB.

Improve cookie SameSite/Secure handling means no more warnings in Firefox’ console and future-proofing.

Tooltip saying email will not be public - small tooltip in frontend, makes users less squeamish about entering E-mail

Require an edit cookie to be able to view a comment before editing - not really necessary since I do not have private posts, but nice to have nonetheless, helps a bit against db scraping directly. From this PR, with more background here.

Larger type on mobile, “Markdown supported” - Just a small adjustment to the comment area

Makefile: Use sassc - Does what it says. nodejs is a madhouse, no need to pull in all those dependencies when sassc can do the job just as well.

Skype for Linux - Disable autostart

Fri, 08 Jan 2021 00:00:00 +0000

TL;DR

Insert "app.autoStartEnabled":false into ~/.config/skypeforlinux/settings.json.

Intro

The native Skype app for Linux - I guess I should be thankful there even is one - auto-starts itself on boot¹, and there is no way to turn off that behaviour as a guest because the Settings pane can only be accessed by logged-in users.

Auto-start works on Linux desktops works by inserting a .desktop file in a config folder, so Skype’s is at ~/.config/autostart/skypeforlinux.desktop. Even if you delete the file, Skype just re-creates it every time it launches. You could just empty and then chmod a-w the desktop file, but that’s lame and Skype might change their auto-start behaviour in the future using systemd units or some other shenanigans; it’s always whack-a-mole since Skype’s business interest lies in having its app always present to create “engagement”.

We have to find a way to access the auto-start setting from outside of the UI then.

Digging into the source

Since every “app” nowadays is just webshit upon turtles-all-the-way-down fragile tech stacks, the whole skypeforlinux app is just a giant pile of minified javascript running on Electron, whose source code you can thankfully inspect.

/usr/share/skypeforlinux/resources/app.asar, is an asar archive² that you can extract using asar extract app.asar app-extracted.

Packages for e.g. Ubuntu or Arch Linux are available.

You’ll find a file named main.js which contains the settings handling and autostart creation code.

I’ve pretty-printed relevant parts using js-beautify, which is available via pip install jsbeautifier or as an Arch Linux package.

The t.AutoStartLinux object contains the main creation logic:

class u extends a.AutoStartBase {
    _enable(e) {
        e ? this._createFile() : this._removeFile()
    }
    _createFile() {
        try {
            s.ensureDir(this._dir), i.writeFileSync(this._file, this._content),
            l.getInstance().info("[AutoStartLinux] Created file " + this._file)
        } catch (e) {
            l.getInstance().error("[AutoStartLinux] Unable to create file " + this._file, e)
        }
    }
    _removeFile() {
        try {
            i.unlinkSync(this._file)
        } catch (e) {
            l.getInstance().error("[AutoStartLinux] Unable to remove file " + this._file, e)
        }
    }
    get _dir() {
        return o.join(r.homedir(), ".config", "autostart")
    }
    get _file() {
        return o.join(this._dir, "skypeforlinux.desktop")
    }
    get _content() {
        return ["[Desktop Entry]", "Name=Skype for Linux",
                "Comment=Skype Internet Telephony", "Exec=/usr/bin/skypeforlinux",
                "Icon=skypeforlinux", "Terminal=false", "Type=Application",
                "StartupNotify=false", "X-GNOME-Autostart-enabled=true"].join("\n")
    }
}
t.AutoStartLinux = u

The AutoStartBase class, which the AutoStartLinux subclass inherits, reads a setting called autoStartEnabled:

t.AutoStartBase = class {
    constructor() {
        this._appSettings = i.getInstance()
    }
    sync() {
        this._appSettings || (i.init(), this._appSettings = i.getInstance());
        const e = !!this._appSettings.get().autoStartEnabled;
        void 0 !== this._isEnabled && this._isEnabled === e || this._enable(e), this._isEnabled = e
    }
}

This setting is translated into a key read from the json file by SettingsKeys:

t.SettingsKeys = {
    AutoStartEnabled: "app.autoStartEnabled",
    OnCloseKeepRunning: "app.onCloseKeepRunning",
    Crashed: "didCrashInLastSession",
    RegisterProtocols: "app.registerProtocols",
    LaunchMinimized: "app.launchMinimized",
    CheckNonAdminUser: "app.checkNonAdmin",
    LoggingLevel: "logging.level",
    LoggingEnabled: "logging.enabled",
    LoggingConsole: "logging.console",
    UpgradedFromDelphi: "app.upgradedFromDelphi",
    UpgradedFromDelphiDate: "app.upgradedFromDelphiDate",
    MainWindowPosition: "main-window.position",
    WindowMaximized: "main-window.isMaximised",
    ZoomLevel: "main-window.zoom-level"
};

Gotcha!

Background

The whole working-from-home fad has reached me as well. Every person in a position of authority insists on choosing their own videoconferencing solution, which wouldn’t be as bad if those products were at least all web-based.

But, because we live in the shittiest of all possible universes, every large company scrambled to push their brittle technology out the door as fast as possible, of course with maximum lock-in and native “apps” to increase user retention, and hopefully sell some of that sweet data.

Color me surprised then to find out Microsoft-owned Skype’s “Skype for Web” does not work at all in both Firefox or Chromium but wants you to install Microsoft Edge (a 1-to-1 clone of Chrome with different lipstick on top). Synergy!

On every session login, not on every Skype login. ↩︎
Yes, of course the javascript people had to “invent” their own archive format. ↩︎

flask-admin: Access row object in LinkRowAction

Mon, 04 Jan 2021 00:00:00 +0000

I have an admin interface with tags that can be associated with any number of entries. As a demonstration, flaskadmintagdemo should give you an idea of what my side looks like.

I wanted to have a link next to each tag pointing to a filtered list of all entries tagged with that specific tag.

The logic for that is in flask-admin > model > template.py

class LinkRowAction(BaseListRowAction):
    def render(self, context, row_id, row):
        m = self._resolve_symbol(context, 'row_actions.link')
        # string_types = (str, unicode)
        if isinstance(self.url, string_types):
            url = self.url.format(row_id=row_id)
        else:
            url = self.url(self, row_id, row)
        return m(self, url)

So, we need to amend the column_extra_row_actions to add a LinkRowAction that points to the filtered listing of entries:

class TagView(ModelView):
    class PostURLObj(str):
        """
        Fake str object that fails isinstance(str, unicode) so that we can
        access the row object
        """
        def __new__(self, cls, row_id, row):
            _url = "/admin/?flt0_9=%s" % row.name
            return str.__new__(self, _url)

    column_extra_row_actions = [
        # Cannot use EndpointLinkRowAction since we need access
        # to the row object
        LinkRowAction(
            "glyphicon glyphicon-eye-open fa fa-eye", PostURLObj
        ),
    ]

Instanciating a new str subclassed object - in this case PostURLObj - will call <cls>.__new__(args), not __init__(). Took me a while to figure that out…

But, instead of fixating on working around LinkRowAction, why not just implement a more suitable action by subclassing BaseListRowAction?

class PostRowAction(BaseListRowAction):
    def __init__(self, icon_class):
        super(PostRowAction, self).__init__()
        self.icon_class = icon_class
    def render(self, context, row_id, row):
        m = self._resolve_symbol(context, 'row_actions.link')
        url = "/admin/?flt0_9=%s" % row.name
        return m(self, url)

class TagView(ModelView):
    column_extra_row_actions = [
        PostRowAction(
            "glyphicon glyphicon-eye-open fa fa-eye",
        ),
    ]

Much easier.

Adding DuckDuckGo properly to Firefox

Sun, 03 Jan 2021 00:00:00 +0000

Needed to re-add DuckDuckGo search to Firefox manually.

Install Add-on

Install Add custom search engine addon from addons.mozilla.org.

Get search parameters

Navigate to DuckDuckGo.com, enter the Settings page, click “Show Bookmarklet and Settings Data”, copy the “Bookmarklet URL”. For me, it looks like: duckduckgo.com/?kah=de…

Enter info in addon

Open the interface to add a new search engine by locating the blue looking glass icon in the menu bar (Not in Firefox Settings! It is an addon with its own interface).

With %s substituted for the search term, add the following as search URL in the addon: duckduckgo.com/?q=%s?kah….

We do not need to use POST queries here.

The icon is available at favicon.ico

Then check “Advanced settings”.

The “Suggest URL” for auto-suggestions in the search bar is https://ac.duckduckgo.com/ac/?q={searchTerms} (Yes, {searchTerms} this time instead of %s).

Give the engine a name, e.g. “DDG”, then hit “Add custom search engine, then, on the next page, follow the instruction to add the search engine. You may also set DDG as the default in Firefox search settings.

Full OpenSearch XML

<?xml version="1.0" encoding="UTF-8"?>
<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/">
<ShortName>DDG</ShortName>
<Description>DuckDuckGo</Description>
<InputEncoding>UTF-8</InputEncoding>
<Image height="16" width="16">https://duckduckgo.com/favicon.ico</Image>
<Url type="text/html" method="GET" template="https://duckduckgo.com/?q={searchTerms}&amp;kah=de-de&amp;kl=de-de&amp;k5=2&amp;k1=-1&amp;kk=-1&amp;kaj=m&amp;kay=b&amp;kak=-1&amp;kax=-1&amp;kaq=-1&amp;kap=-1&amp;kao=-1&amp;kau=-1&amp;kt=n&amp;kae=-1&amp;ks=m&amp;kw=w&amp;ka=n&amp;kd=-1"/>
<Url type="application/x-suggestions+json" method="GET" template="https://ac.duckduckgo.com/ac/?q={searchTerms}"/>
</OpenSearchDescription>

Git tricks

Wed, 27 Feb 2019 20:05:40 +0100

Tips

From twitter.com/zeuxcg

git rebase —autostash stashes and unstashes local changes for you

git commit -v adds diffs to commit draft so that you can inspect them as you write commit msg

git rebase -i Also, you should configure autostash to be on by default, and configure rebase to be default on pull. Pull without -r will screw you every time!

You probably know already but commit messages allow to autosquash commits. fixup! <hash>, squash! <hash> Works best with git rebase —interactive —autostash —autosquash or simply git commit --fixup <hash> (or git commit --squash <hash>) … easier than typing the relevant commit msgs.

I didn’t know those. Thanks. I have 3 aliases to avoid typing: git fixup <hash> git squash <hash> git oops. git oops creates a fixup commit to the current HEAD commit. I use it when I don’t want to amend to keep the history until a later rebase. Used countless times already.

git config rebase.autostash true git config pull.rebase true git add —patch git gui (allows to stage by line)

I’m using git add -i, it’s a superset of -p, allowing to revert changes and add untracked files as well. Oh and git checkout -p, the destructive inverse of add -p.

Misc

git rebase has --committer-date-is-author-date

git rebase -i <commit-to-rebase-on-top-of>
# set all to "edit"
git commit --amend --date=$(date -I) --no-edit
git rebase --continue
# finally
git rebase <commit-to-rebase-on-top-of> --committer-date-is-author-date

split to new folder(not subtree):
from GitHub help

git filter-branch --prune-empty --subdirectory-filter \
  FOLDER-NAME  BRANCH-NAME

cherry-pick commit while ignoring line endings:
e.g. after dos2unix runs

git cherry-pick -X ignore-all-space <commit-id>

Rewrite commit date to author date while rebasing

git filter-branch \
    --env-filter 'export GIT_COMMITTER_DATE="$GIT_AUTHOR_DATE"' \
    <SHA>..HEAD

Find commits in all branches that touch a path

git log --all -- <path>`

Find commits in all branches that contain message

git log --all --grep <msg-or-regex>`

Find code in all branches that matches regex

git grep <regex> $(git rev-list --all)

Might be slow though!

Bash

Tue, 26 Feb 2019 20:05:40 +0100

getopt

my-function() {
    # This is important for getopt or it will fail on the second invocation!
    local OPTIND
    while getopts 'u:b:s:' _flag
    do
        echo "_flag=$_flag"
        case "${_flag}" in
            u) GIT_URL="$OPTARG" ;;
            b) GIT_BRANCH="-b $OPTARG" ;;
            s) SPEC+="$OPTARG " ;;
            *) echo "my-function(): Unexpected option $_flag"; exit 1; ;;
        esac
    done
}

Ubuntu install

Tue, 29 May 2018 00:00:00 +0000

Disclaimer: This is for personal reference, read up on the official documentation before you brick something you did not want bricked…

Create disk

sudo dd if=distro.iso of=/dev/sdX status_progress bs=1M

Boot live cd

Set keymap, then prepare disks

(If in virtualbox, make sure to activate EFI mode)

Partition drive

Use lsblk or blkid, fdisk -l to get an overview of the current situation, then create GPT table and partitions with:

sgdisk --zap-all /dev/sdX
sgdisk --clear \
    --new=1:0:+550MiB --typecode=1:ef00 --change-name=1:EFI \
    --new=2:0:0       --typecode=2:8300 --change-name=2:system \
    /dev/sdX

Or use cgdisk which is nice and interactive.

Create FAT32 filesystem for ESP on first partition:

mkfs.fat -F32 -n EFI /dev/diSk/by-partlabel/EFI

Cryptsetup

cryptsetup luksFormat /dev/sda2 --key-size 512 --iter-time 2000
cryptsetup luksOpen /dev/sda2 cryptroot

(Could also use /dev/disk/by-partlabel/... instead of /dev/sda2…)

mkfs.ext4 /dev/disk/by-partlabel/system

(Using ext4 because the Ubuntu installer ubiquity will create subvolumes for / → @ and /home → @home by default if using btrfs)

Start installation

ubiquity -b

Select language, keymap(de-latin1-nodeadkeys), then for “Installation type” select ”Something else”, use /dev/mapper/cryptroot as / with ext4 and /dev/sda1 as esp(EFI system partition)

Then click through the warnings, enter credentials.

Then DO NOT REBOOT, click “Continue testing”

Fix partitions

mount /dev/mapper/cryptroot /mnt
mkdir -p /mnt/boot/efi
mount LABEL=EFI /mnt/boot/efi

chroot

Before you chroot, mount the following filesystems from the live system:

mount --bind /proc /mnt/proc
mount --bind /sys /mnt/sys
mount --bind /dev /mnt/dev
mount --bind /run /mnt/run
mount --bind /dev/pts /mnt/dev/pts

Otherwise the cryptsetup hook from update-initramfs will fail later.

Finally

chroot /mnt

Fix fstab, crypttab

Find out UUID of root partition(not mounted filesystem!) with blkid.

Remove the swap entry from /etc/fstab, then create /etc/crypttab, replace <UUID from blkid> of course:

#/etc/crypttab
cryptroot UUID=<UUID from blkid> none luks

Check that /dev/mapper/cryptroot is listed correctly in /etc/fstab.

Update initramfs

Regenerate the initramfs:

update-initramfs -k all -u

Should run without any errors; if it complains about not seeing devices you probably forgot to mount /dev into the chroot target.

Then copy to ESP

cp /boot/initrd.img-* /boot/efi/initrd.img
cp /boot/vmlinuz-* /boot/efi/vmlinuz-generic

Bootloader

Install systemd-boot to the ESP

bootctl --path=/boot/efi install

Then check efi vars with efibootmgr -v, in case bootctl failed to install an entry, use

efibootmgr \
--create \
--disk /dev/nvme0n1 \
--gpt \
--loader "\EFI\systemd\systemd-bootx64.efi" \
--label systemdboot \
--part 1 \
--timeout 0 \
--write-signature \
--verbose

Set boot order if necessary, 0010 would be systemd-boot

efibootmgr -o 0010,0020

Configure bootloader in /boot/efi/loader/loader.conf

default ubuntu
# leave timeout to debug
timeout 2
# leave editor to debug
editor 1

Create boot entries

# /boot/efi/loader/entries/ubuntu.conf
title   Ubuntu
initrd  /initrd.img
linux   /vmlinuz-generic
options r0 root=/dev/mapper/cryptroot quiet splash

Finishing

Unmount everything, exit chroot, shutdown, remove USB key, boot, enjoy. If something is broken, hold down space at boot for systemd-boot to halt, allowing editing of the kernel commandline by pressing e on a boot entry.

Outlook

This setup will not install kernel updates by itself, you’ll need a hook that moves the updated initrd and kernel stubs onto the ESP and creates appropriate boot manager entries.

References

Site notice

Wed, 09 May 2018 00:00:00 +0000

Powered by

Made with Hugo, using the Natrium theme.

License

Unless otherwise noted, works on this site are licensed under Creative Commons Attribution-ShareAlike 4.0.

Arch install

Sun, 29 Apr 2018 00:00:00 +0000

Create disk

sudo dd if=distro.iso of=/dev/sdX status_progress bs=1M

Boot arch

Better connect ethernet, else use wifi-menu

loadkeys de-latin1

Partition drive

Use lsblk or blkid, fdisk -l to get an overview of the current situation

sgdisk --zap-all /dev/sdX
sgdisk --clear \
     --new=1:0:+550MiB --typecode=1:ef00 --change-name=1:EFI \
     --new=2:0:0       --typecode=2:8300 --change-name=2:system \
       /dev/sdX

Or use cgdisk which is nice and interactive

mkfs.fat -F32 -n EFI /dev/disK/by-partlabel/EFI

Cryptsetup

cryptsetup luksFormat /dev/sda2 --key-size 512 --iter-time 2000
[--align-payload=8192 -h sha512 -c aus-xts-plain64]
cryptsetup luksOpen /dev/sda2 cryptroot

mkfs.btrfs --force --label system /dev/mapper/cryptroot

Then create btrfs filesystems, optionally with subvolumes

# maybe use compress=lzo but that could be dangerous
o_btrfs=defaults,x-mount.mkdir,compress=zstd,ssd,noatime
mount -t btrfs /dev/mapper/cryptroot /mnt
# Using subvolumes:
btrfs subvolume create /mnt/root
btrfs subvolume create /mnt/home
btrfs subvolume create /mnt/snapshots
umount -R /mnt
# Mount created btrfs filesystem
mount -t btrfs -o $o_btrfs /dev/mapper/cryptroot /mnt
# Or using subvolumes:
mount -t btrfs -o subvol=root,$o_btrfs /dev/mapper/cryptroot /mnt
mount -t btrfs -o subvol=home,$o_btrfs /dev/mapper/cryptroot /mnt/home
mount -t btrfs -o subvol=snapshots,$o_btrfs /dev/mapper/cryptroot /mnt/.snapshots
mkdir -p /mnt/boot/efi
mount LABEL=EFI /mnt/boot/efi

Mirrorlist

Use reflector if possible

cp /etc/pacman.d/{mirrorlist,mirrorlist.backup}
pacman -S reflector
reflector --country France --country Germany --age 12 --protocol https --sort rate --save /etc/pacman.d/mirrorlist

# Old school: uncomment country-specific mirrors in mirrorlist.backup; then
rankmirrors -n 6 /etc/pacman.d/mirrorlist.backup > /etc/pacman.d/mirrorlist

Install base system

pacstrap /mnt base

fstab

genfstab -L -p /mnt >> /mnt/etc/fstab

Should look something like this:

# <file system> <dir> <type> <options> <dump> <pass>
# /dev/sda1 EFI ESP partition
UUID=6285-B457 /boot/efi vfat rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro	0 2

# /dev/sda2 with plain btrfs on top of luks:
# /dev/mapper/cryptroot UUID=XXXXXX
LABEL=cryptroot / btrfs rw,noatime,ssd,compress=zstd,space_cache,subvolid=5,subvol=/ 0 0

# /dev/sda2 with btrfs and subvolumes:
# /dev/mapper/cryptroot UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
/dev/mapper/cryptroot / btrfs rw,noatime,compress=zstd,ssd,space_cache,subvolid=257,subvol=/root,subvol=root	0 0
# /dev/mapper/cryptroot UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
/dev/mapper/cryptroot /home btrfs rw,noatime,compress=zstd,ssd,space_cache,subvolid=258,subvol=/home,subvol=home	0 0
# /dev/mapper/cryptroot UUID=...
/dev/mapper/cryptroot /.snapshots btrfs rw,noatime,compress=zstd,ssd,space_cache,subvolid=259,subvol=/snapshots,subvol=snapshots	0 0

# ext4 for refrence:
# /dev/sda2 with ext4
UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx / ext4 rw,relatime,data=ordered 0 1

Chroot

Chroot into mounted new system, either via systemd-nspawn or chroot: Use systemd-nspawn to use *ctl commands directly in “chroot”

# -b is for boot
systemd-nspawn -bD /mnt
# or
arch-chroot /mnt

Locale

Edit /etc/locale.gen, uncomment de_DE.UTF-8 and en_US.UTF-8, run locale-gen, then use either systemd-firstboot or localectl

systemd-firstboot --prompt-locale
# or
localectl list-locales
↳ [...] en_US.UTF-8 [...]
localectl set-locale LANG=en_US.UTF-8
localectl set-keymap --no-convert de-latin1

Time

Use timedatectl and run configure ntp via systemd-timedated and systemd-timesyncd, ntpd is old

timedatectl set-ntp 1
timedatectl list-timezones
↳ ...
timedatectl set-timezone Europe/Berlin
hwclock --systohc

Hostname

Set via hostnamectl

hostnamectl set-hostname myhostname
hostnamectl set-chassis "laptop"

Append to /etc/hosts

127.0.0.1	localhost.localdomain	localhost
::1		localhost.localdomain	localhost
127.0.1.1	myhostname.localdomain	myhostname
# or via echo
echo "127.0.1.1	myhostname.localdomain myhostname" >> /etc/hosts

Network

Ethernet can be managed by systemd-networkd in case of NetworkManager failure

Install NetworkManager, then use in case of no GUI:

# Scan
nmcli dev wifi
# Add connection, interactively
nmcli con add --ask
↳ Connection type: wifi
↳ No to all others
# Check
nmcli con show
↳ NAME UUID ..., note NAME
# Connect
nmcli con up id 'NAME'

Install packages

pacman -Syu
pacman -S base-devel btrfs-progs gptfdisk
pacman -S adwaita-icon-theme ansible avahi bash binutils btrfs-progs bzip2
cantarell-fonts chrome-gnome-shell coreutils cryptsetup dconf-editor
device-mapper diffutils dosfstools e2fsprogs efibootmgr eog epiphany evince
fakeroot file file-roller filesystem findutils fish freetype2 gawk gcc
gcc-libs gdm gedit glibc gnome-backgrounds gnome-bluetooth gnome-calculator
gnome-clocks gnome-color-manager gnome-control-center gnome-desktop
gnome-icon-theme gnome-icon-theme-extras gnome-keyring gnome-menus
gnome-shell gnome-system-monitor gnome-terminal gnome-tweaks grep gvfs gzip
inetutils iproute2 iputils iw less librsvg libtool linux linux-headers
logrotate man-db man-pages mdadm nano nautilus networkmanager pacman patch
pciutils perl polkit pkg-config procps-ng psmisc pulseaudio sed shadow sudo
sysfsutils systemd-sysvcompat tar tracker ttf-dejavu usbutils util-linux
which xdg-user-dirs-gtk xfsprogs xterm xorg-server xorg-server-xwayland

initramfs

Backup /etc/mkinitcpio.conf, then edit:

# /etc/mkinitcpio.conf
MODULES=(vfat i915)
BINARIES=()
FILES=()
HOOKS=(systemd sd-vconsole autodetect modconf block keymap sd-encrypt btrfs filesystems keyboard)
#HOOKS=(base udev autodetect modconf block filesystems keyboard fsck)

Regenerate the initramfs

mkinitcpio -p linux

Then copy to ESP

cp /boot/initrd-linux.img /boot/efi
cp /boot/initrd-linux-fallback.img /boot/efi
cp /boot/vmlinuz-generic /boot/efi

Security

Set password, add to sudoers/wheel group

passwd
usermod -a -G wheel username

Bootloader

Install systemd-boot to the ESP

bootctl --path=/boot/efi install

Then check efi vars with efibootmgr -v, in case bootctl failed to install an entry, use

efibootmgr \
--create \
--disk /dev/nvme0n1 \
--gpt \
--loader "\EFI\systemd\systemd-bootx64.efi" \
--label systemdboot \
--part 1 \
--timeout 0 \
--write-signature \
--verbose

Set boot order if necessary, 0010 would be systemd-boot

efibootmgr -o 0010,0020

Configure bootloader in /boot/efi/loader/loader.conf

default arch
# leave timeout to debug, can disable later
timeout 2
# leave editor to debug, should disable later(security risk!)
editor 1

Create boot entries:

Get luks-uuid with: sudo cryptsetup luksUUID /dev/nvme0n1p2
Device-mapper name is luksroot here, be consistent

# /boot/efi/loader/entries/arch.conf
title   Arch Linux
#initrd /acpi_override # applying DSDT patch later
initrd  /initrd-linux.img
linux   /vmlinuz-generic
options rd.luks.uuid=<uuid> rd.luks.name=<uuid>=luksroot root=/dev/mapper/luksroot rw quiet splash
# for btrfs subvoumes use rootflags=subvol=...
# Set "dangerous" options later:
#options rw root=/dev/nvme0n1p2 quiet splash acpi.ec_no_wakeup=1 psmouse.synaptics_intertouch=1 mem_sleep_default=deep loglevel=3 i915.modeset=1 i915.fastboot=1 i915.enable_guc=1

# /boot/efi/loader/entries/arch-fallback.conf
title   Arch Linux fallback
initrd  /initrd-linux-fallback.img
linux   /vmlinuz-generic
options rd.luks.uuid=<uuid> rd.luks.name=<uuid>=luksroot root=/dev/mapper/luksroot rw

Might have to fiddle around with root=... and include subvol, encrypt stuff

Rhythmbox

Thu, 09 Feb 2017 00:00:00 +0000

Remove clutter from database

Install lxml (apt-get install python-lxml)
Download removehidden.py and removeignored.py.
Copy rhythmdb.xml over from ~/.local/share/rhythmbox into working directory
Run either script to remove hidden or ignored entries respectively
Copy rhythmdb_new.xml back to the rhythmbox directory

Plugins

MediaServer2 D-Bus interface (included by default in 3.x)
MPRIS D-Bus interface (included by default in 3.x)
Rating filters
Open containing folder

Brother DCP-L2520DW

Mon, 01 Aug 2016 00:00:00 +0000

Download files from the brother printer support page.
Use either the install tool(will complain about missing ia32-libs but works fine) or install the packages manually.
Assign a static ip address to the device and add the printer with CUPS. The device should appear with proper name and abilities.
Do not install the generic packages(named ‘Generic LPR printer driver, Generic CUPSwrapper printer driver’).

(for Arch Linux: Install brscan4 from AUR)

For scanning functionality, use brsaneconfig4. Run brsaneconfig4 -q and confirm that the line 39 "DCP-L2520DW" appears. Then run brsaneconfig4 -a name='Friendly Name' model='DCP-L2520DW' ip=192.168...
Or use nodename=printerhostname where printerhostname might be something like BRWxxxxxxxx.fritz.box, use nmap -sn to find out.
Use simple-scan and select the device, then run a test scan. DPI can be set to up to 2400.

Edit /lib/udev/rules.d/60-libsane.rules and add:

# Brother scanners
ATTRS{idVendor}=="04f9", ENV{libsane_matched}="yes"

For scankey support run brscankey-skey -l. Should it not display anything, killing it with brscankey -t and re-running the previous command might help. (Haven’t gotten it to work yet, and I do not see a use for this.)

Update 2021

On printer: Set static IP, enable ipv6 (nice to have), set printer hostname (“nodename”)

Note: Adding via hostname (even appending “.local”) does not work

CUPS: Add new lpd printer

lpd://192.168.1.50/BINARY_P1

nmap:

sudo nmap -sn 192.168.1.1/24
# or
sudo nmap -p 631 192.168.1.1/24

SANE:

sudo brsaneconfig4 -a name=NETWORK-Brother-DCP-L2520DW model=DCP-L2520DW ip=192.168.1.50

Banshee

Mon, 18 Jul 2016 00:00:00 +0000

Remove deleted files from internal database

Install libdbi-perl and libdbd-sqlite3-perl
Make a backup of ~/.config/banshee-1/banshee.db
Download reap-dead-tracks.pl from this server or GitHub
Run perl reap-dead-tracks.pl <db>, confirm deletion

Reload internal database

sqlite3 ~/.config/banshee-1/banshee.db "update coretracks \
    set LastSyncedStamp = 0 where PrimarySourceID=1"

UI

Use right mouse button to resize time slider

Sync

Make an .is_audio_player file in root of external device, describe capabilites
Do not start banshee wile an mtp device is connected (Just starting up banshee in the background will cause the connection to fail and mess up everything)

Mixxx reference

Thu, 11 Feb 2016 00:00:00 +0000

AutoDJ

Activate with Shift-F12, a little android icon will appear in top left corner and control over crossfader will be automated
Skip track with Shift-F11
Set fade time to negative value for breaks in between songs, don’t forget to turn it off. Setting a negative value will make mixxx not respect cue points!
Use crates to refill cue if necessary
Seeking to cue point on load can be turned off in preferences

Musicality

The BPM detection is insufficicient

Knobs

Any effect will increase audio latence, this includes key lock
Don’t forget to turn off filters after the song has ended
Mind the crossfader
Use the gain slider cautiously, and turn it off immediately afterwards
For effects: select appropriate channel

Recording

CTRL-R to start-stop recording, little icon will appear in top left corner

Organizing

Use playlists for tracks in a specific order
- Tandas are designated T:Genre:Artist(tempo):Singer/mood/etc. (year from-til), where genres are
  - Cd - Candombe
  - Fx - Foxtrot
  - Mi - Milonga (Mi/Cd: Milonga Candombe)
  - Ta - Tango
  - Va - Vals
  - X<Genre> - Fusion (genre), nuevo
- Setlists are designated Deployment:Date:Location
Use crates for collections of tracks

Tagging

Song title: Name | Alternative Name (canta Singer y Another Singer y coro)(en vivo, extra information).
Extra information, in title or otherwhere, is in Spanish
Singer also goes into vocal field(which most players do not display)
Artist: Just the name, omission of ‘Orquesta’ if possible e.g.: Carlos Di Sarli, Orquesta Típica Victor, Quinteto Francisco Canaro(Dir. Antonio D’Alessandro)
Album: Title | Subtitle. If artist is mentioned in album title, include the name in album tag(e.g. Milongueando con Canaro). Disc number goes into a separate tag
Genre is capitalized/title cased, e.g. Milonga Candombe
Year: Either YYYY or YYYY-MM-DD
grouping tag is for eras(guardia vieja, early, late, …), can be displayed in mixxx
Use comment field for danceability information/tagging, trivia etc. goes into my notebook
Tagging-todo:
- Implement rating system from 0 to 100 like on tango-dj.at instead of stars, rate for milonga suitability(and custom tag/commandeer one?)
- Grouping standards(maybe from tejastango)
- Where to put Instrumental tag? In song title(El chamuyo (Instrumental))? Into grouping?

Search

Fuzzy matching: ~key:c#m finds “tracks with harmonically compatible keys to C# minor”
artist:Danger duration:>3m rating:>=4
genre:hip-hop -year:1990 (negate with -)

Building, audio format support

Don’t forget to install libmp4v2-2

for m4a support:

Ubuntu should have m4a support with ubuntu-restricted-extras and ubuntu-restricted-addons installed

If not, use these prebuilt files:

libsoundsourcem4a for 2.0.0
libsoundsourcem4a for 2.1.0 alpha
rename to libsoundsourcem4a.so and move into /usr/lib/mixxx/plugins/soundsource/

or build everything yourself like a madman:

apt-get build-dep mixxx, put deb-src [...] universe in sources.list
install libupower-glib-dev
install libavformat-dev libavcodec-dev for ffmpeg and m4a/aac support
install development build from mixxx.org
scons --PREFIX=<prefix> -j <cores> optimize=native faad=1 ffmpeg=1
go to lin64_build/plugins/lin64_build/m4a/ and copy libsoundsourcem4a.so to /usr/lib/mixxx/plugins/soundsource/
Links
- gmane thread on m4a plugin
- forum thread

Importing

Exact Audio Copy will write id3(?) tags that are hard to get rid of. They do not appear in puddletag, but can be spotted with a hex editor(e.g. bless). Use Tag Tools -> Remove id3/ape Tags in puddletag to remove them, or use mutagen to programatically get rid of those tags and weird output like Junto a tu corazon;Junto a tu corazon (feat. Alberto Podesta)

USB audio

Check with dmesg -w whether the device was recognized as USB AUDIO
Check aplay -l whether the interface(?) appears
Check whether snd_usb_audio kernel module is loaded with lsmod
After a kernel update: reboot and check if issues persist
Install alsa-utils, alsa-firmware, alsa-plugins, pulseaudio-alsa
Manage with pavucontrol

Wishlist

Support for rating system from 0 to 1, in arbitrary increments
Cue out points
Saving metadata to files
Displaying arbitrary tags
Less weird sorting, reliable sorting either by Artist-Album-Discnr-Tracknr or Album-Discnr-Tracknr
Setting loudness, not just via ReplayGain but also manually, to make cortinas more quiet
‘Danceability’ tag(or maybe just use comment)
Showing comment as multiline popover when hovering instead of a long string

Firefox modifications

Sun, 20 Dec 2015 00:00:00 +0000

Addons

Essential

uBlock Origin (with custom filters)
Stylish
Session Manager
GrabMyBooks
I don’t care about cookies
Clean Uninstall
New Tab override(since Firefox doesn’t allow setting newtab.url any more for some reason)
Suspend Tab (actually works, as opposed to UnloadTab)
VimFx(with custom mappings, don’t forget to back up)
wallabag(v2)
Warn Before Quit

Unused

UnloadTab (loses tabs, not recommended)
Super Start (load a custom html page from file:/// instead)
RSVP Reader
ScrapBook
HTML5 Video Everywhere! (works only sporadically)
Resurrect Pages
Tab Mix Plus
Random Agent Spoofer
Content Search X
BetterPrivacy
QuickMark
Privacy Badger (or Disconnect or Ghostery)
NoScript (even with javascript turned on, it still offers some security features)
Mozilla Archive Format
meta-q-override

misc

remove default mail providers
turn off sync, hello, pocket, tiles, weed out search providers, turn off telemetry etc.
set default search engine, disable search suggestions

about:config

browser.search.showOneOffButtons false (deprecated)
browser.newtab.url and browser.startup.homepage to about:blank(obsolete, see new tab override)
browser.pocket.enabled false
beacon.enabled false
browser.visited_color #00CCCC
dom.max_script_run_time 40
dom.max_chrome_script_run_time 40 (from 20)
dom.battery.enabled false (how could anyone think this was a good idea)
dom.event.clipboardevents.enabled false
general.warnOnAboutConfig false
geo.enabled false
network.dns.disablePrefetch true
pdfjs.disable true
plugins.enumerate_names (empty)
privacy.trackingprotection.enabled true
webgl.disabled true (will break fullscreen mode?)
browser.fullscreen.{animateUp,autohide}
dom.serviceWorkers.enabled false (what a dumb idea to let scripts run after closing a tab)
privacy.file_unique_origin=false - See CVE-2019-11730 But I don’t particularly care Needed so that file:// stuff works properly and doesn’t complain about CORS, e.g. for accessing cssRules
devtools.inspector.color-scheme-simulation.enabled=true for testing out prefers-color-scheme, see stackoverflow

Debian cheatsheet

Fri, 16 May 2014 00:00:00 +0000

This has grown into a very useful list of things that make my life easier. Divided into sections for clarity. Mostly applicable to Ubuntu as well.

Nifty shortcuts, one-liners and useful programs

netcat for file transfer receiver: nc -l -p 1234 > somefile.png
sender: nc -w 3 192.168.178.20 1234 < somefile.png
ssh port forwarding ssh -R 80:localhost:8080 user@server.com -i key.pem
multiple x sessions exec startx -- :1 vt8
get random SALT openssl rand -base64 32
use imagemagick to resize pictures convert <infile> -resize 100x100 [<outfile>]
force aspect ratio convert <infile> -thumbnail '100x100>' -background white -alpha remove -gravity center -extent 100x100 [<outfile>]
use imagemagick to create quick thumbnails POSIX, only works with .jpeg, height=200px: for f in *; do; echo $f; convert -thumbnail x200 -background white -alpha remove $f ${f:%.jpeg}_thumb.png; done
with zsh: for f in *; do; echo $f; convert -thumbnail x200 -background white -alpha remove $f ${f:r}_thumb.png; done
where ${f:r} means “remove file extension from $f”, see zsh archive
fast video thumbnails with ffmpegthumbnailer for f in *; do; echo $f; ffmpegthumbnailer -i $f -o ${f:r}_thumb.png -s 400; done
imagemagick: crop and then append images Images are 1080x1920, we want to crop them to the middle, but full height. First, create the needed folders folders, with files named identically: mkdir before now cropped_{before,now}, so that you have e.g. before/1.png, now/1.png, before/2.png, now/2.png and so forth.
Then, figure out the crop dimensions, e.g. 800x1080+580+0 where we want an image 800x1080 in dimension, and we carve it out with the x-offset of 580 and y-offset of 0.
Then, run the following imagemagick commands:
```
# Crop all before images
for f in before/*; do
  i=${f#before/} && echo $i && \
  convert -crop 800x1080+580+0 before/$i cropped_before/cropped_$i
done`
# Crop all after images
for f in now/*; do
  i=${f#now/} && echo $i && \
  convert -crop 800x1080+580+0 now/$i cropped_now/cropped_$i
done`
# Merge the cropped before and after pictures side-by-side
for f in before/*; do
  i=${f#before/} && echo $i && \
  convert +append cropped_before/cropped_$i cropped_now/cropped_$i $i
done`
```
ffmpeg gif optimizer
ffmpeg -i input.mp4 -r 10 -f image2pipe -vcodec ppm - | convert -loop 0 - output.gif
find and remove “~” files from emacs etc. find . -name*\~ | xargs rm
use quvi to download videos etc quvi --exec "wget -O <name.webm> %u" <link>
or just use youtube-dl

set .mp4 metadata from filename

  for f in *.mp4; do
      title="${${f#"Series Name to Remove"}%".mp4"}"
      exiftool "$f" -Title="$title" -Artist=
  done

Where Series Name to Remove is a prefix to remove

count words (or lines) wc [-l]
print all file types in a directory find . -type f -name "*.*" | awk -F. '{print $NF}' | sort -u
or just print number of files of one filetpye: find . -name "*.mp3" -type f | wc -l
count number of files find <dir> -type f | wc -l
generate random number shuf -i 2000-65000 -n 1
recover data from btrfs partitions from superuser
run strings /dev/sda1 | less and then use / to search for the strings that could be in that file. delimit the start of it with “m” then move to the end and use |cat >file to save it to a file
(note: worked on text for me, binary could be difficult)
tre-agrep
cat file.txt | tre-agrep -i(ignorecase) -s(show match cost) -3(max cost of 3) 'Fuzzy match'
- -D: cost of missing characters
- -I: cost of extra characters
- -S: cost of substitutions(should be x2 since deletion and insertion are 2 operations)
load testing use siege --concurrent=100
manipulate systemd units from users as root systemctl --user --machine=<user>@.host list-units where <user> the actual user, but host is written verbatim

Pdf handling

use pdftk to split a document into multiple pages pdftk A=input.pdf cat A1-100 output output-1-100.pdf
use pdftk to join multiple pdfs
pdftk input1.pdf input2.pdf cat output output.pdf
use pdfjam to make presentations printable
pdf90 inputs.pdf pdfjam Inputs_rotated.pdf --nup 2x1 --landscape --outfile Outname.pdf pdf270 Outname.pdf

convenient fish function

function pp -d 'Make presentations printable'
    pdf90 $argv
    set -x pdf_file (echo $argv | cut -f 1 -d '.')
    pdfjam (echo $pdf_file)-rotated90.pdf --nup 2x1 \
    --landscape --outfile print-(echo $pdf_file).pdf
    pdf270 print-(echo $pdf_file).pdf
    rm print-(echo $pdf_file).pdf
    mv print-(echo $pdf_file)-rotated270.pdf print-(echo $pdf_file).pdf
    rm (echo $pdf_file)-rotated90.pdf
end

k2pdfopt
-om 0.3 for margins, -n for disabling ocr and bitmap generation
removing margins
use pdfcrop to remove whitespace automatically
use --margins '5 10 20 30' to remove margins in pts(left, top, right, bottom), use negative values (-50 -50 -50 -50) for actually cropping away
generate blank page
echo "" | ps2pdf -sPAPERSIZE=a4 - blankpage.pdf

Misc

(Difficult-to-remember commands or usage of programs, accomplishing very specific tasks)

tar tar xvzf file to extract file.tar.gz, tar cvzf file.tar.gz files_to_include to create
find find . -iname "*thing*", case insensitive
multiple keyboard locales setxkbmap -option grp:switch,grp:alt_shift_toggle,grp_led:scroll us,de

encrypt harddrive (from hermann-uwe.de)

badblocks -c 10240 -s -w -t random -v /dev/sdX
# create partition using fdisk
cryptsetup --verbose --verify-passphrase luksFormat /dev/sdX1
cryptsetup luksOpen /dev/sdX1 name
mkfs.ext3 -j -m 1 -O dir_index,filetype,sparse_super /dev/mapper/name
# mount it
# write to it
# unmount it
cryptsetup luksClose /dev/mapper/name

invert bash wildcards
shopt (-s)(-u) extglob
ls !(*.zip)
the sudo tee duo sh unprivileged-command | sudo tee output-file >/dev/null
sed instead of head sed 11q file is equal to head file
sed '3q;d' file to get 3rd line of file
bridge terminal and clipboard
echo test | xclip -selection c - put into clipboard
xclip -selection clipboard -o - print out clipboard contents
use notify-send sanely
use notify-send 'Heading' 'Body text' --icon=dialog-information with --hint int:transient:1 for ephemeral notifications that do not clutter the message view
qemu
qemu-img create disk.img 10G
qemu -hda disk.img -boot d -cdrom distro.iso -m 1024
download a whole webpage
wget --recursive --no-clobber --page-requisites --html-extension --convert-links --domains domain.com domain.com
reinstall all python packages
sudo dpkg --get-selections | sed 's:install$::' | sed -e 's/^[ \t]*//' -e 's/[ \t]*$//' | grep -v 'python3' | grep python sudo dpkg --get-selections | sed 's:install$::' | sed -e 's/^[ \t]*//' -e 's/[ \t]*$//' | grep python3 | xargs sudo apt-get --reinstall install
thunderbird: disable annoyances thunderbird for enterprise

upgrade all pip packages
add --local for virtualenvs

  pip install -U `pip list --outdated | awk '{ print $1}'`

Clear systemd journal
sudo journalctl --rotate && sudo journalctl --vacuum-time=1s

use tar to back up a server
crude, but effective

tar -zcvpf /archive/full-backup-`date '+%d-%B-%Y'`.tar.gz --directory / \
    --exclude=mnt --exclude=proc --exclude=var/spool/squid --exclude=sys \
    --exclude=cdrom --exclude=sys --exclude=tmp

or from serverpronto.com

tar cvpzf /backups/backup.tgz --exclude=/proc --exclude=/lost+found \
    --exclude=/backups --exclude=/dev --exclude=/sys --exclude=/boot/grub \
    --exclude=/etc/fstab --exclude=/etc/network/interfaces \
    --exclude=/etc/udev/rules.d/70-persistent-net.rules /

tar cvpzf /root/backups.tgz --exclude=/root --exclude=/proc \
    --exclude=/lost+found --exclude=/backups --exclude=/dev \
    --exclude=/sys --exclude=/boot/grub --exclude=/etc/fstab \
    --exclude=/etc/network/interfaces

rsync

basic rsync -avs --delete -h . user@domain.tld:/home/user/server
copy directory structure only rsync -av -f"+ */" -f"- *" /path/to/src /path/to/dest/
ignore version control files(warning: ignores folders named core) rsync --cvs-exclude

git

set up new repo

git init
git add .
git commit -m "commit message"
git remote add remotename user@domain.tld:/home/user/proj.git
git push --set-upstream remotename master
# on server
git --bare init

update submodules
git pull --recurse-submodules
with proper commits: git submodule update --recursive
change author info
from github:
git clone --bare https://github.com/user/repo.git
cd repo.git
run git-author-rewrite.sh, check logs
git push --force --tags origin 'refs/heads/*'

Media

convert ogg to mp3
for name in *.ogg; do ffmpeg -i "$name" -ab 128k -map_metadata 0:s:0 "${name/.ogg/.mp3}"; done;

convert flac to mp3(fish shell)

for i in (ls *.flac)
    ffmpeg -i $i -ab 320k -map_metadata 0 -id3v2_version 3 $i.flac
end

ffmpeg: reduce video size
ffmpeg -i input.mp4 -preset medium -c:v libx265 -crf 28 -c:a copy output.mp4
(crf: low is better quality, high is worse. 20-28 are sane values, 23 is default, see stackoverflow)
ffmpeg: convert to x264 with good quality
ffmpeg -i input.mp4 -c:a copy -vcodec libx264 -crf 20 output.mp4

ffmpeg: use VAAPI acceleration for converting

ffmpeg -init_hw_device vaapi=default_vaapi:/dev/dri/renderD128 -hwaccel vaapi \
  -hwaccel_output_format vaapi -hwaccel_device default_vaapi \
  -i input.mkv -filter_hw_device default_vaapi -vf 'format=nv12|vaapi,hwupload' \
  -preset medium -c:v hevc_vaapi -crf 25 -c:a copy output.mkv

Typography

create specific unicode characters CTRL-SHIFT-U <NUMBER> ENTER
via compose key Set Right control(makes most sense) as compose key using gnome-tweak-tool, then either look up the corresponding sequence here or try to guess it, the mappings make sense most of the time
(No need to keep pressing the compose modifier while entering)
Examples: - . . for … (ellipsis) - ~ n for ñ(eñe) - ? ? for ¿(inverted ?) - C C C P(uppercase!) for ☭ (hammer & sickle)

Display management

start display manager (xmonad, …)
in .xinitrc
```
xrandr -s 0
xmonad &
wait
```
set brightness xrandr --output DVI0 --brightness 0.5
also use arandr
turn off display xset dpms force off, combine with lock: xset dpms force off; slock

System deployment

manipulate installed packages
sudo dpkg --get-selections > packages.txt to back up
then sudo dpkg --clear-selections to set all to deinstall
sudo dpkg --set-selections < packages.txt to import the saved package list
sudo aptitude install or sudo apt-get -u dselect-upgrade(much faster)

fish shell

‘clear cache’ and speed up: remove ~/.config/fish/fishd*
save alias: alias la 'ls -la' and then funcsave <func>
clipboard: kill with CTRL-w or ALT-BCKSP -> kill ring.
Paste:CTRL-y, cycle: ALT-y
man dmenu has a good explanation of movement
use ALT and left/right to move through directory history
use z with tab completion for a fuzzy ncurses-like file manager

Shortcuts from emacs-mode

Shortcut	Action
`TAB` and `SHIFT`+`TAB`	completion cycling
`CTRL`+`a` and `CTRL`+`e`	jump to beginning/end of line
forward: `ALT`+`f` backward: `ALT`+`b`	move by words
left: `CTRL`+`b` right: `CTRL`+`f`	moving, autosuggestions
up: `CTRL`+`p` down: `CTRL`+`n`	searching command history (`ALT` with up/down only for tokens)
`CTRL`+`h` `CTRL`+`d` `CTRL`+`j`	backspace forward delete enter
`CTRL`+`c` `CTRL`+`k` `CTRL`+`u`	deletes whole line cursor to end beginning to cursor
`CTRL`+`w` `ALT`+`d`	deletes the next part deletes last part
`ALT`+`l`	list contents of current dir
`ALT`+`p`	pipe output to `less` for paged output
`ALT`+`w`	show command description
`CTRL`+`t` `ALT`+`t`	switches (transposes) the last two characters the last two words

Fix specific application problems or configure for desired mode of operation

apt-get without diffs diff files are great for low bandwidth connections but they are not needed most of the time
either run apt-get update with -o Acquire::Pdiffs=false
or edit /etc/apt/apt.conf to include Acquire::PDiffs "false";
apt: disable automatic updates edit /etc/apt/apt.conf.d/10periodic and set APT::Periodic::Update-Package-Lists "0";
fix alsa problems (sudo) alsactl init
pandoc generate proper html with utf-8 encoding pandoc -f markdown infile.mkd -t html -s -o outfile.html
pandoc: website to epub pandoc -s -r html http://url.tld/page -o epub
try to explicitly log into an ssh server using password instead of an ssh key ssh -o PreferredAuthentications=keyboard-interactive -o \ PubkeyAuthentication=no user@host
gnome, systemd dependency madness on ubuntu from shallowsky.com
install equivs package
run equivs-control, edit Package to package name, Version to 2:42 or something high, Maintainer to some name and <email@address.com>
run equivs-build and install with dpkg -i <packagename>
Use this to remove e.g. evolution-data-server
package that a file belongs to dpkg -S file
disable bootup of services on ubuntu update-rc.d cups remove (note: at least for now, systemd might change this)
let apt ignore dependencies edit /var/lib/dpkg/status and remove Depends: entries (note: only short-term, this will get overwritten)
flask install directly from github tarball to use flask * commands
use sudo without entering your password all the time sudo visudo
user ALL=NOPASSWD: ALL
however, this is very insecure, so it is better to just whitelist specific commands:
user ALL=(root) NOPASSWD: /usr/bin/command [args]
%group ALL=(root) NOPASSWD: /path/to/comand [args]
sudo password asterisks Defaults env_reset,pwfeedback in visudo
pip problems cryptography need libssl-dev on ubuntu
cffi needs python-cffi
Pillow needs libjpeg-dev
stop mongodb from allocating 3GB of space (shutdown mongodb), edit /etc/mongodb.conf and insert smallfiles=true
remove /var/lib/mongodb/journal/\*, (startup mongodb)
partition woes
- partprobe /dev/sdX or kpartx /dev/sdX
- try hdparm -Z /dev/sdX or sfdisk -R /dev/sdX
QT5
- set ENVVAR with export QT_AUTO_SCREEN_SCALE_FACTOR=0.8 in /etc/profile.d/qt.sh or try with .bash_profile, .profile, .bashrc etc.
- install libqt5libqgtk2 qt5-image-formats-plugins qtwayland5 qt5-qmltooling-plugins
- run qt5ct

Power management and laptop handling

power saving
- install laptop-mode-tools or, preferably tlp
- install powertop and set toggles
- utilize your desktop environment’s built-in solution, or run pm-suspend, pm-hibernate or pm-suspend-hybrid like an animal
- edit /etc/systemd/logind.conf and set HandleLidSwitch=ignore, then run systemctl restart systemd-logind.service clean up boot with update-rc.d or disable systemd units, use the Startup applications program to manage autostarts

gsettings

Reset app grid
gsettings reset org.gnome.shell app-picker-layout
List tracker settings
gsettings list-recursively | grep -i org.freedesktop.Tracker | sort | uniq

systemctl –user

Get DBus access as su-d normal user
export DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/$(id -u)/bus See Start systemd units at boot time

Fix problems pertaining to specific hardware configurations

wifi on Asus EEE PC 1000H

install firmware-ralink

edit /etc/network/interfaces

  auto wlan0
  iface wlan0 inet dhcp
      wpa-ssid YOUR-SSID-HERE
      wpa-psk YOUR-PASSWORD-HERE
  /etc/init.d/networking restart
  iwlist
  iwconfig -a
  iwconfig wlan0 down
  ifconfig wlan0 hw ether 01:23:45:67:89:00
  ifconfig wlan0 up

I don’t use my old netbook any more, I hear OpenBSD has good drivers for wireless networking. Go have fun installing that one.

mitigate problems with intel ac 7260

edit /etc/modprobe.d/iwlwifi-11n.conf and add options iwlwifi 11n_disable=8 (or disable=1)
(at runtime) run sudo rmmod {iwlmvm,iwlwifi}, then sudo modprobe iwlwifi 11n_disable=1, sudo modprobe iwlmvm
edit /etc/default/crda and change REGDOMAIN= to your country code
note: this still does not fix all problems, especially eduroam is still wonky
bluetooth
purge gnome-bluetooth
install bluez-tools bluetooth blueman bluez-hcidump python-bluez and pulseaudio-module-blueooth (for audio)
run sudo hciconfig hci0 reset and sudo pactl load-module module-blueotooth-discover
if applicable, edit /etc/bluetooth/main.conf

Vim cheat sheet

Fri, 16 May 2014 00:00:00 +0000

Folds

za to toggle current fold
zm to close, zr to open

Windows

from windows

A buffer is the in-memory text of a file.
A window is a viewport on a buffer.
A tab page is a collection of windows.

Opening: CTRL-W and v(vert), s(split/duplicate), n(horiz)
Closing: CTRL-W and q(quit), c(close)
Moving cursor: CTRL-W and up, down, left, right
Moving windows: CTRL-W CTRL-O make window the only(maximize)

Plugin mappings

sensible: CTRL-u in insert mode: delete backwards, for real
sneak: s or S and then two letters for faster jumping, ; to repeat
table-mode: <leader>tm to toggle, <leader>tr to realign, SHIFT-jklö to jump table cells
multiple cursors: CTRL-n to select word, CTRL-p for undo, CTRL-x to skip the current one

Settings in vimrc

:::vim
" vimrc
" Remember last position
if has("autocmd")
  au BufReadPost * if line("'\"") > 1 && line("'\"")
  \ <= line("$") | exe "normal! g'\"" | endif
endif
" set pyton3
let g:python3_host_prog = '/usr/bin/python3'
" highlighting
hi Search cterm=black ctermfg=white ctermbg=red
" German keyboard layout
nnoremap j h
nnoremap k j
nnoremap l k
nnoremap ö l

Movement

see scroll

scroll line for line: CTRL-e(up) CTRL-y(down)
scroll window half a screen: CTRL-u(up) CTRL-d(down)
scroll full windows: SHIFT-UP and SHITFT-DOWN
move line to top of window: z-ENTER
Jump back CTRL-o (and CTRL-i for forward)
Move to last position '' or use double backticks
delete(in insert move): like in fish:
CTRL-k to end, CTRL-u to start, CTRL-w to delete word
CTRL-h for backspace, CTRL-j for enter
run single command in insert mode: CTRL-O
deleting(normal)(swap a for i for inner):
swap c for d: deletng, for v: visual select, for y: ‘copy’
- ciw: change in word
- cis: change in sentence
- cib and diB: change () and {} blocks
- cip: change in paragraph
- cit: change in tags(html)
D to delete from cursor to end of line
move by whole words(no whitespace): W
(), {}: jump sentences, paragraphs
H M L: move to top, middle, bottom of viewport
marks: m(register): mark point (here: x)
- 'x(single quote): jump to point(my mapping, default is backtick)
pasting: p after cursor, P before cursor
CTRL-r-x to insert from register x while in insert mode

Very useful

wrap lines visual select and then gq
use previous selection gv for selecting previous visual selection gi to go into insert mode in previous position
o in visual mode jumps to other end of selection
= to reformat text
ciw''P: enclose in quotes

General

get out of “completion mode” / resume flow control CTRL-q
save and quit vim ZZ, to only quit: ZQ, to write all files and quit: :xa
escape CTRL-c
indents use CTRL-i(indent) and CTRL-d(remove indent) in insert move,
use > and < in visual mode(once) and in normal(press twice)
jump to line [line nr], G
recording
q [letter] to start recording
q to stop, @ [letter] to play back
parenthesis ([{ jumping %
manipulate multiple lines use visual selection and then :norm <command>, e.g. :'<,'>norm A,
or block(CTRL-v) select and then $A<text> to append to all lines
numbers CTRL-a and CTRL-x increment and decrement
netrw use CTRL-o and CTRL-i to switch, use % to create a file
replacing/‘substituting’
set incsearch and then search /oldstring, replace with %s//newstring/
:s/oldstring/newstring (only once)
:s/oldstring/newstring/gic: global, ignore case and confirm each :[startline],[endline]s/oldstring/newstring/g replace only between specified lines
&: **repeat last subsitution
writing
visual selection only writes the selected part
wr :r loads file, can be combined with !<command>
windows CTRL-W CTRL-W switch windows
neat
- CTRL-X [o p n] auto-complete and more
- ~ toggle upper/lowercase, in visual mode: u and U for lower/uppercasing
- :bro ol browse oldfiles
dictionary
use zg on a word to add it to dictionary
located in ~/<vimdir>/spell/en.utf-8.add
fix encoding
:set bomb
:set fileencoding(s)=utf-8
:w
whitespace
%le for removing indents
g/^$/d for removing blank lines
:%s/\s\+$// for removing trailing whitespace
centering :ce columns: center with columns width
abbreviations
ab lw Long word: expand ’lw’ to “Long word” una lw: unset the abbreviation
tabs to spaces
:retab

Mappings

::vim
:nmap - Display normal mode maps
:imap - Display insert mode maps
:vmap - Display visual and select mode maps
:smap - Display select mode maps
:xmap - Display visual mode maps
:cmap - Display command-line mode maps
:omap - Display operator pending mode maps

Todo

Use registers more often
Use ci{ and the like more often
Jump by s and f instead of hjkl
Use CTRL-h and CTRL-j for backspace and enter
map NOP to unused extension mappings for uncluttering :map and no surprises
Explain d/D, y/Y, c/C, p/P, b/B, i/I, … in various modes