Languages

February 7, 2021

For reasons that (I swear!) have nothing to do with my nationality, I have always avoided acquiring any modicum of French language skills. I took Latin in high school and while at university, Spanish sounded like the clearly superior investment, especially for my social life. Hearing people speak French, I was having trouble holding it together, because, obviously this was a fake language designed to make you emit as many silly noncoherent noises as possible.

Similarly, I had put off learning Javascript until very recently. The language of “sprinkle some ads on it”, the language of ADHD-riddled kids who import a package to left-align a number, the language that makes the web an increasingly unpleasant experience. Javascript did not appeal to me.

l’ordinateur - the computer

$ 'wtf' instanceof String
> false
$ typeof 'wtf'
> 'string'

As happens from time to time, it is not by choice but rather by necessity that you have to pick up a skill. So while I am now forced to learn French for future foreign assignments, I also had to delve into the murky waters that are the Javascript “ecosystem”.

Ubuntu Touch & just a small bug

Flashing custom firmware on mobile phones has historically been a manual and painful process. All the more was I happy to hear that my buddy Konrad had worked hard to submit my favourite device, the Sony Xperia XZ, for inclusion in Ubuntu Touch’s fabulous interactive GUI installer.

There was just a small issue with the (Electron/Javascript-based) installer failing to verify a specific file.

error: Error: core:manual_download: Error: checksum mismatch

I know about Android, Linux administration and hardware debugging. I know where to look and I can solve issues rather quickly. From investing quite a bit of time, I’ve seen a lot of errors and know how to look for the part in the whole source tree that might be responsible.

With Javascript however - or any new and unfamiliar environment - hunting down bugs takes me ages. It’s incredibly frustrating, like you’re a preschooler being made to navigate complex algebra.

le magazine - the magazine; le magasin - the store

There are so many convenitions, syntax oddities and build/ecosystem tooling to learn, none of which I could give a rat’s ass about. I am not interested in any of the individual tools involved, I just want to solve a problem that has been caused by them.

Not to be deterred though, I dug in. This, treasured reader, is the story of that problem.

Promises and Lies

The code I am about to reference makes heavy use of asynchronous Promise functionality.

A basic usage pattern would look like this:

const prom = new Promise( (resolutionFunc, rejectionFunc) => {
    console.log("Look, ma!");
    return true;
});

With this block, a new Promise object would be created (and immediately executed). It executes the log statement and calls back to one of the two callback functions, one for success (fulfilled) and one for failure (rejected).

More commonly, developers use .then to handle promise resolution:

const myPromise =
  (new Promise(resolutionFunc))
  .then(() =>
    // handle first fulfilled, return second fulfilled
    dostuff;
  )
  .then(() =>
    // handle second fulfilled, return something
    domorestuff;
  )
  .then(resolved => {
    // handle domorestuff success
  }, rejected {
    // handle domorestuff failure
  })

This is called a “chained promise” and is basically a try/catch, but for Javascript and asynchrooonouuuuus.

in, an, en, eau, eux - all pronounced pretty much the same

Having learned the semantics of promises, this block of code looked perfectly innocent to me:

return new Promise(resolve => {
  // get user input for path - simplified here
  return "/home/user/path/file.zip";
})
.then(downloadedFilePath => {
  fs.ensureDir(
      path.join(this.cachePath, this.props.config.codename, group)
    )
    .then(() =>
      fs.copyFile(
        downloadedFilePath,
        path.join(
          this.cachePath,
          this.props.config.codename,
          group,
          file.name
        )
      )
    );
})
.then(() =>
  checkFile(
    {
      checksum: file.checksum,
      path: path.join(
        this.cachePath,
        this.props.config.codename,
        group,
        file.name
      )
    },
    true
  )
)
.then(ok => {
  if (ok) {
    return ok;
  } else {
    throw new Error("checksum mismatch");
  }
});

Simple, right? The first promise passes the downloadedFilePath to the ensureDir func, which then triggers copyFile. Once the block is resolved, checkFile is run on the copied file. All functions politely wait for the previous func to resolve via the .then() statement.

Or do they? Has this all been a lie?

Les Nombres

Similarly, lulled into a false sense of security, I thought I might approach learning French numbers above the single-digit range. Une, deux, trois… that’s practically Spanish, right? So how hard can those until 100 be?

4, 5, 6, 7, 8, 9, 10 - quatre, cinq, six, sept, huit, neuf, dix

So far, so good.

11, 12, 13, 14, 15, 16 - onze, douze, treize, quatorze, quinze, seize

Great!

17, 18, 19, 20, 21 - dix-sept, dix-huit, dix-neuf, vingt, vingt-et-un

Okay, small inconsistency, but I can get used to that…

30, 40, 50 - trente, quarante, cinquante

So from 20 onwards, there’s a nice pattern again. Sweet.

Continuing on with Javascript.

The intended flow of the specific part of the program I was looking at was:

  • If file already present, checksum and compare with predefined checksum, on success skip all the next steps
  • If checksum mismatch or file not present:
  • Let user pick file with system file picker
  • Return path of picked file as promise
  • Ensure target dir inside cache is present
  • Copy user-picked file to cache location (~/.cache/ubports/<device>/firmware/)
  • Checksum newly copied file
  • Unpack newly copied file (extract .img from zip file)

To reiterate, the bug which I was encountering looked like this:

error: Error: core:manual_download: Error: checksum mismatch

I quickly narrowed down the issue to a call of checkFile() in src/core/plugins/core/plugin.js.

At first, I thought maybe the crypto library was failing to call the right hash functions, but when a file was already in the right place, the hashes were verified as correct.

I then quickly disabled the checksum verification after the file had been copied and was met with a failure of the unpack step, which complained about a missing input file. The file was being copied alright, but it seems the checksum and unpack steps were winning a race against the copyFile() operation even though I thought they were running in sequence.

After some further digging, maybe what I needed was the synchronous variant of copyFile? Swapped it for copyFileSync, but no change in results.

I added an artificial delay of 100ms after the whole ensuredir/copy block and before the checksum, and everything went smoothly. So, why was the copy operation not being finished in time?

Having spent way too much time on this issue already, there must’ve been something wrong with all this asynchronous promising going on…

Shadows looming

Helpfully, Mozilla’s excellent MDN includes an admonition about the subtle ways you can shoot yourself in the foot:

An action can be assigned to an already “settled” promise. In that case, the action (if appropriate) will be performed at the first asynchronous opportunity. Note that promises are guaranteed to be asynchronous. Therefore, an action for an already “settled” promise will occur only after the stack has cleared and a clock-tick has passed. The effect is much like that of setTimeout(action,10).

and this little snippet:

const promiseA = new Promise( (resolutionFunc,rejectionFunc) => {
    resolutionFunc(777);
});
// At this point, "promiseA" is already settled.
promiseA.then( (val) => console.log("asynchronous logging has val:",val) );
console.log("immediate logging");

// produces output in this order:
// immediate logging
// asynchronous logging has val: 777

Not quite the problem I was facing, but a harbinger of things to come.

Now for the final nail in the coffin: Implied return statements.

This:

() => 1 // returns 1

is the same as this:

() => { return 1; } // returns 1

but not the same as this:

() => { 1; } // returns nothing

With all the pieces of the puzzle collected, we are ready to solve this riddle.

But not before going back again to the French numbers. We left off at 50 and I’m excited for what lies ahead:

60 - soixante

Excellent. This is beginning to make a lot of sense to me.

70 - soixante-dix (60 + 10)

What. The. Fuck.

71 - soixante et onze (60 + 11)

So from 70 on, they just use 60 as a base? Alright, very confusing, but I’m in it for the long haul, I can deal with that. How do the numbers continue after 70?

80 - quatre-vingts (4 x 20)

At this point I’m scrambling to cross-reference, because clear as day my study material has been written by a prankster. That can’t possibly be right, no?

90 - quatre-vingt-dix ((4 x 20) + 10)

By now I have consulted several sources and confirmed this is indeed how the French count. We might have been premature in mocking the Anglos for their imperial units. This is worse.

But I’ll leave les nombres be for now.

Deadly void

Aaaand back into the joyous world of machines! Let’s look at the mutinous code block again.

return new Promise(resolve => {
  // Return downloadedFilePath
  return "/home/user/path/file.zip";
})
// block start
.then(downloadedFilePath => {
  fs.ensureDir("/home/user/path")
    .then(() =>
      fs.copyFile(
        downloadedFilePath,
        "/home/user/.cache/path/file.zip"
      )
    );
}) // block end
.then(() =>
  checkFile();
)
.then(ok => {
  if (ok) { // handle stuff
}

I’ve simplified it for better readibility. Astute readers will have spotted the error by now:

  • The whole “block” was never going to resolve its promise with an actual return value (inside curly braces but no return statement)
  • checkFile() was being called before any of the promise functions inside the previous block had the chance to resolve
  • The copied file was not yet available
  • checkFile tried to verify a non-existent path
  • Checksum mismatch error.

This is the sort of bug that makes you want to strangle a kitten. To shout obscenities à la Captain Haddock. To permanently disable the power grid and return to farming sheep.

parle, parles, parle - I speak, you speak, he speaks - all pronounced the same

Solution & Reflections

The first solution I landed on was to make the whole block explicitly return something, in this case the value of the last chained promise:

.then(downloadedFilePath => {
  // Added return statement below
  return fs.ensureDir("/home/user/path")
    .then(() =>
      fs.copyFile(
        downloadedFilePath,
        "/home/user/.cache/path/file.zip"
      )
    );
})

I finally settled (after consultation with Marijn) on removing the curly braces to convert the block into an implied return, as that would fit better with the syntax of the surrounding code:

// Removed curly braces:
.then(downloadedFilePath =>
  fs.ensureDir("/home/user/path")
    .then(() =>
      fs.copyFile(
        downloadedFilePath,
        "/home/user/.cache/path/file.zip"
      )
    )
)

I submitted a Pull Request, as you do nowadays, and waited for the fix to propagate to the users.

The saddest part about all this is that none of these functions need to be asynchronous. They are executed in sequence and the UI blocks until they return anyway.

The developer who wrote all this, Jan Sprinz, is a perfectly nice guy, and if it wasn’t for him, people would probably still have to fumble around with arcane command line tools to install Ubuntu Touch. I can’t fault him for seeing common patterns in Javascript development and, thinking they’re good practices, writing the whole function using promises.

In conclusion, programming is terrible, hug your loved ones, go outside and enjoy the sun.

P.S.

Huge thanks to my buddy Marijn for proofreading and giving corrections!

← Customizing Isso for my needs
Netgear R6120 OpenWRT →