Shell scripts as a poor man's AppImage

The best application bundle format is .sh
Reading time: about 3 minutes

I love applications that can be packaged as a single executable file. A lot of languages can create files like that now, including C, C++, Rust, Go, and even C#. But not every language can compile to a single executable like that, even even ones that do might need some extra files bundled with the executable.

There is a lot of overlap between this topic and self-extracting archives. Check those out for ideas if you are going to roll your own solution.

AppImages

One popular method I see these days is AppImages. AppImage is a format for bundling Linux applications, along with their dependencies, in order to create applications that work without installation or root, and the whole thing gets packaged in a single file.

They are extremely convenient, and they work pretty well. We’ve deployed AppImages to production and used them to bundle tools and dependencies without messing with developer or server systems.

An AppImage is basically a directory that contains all the files for an application. This directory is turned into a compressed disk image with squashfs. A small runtime that is prepended to the squashfs image handles executing the application. This is done by mounting the image as a temporary mount point, and then executing the application inside it. This runtime also handles things like extracting the disk image to execute it without mounting.

In the time we used AppImages, we ran into two small problems.

The first one is the dependence on libfuse. Running an AppImage normally, without extracting it first with --appimage-extract, requires FUSE and libfuse. A huge appeal of AppImages for us was being able to deploy applications without any superuser privileges and without modifying the system in any way. This dependency on libfuse means either we need to modify the system to install stuff, or we need to extract the image and get rid of the “single-file executable” benefits.

The other downside is also related to mounts. In order to make startups faster, and not require extra disk space or RAM when executing the application, AppImages create a temporary mount instead of extracting themselves every time you execute them. This is a good way to get free performance.

But we ran into some commercial system monitoring tools that give alerts to sysadmins when a new mount is created. This is meant for checking the disks, stuff like making sure everything is in /etc/fstab and will be mounted again on reboots etc. But this system doesn’t understand temporary mounts, and thinks everything is a physical drive. When we execute AppImages regularly, that monitoring system creates useless alerts. It’s not something we can disable, and it’s not a bug we can fix ourselves. And frankly mounting stuff feels too much like modifying the system, even if FUSE lets us do it without extra permissions.

Those two minor issues inspired me to come up with an alternative solution.

An ugly and/or great solution

I wrote a Shell script generator in Python. It takes the application to bundle as a directory, just like AppImages. The output is a single executable that runs the application, just like AppImages. But the internals? That’s very much unlike AppImages.

We first take a statically-compiled binary of the zstd compressor. We encode this binary as base64, and embed this into our Shell script as a heredoc. The script un-base64’s that into a temporary file, and makes it executable.

We then tar our application folder, pipe that through zstd to compress it, encode it as base64, and embed it into our Shell script. The script pipes that through a base64 decoder, decompresses is with the zstd binary we unpacked earlier, and un-tars it into another temporary directory.

After this, the application is executed normally from this temp directory. After our shell scripts exits, all the temp files are deleted from the system.

All of this works, and we’ve deployed applications with this strategy. The only downside seems to be a small latency when executing the application, and this is due to the decompression and extraction of the embedded tar file.

Future work

This project was both fun and useful in real life. If I come back to it in the future, I’m planning to investigate some of the things below.

  • Instead of base64, embed binary data after the script and unpack with dd.
  • Instead of a shell script with embedded data, make a statically compiled executable that can be prepended to archives.
  • Instead of extracting a zstd binary and using it to decompress data, compile the decompressor into the “runtime” as a static library.
  • Allow stuff like --appimage-extract to extract application files somewhere without executing them.

The following pages link here

Citation

If you find this work useful, please cite it as:
@article{yaltirakli202304poormansappimage,
  title   = "Shell scripts as a poor man's AppImage",
  author  = "Yaltirakli, Gokberk",
  journal = "gkbrk.com",
  year    = "2023",
  url     = "https://www.gkbrk.com/2023/04/poor-mans-appimage/"
}
Not using BibTeX? Click here for more citation styles.
IEEE Citation
Gokberk Yaltirakli, "Shell scripts as a poor man's AppImage", April, 2023. [Online]. Available: https://www.gkbrk.com/2023/04/poor-mans-appimage/. [Accessed Nov. 12, 2024].
APA Style
Yaltirakli, G. (2023, April 28). Shell scripts as a poor man's AppImage. https://www.gkbrk.com/2023/04/poor-mans-appimage/
Bluebook Style
Gokberk Yaltirakli, Shell scripts as a poor man's AppImage, GKBRK.COM (Apr. 28, 2023), https://www.gkbrk.com/2023/04/poor-mans-appimage/

Comments

Comment by admin
2023-12-18 at 12:48
Spam probability: 0.15%

Hey Benjamin, One reason I went with a standalone, all dependencies included approach is because I want the application to run on many Linux distros that might be old or not updated for a long time. And I can't depend on an internet connection to install dependencies from pip, as the servers that are deployment targets can often not have internet connectivity. The installer method, or at least extracting once to some hidden directory with the hash of the file, is something I'm considering as well. It should greatly reduce startup time, and prevent multiple copies of the application ending up in /tmp if the user executes multiple copies. One reason I haven't done that was to prevent the user or the application itself from modifying the files and accumulating state that diverges from whatever was packaged, but maybe this is not so important compared to startup times and making /tmp less cluttered.

Comment by Benjamin Furstenwerth
2023-11-06 at 02:52
Spam probability: 0.158%

I have made something similar as my installer builder using cython, makeself and shc. I went the installer method and have the client install a python venv and grab all the pip stuff on their end. Standalone can be huge otherwise.

© 2024 Gokberk Yaltirakli