Defcon Kaklkaklkekl (Wiki)

====== Creating a Read only Debian Lenny system ====== The task here is: at work, we have these cute little Vesa-mount-sized computers originally manufactured by DMP Electronics as the [[http://www.compactpc.com.tw/ebox-4300.htm|eBOX 4310]], rebranded as [[http://www.norhtec.com/|NorhTec]] [[http://www.norhtec.com/products/mcsr/index.html|MicroClient Sr]], that we are going to use backpack-mounted on large-screen HDTV's for our internal digital signage project. These little boxes are to have no spinning disk, and an as reliable as possible system. To get no moving parts, the boxes skip using a hard drive, and we are setting them up with Compact Flash (CF) as the main storage. But, as any documentation will tell you, CF has a limited number of write-cycles, and as a result of that, it is desirable to have the system running with its filesystem read-only once booted. Internally we normally standardize on CentOS for servers, and Ubuntu+Fedora as supported desktop Linux'es. But none of these are really "dead simple" to make read-only-rootfs, and to be honest comes with too much bloat in my opinion. So I am trying to do this using Debian Lenny. Debian is stock, standard, known tech, easily modified (ref. Pebble, LEAF, DSL, Ubuntu, Mint), and supportable. The base install is also fairly easy to make small. ===== Install ===== I hooked up one of the eBOX'es, PXE-booted the Debian Lenny installer, and installed onto the CF. My magic during install was close to none: * Partitioning: Manual, one partition, no swap * Tasksel Install: Base system only. * Non-root user: kitteh Post install: <code> apt-get -y install less bzip2 screen vim-nox openssh-server halt </code> ===== CF Image file ===== After installing, I removed the card from the eBOX, put it in a CF-reader on my workstation, created an image of it, and made a backup. <code> sudo cat /dev/sdb > sdb_dump1 # Raw disk-dump of the CF card.. cp sdb_dump1 lenny_testimg1.img # This copy is what I will be working on, sudo bzip2 sdb_dump1 # while this file is my backup. </code> So, should I ever need to restore the original Lenny install, all I'll have to do is bunzip the sdb_dump1.bz2 file back onto a CF card of the same size (or larger). After waiting for a few ages (dumping 8GB, copying 8GB and fibnally bzipping 8GB down to 586MB takes a SOLID time), I mounted up the image using loopback mounting, and continued customizing the install. ==== Loopmounting image ==== As the CF dump is a dump of the complete CF card, and Debian Installer has treated it as a harddisk, it contains a partition table. So a simple "mount -o loop imagefile" will not work. First, get a hold of where the partition to mount starts: <code> fdisk -lu lenny_testimg1.img </code> <code> You must set cylinders. You can do this from the extra functions menu. Disk lenny_testimg1.img: 0 MB, 0 bytes 255 heads, 63 sectors/track, 0 cylinders, total 0 sectors Units = sectors of 1 * 512 = 512 bytes Disk identifier: 0x000bdb20 Device Boot Start End Blocks Id System lenny_testimg1.img1 63 15615179 7807558+ 83 Linux </code> This image opnly contains one partition, and it starts at sector number 63. Each sector is 512 bytes, so the mount command becomes: <code> sudo mount -o loop,offset=$((63*512)) lenny_testimg1.img /mnt/ </code> This mounts the file system that starts at sector 63 onto /mnt. If you get the mount command wrong, it will tell you: <code> mount: you must specify the filesystem type </code> If you get this, adjust your command and try again .... You correct info __is__ given in the "fdisk -lu" output. ===== Customization ===== Jump into the file-system from the host computer: <code> sudo mount -t proc proc /mnt/proc/ sudo mount -o bind /dev/ /mnt/dev/ sudo LC_ALL=C chroot /mnt /bin/bash </code> Remove /etc/mtab and symlink it to /proc/mounts. This special file also describes the mounted filesystems, and can replace mtab without needing a write access on it. <code> rm /etc/mtab ln -s /proc/mounts /etc/mtab </code> Setting up /etc/fstab so that all locations where we will write data is a tempfs, in other words a memory filesystem. If the directories that are set up as tempfs here need to contain data after boot, those data will either be copied across by /etc/rcS.d/S36rosystem-fix or /etc/rc.local <code> # <file system> <mount point> <type> <options> <dump> <pass> /dev/hdc1 / ext3 defaults,noatime 0 0 proc /proc proc defaults 0 0 tmpfs /var/run tmpfs defaults 0 0 tmpfs /var/lock tmpfs defaults 0 0 tmpfs /var/local/data tmpfs defaults 0 0 tmpfs /var/log tmpfs defaults 0 0 tmpfs /tmp tmpfs defaults 0 0 tmpfs /var/lib/urandom tmpfs defaults 0 0 tmpfs /var/lib/dhcp3 tmpfs defaults 0 0 tmpfs /home tmpfs defaults 0 0 </code> Now, we need resolv.conf to be writable for dhclient, so it needs to be located somewhere outside of /etc <code> mkdir /var/local/data mv resolv.conf /var/local/data/ ln -s /var/local/data/resolv.conf /etc/resolv.conf </code> Now, we need to have the dhclient-script (runs while/after dhclient is run) update the writable resolv.conf, instead of trying to replace our link on the read-only part of the system. This is sed-magic, to really see what this does, make a copy of /sbin/dhclient-script, and diff it afterwards... <code> sed \ -e '/local new_/s/=.*/=\/tmp\/resolv.conf.dhclient-new/' \ -e '/mv -f/s/mv -f.*/cat $new_resolv_conf > \/etc\/resolv.conf/' \ -i /sbin/dhclient-script </code> Just verify that network settings are appropriate for our/your/the network. I am using DHCP to configure, so i end up with: <code> cat /etc/network/interfaces </code> <code> auto lo iface lo inet loopback allow-hotplug eth0 iface eth0 inet dhcp </code> ifupdown insists on writing a status file to /etc/network/run, so that location needs to be redirected to a tmpfs location. Note that the directory /var/local/data/network_run is created on boot by /etc/rcS.d/S36rosystem-fix, which will be created later on.. <code> rm /etc/network/run mkdir /var/local/data/network_run ln -s /var/local/data/network_run /etc/network/run rmdir /var/local/data/network_run </code> Debian uses udev and hotplug since, well, a long time now. It is set up to assign each ethernet adaper a separate device name, based on the MAC-address and/or the kernel-driver used. We do not want to save any persistent ethernet device data, and we want to use eth0 as the networking device. So, we have to remove the udev-definitions of “persistent devicenames”. The definition for persistent net device names are located in /etc/udev/rules.d/70-persistent-net.rules. Edit this file by hand, or use the following sed line: <code> sed -e '/^SUBSYSTEM=="net".*eth/D' -e '/^$/D' \ -e '/^# PCI device/D' \ -e '/^$/D' \ -i /etc/udev/rules.d/70-persistent-net.rules </code> We install syslogd for logging, and remove all references to xconsole from the configuration. At a later stage, it will be configured to send its log data to a central logging server, and not buffer any logging locally. <code> apt-get install syslogd sed -e '/xconsole/s/^/#/' -i /etc/syslog.conf </code> In the fstab listed above, the root filesystem is mounted with "defaults,noatime". That normally implies that the file system is mounted read-write. But we want read-only. This is fixed by changing the default "rootmode" in /etc/init.d/checkroot.sh from "rw" to "ro". It _could_ also be done by setting "ro" on the kernel boot lines in /boot/grub/menu.lst, but previous experience shows that this may sometimes not be honored. So, forcing it is perhaps not the best way, but still the most "stable" way((Most stable as long as we do not upgrade to a new release of Debian, that is...)). <code> sed -e 's/rootmode=rw/rootmode=ro/' -i /etc/init.d/checkroot.sh </code> I have referenced the file /etc/rcS.d/S36rosystem-fix a few times now. This file is a init-script-hack that sets up a few locations and files that are needed right after the mount of local filesystems. I'm not listing the entire file here, to examine the content, and what it does, check out the current version at http://dilbert.hig.no/jonl/rosystem-fix.txt The file is placed in /etc/init.d as normal, set executable, and symlinked into /etc/rcS.d. The number is chosen so that it is run right after mountlocal-bootmisc.sh. <code> wget http://dilbert.hig.no/jonl/rosystem-fix.txt -O /etc/init.d/rosystem-fix chmod 755 /etc/init.d/rosystem-fix ln -s /etc/init.d/rosystem-fix /etc/rcS.d/S36rosystem-fix </code> The final changes done to the prototype image, is to set up a pair of handy aliases to remount the root filesystem in read-write and read-only setups, respectively. <code> echo "alias ro='mount -o remount,ro /'" >> /root/.bashrc echo "alias rw='mount -o remount,rw /'" >> /root/.bashrc </code> ===== Transferring back ===== Transferring the image back is simply a task of unmounting the loopmount, and dumping the resulting image back onto the CF card. Unmounting first... <code> exit cd ~ sudo umount -l /mnt/proc sudo umount -l /mnt/dev/ sudo umount /mnt/ </code> And dumping the image back. Yes, I am using 'cat'. I should have used 'dd'. But both tools do the same job, so, why bother with complexity when you can have simplicity... <code> sudo cat lenny_testimg1.img > /dev/sdb </code> ===== Booting up, verifying ===== After tanking up the CF image, the card is moved back to the eBOX, and the eBOX is booted up. While the system is booting, check closely for stuff that fails, and verify that all essential systems start up OK. In the first iteration, I had a whole lot of errors. I went though them and corrected all the critical ones, and updated this document to reflect the corrections. The corrections that I was not able to fix by cleanly modifying existing startup scripts, or doing bind-mounts or tempfs mounts, are fixed in the rosystem-fix startup script. So, if you follow this walkthrough, and have errors, the likely places to put your fixes will be in /etc/fstab and /etc/init.d/rosystem-fix. After booting up, the next task is to make this a kind of Kiosk; i.e. starting a web browser in full screen... ===== X Server installation ===== I decided to try to keep the X Server installation small, but not super-small ;) By using apt, a bunch of dependencies that are not really needed are pulled in, but at least it is not as bloated as it would be, had I pulled in a complete desktop meta-package. First: set the filesystem writable.... <code> rw </code> <code> apt-get install pci-utils dbus dbus-x11 defoma x11-apps x11-session-utils x11-utils \ x11-xserver-utils xfonts-base xserver-xorg-input-kbd \ xserver-xorg-input-mouse xserver-xorg-video-dummy xserver-xorg-video-openchrome \ xserver-xorg-video-vesa xserver-xorg-video-vga xinit cpp cpp-4.3 xserver-xorg \ xserver-xorg-core xfs xfonts-100dpi xfonts-75dpi xfonts-scalable ttf-dejavu \ ttf-freefont gtk2-engines xterm blackbox iceweasel unclutter </code> We need some additional fonts for our setup. Note that we add the "contrib" package of Microsoft Core fonts. No religious views here, we allow non-free stuff in... Klavika is a font that is used extensively in the GUC visual profile. <code> echo "deb http://ftp.no.debian.org/debian/ lenny contrib" >> /etc/apt/sources.list echo "deb http://ftp.no.debian.org/debian/ lenny non-free" >> /etc/apt/sources.list apt-get update apt-get install msttcorefonts cd /tmp wget http://dilbert.hig.no/jonl/klavika.tar.bz2 cd /usr/share/fonts tar jxvf /tmp/klavika.tar.bz2 fc-cache -f -v fc-list | grep Kl # to confirm the install </code> The Xorg configuration will need to be tweaked to the display the Kiosk is to run with. Finally, the filesystem is set read-only again. <code> ro </code> ===== Setting up the Kiosk function... ===== With X up and running, and the filesystem back in read-only mode, make sure the non-root user crated during install is available for use, by setting up the home directory: <code> mkdir /home/kitteh chown kitteh.kittteh /home/kitteh </code> Now, either switch to a different VT, or log out root. Log in as the non-root user created during install, create a simple ~/.xinitrc: <code> # File: /home/kitteh/.xinitrc xterm & blackbox </code> Start up X: <code> startx </code> Now, start up iceweasel (yeah, firefox is not called firefox, but iceweasel on Debian, due to licensing issues). Install the Autohide extension from http://www.krickelkrackel.de/autohide/, and configure it for the local user. By enabling this extension, and exiting iceweasel while it is displayed fullscreen, iceweasel will "always" start in full-screen mode. After setting iceweasel up, update the .xinitrc: <code> # File: /home/kitteh/.xinitrc xsetroot -solid white & xset -dpms s off & unclutter & xterm -iconic & iceweasel http://the.url.of.the.kiosk.example.com & blackbox </code> After this, exit X11, test that it works OK, and exit the non-root login completely. Now, we need to retain the settings made from boot to boot. /home is mounted as a tmpfs, so we need to clone the kitteh home directory somewhere else on the file system, and push it back in place on boot. Log in as root.. <code> rw cp -pRv /home/kitteh /usr/local/kitteh </code> Add the following to /etc/rc.local to clone the saved homedir content on boot: <code> cp -R /usr/local/kitteh /home/kitteh chown -R kitteh.kitteh /home/kitteh </code> I have opted to force X to start on boot by running it from /etc/rc.local as well, by adding this line just above "exit 0": <code> su -c "startx" kitteh </code> Now, X will normally not start unless the user starting it is also controlling the console. As I am starting X from an rc-script, that is not the case. So, to allow rc.local starting X as the user "kitteh", put the following content in /etc/X11/Xwrapper.config <code> # Keeping the old setting as a comment, for clarity... # allowed_users=console # # This is not really seen as "safe", but in our case, it is needed.. allowed_users=anybody # # Niceness value? Nah.... nice_value=0 </code> That should really be enough for starters. Close everything up, set the file system as read only again, and reboot. <code> ro reboot </code>