More space for packages with extroot on your OpenWrt router

Sam Hobbs ·

OpenWrt router EXTroot

If you would like to install extra packages on OpenWrt, but you have run out of space on your router’s internal flash memory, then this tutorial is for you.

The plan is to copy the OpenWrt’s root filesystem onto an external USB flash drive, and tell the router to switch to that when it boots up.

All you need is a standard USB flash drive, a USB capable router running OpenWrt, and about 30 mins.

Theory

The OpenWrt wiki pages for this subject are very good. If you’re interested in reading about all the different options, then take a look (for example, it’s possible to mount the USB drive on /opt and install extras here, but some packages expect to be installed in root and won’t behave nicely).

In case you’d like to look it up later, the type of extroot that I’m talking about here is external root (aka pivot root), not external overlay (aka pivot-overlay).

Process

This assumes that you have already formatted your external USB drive with a journaling filesystem (like EXT3 or EXT4). If you haven’t, do this first (the partition managers in Ubuntu and Kubuntu do this very easily, do a google search for a tutorial if you’re unsure how to do this).

To use pivot root, you must be using a version of OpenWrt that is newer than v12 (so 12.09 Attitude Adjustment is fine, but Backfire is not).

First, connect to OpenWrt via ssh or telnet.

Install some packages

First, install the package that will swap your root from the router’s inbuilt flash memory to the external USB flash device:

opkg update
opkg install block-mount

Copy your current root filesystem to the USB flash drive

Before you do this, make sure your router is secure (e.g. SSH login isn’t possible from WAN, wifi interfaces are password protected etc).

This is important because if something goes wrong when your router boots up then OpenWrt will be unable to boot to the USB flash drive, leaving you with whatever your present setup is until you can make it boot to the USB drive properly.

First, install fdisk, a tool to tell you info about attached devices:

opkg update
opkg install fdisk

Now, insert your USB drive and run the following command to get some details about it:

root@OpenWrt:~# fdisk -l
...
Disk /dev/sda: 4229 MB, 4229496832 bytes
255 heads, 63 sectors/track, 514 cylinders, total 8260736 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00933453

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1              63     8257409     4128673+  83  Linux

You see your USB drive listed under “Device”. Now make a mount point for the drive. You can choose whatever you like instead of sda1, just remember what you chose later!

mkdir /mnt/sda1

Mount the drive on the mount point you just created (replace /dev/sda2 with whatever you got from the fdisk command, and /mnt/sda1 with whatever you chose as your mount point):

mount /dev/sda2 /mnt/sda1

Now copy the root filesystem from the router’s inbuilt flash to the USB drive with these commands (if you chose a name other than /mnt/sda1 in the previous step, then replace /mnt/sda1 in line 3):

mkdir -p /tmp/cproot
mount --bind / /tmp/cproot
tar -C /tmp/cproot -cvf - . | tar -C /mnt/sda1 -xf -
umount /tmp/cproot

You now have a USB drive with a copy of your router’s filesystem on it. The next step is to make it automatically mount when you boot up, and use it as root.

Configure /etc/config/fstab

Open /etc/config/fstab with your favourite text editor:

nano /etc/config/fstab

If nano is not installed, you can use vi (which is installed by default, but is horrible!) or you can just install nano with:

opkg update
opkg install nano

Add the following to the file:

config mount
        option target        /
        option device        /dev/sda1
        option fstype        ext3
        option options       rw,sync
        option enabled       1
        option enabled_fsck  0

Reboot and Check

When you reboot, you should now be running OpenWrt from your USB stick.

Check your current mounts with this command:

mount

Output should be something similar to the following:

root@OpenWrt:~# mount
rootfs on / type rootfs (rw)
/dev/root on /rom type squashfs (ro,relatime)
proc on /proc type proc (rw,noatime)
sysfs on /sys type sysfs (rw,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime,size=63340k)
tmpfs on /dev type tmpfs (rw,noatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,noatime,mode=600)
/dev/sda1 on / type ext4 (rw,sync,relatime,user_xattr,barrier=1,data=ordered)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
none on /proc/bus/usb type usbfs (rw,relatime)

See how much space you’ve created for exciting packages with this command:

df -h

Which should show you something like this:

root@OpenWrt:~# df -h
Filesystem                Size      Used Available Use% Mounted on
rootfs                    3.9G    157.1M      3.6G   4% /
/dev/root                 2.0M      2.0M         0 100% /rom
tmpfs                    61.9M    932.0K     60.9M   1% /tmp
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/sda1                 3.9G    157.1M      3.6G   4% /

You’re all done. Have fun building your personalised OpenWrt cathedral!

Tags: 
openwrt
opkg
USB
partitioning
Category: 
DD-WRT and OpenWrt
Type: 
Tutorial

Comments

Marius Boeru ·

Hello,
Thank you for the great article. I wanted to add some things that I had to do extra on my router TP-LINK MR3040, that may need to be added to the article in order for it to be more generic.

1. I had to install via opkg the package kmod-scsi-core, because otherwise, even though I saw the USB device using lsusb, the ext4 device was nowhere to be seen when doing fdisk -l.

2. My /etc/config/fstab file was completly empty and after adding the whole "config mount", after reboot nothing happened. What I did in the end was run #block detect and copy the "config global" part into /etc/config/fstab above the "config mount" part

3. This is what I now run in order to install everything I need:

opkg update
opkg install kmod-scsi-core kmod-usb-core kmod-usb2 kmod-usb-storage kmod-fs-ext4 fdisk block-mount

Cheers!

Roy ·

Hi
I'm trying to follow your advice however the symlinks are not being copied:

tar: can't create symlink from ./etc/fstab to /tmp/fstab: Operation not permitted

Is it important?

Sam Hobbs ·

Hi Roy,

I can't remember what happened when I followed the instructions, it was about a year ago.

No harm in trying though, if you boot the router without the USB drive in it will fall back to a "normal" boot from the ROM. So... you can try it with it in, see what happens, if the boot fails just take the USB drive out and do a normal boot of the router, put the USB in your laptop and take a look at the syslog to see what went wrong.

Sam

Roy ·

Hi,
The Arduino Yun (running OpenWRT) has a nice tool for expanding the disk size: http://arduino.cc/en/Tutorial/ExpandingYunDiskSpace
Looking inside it for the console commands, it formats the new disk to ext4 (mkfs.ext4) and then copies the files with rsync.
Maybe it'll be helpful for someone who gets to this page...

J ·

worked like a charm, i followed a few other guides with no luck, works well, should i create a swap partition on sda1 or leave with no swap, thanks again for the efforts here and time spent creating this tutorial

Sam Hobbs ·

Thanks for your comment, I wrote this a year ago and haven't done it since... It's nice to know that it still works!

I wouldn't bother with swap, if you run out of RAM you're probably screwed in terms of performance, and putting swap on a flash drive will cause fairly rapid deterioration of the drive. I learned that one the hard way, for about ten months this site was hosted on a raspberry pi with the root FS on a flash drive, and one day it just died spectacularly!

Do you have any reason to think you're running out of RAM?

Sam

Fabio ·

Hi,
EXCELLENT guide! Thx!
But... I DO need a swap partition, because transmission-daemon is eating my memory so fast and stops itself after a while... I'm running OpenWRT on TL-WR710N; I can't mount swap partition, whatever I do. I get only this message: "block: failed to swapon /dev/sda5". Do you have any advice?
Thanks!

Sam Hobbs ·

You're welcome!

See my reply to a comment above about swap on a flash drive, I wouldn't recommend it.

To be honest, I don't think running a bittorrent client on a router is a great idea - router hardware isn't really up to the task, as you have discovered! My router only has 128MB of RAM (less than a raspberry pi!). I'd move the client onto a different machine if I were you.

Thanks for commenting!

Sam

Saldanha ·

Sam,
Thank you very much !
Your tutorial worked fine ! Now I able to install openvpn ! and make a backup of the entire router system !

Alex ·

Hi Sam!
I have a TP-Link 3420 (HW 1.1) with OpenWrt Chaos Calmer 15.05 / LuCI (git-15.248.30277-3836b45)
I'm trying to follow your instructions but, after installing block-mount, I cannot install fdisk because of lack of free space.
So i hoped to use
opkg install fdisk -d USB_path but I cannot mount the USB device (/dev/sda1) because my filesystem is read-only.
----
~# mount
rootfs on / type rootfs (rw)
/dev/root on /rom type squashfs (ro,relatime)
proc on /proc type proc (rw,noatime)
sysfs on /sys type sysfs (rw,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
/dev/mtdblock3 on /overlay type jffs2 (rw,noatime)
overlayfs:/overlay on / type overlay (ro,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
tmpfs on /dev type tmpfs (rw,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,relatime,mode=600)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)
---

Please help!!
Alex

Alex ·

Hi again SAM!
I tried installing fdisk in ram..

opkg install fdisk -d ram

Installing fdisk (2.25.2-4) to ram...
Downloading http://downloads.openwrt.org/chaos_calmer/15.05/ar71xx/generic/packages/....
Installing libblkid (2.25.2-4) to ram...
Downloading http://downloads.openwrt.org/chaos_calmer/15.05/ar71xx/generic/packages/....
Installing libuuid (2.25.2-4) to ram...
Downloading http://downloads.openwrt.org/chaos_calmer/15.05/ar71xx/generic/packages/....
Installing libsmartcols (2.25.2-4) to ram...
Downloading http://downloads.openwrt.org/chaos_calmer/15.05/ar71xx/generic/packages/....
Configuring libuuid.
grep: /usr/lib/opkg/info/libuuid.control: No such file or directory
cat: can't open '/usr/lib/opkg/info/libuuid.list': No such file or directory
Configuring libblkid.
cat: can't open '/usr/lib/opkg/info/libblkid.list': No such file or directory
Configuring libsmartcols.
grep: /usr/lib/opkg/info/libsmartcols.control: No such file or directory
cat: can't open '/usr/lib/opkg/info/libsmartcols.list': No such file or directory
Configuring fdisk.
grep: /usr/lib/opkg/info/fdisk.control: No such file or directory
cat: can't open '/usr/lib/opkg/info/fdisk.list': No such file or directory

Thanks,
Alex

Sam Hobbs ·

Hi Alex,

I guess it can't be installed to RAM because it needs to write some configuration files.

I think you're right that the problem is that you have a read only filesystem. Looking at your mount output again, I saw this:

overlayfs:/overlay on / type overlay (ro,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)

If you look on the OpenWrt wiki page describing flash layout under the "mount points" section, it says that the purpose of /overlay is to contain the writable part of the filesystem, so I have no idea why it's read only on yours!

"/overlay" is the writable part of the file system that gets merged with /rom to create a uniform /-tree. It contains anything that was written to the router after installation, e.g. changed configuration files, additional packages installed with OPKG, etc. It is formated with JFFS2.

Anyway, I guess we can just try re-mounting it read-writable! I'm no mount guru, but I think this will work:

mount -o remount,rw /overlay

I think mount reads the default options in /etc/fstab and merges them with whatever you specify on the commandline, so I don't think it's necessary to list all the options (lowerdir, upperdir etc.) again, but I'm not certain.

Finally, are you sure you want to remove wpad-mini? Seems like an important package. If it was me I think I would remove some luci packages temporarily (so you'd lose the web interface) because they're not required for the router to function, and you can still interact with it via SSH.

Sam

Alex ·

Hi Sam!
Thanks for your reply !

I tried removing some package (odhcp6c and wpad-mini, for example) with
opkg remove --force-remove but nothing happens.

It says:
Removing package wpad-mini from root...

but nothing happens (df -h has the same values..). I believe it's about READ-ONLY filesystem..

This is the opkg -list-installed output.

base-files - 157-r46767
block-mount - 2015-05-24-09027fc86babc3986027a0e677aca1b6999a9e14
busybox - 1.23.2-1
dnsmasq - 2.73-1
dropbear - 2015.67-1
firewall - 2015-07-27
fstools - 2015-05-24-09027fc86babc3986027a0e677aca1b6999a9e14
hostapd-common - 2015-03-25-1
ip6tables - 1.4.21-1
iptables - 1.4.21-1
iw - 3.17-1
jshn - 2015-06-14-d1c66ef1131d14f0ed197b368d03f71b964e45f8
jsonfilter - 2014-06-19-cdc760c58077f44fc40adbbe41e1556a67c1b9a9
kernel - 3.18.20-1-7bed08fa9c06eb8089e82c200340ec66
kmod-ath - 3.18.20+2015-03-09-3
kmod-ath9k - 3.18.20+2015-03-09-3
kmod-ath9k-common - 3.18.20+2015-03-09-3
kmod-cfg80211 - 3.18.20+2015-03-09-3
kmod-crypto-aes - 3.18.20-1
kmod-crypto-arc4 - 3.18.20-1
kmod-crypto-core - 3.18.20-1
kmod-gpio-button-hotplug - 3.18.20-1
kmod-ip6tables - 3.18.20-1
kmod-ipt-conntrack - 3.18.20-1
kmod-ipt-core - 3.18.20-1
kmod-ipt-nat - 3.18.20-1
kmod-ipv6 - 3.18.20-1
kmod-ledtrig-usbdev - 3.18.20-1
kmod-lib-crc-ccitt - 3.18.20-1
kmod-mac80211 - 3.18.20+2015-03-09-3
kmod-nf-conntrack - 3.18.20-1
kmod-nf-conntrack6 - 3.18.20-1
kmod-nf-ipt - 3.18.20-1
kmod-nf-ipt6 - 3.18.20-1
kmod-nf-nat - 3.18.20-1
kmod-nf-nathelper - 3.18.20-1
kmod-nls-base - 3.18.20-1
kmod-ppp - 3.18.20-1
kmod-pppoe - 3.18.20-1
kmod-pppox - 3.18.20-1
kmod-scsi-core - 3.18.20-1
kmod-slhc - 3.18.20-1
kmod-usb-core - 3.18.20-1
kmod-usb-storage - 3.18.20-1
kmod-usb-storage-extras - 3.18.20-1
kmod-usb2 - 3.18.20-1
libblobmsg-json - 2015-06-14-d1c66ef1131d14f0ed197b368d03f71b964e45f8
libc - 0.9.33.2-1
libgcc - 4.8-linaro-1
libip4tc - 1.4.21-1
libip6tc - 1.4.21-1
libiwinfo - 2015-06-01-ade8b1b299cbd5748db1acf80dd3e9f567938371
libiwinfo-lua - 2015-06-01-ade8b1b299cbd5748db1acf80dd3e9f567938371
libjson-c - 0.12-1
libjson-script - 2015-06-14-d1c66ef1131d14f0ed197b368d03f71b964e45f8
liblua - 5.1.5-1
libnl-tiny - 0.1-4
libpcre - 8.37-2
libubox - 2015-06-14-d1c66ef1131d14f0ed197b368d03f71b964e45f8
libubus - 2015-05-25-f361bfa5fcb2daadf3b160583ce665024f8d108e
libubus-lua - 2015-05-25-f361bfa5fcb2daadf3b160583ce665024f8d108e
libuci - 2015-04-09.1-1
libuci-lua - 2015-04-09.1-1
libxtables - 1.4.21-1
lua - 5.1.5-1
luci - git-15.248.30277-3836b45-1
luci-app-firewall - git-15.248.30277-3836b45-1
luci-base - git-15.248.30277-3836b45-1
luci-lib-ip - git-15.248.30277-3836b45-1
luci-lib-nixio - git-15.248.30277-3836b45-1
luci-mod-admin-full - git-15.248.30277-3836b45-1
luci-proto-ipv6 - git-15.248.30277-3836b45-1
luci-proto-ppp - git-15.248.30277-3836b45-1
luci-theme-bootstrap - git-15.248.30277-3836b45-1
mtd - 21
netifd - 2015-06-08-8795f9ef89626cd658f615c78c6a17e990c0dcaa
odhcp6c - 2015-07-13-024525798c5f6aba3af9b2ef7b3af2f3c14f1db8
odhcpd - 2015-05-21-2ebf6c8216287983779c8ec6597d30893b914a7c
opkg - 9c97d5ecd795709c8584e972bfdf3aee3a5b846d-7
ppp - 2.4.7-6
ppp-mod-pppoe - 2.4.7-6
procd - 2015-08-16-0da5bf2ff222d1a499172a6e09507388676b5a08
rpcd - 2015-05-17-3d655417ab44d93aad56a6d4a668daf24b127b84
swconfig - 10
uboot-envtools - 2014.10-2
ubox - 2015-07-14-907d046c8929fb74e5a3502a9498198695e62ad8
ubus - 2015-05-25-f361bfa5fcb2daadf3b160583ce665024f8d108e
ubusd - 2015-05-25-f361bfa5fcb2daadf3b160583ce665024f8d108e
uci - 2015-04-09.1-1
uhttpd - 2015-08-17-f91788b809d9726126e9cf4384fedbbb0c5b8a73
uhttpd-mod-ubus - 2015-08-17-f91788b809d9726126e9cf4384fedbbb0c5b8a73
usign - 2015-05-08-cf8dcdb8a4e874c77f3e9a8e9b643e8c17b19131
wpad-mini - 2015-03-25-1

Thanks for any other help.
Alex

diego ·

Thanks this opens a world of possibilities! Although I had to build my own minimal image without luci, then follow your steps and then install luci and everything else.

KingBeetle ·

Thank you so much for the great tutorial! Your tutorial just made my $12 toy something I can actually use.
Out of the box (so to speak) a fresh firmware flash on the VoCore2-Lite leaves you with a whopping 916.0K of available space on root:
Filesystem Size Used Available Use% Mounted on
rootfs 1.1M 236.0K 916.0K 20% /

After formatting a 16GB USB flashdrive and following your tutorial:
Filesystem Size Used Available Use% Mounted on
rootfs 14.1G 57.9M 13.3G 0% /

Now I can actually install more then three packages without running out of space! Thanks!

Narasing ·

Hi,
I've LG01 gateway runs openWrt system. I want to attach USB flash device for that i followed your article and https://openwrt.org/docs/guide-user/storage/usb-drives-quickstart artical.
Now USB flash device is mounted.

root@dragino-1c0b14:~# df -h
Filesystem Size Used Available Use% Mounted on
rootfs 3.5M 632.0K 2.9M 18% /
/dev/root 11.0M 11.0M 0 100% /rom
tmpfs 29.8M 484.0K 29.4M 2% /tmp
/dev/mtdblock4 3.5M 632.0K 2.9M 18% /overlay
overlayfs:/overlay 3.5M 632.0K 2.9M 18% /
tmpfs 512.0K 0 512.0K 0% /dev
/dev/sda1 7.0G 16.3M 6.6G 0% /mnt/sda1

now i wanted to copy some file to USB flash device. How to do it, which place i've to store it and run .

Thanks

André ·

Hey.

Can you update the howto for OpenWRT 18.06.4?

I get the warning "block-mount is deprecated".

And it triggered a boot loop.

Add new comment