Sam Hobbs ·
If you would like to install extra packages on OpenWrt, but you have run out of space on your router’s internal flash memory, then this tutorial is for you.
The plan is to copy the OpenWrt’s root filesystem onto an external USB flash drive, and tell the router to switch to that when it boots up.
All you need is a standard USB flash drive, a USB capable router running OpenWrt, and about 30 mins.
Theory
The OpenWrt wiki pages for this subject are very good. If you’re interested in reading about all the different options, then take a look (for example, it’s possible to mount the USB drive on /opt and install extras here, but some packages expect to be installed in root and won’t behave nicely).
In case you’d like to look it up later, the type of extroot that I’m talking about here is external root (aka pivot root), not external overlay (aka pivot-overlay).
Process
This assumes that you have already formatted your external USB drive with a journaling filesystem (like EXT3 or EXT4). If you haven’t, do this first (the partition managers in Ubuntu and Kubuntu do this very easily, do a google search for a tutorial if you’re unsure how to do this).
To use pivot root, you must be using a version of OpenWrt that is newer than v12 (so 12.09 Attitude Adjustment is fine, but Backfire is not).
First, connect to OpenWrt via ssh or telnet.
Install some packages
First, install the package that will swap your root from the router’s inbuilt flash memory to the external USB flash device:
opkg update opkg install block-mount
Copy your current root filesystem to the USB flash drive
Before you do this, make sure your router is secure (e.g. SSH login isn’t possible from WAN, wifi interfaces are password protected etc).
This is important because if something goes wrong when your router boots up then OpenWrt will be unable to boot to the USB flash drive, leaving you with whatever your present setup is until you can make it boot to the USB drive properly.
First, install fdisk, a tool to tell you info about attached devices:
opkg update opkg install fdisk
Now, insert your USB drive and run the following command to get some details about it:
root@OpenWrt:~# fdisk -l ... Disk /dev/sda: 4229 MB, 4229496832 bytes 255 heads, 63 sectors/track, 514 cylinders, total 8260736 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00933453 Device Boot Start End Blocks Id System /dev/sda1 63 8257409 4128673+ 83 Linux
You see your USB drive listed under “Device”. Now make a mount point for the drive. You can choose whatever you like instead of sda1, just remember what you chose later!
mkdir /mnt/sda1
Mount the drive on the mount point you just created (replace /dev/sda2 with whatever you got from the fdisk command, and /mnt/sda1 with whatever you chose as your mount point):
mount /dev/sda2 /mnt/sda1
Now copy the root filesystem from the router’s inbuilt flash to the USB drive with these commands (if you chose a name other than /mnt/sda1 in the previous step, then replace /mnt/sda1 in line 3):
mkdir -p /tmp/cproot mount --bind / /tmp/cproot tar -C /tmp/cproot -cvf - . | tar -C /mnt/sda1 -xf - umount /tmp/cproot
You now have a USB drive with a copy of your router’s filesystem on it. The next step is to make it automatically mount when you boot up, and use it as root.
Configure /etc/config/fstab
Open /etc/config/fstab with your favourite text editor:
nano /etc/config/fstab
If nano is not installed, you can use vi (which is installed by default, but is horrible!) or you can just install nano with:
opkg update opkg install nano
Add the following to the file:
config mount option target / option device /dev/sda1 option fstype ext3 option options rw,sync option enabled 1 option enabled_fsck 0
Reboot and Check
When you reboot, you should now be running OpenWrt from your USB stick.
Check your current mounts with this command:
mount
Output should be something similar to the following:
root@OpenWrt:~# mount rootfs on / type rootfs (rw) /dev/root on /rom type squashfs (ro,relatime) proc on /proc type proc (rw,noatime) sysfs on /sys type sysfs (rw,noatime) tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime,size=63340k) tmpfs on /dev type tmpfs (rw,noatime,size=512k,mode=755) devpts on /dev/pts type devpts (rw,noatime,mode=600) /dev/sda1 on / type ext4 (rw,sync,relatime,user_xattr,barrier=1,data=ordered) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /proc/bus/usb type usbfs (rw,relatime)
See how much space you’ve created for exciting packages with this command:
df -h
Which should show you something like this:
root@OpenWrt:~# df -h Filesystem Size Used Available Use% Mounted on rootfs 3.9G 157.1M 3.6G 4% / /dev/root 2.0M 2.0M 0 100% /rom tmpfs 61.9M 932.0K 60.9M 1% /tmp tmpfs 512.0K 0 512.0K 0% /dev /dev/sda1 3.9G 157.1M 3.6G 4% /
You’re all done. Have fun building your personalised OpenWrt cathedral!
Comments
Marius Boeru · Permalink
additions
Hello,
Thank you for the great article. I wanted to add some things that I had to do extra on my router TP-LINK MR3040, that may need to be added to the article in order for it to be more generic.
1. I had to install via opkg the package kmod-scsi-core, because otherwise, even though I saw the USB device using lsusb, the ext4 device was nowhere to be seen when doing fdisk -l.
2. My /etc/config/fstab file was completly empty and after adding the whole "config mount", after reboot nothing happened. What I did in the end was run #block detect and copy the "config global" part into /etc/config/fstab above the "config mount" part
3. This is what I now run in order to install everything I need:
Cheers!
Sam Hobbs · Permalink
Thank you for leaving such a
Thank you for leaving such a useful comment :)
Roy · Permalink
Symlinks not copied in the tar ?
Hi
I'm trying to follow your advice however the symlinks are not being copied:
tar: can't create symlink from ./etc/fstab to /tmp/fstab: Operation not permitted
Is it important?
Sam Hobbs · Permalink
Hi Roy,
Hi Roy,
I can't remember what happened when I followed the instructions, it was about a year ago.
No harm in trying though, if you boot the router without the USB drive in it will fall back to a "normal" boot from the ROM. So... you can try it with it in, see what happens, if the boot fails just take the USB drive out and do a normal boot of the router, put the USB in your laptop and take a look at the syslog to see what went wrong.
Sam
Roy · Permalink
Hi,
Hi,
The Arduino Yun (running OpenWRT) has a nice tool for expanding the disk size: http://arduino.cc/en/Tutorial/ExpandingYunDiskSpace
Looking inside it for the console commands, it formats the new disk to ext4 (mkfs.ext4) and then copies the files with rsync.
Maybe it'll be helpful for someone who gets to this page...
J · Permalink
Fantastic
worked like a charm, i followed a few other guides with no luck, works well, should i create a swap partition on sda1 or leave with no swap, thanks again for the efforts here and time spent creating this tutorial
Sam Hobbs · Permalink
I wouldn't bother with SWAP
Thanks for your comment, I wrote this a year ago and haven't done it since... It's nice to know that it still works!
I wouldn't bother with swap, if you run out of RAM you're probably screwed in terms of performance, and putting swap on a flash drive will cause fairly rapid deterioration of the drive. I learned that one the hard way, for about ten months this site was hosted on a raspberry pi with the root FS on a flash drive, and one day it just died spectacularly!
Do you have any reason to think you're running out of RAM?
Sam
Fabio · Permalink
Swap partition
Hi,
EXCELLENT guide! Thx!
But... I DO need a swap partition, because transmission-daemon is eating my memory so fast and stops itself after a while... I'm running OpenWRT on TL-WR710N; I can't mount swap partition, whatever I do. I get only this message: "block: failed to swapon /dev/sda5". Do you have any advice?
Thanks!
Sam Hobbs · Permalink
You're welcome!
You're welcome!
See my reply to a comment above about swap on a flash drive, I wouldn't recommend it.
To be honest, I don't think running a bittorrent client on a router is a great idea - router hardware isn't really up to the task, as you have discovered! My router only has 128MB of RAM (less than a raspberry pi!). I'd move the client onto a different machine if I were you.
Thanks for commenting!
Sam
Saldanha · Permalink
Thanks !
Sam,
Thank you very much !
Your tutorial worked fine ! Now I able to install openvpn ! and make a backup of the entire router system !
Alex · Permalink
Help!
Hi Sam!
I have a TP-Link 3420 (HW 1.1) with OpenWrt Chaos Calmer 15.05 / LuCI (git-15.248.30277-3836b45)
I'm trying to follow your instructions but, after installing block-mount, I cannot install fdisk because of lack of free space.
So i hoped to use
opkg install fdisk -d USB_path but I cannot mount the USB device (/dev/sda1) because my filesystem is read-only.
----
~# mount
rootfs on / type rootfs (rw)
/dev/root on /rom type squashfs (ro,relatime)
proc on /proc type proc (rw,noatime)
sysfs on /sys type sysfs (rw,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
/dev/mtdblock3 on /overlay type jffs2 (rw,noatime)
overlayfs:/overlay on / type overlay (ro,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
tmpfs on /dev type tmpfs (rw,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,relatime,mode=600)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)
---
Please help!!
Alex
Sam Hobbs · Permalink
Any packages you could remove?
Hi Alex,
Are there any other packages you don't need that you could temporarily remove to make space for fdisk?
Sam
Alex · Permalink
Hi again SAM!
Hi again SAM!
I tried installing fdisk in ram..
opkg install fdisk -d ram
Installing fdisk (2.25.2-4) to ram...
Downloading http://downloads.openwrt.org/chaos_calmer/15.05/ar71xx/generic/packages/....
Installing libblkid (2.25.2-4) to ram...
Downloading http://downloads.openwrt.org/chaos_calmer/15.05/ar71xx/generic/packages/....
Installing libuuid (2.25.2-4) to ram...
Downloading http://downloads.openwrt.org/chaos_calmer/15.05/ar71xx/generic/packages/....
Installing libsmartcols (2.25.2-4) to ram...
Downloading http://downloads.openwrt.org/chaos_calmer/15.05/ar71xx/generic/packages/....
Configuring libuuid.
grep: /usr/lib/opkg/info/libuuid.control: No such file or directory
cat: can't open '/usr/lib/opkg/info/libuuid.list': No such file or directory
Configuring libblkid.
cat: can't open '/usr/lib/opkg/info/libblkid.list': No such file or directory
Configuring libsmartcols.
grep: /usr/lib/opkg/info/libsmartcols.control: No such file or directory
cat: can't open '/usr/lib/opkg/info/libsmartcols.list': No such file or directory
Configuring fdisk.
grep: /usr/lib/opkg/info/fdisk.control: No such file or directory
cat: can't open '/usr/lib/opkg/info/fdisk.list': No such file or directory
Thanks,
Alex
Sam Hobbs · Permalink
remount overlay read-write
Hi Alex,
I guess it can't be installed to RAM because it needs to write some configuration files.
I think you're right that the problem is that you have a read only filesystem. Looking at your
mount
output again, I saw this:If you look on the OpenWrt wiki page describing flash layout under the "mount points" section, it says that the purpose of
/overlay
is to contain the writable part of the filesystem, so I have no idea why it's read only on yours!Anyway, I guess we can just try re-mounting it read-writable! I'm no
mount
guru, but I think this will work:I think mount reads the default options in /etc/fstab and merges them with whatever you specify on the commandline, so I don't think it's necessary to list all the options (lowerdir, upperdir etc.) again, but I'm not certain.
Finally, are you sure you want to remove
wpad-mini
? Seems like an important package. If it was me I think I would remove some luci packages temporarily (so you'd lose the web interface) because they're not required for the router to function, and you can still interact with it via SSH.Sam
Alex · Permalink
Package list
Hi Sam!
Thanks for your reply !
I tried removing some package (odhcp6c and wpad-mini, for example) with
opkg remove --force-remove but nothing happens.
It says:
Removing package wpad-mini from root...
but nothing happens (df -h has the same values..). I believe it's about READ-ONLY filesystem..
This is the opkg -list-installed output.
base-files - 157-r46767
block-mount - 2015-05-24-09027fc86babc3986027a0e677aca1b6999a9e14
busybox - 1.23.2-1
dnsmasq - 2.73-1
dropbear - 2015.67-1
firewall - 2015-07-27
fstools - 2015-05-24-09027fc86babc3986027a0e677aca1b6999a9e14
hostapd-common - 2015-03-25-1
ip6tables - 1.4.21-1
iptables - 1.4.21-1
iw - 3.17-1
jshn - 2015-06-14-d1c66ef1131d14f0ed197b368d03f71b964e45f8
jsonfilter - 2014-06-19-cdc760c58077f44fc40adbbe41e1556a67c1b9a9
kernel - 3.18.20-1-7bed08fa9c06eb8089e82c200340ec66
kmod-ath - 3.18.20+2015-03-09-3
kmod-ath9k - 3.18.20+2015-03-09-3
kmod-ath9k-common - 3.18.20+2015-03-09-3
kmod-cfg80211 - 3.18.20+2015-03-09-3
kmod-crypto-aes - 3.18.20-1
kmod-crypto-arc4 - 3.18.20-1
kmod-crypto-core - 3.18.20-1
kmod-gpio-button-hotplug - 3.18.20-1
kmod-ip6tables - 3.18.20-1
kmod-ipt-conntrack - 3.18.20-1
kmod-ipt-core - 3.18.20-1
kmod-ipt-nat - 3.18.20-1
kmod-ipv6 - 3.18.20-1
kmod-ledtrig-usbdev - 3.18.20-1
kmod-lib-crc-ccitt - 3.18.20-1
kmod-mac80211 - 3.18.20+2015-03-09-3
kmod-nf-conntrack - 3.18.20-1
kmod-nf-conntrack6 - 3.18.20-1
kmod-nf-ipt - 3.18.20-1
kmod-nf-ipt6 - 3.18.20-1
kmod-nf-nat - 3.18.20-1
kmod-nf-nathelper - 3.18.20-1
kmod-nls-base - 3.18.20-1
kmod-ppp - 3.18.20-1
kmod-pppoe - 3.18.20-1
kmod-pppox - 3.18.20-1
kmod-scsi-core - 3.18.20-1
kmod-slhc - 3.18.20-1
kmod-usb-core - 3.18.20-1
kmod-usb-storage - 3.18.20-1
kmod-usb-storage-extras - 3.18.20-1
kmod-usb2 - 3.18.20-1
libblobmsg-json - 2015-06-14-d1c66ef1131d14f0ed197b368d03f71b964e45f8
libc - 0.9.33.2-1
libgcc - 4.8-linaro-1
libip4tc - 1.4.21-1
libip6tc - 1.4.21-1
libiwinfo - 2015-06-01-ade8b1b299cbd5748db1acf80dd3e9f567938371
libiwinfo-lua - 2015-06-01-ade8b1b299cbd5748db1acf80dd3e9f567938371
libjson-c - 0.12-1
libjson-script - 2015-06-14-d1c66ef1131d14f0ed197b368d03f71b964e45f8
liblua - 5.1.5-1
libnl-tiny - 0.1-4
libpcre - 8.37-2
libubox - 2015-06-14-d1c66ef1131d14f0ed197b368d03f71b964e45f8
libubus - 2015-05-25-f361bfa5fcb2daadf3b160583ce665024f8d108e
libubus-lua - 2015-05-25-f361bfa5fcb2daadf3b160583ce665024f8d108e
libuci - 2015-04-09.1-1
libuci-lua - 2015-04-09.1-1
libxtables - 1.4.21-1
lua - 5.1.5-1
luci - git-15.248.30277-3836b45-1
luci-app-firewall - git-15.248.30277-3836b45-1
luci-base - git-15.248.30277-3836b45-1
luci-lib-ip - git-15.248.30277-3836b45-1
luci-lib-nixio - git-15.248.30277-3836b45-1
luci-mod-admin-full - git-15.248.30277-3836b45-1
luci-proto-ipv6 - git-15.248.30277-3836b45-1
luci-proto-ppp - git-15.248.30277-3836b45-1
luci-theme-bootstrap - git-15.248.30277-3836b45-1
mtd - 21
netifd - 2015-06-08-8795f9ef89626cd658f615c78c6a17e990c0dcaa
odhcp6c - 2015-07-13-024525798c5f6aba3af9b2ef7b3af2f3c14f1db8
odhcpd - 2015-05-21-2ebf6c8216287983779c8ec6597d30893b914a7c
opkg - 9c97d5ecd795709c8584e972bfdf3aee3a5b846d-7
ppp - 2.4.7-6
ppp-mod-pppoe - 2.4.7-6
procd - 2015-08-16-0da5bf2ff222d1a499172a6e09507388676b5a08
rpcd - 2015-05-17-3d655417ab44d93aad56a6d4a668daf24b127b84
swconfig - 10
uboot-envtools - 2014.10-2
ubox - 2015-07-14-907d046c8929fb74e5a3502a9498198695e62ad8
ubus - 2015-05-25-f361bfa5fcb2daadf3b160583ce665024f8d108e
ubusd - 2015-05-25-f361bfa5fcb2daadf3b160583ce665024f8d108e
uci - 2015-04-09.1-1
uhttpd - 2015-08-17-f91788b809d9726126e9cf4384fedbbb0c5b8a73
uhttpd-mod-ubus - 2015-08-17-f91788b809d9726126e9cf4384fedbbb0c5b8a73
usign - 2015-05-08-cf8dcdb8a4e874c77f3e9a8e9b643e8c17b19131
wpad-mini - 2015-03-25-1
Thanks for any other help.
Alex
diego · Permalink
Thanks this opens a world of
Thanks this opens a world of possibilities! Although I had to build my own minimal image without luci, then follow your steps and then install luci and everything else.
KingBeetle · Permalink
You made my $12 64MB VoCore2-Lite a usable piece of hardware!
Thank you so much for the great tutorial! Your tutorial just made my $12 toy something I can actually use.
Out of the box (so to speak) a fresh firmware flash on the VoCore2-Lite leaves you with a whopping 916.0K of available space on root:
Filesystem Size Used Available Use% Mounted on
rootfs 1.1M 236.0K 916.0K 20% /
After formatting a 16GB USB flashdrive and following your tutorial:
Filesystem Size Used Available Use% Mounted on
rootfs 14.1G 57.9M 13.3G 0% /
Now I can actually install more then three packages without running out of space! Thanks!
Sam Hobbs · Permalink
Happy Christmas from me in 2013
Haha, Happy Christmas from me in 2013! Glad it was useful.
Narasing · Permalink
copy data to USB flash device
Hi,
I've LG01 gateway runs openWrt system. I want to attach USB flash device for that i followed your article and https://openwrt.org/docs/guide-user/storage/usb-drives-quickstart artical.
Now USB flash device is mounted.
root@dragino-1c0b14:~# df -h
Filesystem Size Used Available Use% Mounted on
rootfs 3.5M 632.0K 2.9M 18% /
/dev/root 11.0M 11.0M 0 100% /rom
tmpfs 29.8M 484.0K 29.4M 2% /tmp
/dev/mtdblock4 3.5M 632.0K 2.9M 18% /overlay
overlayfs:/overlay 3.5M 632.0K 2.9M 18% /
tmpfs 512.0K 0 512.0K 0% /dev
/dev/sda1 7.0G 16.3M 6.6G 0% /mnt/sda1
now i wanted to copy some file to USB flash device. How to do it, which place i've to store it and run .
Thanks
André · Permalink
Update HowTo
Hey.
Can you update the howto for OpenWRT 18.06.4?
I get the warning "block-mount is deprecated".
And it triggered a boot loop.
Add new comment