EasyTomato 0.8 Released


WRD - EasyTomato

WRD - EasyTomato

LI Guru
Member
Took us long enough but EasyTomato 0.8 is out. Lots of changes in this one. Big updates are...
  • New Bandwidth Graphing (see below)
  • Blocking HTTPS websites
  • Direct Flashing, no DD-WRT for RT-N16 (Thanks RMerlin!)
  • Integrated Adblocking (Thanks mstombs and the All-U-Need ad blocking folks)
  • Upgraded to Toastman v1.28.0502.7
bandwidth-graphing.png

You can make your way over the the EasyTomato blog for more info and a download link. If you find any bugs, please report them here.

There is still lots to do on the EasyTomato front, but our next to areas of focus are going to be on an EasyQoS and nvram compression (some schools are having 100+ kids on an ET router and their nvram fills).
 
I like where your project is heading. NVRAM compression is much overdue in EasyTomato. During my testing of RC2, I had about 3% of NVRAM left after I loaded all my settings. This is before I even created the groups feature. I essentially had to erase other settings to test the group feature which works as advertised.
 
I wonder if requiring the use of a USB drive would be within the realm of possibility, since then you could remove settings from NVRAM and place them in configuration files which would live on the USB drive. Flash drives are available for as little as $5, and even 2GB or 4GB of space would present an enormous increase in space.
 
The N16 has plenty of flash, so a few MB dedicated for settings would be fine I think. Tomato, however, is a bit fiddly with creating jffs -it which occupies the whole of free-space, which means that contents have to be backed up jffs removed before upgrade, then it all put back. I would like to be able to allocate an mtdblock towards the end of flash, before the nvram which could be less volatile. I have seen in a firmware for something a pre-formatted jffs partition but couldn't find how it was done. Due to squashfs compression a blank formatted block won't take up much space in the firmware file! But flash is precious and has finite life, if something that can write to it goes wrong it could wear a bit out.

Meanwhile dd-wrt are testing a universal 64kB nvram solution. Their code will identify what the CFE allocates, then I guess it uses the otherwise wasted flash blocks in the same erase block as extra space. ray123 tried this sort of thing for Tomato, but it is a bit risky - it appears the Broadcom binaries for Ethernet and Wireless do their own low level access to nvram, so their variables must always be stored in the original location etc etc.

Therefore I agree - why not use a usb stick?, we could also store the adblock lists there!
 
mstombs said:
Therefore I agree - why not use a usb stick?, we could also store the adblock lists there!
Click to expand...

With some semi-clever coding, you probably wouldn't have to make it mandatory, but with EasyTomato only officially supporting the RTN16 I could see taking advantage of the jffs/usb capabilities. Biggest disadvantage of jffs is it has to be backed up/disabled/restored when upgrading firmware. Biggest disadvantage of usb is it could be pulled from the router and walked away with - so no passwords/keys/etc should be stored there without dire warnings.

Some form of backup/restore GUI functionality would be needed for either jffs or usb, but with jffs a less than brilliant admin could click through things, "confirm" a backup had been saved when it hadn't and blow away who knows what. They'd get what they deserve, but who wants the support headache? We generally email backups at least weekly to ourselves, might require that as an option for jffs settings.

Startup sequencing would need to be looked at to make sure jffs/usb has been mounted, or at least have some form of default automount script to reload everything that needs updated config from usb/jffs.
 
Monk E. Boy said:
I wonder if requiring the use of a USB drive would be within the realm of possibility, since then you could remove settings from NVRAM and place them in configuration files which would live on the USB drive. Flash drives are available for as little as $5, and even 2GB or 4GB of space would present an enormous increase in space.
Click to expand...

USB drives get mounted rather late in the boot process. Relying on JFFS would be a better solution IMHO.
 
jerrm said:
With some semi-clever coding, you probably wouldn't have to make it mandatory, but with EasyTomato only officially supporting the RTN16 I could see taking advantage of the jffs/usb capabilities. Biggest disadvantage of jffs is it has to be backed up/disabled/restored when upgrading firmware. Biggest disadvantage of usb is it could be pulled from the router and walked away with - so no passwords/keys/etc should be stored there without dire warnings.
Click to expand...

That can be avoided by using a static jffs location at the top of the available flash. That way, only a firmware that would totally fill all the flash space would overwrite it.
 
RMerlin said:
That can be avoided by using a static jffs location at the top of the available flash. That way, only a firmware that would totally fill all the flash space would overwrite it.
Click to expand...
Cool. Sounds like a worthwhile feature for any version for routers with larger flashes. I could do a lot with 2MB or so of space without update worries.
 
For now, we just trying to do NVRAM compression on a couple EasyTomato specific vars. Our groups vars can get huge with 100+ computers on an ET router. We checked one from a school in Rwanda and the groups var was about 11k alone!

Moving forward we'd love a way to get more space, be it baking in JFFS, USB, or other. We very much welcome all ideas or code/help from you all!
 
jerrm said:
iggest disadvantage of usb is it could be pulled from the router and walked away with - so no passwords/keys/etc should be stored there without dire warnings.
Click to expand...
As a general rule I think that if a flash drive is going to walk off, the entire router is going to walk off. If you can be unobserved for long enough to grab a flash drive, it just takes a couple more seconds to unplug the entire router and stuff it in an open backpack. That being said, I do purchase physically small drives for routers, in part to prevent them from walking, but also because it's hard for anyone to dislodge them accidentally. They still end up being under $10. The last batch of 4GB drives I bought were $8 each.

While I'm not suggesting that all NVRAM variables can be placed in a configuration file on USB, if you can adjust the boot process so that items load later in the process - after USB has been initialized - then it should be possible to place the configuration files for those items on USB. While I like JFFS, having to copy the contents off before a firmware upgrade (and restore the contents after) is probably beyond most of ET's installed base.

I wish you guys luck in any case, I like the progress you've made.
 
Monk E. Boy said:
As a general rule I think that if a flash drive is going to walk off, the entire router is going to walk off. If you can be unobserved for long enough to grab a flash drive, it just takes a couple more seconds to unplug the entire router and stuff it in an open backpack.
Click to expand...
If the entire router walks, the admin knows something is wrong.

With a usb, the drive could be unplugged, the few small files that matter copied, replaced, and the router rebooted within just a few minutes. By the time the "is there a problem with the internet?" question gets to the admin, everything is working OK again. Most should probably investigate what really happened, but won't. Avoiding sensitive data on USB and/or some rather stern warnings would only be prudent (even if the "admin" doesn't really care).

Sensitive data could also be encrypted using an nvram variable for the key, but I would still think the warnings would be needed.
 
Monk E. Boy said:
While I like JFFS, having to copy the contents off before a firmware upgrade (and restore the contents after) is probably beyond most of ET's installed base.
Click to expand...


This is only an issue if you configure your JFFS partition right after the FW partition - meaning that a new firmware requiring more space might then overwrite the JFFS partition. But if you keep the JFFS partition small and at the top of the flash space, you will be fine. For example take an RT-N66U with 32 MB of flash: if you only use the top 1 MB of it for storing settings, it is highly unlikely that this will ever be overwritten.

You could also go for an hybrid solution: require users to have a USB disk connected, but in reality rely on the JFFS partition. Have the router automatically backup the JFFS to the flash disk any time you change a setting that is normally stored in JFFS. If at boot time the partition is missing/corrupted then wait for the USB drive to be available, and restore a backup from it.

Since the Easy Tomato devs control the size of their firmware, they only have to ensure to never get anywhere close to the last 1 MB of flash space. They could even create two separate JFFS partition: one using all free space until the last 1 MB, and a second one using the last 1 MB, dedicated to settings.
 
jerrm said:
If the entire router walks, the admin knows something is wrong.
Click to expand...
To be honest, on the couple occasions I've had where a flash drive has been dislodged, failed, or was otherwise accidentally dismounted, I knew something was wrong almost immediately since it started passing data at a slugs pace. I stick all logs and related files onto the flash drive, and when Tomato can't write to its log it gets very, very upset.

RMerlin said:
Since the Easy Tomato devs control the size of their firmware, they only have to ensure to never get anywhere close to the last 1 MB of flash space. They could even create two separate JFFS partition: one using all free space until the last 1 MB, and a second one using the last 1 MB, dedicated to settings.
Click to expand...
Very good point.

I really just wanted to throw the USB idea out there, since NVRAM compression could end up being a pretty big project for them.
 
I really like the idea of having a generic JFFS space and if time permits, will look into that more.

However, the compression stuff doesn't seem too difficult and I think I've already got a good start on it (need to debug a bit and do a lot more testing). Here's the code I've whipped up so far, it just makes use of some temp files and the fact that gzip is already available. I've also just focused on easytomato_groups (the worst offender for space) and will then include the next couple largest variables. Our JSON objects compress very nicely.

Code: 
diff --git a/release/src/router/httpd/nvram.c b/release/src/router/httpd/nvram.c
index 41b6e8e..11f7624 100644
--- a/release/src/router/httpd/nvram.c
+++ b/release/src/router/httpd/nvram.c
@@ -12,6 +12,34 @@
#endif
 
 
+static void print_compressed_var(const char *varName) {
+  char tmp_gz[] = "/tmp/tmp.XXXXXX.gz";
+  char tmp[] = "/tmp/tmp.XXXXXX";
+  char s[128];
+  int r;
+ 
+  // Create temporary files
+  mktemp(tmp);
+  memcpy(tmp_gz, tmp, strlen(tmp));
+  // Copy out of nvram into the tmp_gz file
+  sprintf(s, "nvram getfb64 %s %s", varName, tmp_gz);
+  r = system(s);
+  if (r) web_puts("Error fetching easytomato_groups!");
+  // Unzip the file
+  sprintf(s, "gunzip -c %s > %s", tmp_gz, tmp);
+  r = system(s);
+  if (r) {
+    web_puts("Error unzipping easytomato_groups!");
+  } else {
+    // Read out the characters from the file and web_putj them
+    web_putfile(tmp, WOF_JAVASCRIPT);
+  }
+ 
+  unlink(tmp);
+  unlink(tmp_gz);
+}         
+
+
static int print_wlnv(int idx, int unit, int subunit, void *param)
{
    char *k = param;
@@ -98,7 +126,14 @@ void asp_nvram2(int argc, char **argv)
            continue;
 
        web_printf("\t\"%s\": \"", k); // AB multiSSID
-        web_putj(nvram_safe_get(k));
+
+        if (strcmp(k, "easytomato_groups") == 0) {
+          // easytomato_groups must be decompressed before returning data
+          print_compressed_var(k);
+        } else {
+          web_putj(nvram_safe_get(k));
+        }
+
        web_puts("\",\n");
 
        if (strncmp(k, "wl_", 3) == 0) {
diff --git a/release/src/router/httpd/tomato.c b/release/src/router/httpd/tomato.c
index 0581c35..0309aa8 100644
--- a/release/src/router/httpd/tomato.c
+++ b/release/src/router/httpd/tomato.c
@@ -1882,6 +1882,31 @@ static int save_variables(int write)
        }
    }
 
+    p = NULL;
+    p = webcgi_get("easytomato_groups");
+    if (p != NULL) {
+      // Need to compress the variable and save it
+      char tmp[] = "/tmp/tmp.XXXXXX";
+      char cmd[16*1024]; // Needs to be large enough to hold all the data
+      int r;
+
+      // Create a temporary file
+      mktemp(tmp);
+     
+      // This command will compress the data and save it into a file
+      sprintf(cmd, "echo \"%s\" | gzip -c > %s", p, tmp);
+      r = system(cmd);
+     
+      // Save that file into nvram
+      sprintf(cmd, "nvram setfb64 %s %s", "easytomato_groups", tmp);
+      r = system(cmd);
+
+      // Delete the tmp file
+      unlink(tmp);
+      dirty = 1;
+    }
+
+
    return (write) ? dirty : 1;
}
 
You must log in or register to reply here.

Back
Top