Virgin Media UK cable internet connections are subject to a traffic management policy (commonly referred to as STM) that cuts your bandwidth by 50-75% if you download/upload a certain amount of data between certain hours of the day, as described here. When this happens, your QOS settings become useless and your connection usually slows to a crawl. Over the last week or so I've been working on a script to monitor bandwidth usage (via rstats) and automatically reconfigure QOS if I go over my STM limit. It seems to work ok, so I thought I'd share it to see if anyone else was interested. I've set up a small web page here describing it and providing instructions. The script itself is available here. You can enable the script on your router by adding the following to your "WAN Up" script and rebooting: Code: wget -O /tmp/stm-monitor.sh http://www.tievolu.co.uk/stm/stm-monitor.sh chmod 755 /tmp/stm-monitor.sh cru a STM-Monitor "1-59/2 * * * * /tmp/stm-monitor.sh [b][broadband type][/b]" logger -t STM-Monitor "Downloaded `head /tmp/stm-monitor.sh | grep \"STM Monitor\" | sed 's/# //g'`" Where [broadband type] = M, M+, L or XL Here's the first version of the script, just so you can see what's involved: Code: #!/bin/sh -x # # Tomato STM Monitor v1.00 # Written by tievolu # http://tievolu.googlepages.com/stm-monitor.html # # This script is for use on Virgin Media UK cable internet connections, # which implement Subscriber Traffic Management (STM) i.e. the WAN # bandwidth is limited for a set time when an predefined amount of data # is transferred between certain hours of the day. # # It allows the WAN bandwidth usage to be automatically monitored and # suitable QOS settings applied if an STM limit is exceeded. When the # STM sentence has been served the QOS settings return to normal. # # Add the following to your "WAN Up" script to download the script and # set it to run every two minutes: # # wget -O /tmp/stm-monitor.sh http://tievolu.googlepages.com/stm-monitor.sh # chmod 755 /tmp/stm-monitor.sh # cru a STM-Monitor "1-59/2 * * * * /tmp/stm-monitor.sh [broadband type]" # logger -t STM-Monitor "Downloaded `head /tmp/stm-monitor.sh | grep \"STM Monitor\" | sed 's/# //g'`" # # Where [broadband type] = "M", "L" or "XL" (no quotes) # # Events are recorded in the system log, and details are displayed in a # small web page located at: # # http://[router IP address]/ext/stm-monitor.htm # # The script assumes that the time on your router is accurate to within # a minute or so, so make sure your NTP settings are correct! It also only # works correcly if your router is left on throughout each STM period, # because rebooting destroys the Tomato bandwidth stats this script relies on. # Note that the actions of this script (i.e. setting QOS to deal with an STM # sentence) use NVRAM variables so they *will* survive a reboot (but not a # power cycle, because the NVRAM variables are not committed to flash). # Logging command LOGGER="logger -t STM-Monitor" # STM Monitor version stm_monitor_version=`head /tmp/stm-monitor.sh | grep "STM Monitor" | sed 's/# //g'` # Check command line argument if [[ "$#" -ne 1 && "$1" != "M" && "$1" != "L" && "$1" != "XL" ]] then $LOGGER "Usage: stm-monitor.sh [M|L|XL]" exit fi # The upload QOS bandwidth values are set at ~91% of the cable modem # bandwidth limiter (download set at 100%), which works well for my # connection. Cable connections tend to be pretty stable so they should work # ok for most users. The STM related variables are based on the information # published by VM here: http://allyours.virginmedia.com/html/internet/traffic.html # STM period definitions (time values are hours of the day e.g. 16 = 16:00 = 4pm) stm_period_name_1="Daytime Downstream" stm_period_start_1=10 stm_period_end_1=15 stm_period_direction_1="RX" stm_period_name_2="Evening Downstream" stm_period_start_2=16 stm_period_end_2=21 stm_period_direction_2="RX" stm_period_name_3="Evening Upstream" stm_period_start_3=15 stm_period_end_3=20 stm_period_direction_3="TX" # STM sentence length (i.e. how long STM lasts for once it's triggered) stm_sentence_length_hours=5 stm_sentence_length_seconds=`expr $stm_sentence_length_hours \* 3600` # Settings specific to XL (20Mb down, 768Kb up) if [[ $1 = "XL" ]] then # STM limits (values in MB) stm_period_bw_limit_mb_1=6000 stm_period_bw_limit_mb_2=3000 stm_period_bw_limit_mb_3=1400 # QOS settings (values in Kbits/s) inbound_bandwidth_without_stm=20480 inbound_bandwidth_with_stm=5120 outbound_bandwidth_without_stm=700 outbound_bandwidth_with_stm=174 fi # Settings specific to L (10Mb down, 512Kb up) if [[ $1 = "L" ]] then # STM limits (values in MB) stm_period_bw_limit_mb_1=2400 stm_period_bw_limit_mb_2=1200 stm_period_bw_limit_mb_3=700 # QOS settings (values in Kbits/s) inbound_bandwidth_without_stm=10240 inbound_bandwidth_with_stm=2560 outbound_bandwidth_without_stm=465 outbound_bandwidth_with_stm=116 fi # Settings specific to M (2Mb down, 256Kb up) if [[ $1 = "M" ]] then # STM limits (values in MB) stm_period_bw_limit_mb_1=1000 stm_period_bw_limit_mb_2=500 stm_period_bw_limit_mb_3=200 # QOS settings (values in Kbits/s) inbound_bandwidth_without_stm=2048 inbound_bandwidth_with_stm=1024 outbound_bandwidth_without_stm=233 outbound_bandwidth_with_stm=116 fi # Set up web page. Use a string to store web page content and pipe # it to /var/wwwext/stm-monitor.htm when the script exits if [[ ! -e "/var/wwwext" ]] then mkdir /var/wwwext fi web_page="" # Grab the current time current_hours=`date +%H` current_minutes=`date +%M` current_unix_time=`date +%s` # Check whether we are still serving an STM sentence that has completed if [[ "`nvram get stm_sentence_active`" = 1 && $current_unix_time -gt "`nvram get stm_sentence_end_unix_time`" ]] then # Return QOS configuration to non-STM settings $LOGGER "STM sentence served: applying normal QOS configuration" nvram set qos_obw=$outbound_bandwidth_without_stm nvram set qos_ibw=$inbound_bandwidth_without_stm # Restart QOS service qos restart # Unset our NVRAM variables nvram unset stm_sentence_active nvram unset stm_sentence_end_time nvram unset stm_sentence_end_unix_time fi # Grab current statistics from rstats and wait for the file to be written killall -USR1 rstats sleep 2 stats_file=/var/spool/rstats-speed.js # Read the stats file and extract the parts we're interested in (vlan1 RX and TX) read_next_rx_tx=0 while read line do case $line in *vlan1*) read_next_rx_tx=1 ;; esac if [[ $read_next_rx_tx = 1 ]] then case $line in *rx:*) rx_data=$line ;; *tx:*) tx_data=$line ; read_next_rx_tx=0 ;; esac fi done < $stats_file rm $stats_file # Check which STM periods are active, check total TX/RX against # STM limits and note if an STM limit was exceeded stm_sentence_triggered=0 web_page="${web_page}<table cellpadding=9 cellspacing=3 border=0>" web_page="${web_page}<tr align=\"center\" bgcolor=\"DDDDFF\"><th>STM Description<th>Start</th><th>End</th><th>Bandwidth Limit</th><th>Bandwidth Used</th><th>Average Rate</th></tr>" for i in 1 2 3; do stm_period_name=`eval echo "\\$stm_period_name_$i"` stm_period_start=`eval echo "\\$stm_period_start_$i"` stm_period_end=`eval echo "\\$stm_period_end_$i"` stm_period_direction=`eval echo "\\$stm_period_direction_$i"` stm_period_bw_limit_mb=`eval echo "\\$stm_period_bw_limit_mb_$i"` if [[ $current_hours -ge $stm_period_start && $current_hours -lt $stm_period_end ]] then web_page="${web_page}<tr bgcolor=\"EEEEEE\" align=\"center\">" web_page="${web_page}<td align=\"left\">${stm_period_name} (${1})</td><td>${stm_period_start}:00</td><td>${stm_period_end}:00</td><td>${stm_period_bw_limit_mb} MB</td>" # Calculate the number of values to take from the rstats data (each value covers two minutes) minutes_since_stm_period_started=`echo "" | awk '{ print((((hours - stm_start) * 60) + minutes)); }' hours=$current_hours minutes=$current_minutes stm_start=${stm_period_start}` number_of_values=`expr $minutes_since_stm_period_started / 2` if [[ $stm_period_direction = "RX" ]] then data=$rx_data else data=$tx_data fi if [[ $number_of_values != 0 ]] then # Sum the last XXX values of the 720 in the rstats data to get the # total bytes transferred since the STM period started total_transferred_bytes=`echo $data | awk ' { values_string = substr($0, index($0, "[") + 1, index($0, "]") - 6); split(values_string, values_array, ","); for (i=(721-number_of_values); i<=720; i++) { total += values_array[i]; } printf("%0.f" total); }' number_of_values=$number_of_values total=0` total_transferred_mb=`echo "" | awk '{ printf("%.2f", bytes / 1048576) }' bytes=$total_transferred_bytes` average_rate=`echo "" | awk '{ printf("%.2f", bytes / (number_of_values * 2 * 60 * 1024)) }' number_of_values=$number_of_values bytes=$total_transferred_bytes` else total_transferred_bytes="0" total_transferred_mb="0.00" everage_rate="0.00" fi # Calculate maximum average rate (i.e. rate required to trigger STM), # then compare this to our actual rate. If the actual rate is equal or # higher, we paint the cell red, if it's greater that 75% of the max # we paint it amber, and if it's anything else we paint it green. stm_period_max_ave_rate=`echo "" | awk '{printf((limit * 1024) / ((end - start)*3600))}' limit=$stm_period_bw_limit_mb start=$stm_period_start end=$stm_period_end` cell_color=`echo | awk '{ if (rate >= max) {print("FF6633")} else if (rate > (0.75 * max)) {print("FFCC33")} else {print("66FF33")}}' rate=$average_rate max=$stm_period_max_ave_rate` # Add data to web page web_page="${web_page}<td>$total_transferred_mb MB</td><td bgcolor=\"${cell_color}\">$average_rate KB/s</td>" # Check whether STM limit has been exceeded stm_period_bw_limit_bytes=`expr $stm_period_bw_limit_mb \* 1048576` result=`echo "" | awk '{ if (total > limit) {print(1)} else {print(0)}}' total=$total_transferred_bytes limit=$stm_period_bw_limit_bytes` if [[ $result = 1 && "`nvram get stm_sentence_active`" != 1 ]] then stm_sentence_triggered=1 nvram set stm_sentence_active=1 $LOGGER "STM Triggered: ${stm_period_name} (${stm_period_start}:00 -> ${stm_period_end}:00, ${stm_period_direction}, ${stm_period_bw_limit_mb} MB)" fi else web_page="${web_page}<tr bgcolor=\"EEEEEE\" style=\"color:AAAAAA\" align=\"center\">" web_page="${web_page}<td align=\"left\">${stm_period_name} (${1})</td><td>${stm_period_start}:00</td><td>${stm_period_end}:00</td><td>${stm_period_bw_limit_mb} MB</td><td>N/A</td><td>N/A</td>" fi web_page="${web_page}</tr>" done web_page="${web_page}</table>" # If STM is active, check whether QOS is set accordingly if [[ "`nvram get stm_sentence_active`" = 1 && "`nvram get qos_obw`" != $outbound_bandwidth_with_stm ]] then if [[ $stm_sentence_triggered = 1 ]] then # We must have triggered an STM sentence on this run. # Work out the hour component of the time the STM sentence will end # (for logging purposes only - we will use unix time to track the # STM sentence accurately) stm_end_hour=`expr $current_hours + $stm_sentence_length_hours` if [[ $stm_end_hour -gt 23 ]] then stm_end_hour="0`expr $stm_end_hour - 24`" fi # Set NVRAM variables to describe the STM sentence being served nvram set stm_sentence_end_time="$stm_end_hour:$current_minutes" nvram set stm_sentence_end_unix_time=`expr $current_unix_time + $stm_sentence_length_seconds` # Send information to log $LOGGER "STM sentence started: applying STM QOS configuration (until $stm_end_hour:$current_minutes)" else # An STM sentence is active and we didn't trigger it on this run, # but the QOS settings are wrong. We must have rebooted, or someone # messed with the QOS settings. So we want to set the QOS settings # to deal with the STM, but we don't want to overwrite our NVRAM # variables. # Send information to log $LOGGER "STM sentence resumed: applying STM QOS configuration (until `nvram get stm_sentence_end_time`)" fi # Set QOS NVRAM variables nvram set qos_obw=$outbound_bandwidth_with_stm nvram set qos_ibw=$inbound_bandwidth_with_stm # Restart QOS service qos restart fi # Add QOS settings to web page web_page="${web_page}<p>Current QOS settings - <b>`nvram get qos_ibw`</b> kbits/s down, <b>`nvram get qos_obw`</b> kbits/s up</p>" # Add latest log entries to web page web_page="${web_page}<h3>Latest Log Entries</h3>" log="" while read line do case $line in *STM-Monitor*) log="${log}<nobr><tt>${line}</tt></nobr><br>" ;; esac done < /tmp/var/log/messages web_page="${web_page}$log" # Publish web page # Write the title and status now, so that they accurately relfect the data in the table web_page_header="<html><head><title>${stm_monitor_version}</title><meta http-equiv=\"refresh\" content=\"120\"</head>" web_page_header="${web_page_header}<body style=\"font-family:verdana\"><h2>${stm_monitor_version}</h2>" web_page_header="${web_page_header}<h3>Status at ${current_hours}:${current_minutes} - " if [[ "`nvram get stm_sentence_active`" = 1 ]] then web_page_header="${web_page_header}<b style=\"color:red\">STM has been triggered - the sentence will end at `nvram get stm_sentence_end_time`</b></h3>" else web_page_header="${web_page_header}<b style=\"color:green\">STM has not been triggered</b></h3>" fi web_page="${web_page_header}${web_page}</body></html>" echo $web_page > /var/wwwext/stm-monitor.htm Even if you're not interested in this particular functionality, there may well be stuff in the script that you find useful for other scripts. I learned quite a lot while writing it. Any feedback or suggestions most welcome.
Tiev, thanks for the script, but I couldn't get it to work. Could you please verify I have the following correct for a VM L service wget -O /tmp/stm-monitor.sh http://tievolu.googlepages.com/stm-monitor.sh chmod 755 /tmp/stm-monitor.sh cru a STM-Monitor "1-59/2 * * * * /tmp/stm-monitor.sh L" logger -t STM-Monitor "Downloaded `head stm-monitor.sh | grep \"STM Monitor\" | sed 's/# //g'`" I couldn't call the web page either http://192.168.1.1/ext/stm-monitor.sh I have 'logging' enabled. I could see your script was downloaded when i looked the the log thx
Can you log into the router and run "ls -l /tmp" and paste the output? If the script is there, try running it directly and tell me what happens i.e. run: /tmp/stm-monitor.sh L EDIT: The web page link will be http://192.168.1.1/ext/stm-monitor.htm (not .sh) That's probably your only problem - the WAN Up code looks fine (that was my fault btw - I've just corrected the web page - d'oh!)
tievolu, I managed to get this working and this script is fantastic, also great web interface. If I read correctly does the STM 5 hour period start from when you overstep the limits? Is it possible to script this so that i include the whole file in my wan-up code? (i.e. it doesn't go off to load a file from a server). For my security i'd like to know what is within a file everytime my router boots. Great work!! Thanks
Glad you like it While I promise not to do anything dodgy with the script, the possibility that I could if I was that way inclined is definitely there, so I completely understand your concerns about downloading it automatically. You could build the script entirely within the WAN Up script, but it would be a bit ugly - lots of echo commands, with the need to escape all the special characters in the correct way. It would be much easier to mount a JFFS partition and store a copy of the script there permanently. So, assuming your router has enough flash, and you like the script the way it is right now, create and mount your JFFS partition using the Tomato GUI (Administration -> JFFS2) and then issue the following commands (i.e. just telnet in and run them - don't add them to your WAN Up script): Code: wget -O /jffs/stm-monitor.sh http://tievolu.googlepages.com/stm-monitor.sh chmod 755 /jffs/stm-monitor.sh Then you would just need the following command in your WAN Up or startup script to set up the cron job when the router boots: Code: cru a STM-Monitor "1-59/2 * * * * /jffs/stm-monitor.sh [M|L|XL]" Hope this helps.
Just realised I didn't answer this before... Yes, the 5 hour punishment period begins when you go over the STM threshold. So, if you went over the limit for the 16:00 -> 21:00 period at, say, 18:30, you'd have your bandwidth reduced for five hours from that point i.e. until 23:30. Some would argue that it's a bit harsh, but that's the way it works.
This script is excellent. Thank you tievolu. Although I do not really use QOS I have found the STM monitor page extremely useful. I live in a student house with 5 other people so we frequently go over the limit, usually within the last hour of the evening downstream period A suggestion I would like to make would be to have the QOS settings become stricter according to the amount of time and bandwidth left to prevent the limit being reached in the last couple of hours. I will expand upon this further if you need me to, but for know I am struggling to turn my thoughts into words Tanks again
I've thought about this too - it would be nice to have the option of preventing STM from kicking in, rather than merely coping with it when it does. I don't think it would be too hard to implement. I'll have a look into it...
Ok, here we go. There are now two modes of operation: STM Mitigation - This is the default mode. If an STM limit is exceeded, QOS limits are applied in accordance with the bandwidth limits imposed by STM. STM Prevention - If bandwidth usage is on target to exceed an STM limit, QOS limits are applied to prevent that from happening. Activated by the "-p" option (see below). Usage is as follows: Code: stm-monitor.sh [broadband type] [-p] With STM prevention enabled, the script monitors the average rate, checking whether it has exceeded the maximum rate (i.e. when the cell is red and you're on course to trigger STM). If this happens, the script calculates how much bandwidth you have left in this STM period, and limits the inbound or outbound QOS such that you will only be able to transfer 90% of that bandwidth before the STM period ends. The QOS limit returns to normal if the average rate drops below 90% of the maximum, or when the STM period ends. The end result of STM prevention is that you may take a more severe bandwidth cut than if you triggered STM, but that cut will usually last for a much shorter period of time. Personally, I'm not really sure whether prevention is better than mitigation, but YMMV. I've also modified the status page slightly. See this page for an example. Code: #!/bin/sh -x # # Tomato STM Monitor v1.01 # Written by tievolu # http://tievolu.googlepages.com/stm-monitor.html # # This script is for use on Virgin Media UK cable internet connections, # which implement Subscriber Traffic Management (STM) i.e. the WAN # bandwidth is limited for a set time when an predefined amount of data # is transferred between certain hours of the day. # # It allows the WAN bandwidth usage to be automatically monitored and # suitable QOS settings applied. There are two modes of operation: # # 1. STM Mitigation # # This is the default mode. If an STM limit is exceeded, QOS limits # are applied in accordance with the bandwidth limits imposed by STM. # # 2. STM Prevention # # If bandwidth usage is on target to exceed an STM limit, QOS limits # are applied to prevent that from happening. Activated by the "-p" # option (see below). # # Add the following to your "WAN Up" script to download the script and # set it to run every two minutes: # # wget -O /tmp/stm-monitor.sh http://tievolu.googlepages.com/stm-monitor.sh # chmod 755 /tmp/stm-monitor.sh # cru a STM-Monitor "1-59/2 * * * * /tmp/stm-monitor.sh [broadband type] [-p]" # logger -t STM-Monitor "Downloaded `head /tmp/stm-monitor.sh | grep \"STM Monitor\" | sed 's/# //g'`" # # Where [broadband type] = "M", "L" or "XL" (no quotes) # [-p] = Activates STM prevention # # Events are recorded in the system log, and details are displayed in a # small web page located at: # # http://[router IP address]/ext/stm-monitor.htm # # The script assumes that the time on your router is accurate to within # a minute or so, so make sure your NTP settings are correct! It also only # works correcly if your router is left on throughout each STM period, # because rebooting destroys the Tomato bandwidth stats this script relies on. # Logging command LOGGER="logger -t STM-Monitor" # STM Monitor version stm_monitor_version=`head /tmp/stm-monitor.sh | grep "STM Monitor" | sed 's/# //g'` # Check command line arguments if [[ "$#" -eq 1 && "$1" != "M" && "$1" != "L" && "$1" != "XL" ]] then $LOGGER "Usage: stm-monitor.sh [M|L|XL] [-p]" exit fi if [[ "$#" -eq 2 && "$2" != "-p" ]] then $LOGGER "Usage: stm-monitor.sh [M|L|XL] [-p]" exit fi # For testing purposes if [[ "$2" = "-p" ]] then preventative_mode=1 fi # The upload QOS bandwidth values are set at ~91% of the cable modem # bandwidth limiter (download set at 100%), which works well for my # connection. Cable connections tend to be pretty stable so they should work # ok for most users. The STM related variables are based on the information # published by VM here: http://allyours.virginmedia.com/html/internet/traffic.html # STM period definitions (time values are hours of the day e.g. 16 = 16:00 = 4pm) stm_period_name_1="Daytime Downstream" stm_period_start_1=10 stm_period_end_1=15 stm_period_direction_1="RX" stm_period_name_2="Evening Downstream" stm_period_start_2=16 stm_period_end_2=21 stm_period_direction_2="RX" stm_period_name_3="Evening Upstream" stm_period_start_3=15 stm_period_end_3=20 stm_period_direction_3="TX" # STM sentence length (i.e. how long STM lasts for once it's triggered) stm_sentence_length_hours=5 stm_sentence_length_seconds=`expr $stm_sentence_length_hours \* 3600` # Settings specific to XL (20Mb down, 768Kb up) if [[ $1 = "XL" ]] then # STM limits (values in MB) stm_period_bw_limit_mb_1=6000 stm_period_bw_limit_mb_2=3000 stm_period_bw_limit_mb_3=1400 # QOS settings (values in Kbits/s) inbound_bandwidth_without_stm=20480 inbound_bandwidth_with_stm=5120 outbound_bandwidth_without_stm=700 outbound_bandwidth_with_stm=174 fi # Settings specific to L (10Mb down, 512Kb up) if [[ $1 = "L" ]] then # STM limits (values in MB) stm_period_bw_limit_mb_1=2400 stm_period_bw_limit_mb_2=1200 stm_period_bw_limit_mb_3=700 # QOS settings (values in Kbits/s) inbound_bandwidth_without_stm=10240 inbound_bandwidth_with_stm=2560 outbound_bandwidth_without_stm=465 outbound_bandwidth_with_stm=116 fi # Settings specific to M (2Mb down, 256Kb up) if [[ $1 = "M" ]] then # STM limits (values in MB) stm_period_bw_limit_mb_1=1000 stm_period_bw_limit_mb_2=500 stm_period_bw_limit_mb_3=200 # QOS settings (values in Kbits/s) inbound_bandwidth_without_stm=2048 inbound_bandwidth_with_stm=1024 outbound_bandwidth_without_stm=233 outbound_bandwidth_with_stm=116 fi # Grab the current time current_hours=`date +%H` current_minutes=`date +%M` current_unix_time=`date +%s` # Set up web page. Use a string to store web page content and pipe # it to /var/wwwext/stm-monitor.htm when the script exits if [[ ! -e "/var/wwwext" ]] then mkdir /var/wwwext fi web_page="<html><head><title>${stm_monitor_version} - Status at ${current_hours}:${current_minutes}</title><meta http-equiv=\"refresh\" content=\"120\"</head>" web_page="${web_page}<body style=\"font-family:verdana\"><h2>${stm_monitor_version}</h2>" web_page="${web_page}<h3>Status at ${current_hours}:${current_minutes}</h3>" # Check whether we are still serving an STM sentence that has completed if [[ "`nvram get stm_sentence_active`" = 1 && $current_unix_time -gt "`nvram get stm_sentence_end_unix_time`" ]] then # Return QOS configuration to non-STM settings $LOGGER "STM sentence served: applying normal QOS configuration" nvram set qos_obw=$outbound_bandwidth_without_stm nvram set qos_ibw=$inbound_bandwidth_without_stm # Restart QOS service qos restart # Unset our NVRAM variables nvram unset stm_sentence_active nvram unset stm_sentence_end_time nvram unset stm_sentence_end_unix_time fi # Grab current statistics from rstats and wait for the file to be written killall -USR1 rstats sleep 2 stats_file=/var/spool/rstats-speed.js # Read the stats file and extract the parts we're interested in (vlan1 RX and TX) read_next_rx_tx=0 while read line do case $line in *vlan1*) read_next_rx_tx=1 ;; esac if [[ $read_next_rx_tx = 1 ]] then case $line in *rx:*) rx_data=$line ;; *tx:*) tx_data=$line ; read_next_rx_tx=0 ;; esac fi done < $stats_file rm $stats_file # Check which STM periods are active, check total TX/RX against # STM limits and note if an STM limit was exceeded stm_sentence_triggered=0 web_page="${web_page}<table cellpadding=9 cellspacing=3 border=0>" web_page="${web_page}<tr align=\"center\" bgcolor=\"DDDDFF\"><th>STM Description<th>Start</th><th>End</th><th>Bandwidth Limit</th><th>Bandwidth Used</th><th>Average Rate</th></tr>" for i in 1 2 3; do stm_period_name=`eval echo "\\$stm_period_name_$i"` stm_period_start=`eval echo "\\$stm_period_start_$i"` stm_period_end=`eval echo "\\$stm_period_end_$i"` stm_period_direction=`eval echo "\\$stm_period_direction_$i"` stm_period_bw_limit_mb=`eval echo "\\$stm_period_bw_limit_mb_$i"` if [[ $current_hours -ge $stm_period_start && $current_hours -lt $stm_period_end ]] then # This STM period is active. web_page="${web_page}<tr bgcolor=\"EEEEEE\" align=\"center\">" web_page="${web_page}<td align=\"left\">${stm_period_name} (${1})</td><td>${stm_period_start}:00</td><td>${stm_period_end}:00</td><td>${stm_period_bw_limit_mb} MB</td>" # Calculate the number of values to take from the rstats data (each value covers two minutes) minutes_since_stm_period_started=`echo "" | awk '{ print((((hours - stm_start) * 60) + minutes)); }' hours=$current_hours minutes=$current_minutes stm_start=${stm_period_start}` number_of_values=`expr $minutes_since_stm_period_started / 2` if [[ $stm_period_direction = "RX" ]] then data=$rx_data stm_period_rx_active=1 else data=$tx_data stm_period_tx_active=1 fi if [[ $number_of_values != 0 ]] then # Sum the last XXX values of the 720 in the rstats data to get the # total bytes transferred since the STM period started total_transferred_bytes=`echo $data | awk ' { values_string = substr($0, index($0, "[") + 1, index($0, "]") - 6); split(values_string, values_array, ","); for (i=(721-number_of_values); i<=720; i++) { total += values_array[i]; } printf("%0.f" total); }' number_of_values=$number_of_values total=0` total_transferred_mb=`echo "" | awk '{ printf("%.2f", bytes / 1048576) }' bytes=$total_transferred_bytes` average_rate=`echo "" | awk '{ printf("%.2f", bytes / (number_of_values * 2 * 60 * 1024)) }' number_of_values=$number_of_values bytes=$total_transferred_bytes` average_rate_kilobits=`echo "" | awk '{printf("%.2f", rate_kilobytes * 8)}' rate_kilobytes=$average_rate` else total_transferred_bytes="0" total_transferred_mb="0.00" average_rate="0.00" average_rate_kilobits="0.00" fi # Calculate maximum average rate (i.e. rate required to trigger STM), # then compare this to our actual rate. If the actual rate is equal or # higher, we paint the cell red, if it's greater than 90% of the max # we paint it amber, and if it's anything else we paint it green. stm_period_max_ave_rate=`echo "" | awk '{printf((limit * 1024) / ((end - start)*3600))}' limit=$stm_period_bw_limit_mb start=$stm_period_start end=$stm_period_end` cell_color=`echo | awk '{ if (rate >= max) {print("FF6633")} else if (rate > (0.9 * max)) {print("FFCC33")} else {print("66FF33")}}' rate=$average_rate max=$stm_period_max_ave_rate` # STM Prevention code if [[ $preventative_mode = 1 ]] then if [[ $cell_color = "FF6633" ]] then # Cell is red => calculate preventative QOS limit # Calculate the number of kilobits we have left before we trigger STM total_kilobits_remaining=`echo "" | awk '{print(((limit*1048576)-total)/128)}' limit=$stm_period_bw_limit_mb total=$total_transferred_bytes` # Calculate time remaining until end of STM period seconds_remaining=`echo "" | awk '{print(((end_hour - current_hour - 1) * 3600) + (3600 - (60 * current_minute)))}' end_hour=$stm_period_end current_hour=$current_hours current_minute=$current_minutes` # Calculate a QOS limit such that we'll download a maxmum of 90% of this in the remaining time preventative_qos_limit=`echo "" | awk '{printf("%.0f", 0.9*(kilobits/secs))}' kilobits=$total_kilobits_remaining secs=$seconds_remaining` if [[ $stm_period_direction = "RX" && "`nvram get qos_ibw`" = $inbound_bandwidth_without_stm ]] then $LOGGER "Setting preventative inbound QOS limit of $preventative_qos_limit kbits/s" nvram set qos_ibw=$preventative_qos_limit # We must set the inbound rate limits to 100% or less for this to work ("None" will not work # because the rate will not be limited). To make sure this does work, we'll change the # settings to 100% across the board and restore the old settings when we remove the # temporary QOS limit. nvram set qos_irates_old=`nvram get qos_irates` nvram set qos_irates=100,100,100,100,100,100,100,100,100,100 service qos restart fi if [[ $stm_period_direction = "TX" && "`nvram get qos_obw`" = $outbound_bandwidth_without_stm ]] then $LOGGER "Setting preventative outbound QOS limit of $preventative_qos_limit kbits/s" nvram set qos_obw=$preventative_qos_limit service qos restart fi fi if [[ $cell_color = "66FF33" ]] then # Cell is green => apply normal QOS if necessary if [[ $stm_period_direction = "RX" && "`nvram get qos_ibw`" != $inbound_bandwidth_without_stm ]] then $LOGGER "Removing preventative inbound QOS limit" nvram set qos_ibw=$inbound_bandwidth_without_stm nvram set qos_irates=`nvram get qos_irates_old` nvram unset qos_irates_old service qos restart fi if [[ $stm_period_direction = "TX" && "`nvram get qos_obw`" != $outbound_bandwidth_without_stm ]] then $LOGGER "Removing preventative outbound QOS limit" nvram set qos_obw=$outbound_bandwidth_without_stm service qos restart fi fi fi # End of STM prevention code # Add data to web page web_page="${web_page}<td>$total_transferred_mb MB</td><td bgcolor=\"${cell_color}\">$average_rate KB/s ($average_rate_kilobits kb/s)</td>" # Check whether STM limit has been exceeded stm_period_bw_limit_bytes=`expr $stm_period_bw_limit_mb \* 1048576` result=`echo "" | awk '{ if (total > limit) {print(1)} else {print(0)}}' total=$total_transferred_bytes limit=$stm_period_bw_limit_bytes` if [[ $result = 1 && "`nvram get stm_sentence_active`" != 1 ]] then stm_sentence_triggered=1 nvram set stm_sentence_active=1 $LOGGER "STM Triggered: ${stm_period_name} (${stm_period_start}:00 -> ${stm_period_end}:00, ${stm_period_direction}, ${stm_period_bw_limit_mb} MB)" fi else # This STM period is not currently active web_page="${web_page}<tr bgcolor=\"EEEEEE\" style=\"color:AAAAAA\" align=\"center\">" web_page="${web_page}<td align=\"left\">${stm_period_name} (${1})</td><td>${stm_period_start}:00</td><td>${stm_period_end}:00</td><td>${stm_period_bw_limit_mb} MB</td><td>N/A</td><td>N/A</td>" fi web_page="${web_page}</tr>" done web_page="${web_page}</table>" # Remove RX/TX STM Prevention QOS limits if necessary. This code makes sure the temporary limits # are removed when they are not relevant (i.e. when we're not in an STM period and STM has not # been triggered) if [[ $preventative_mode = 1 ]] then if [[ "`nvram get stm_sentence_active`" != 1 && "$stm_period_rx_active" != 1 && "`nvram get qos_ibw`" != $inbound_bandwidth_without_stm ]] then $LOGGER "Removing preventative inbound QOS limit" nvram set qos_ibw=$inbound_bandwidth_without_stm nvram set qos_irates=`nvram get qos_irates_old` nvram unset qos_irates_old service qos restart fi if [[ "`nvram get stm_sentence_active`" != 1 && "$stm_period_tx_active" != 1 && "`nvram get qos_obw`" != $outbound_bandwidth_without_stm ]] then $LOGGER "Removing preventative outbound QOS limit" nvram set qos_obw=$outbound_bandwidth_without_stm service qos restart fi fi # If STM is active, check whether QOS is set accordingly if [[ "`nvram get stm_sentence_active`" = 1 && "`nvram get qos_obw`" != $outbound_bandwidth_with_stm ]] then if [[ $stm_sentence_triggered = 1 ]] then # We must have triggered an STM sentence on this run. # Work out the hour component of the time the STM sentence will end # (for logging purposes only - we will use unix time to track the # STM sentence accurately) stm_end_hour=`expr $current_hours + $stm_sentence_length_hours` if [[ $stm_end_hour -gt 23 ]] then stm_end_hour="0`expr $stm_end_hour - 24`" fi # Set NVRAM variables to describe the STM sentence being served nvram set stm_sentence_end_time="$stm_end_hour:$current_minutes" nvram set stm_sentence_end_unix_time=`expr $current_unix_time + $stm_sentence_length_seconds` # Send information to log $LOGGER "STM sentence started: applying STM QOS configuration (until $stm_end_hour:$current_minutes)" else # An STM sentence is active and we didn't trigger it on this run, # but the QOS settings are wrong. We must have rebooted, or someone # messed with the QOS settings. So we want to set the QOS settings # to deal with the STM, but we don't want to overwrite our NVRAM # variables. # Send information to log $LOGGER "STM sentence resumed: applying STM QOS configuration (until `nvram get stm_sentence_end_time`)" fi # Set QOS NVRAM variables nvram set qos_obw=$outbound_bandwidth_with_stm nvram set qos_ibw=$inbound_bandwidth_with_stm # Restart QOS service qos restart fi # Add status to the web page web_page="${web_page}<h4>STM Prevention: " if [[ $preventative_mode = 1 ]] then if [[ "`nvram get qos_ibw`" = $inbound_bandwidth_without_stm ]] then web_page="${web_page}<b style=\"color:green\">Inactive</b></h4>" else web_page="${web_page}<b style=\"color:red\">Active</b></h4>" fi else web_page="${web_page}<b style=\"color:CCCCCC\">Not enabled</b></h4>" fi web_page="${web_page}<h4>STM Mitigation: " if [[ "`nvram get stm_sentence_active`" = 1 ]] then web_page="${web_page}<b style=\"color:red\">Active - STM sentence will end at `nvram get stm_sentence_end_time`</b></h4>" else web_page="${web_page}<b style=\"color:green\">Inactive</b></h4>" fi web_page="${web_page}<h4>Current QOS settings: <b style=\"color:blue\">`nvram get qos_ibw`</b> kb/s down, <b style=\"color:blue\">`nvram get qos_obw`</b> kb/s up</h4>" # Add latest log entries to web page web_page="${web_page}<h3>Latest Log Entries</h3>" log="" while read line do case $line in *STM-Monitor*) log="${log}<nobr><tt>${line}</tt></nobr><br>" ;; esac done < /tmp/var/log/messages web_page="${web_page}$log" # Publish web page web_page="${web_page}</body></html>" echo $web_page > /var/wwwext/stm-monitor.htm I've done as much testing as I had time for but there may still be bugs. Feedback is most welcome as always.
I have been using version 1.01 for a few days now and not found any problems other then at first I used -P instead of -p but that became apparent when I looked in the log . Recently we haven't been STM'd though as I've found that by showing the other house mates how to check the STM monitor page they have curbed their downloads during STM period. I'm finding at the moment my bandwidth to be fairly unstable sometimes as low as 365 kB/s during peak but usually between 500 - 980 kB/s (Damn you virginmedia!). This really does make my QOS almost useless so I may have trouble testing the scripts full capabilities, but I should think its beyond this scripts scope to do bandwidth tests so will take it to the tomato feature request thread. Thanks for updating!
Hi Tievolu, I have a small problem with the bandwidth totals, it seems they're not working. Here's my STM monitor page output and as you can see the values are zero. I'm guessing this might affect the STM triggering too ? * I'm using vanilla tomato 1.23. TIA, Dave Tomato STM Monitor v1.01 Status at 11:07 STM Description Start End Bandwidth Limit Bandwidth Used Average Rate Daytime Downstream (L) 10:00 15:00 2400 MB 0.00 MB 0.00 KB/s (0.00 kb/s) Evening Downstream (L) 16:00 21:00 1200 MB N/A N/A Evening Upstream (L) 15:00 20:00 700 MB N/A N/A STM Prevention: Not enabled STM Mitigation: Inactive Current QOS settings: 102400 kb/s down, 465 kb/s up Latest Log Entries * Thanks for taking the time to write this script
davemuk, the script uses 'rstats' so please check if you have 'Bandwidth monitoring' enabled under Admin. If that fails try enabling loggin aswell.
This is amazing, really good idea Tievolu. Will try this out when I get home. Thanks again for sharing your work.
Tievolu, I moved to version 1.01 this week, previously was happy with version 1. 1.01 works very well and i managed to activate the STM mitigation on purpose to test the logic. Its a much improved version as the user no longer needs to actively monitor the bandwidth used. Some feedback and hopefully you can add the following,,, I have a virgin 'L' service and try to download/use as much of the given bandwidth. For example between 10am-3pm i get 2400mb for which i may download 2 files of size 999mb in succession. I feel the STM prevention kicks in too soon. The STM prevention should only kick-in after a certain limit has been achieved (which may be user specified through the script), i.e. 85% (which is about 2000mb for my service). This would therefore give me a fast download for the 2 files and start STM prevention when bandwidth is almost used up. I look forward to any updates you make. thanks
Unfortunately I've had to change the location of this script, as Google are killing off their Google Pages service. The script can now be found here: http://host.blucube.co.uk/~tievolu/stm/stm-monitor.sh and the (limited) documentation can be found here: http://host.blucube.co.uk/~tievolu/stm/stm-monitor.html The old Google Pages site will be killed in June, so update your WAN Up scripts as soon as you can... On a more postitive note, version 1.02 is now available, which adds an STM prevention threshold to prevent STM prevention kicking in too early. In v1.02, STM prevention won't become active until you've transferred 80% of an STM limit. Thanks to kardzzz for suggesting this feature and testing my bungled attempts at implementing it In addition you can now customise some of the settings, including the STM prevention threshold. More details here: http://host.blucube.co.uk/~tievolu/stm/stm-monitor.html#CUSTOMISE Let me know if you have any questions.
I've got a good feature Idea, If STM prevention is on, recalculate the speed you can download every time the script is run. EG, when I've downloaded 2.4GB (so have 600mb remaining) and have 2 hours left in the stm period, it may set 200kb/s as my downrate. but if I don't download anything for an hour I've still got the same speed of 200kb/s set but since I still have the same 600mb to use in the 1 hour remaining I should be at 400kb/s. I'm loving your work so far.
Thanks for the feedback! You're pretty much describing the way the script currently works, although it's slightly more conservative. Each time it runs, the script checks two things if STM prevention is enabled: 1. Are we downloading/uploading faster than the maximum permissable rate (i.e. is the Average Rate cell red)? If so, calculate the remaining bandwidth available in this STM period and set the QOS limit so that only 90% of that can be downloaded before the STM period is over. 2. If a temporary preventative QOS limit has been applied, check whether our upload/download rate has dropped to an acceptable level (i.e. Average Rate is less than 90% of the maximum permissible rate - cell is green). If so, remove the QOS limit. So if you trigger STM prevention by satisfying condition 1, but then don't download anything for a while, you should satisfy condition two at some point and your QOS limits will return to the normal non-STM settings. So to take your example: - Evening downstream (16:00 - 21:00 3000MB) - Download 2.4GB in three hours - average rate = 227.56KB/s - Maximum permissible rate = 170.67KB/s - => STM prevention kicks in and sets rate to 56.89KB/s - ** Don't download anything for an hour ** - => Downloaded 2.4GB in four hours - average rate 170.67KB/s So you're actually still at the maximum permissible rate overall. The script will wait until the overall Average Rate for the entire STM period has dropped to less than 90% of the maximum permissable rate before removing the STM prevention limits. For this particular example, that means another half an hour or so would have to pass, at which point the overall Average Rate would drop to ~153KB/s and the QOS limits would be removed. It's not perfect, but it works. I think I have to take this conservative approach to ensure that STM prevention actually does its job - i.e. to make sure it definitely prevents STM from kicking in, every time. I could change the script to cut it much finer (e.g. restore the normal QOS settings as soon as your transfer rate has dropped below the maximum permissible rate), but as it only runs every two minutes it might not quite work and you could end up just going over the STM limit in the last few minutes and then being punished for another five hours. I could make it slightly less conservative. Maybe remove the STM prevention limit when you drop below 95% of the maximum permissable rate. In the example above this would mean normal QOS settings would be restored after an extra quarter of an hour rather than half an hour (so in the end it doesn't actually make a huge difference). I could make a test version available with slightly different settings if you'd be interested in testing it? I'd also be interested in other people's opinions on this. T
Can you explain / show the STM Rate calculation for this example? (I need it to further explain what I mean)
I may have confused you by getting one of the calculations wrong (the crucial STM prevention calculation!). Here are the correct values: - Evening downstream (16:00 - 21:00 3000MB) - Download 2.4GB in three hours - average rate = 2400 MB / 180 mins = 227.56KB/s - Maximum permissible rate = 3000 MB / 2300 min = 170.67KB/s - => STM prevention kicks in and sets rate to 0.9 * (600 MB / 120 min) = 76.8KB/s - ** Don't download anything for an hour ** - => Downloaded 2.4GB in four hours - average rate = 2400 MB / 240 mins = 170.67KB/s Let me know if that still doesn't make sense
I just posted version 1.03. This updates the STM limits with the new values introduced by VM on May 12, and also adds the S broadband offering (although I very much doubt anyone here is using "S"!). The new limits consist of a slight increase for L and XL. On the downstream side, L gets an extra 600Mb during the day and 300Mb in the evening. The evening upstream limit has also been increased by 100MB. XL gets an extra 1000Mb downstream during the day and 500Mb downstream in the evening. Upstream limits are unchanged.
Firstly thanks for this script. I've been using it successfully for months, for managing my QoS settings, and it works great. Just thinking through the STM prevention mode. For example, on M+ I am allowed 750MB between 1600-2100 before throttling. If I want to download "lots", I can: enable STM prevention. Download 750MB between 1600-2100 (at a pedestrian 0.3Mbit), and then have full 10Mbit from 2100 until 1000. That is 13 hours of 1.25Mbytes/sec . Total of 750MB + 58.5GB or at 1600, trigger STM after 750MB (only ten minutes!). Then have 2.5Mbit for next five hours. Then at 2110 my STM finishes (is this right?) and I have full speed for only ten minutes less than previous case. Total of 5.6GB + 58.5GB So the optimal strategy is to trigger STM as soon as possible, meaning it doesn't eat into the evening's download period.
tievolu, thanks for v1.03... you're doing a fantastic job! I have a suggestion which may prove useful and user selectable: Say I have 2.9gb between 10am-3pm and I quickly use up 2.5gb by 12noon. The script would enforce a limit of perhaps 450kb/s down for 3 hours which is much worst than the 2560kb/s if i were capped and i would rather take 2 additional hours sentencing. Therefore I would like to add setup parameters upfront and to say, if we have >2.5hours remaining then down >= 2560 (the sentence rate). In summary, we force a setence to get a better download rate. What do other users think?
At the moment STM Prevention kicks in once you've transferred 80% of the bandwidth limit for an STM period and your Average Rate exceeds the maximum permissable rate (i.e. if you kept on downloading at that speed you would trigger STM). So, for your example, on M+ in the 1600-2100 evening downstream STM period, with STM Prevention enabled with the default 80% threshold, you could download 600MB (80% of 750MB) at full speed before STM Prevention kicked in. Assuming you downloaded that amount at the full 10Mb, STM Prevention would kick in at 16:08 and limit your bandwidth to 7.9KB/s so that you could only download 90% of the remaining 150MB over the next 4 hours 52 mins. If STM Prevention was not enabled you'd be STM'd two minutes later at 16:10 and receive five hours limited to 320KB/s. Clearly STM Prevention is much worse in this scenario! If this is the way you plan to use your connection I wouldn't recommend using STM prevention in its current form. Just run in the default mode to automatically reconfigure your QOS settings when you get STM'd. I knew STM Prevention wasn't one-size-fits-all as soon as I started implementing it. Clearly it's only really worthwhile if you don't download constantly and you only rarely get STM'd. It's especially useful if you usually only get STM'd in the last few minutes of an STM period. In that scenario taking the more extreme hit of STM Prevention over a short period is better than taking a five hour hit of the less extreme STM punishment. This is exactly what you and kardzzz are getting at. I reckon I can automate the decision in the script by comparing the options like this: if STM Prevention is about to be triggered, work out when the STM limit would be exceeded if we carried on as we are going, and calculate how much data would be transferred over the following five hours from that point for both scenarios - i.e. being STM'd or limiting bandwidth to prevent STM. The script could then go with whichever option would allow more data to be transferred. Does that sound sensible?
Yes that sounds good. If current STM prevention is going to limit download rate, you should do your normal throttling if we are close to the end of the STM period (i.e. 20:45). If it's closer to start of the interval and I'm downloading at 10Mbits then let the STM kick in, and carry downloading at maximum rate possible. I guess you work out how much data can be downloaded in total in each case from now until STM is lifted (~ next 5 hours), and pick the bigger one.
The more I look into this the better I think it will work. I think once this is implemented we can do away with the STM Prevention threshold and let the script decide what's best. I'll let you know when I have something good enough for you to test.
I like STM prevention, please don't remove the feature, I don't want to be flagged as a user that exceeds stm every day. This is what I was thinking So when STM Prevention kicks in say we've downloaded 2.4GB and have 600MB remaining. 1) Store the Value (600MB*0.1) (the amount you DON'T want to use) 2) each time the script is run work do the following sum:- (Total Allowance - Stored Value - used bandwidth) / seconds remaining That above will never use more than 90% of the remaining bandwidth, and it gets faster each time it's run if they're not downloading at full STM restricted speed. - Anthony
I'm not going to remove STM Prevention. I'm going to modify the script so that when you are on target to trigger STM, it works out which is the best course of action (i.e. which will allow you to transfer more data in the long term) - Prevention or Mitigation. I'll probably make it a new option so that people can carry on using the script exactly the way it is now if they wish. In my last post above I was talking about removing the STM Prevention threshold (i.e. the setting that controls when STM Prevention can kick in - default is after 80% of the STM limit has been transferred). I don't think that setting a threshold will be necessary or desirable with the new implementation - the whole point is to let the script decide what's best by forecasting the results of each option. Artificially skewing those results by introducing an arbitrary threshold would defeat the object of the exercise. I did consider this type of approach, but it means that QOS will probably be restarted every time the script runs when STM Prevention has been triggered. I don't know if you noticed, but restarting QOS isn't completely free, for me at least. It seems to disrupt my connection for a short time. I for one wouldn't want that happening every two minutes. The way STM Prevention works at the moment means that if you don't download/upload at the full restricted rate, prevention will eventually be removed and you will be given full bandwidth again. The temporary STM Prevention QOS limits will be removed if/when your total Average Rate for the STM period drops to below 90% of the maximum permissable rate for that period. I implemented it that way to ensure that QOS is restarted as little as possible.
Ok, v1.04 is now available, with Intelligent STM Prevention/Mitigation implemented as we've been discussing. This new mode is enabled on the command line by using the -i option (instead of -p). This is how it works: If the average transfer rate indicates that STM will be triggered (i.e. the cell in the table is red), calculate how long it will be before that happens. Calculate how much data could be transferred over that time at the uncapped rate, plus the amount of data that could be transferred during the STM sentence at the capped rate. Calculate how much data could be transferred over the same total time period if we enabled STM Prevention to prevent STM from being triggered. Compare the values from (2) and (3), and proceed with the option that will allow more data to be transferred. The details of the algorthim's decisions are written to the system log and can be seen in the Latest Log Entries section of the status page. I decided to leave the Prevention threshold alone for now, so none of this is calculated until at least 80% of the STM period's bandwidth limit has been transferred. While working on this and testing it I also noticed that inbound STM Prevention wasn't actually working, due to the amount of extra data that arrives on the WAN side of the router over and above the QOS limit we've set. This extra data is counted by rstats (and Virgin Media), even though it is ultimately discarded as a result of the temporary QOS settings. So we need to ensure that the incoming bandwidth before routing is low enough to prevent STM. In previous versions, the script worked out how much bandwidth was left before STM would be triggered and set QOS such that only 90% of that could be downloaded in the remaining time. I did some testing and found that the rate actually has to be based on downloading only 50% of the remaining bandwidth. This is obviously quite a big difference, but it has to be done - there's not much point in STM Prevention if it doesn't prevent STM... (Note that outbound STM Prevention is unchanged. 90% is just fine because the router has complete control over how much data is sent out.) This means that the Intelligent algorithm is even more important than before, because inbound STM Prevention is now more harsh on your transfer rate. Anyway, give it a go and let me know how it goes. T
Thanks for that. I've updated and things seem okay. Haven't triggered STM yet. I do get this in the logs: May 25 01:03:01 unknown cron.err crond[103]: USER root pid 10082 cmd /tmp/stm-monitor.sh M+ -i every 2 minutes. I think I've always got that - is it normal?
I didn't have cron logging enabled so I never saw this before. I'm not sure what the problem is, but this message seems to be printed when any cronjob is executed. Here's an extract from my log: Code: May 25 09:59:01 xxxxxxxxxxxx cron.err crond[114]: USER root pid 516 cmd /tmp/stm-monitor.sh XL -i May 25 10:00:01 xxxxxxxxxxxx cron.err crond[114]: USER root pid 678 cmd logger -p syslog.info -- -- MARK -- May 25 10:00:01 xxxxxxxxxxxx cron.err crond[114]: USER root pid 679 cmd rcheck --cron May 25 10:00:01 xxxxxxxxxxxx syslog.info root: -- MARK -- May 25 10:01:01 xxxxxxxxxxxx cron.err crond[114]: USER root pid 682 cmd /tmp/stm-monitor.sh XL -i May 25 10:03:01 xxxxxxxxxxxx cron.err crond[114]: USER root pid 875 cmd /tmp/stm-monitor.sh XL -i May 25 10:05:01 xxxxxxxxxxxx cron.err crond[114]: USER root pid 1107 cmd /tmp/stm-monitor.sh XL -i May 25 10:07:01 xxxxxxxxxxxx cron.err crond[114]: USER root pid 1410 cmd /tmp/stm-monitor.sh XL -i May 25 10:09:01 xxxxxxxxxxxx cron.err crond[114]: USER root pid 1700 cmd /tmp/stm-monitor.sh XL -i May 25 10:09:01 xxxxxxxxxxxx cron.err crond[114]: USER root pid 1701 cmd logger `date` May 25 10:09:01 xxxxxxxxxxxx user.notice root: Mon May 25 10:09:01 BST 2009 As you can see from the last couple of entries in the log, even a simple cronjob command such as "logger `date`" causes an error. I'm not sure what's going on but I don't think it's anything to do with my script. The good news is that these weird errors seem to have no impact at all on the commands executed by cron. I'm probably just going to disable cron logging and forget about them. EDIT: Just had a thought. Perhaps these messages from cron are simply notifications, to log which commands were executed at what times, and what their PIDs were. In other words, maybe they're not errors at all.
Sorry to bug you, but prevention was working, but after a while (over a week) it's stopped:- STM Description Start End Bandwidth Limit Bandwidth Used Average Rate Daytime Downstream (XL) 10:00 15:00 6000 MB 7253.78 MB 423.97 KB/s (3391.76 kb/s) However, mitigation has kicked in ok. Current QOS bandwidth: 5120 kb/s down, 174 kb/s up Any ideas what it could be? - Anthony
Excellent work tievolu! Working as expected. I will change the 80% threshold though as today we hit that at 6pm so internet has been painfully slow for last couple of hours. Thanks!
I've sorted out a domain name - the script is now located here: http://www.tievolu.co.uk/stm/stm-monitor.html http://www.tievolu.co.uk/stm/stm-monitor.sh Might be 48hrs before it's all working properly. In the meantime the old links will work (and will always work).
Script is excellent, been using it a while now. Thanks a lot Tievolu. Had a strange thing happen today though. I'm on M+ so 750MB downstream in the evening. Set iPlayer off on a 1.3 GB file. I have STM monitor in -i mode, so it works out what's best. What I see in the logs is something odd! Customisable settings: STM Prevention Threshold: 70% Normal Inbound Bandwidth: 10240 kb/s STM Inbound Bandwidth: 2560 kb/s Normal Outbound Bandwidth: 467 kb/s STM Outbound Bandwidth: 118 kb/s Jul 14 20:49:10 router user.notice STM-Monitor: STM (RX) will trigger in 00h05m51s. Potential RX over 05h05m51s with/without STM Prevention: 22118 MB / 6064 MB => STM Prevention is beneficial. Jul 14 20:49:10 router user.notice STM-Monitor: Setting preventative inbound QOS limit of 56 kbits/s Jul 14 20:51:08 router user.notice STM-Monitor: STM (RX) will trigger in 23h33m52s. Potential RX over 04h33m52s with/without STM Prevention: 19843 MB / 3665 MB => STM Prevention is beneficial. Jul 14 20:51:08 router user.notice STM-Monitor: Setting preventative inbound QOS limit of -338 kbits/s Jul 14 20:51:13 router user.notice STM-Monitor: STM Triggered: Evening Downstream (16:00 -> 21:00, RX, 750 MB) Jul 14 20:51:14 router user.notice STM-Monitor: STM sentence started: applying STM QOS configuration (until 01:51) Any ideas why it thought preventative was better, and to set QOS inbound to -338 kb/s? Obviously a few moments later (5 seconds, instead of 2 minutes) it hit the limit, and that was that!
(This post ended up as a bit of a brain dump - I hope it all makes sense!) I've always known there was at least one loophole in the script, and you've found it :biggrin: The problem has two factors. Firstly, the rstats sampling period of two minutes is really a bit too long for our purposes. The situation the script is looking at can change massively in a two minute period, especially when you're close to the end of an STM period, and the effect is enhanced when you're on one of the lower tiers with a small bandwidth limit relative to your maximum download rate. Secondly, there is a delay of just over a minute between the rstats data being produced and the script acting on it. The rstats data is produced on the even minutes (19:00, 19:02, 19:04 etc.), while the STM Monitor runs on every odd minute (19:01, 19:03, 19:05 etc.). This, combined with problem 1, means that it usually takes a couple of iterations for STM Prevention to find a suitable value. Basically, it calculates a value every iteration and then undercuts it slightly to make sure it will work. If a subsequent iteration, based on new data, indicates that a lower prevention limit is needed, the script goes with the new, lower value. If the value is low enough, every iteration after that will calculate a progressively higher prevention limit as the triggering of STM is gradually averted... With that in mind, you can see that it's possible to trigger STM before the prevention algorithm has sorted itself out - that seems to be exactly what's happened to you. You are right at the end of an STM period, so there's not much time for the script to do anything if you start transferring at a high rate (which is exactly what you did!) - especially as you can transfer at maximum speed for three minutes before the script knows that's what you're doing. As soon as it does see what you're doing, it tries to set the preventative limit to stop you triggering STM. (Prevention is always going to be beneficial here because we're so close to the end of the STM period - ten mins at a very slow speed is going to be better than the five hour STM punishment.) However, the initial prevention limit is not low enough (due to the second effect described above), and by the next iteration of the script it's too late - the STM limit has already been exceeded. Unfortunately the script isn't prepared for this state of affairs - it assumes that STM Prevention is always going to work i.e. that you'll never exceed the limit when STM prevention is activated. So it goes ahead and tries to tweak the prevention limit, and calculates a negative value (because you've already blown the STM limit). This of course doesn't make any sense, but it's a very short lived screw up because the script then immediately notices that the STM limit has been exceeded and applies the STM bandwidth limits instead. The only thing that could be done to improve this situation is to change the cron job to run the script on the even minutes, and introduce a short delay at the start of the script (maybe 10 seconds) to ensure that rstats has done its business before we start calculating anything. This would improve things, but it would be a little messier and it wouldn't eliminate the problem completely - there is still going to be a delay between the generation of the rstats data and the running of the script, and the rstats data is still produced only every two minutes. So the inherent lag in the script could potentially be reduced from three minutes to two minutes ten seconds, which would clearly help, but I'm not sure it would make a huge difference. The loophole would still exist... The only way to prevent this from happening completely is to avoid starting very large downloads within the last few minutes of an STM period. Awkward I know, but it's the only reliable solution
tievolu, been a while since i looked at this thread... i'm still running version 4 (intelligent) but at 99% on my large VM. What I didn't like was if the threshold is set at 80% and you hit that level at say 6pm, the script will determine what rate is best and maybe go 100 kb/s down 465 kb/s up (up untouched). I would prefer it if the script compares the 'calculated rate' and if its lower than the 'STM active rate', run with the 'STM active rate'. basically I'd rather have a cap at the virgin limit than have a crawling network and if i eventually trigger the STM I won't feel change Do you think its do-able for v 1.05? or can i do that already in the current version. Thanks.
Hi kardzzz. The problem with your suggestion is that the harsh bandwidth limits are essential if you want to prevent STM from kicking in. If you only reduce bandwidth to the level of the STM limit, STM will not be prevented. This defeats the object of "STM Prevention", and I don't think it offers any benefit over simple STM Mitigation either (your proposal would just mean you'd be limited to the STM bandwidth for a longer period of time), so it wouldn't make sense for the Intelligent algorithm to limit the bandwidth in this way rather than choosing Prevention or Mitigation. For what it's worth, I don't use the Prevention/Intelligent algorithms myself for the same reason . I hate having the crawling network. I'd rather trigger STM so long as my QOS settings are aware of the cap and adapt to it. So I stick with the original implementation - STM Mitigation. But I can see why some people would prefer to take the temporary bandwidth hit and prevent STM from kicking in, because in the long term it means you can download/upload more data.
tievolu, you make perfect sense. I think i'll stick with the original mitigation code or keep it to how I have it now. You did a lot of great work on this script and its made a huge difference over the last year that I now take it for granted not having crawling network (unless i trigger it intentionally by rebooting and downloading lots). thx
Hi, great script and it found it works really well. One question i have though is that when STM is in force and the router is rebooted - the preventative qos settings are removed and default qos bandwidth settings are put back. Now i know that rebooting the router resets the data transferred logs so everything shows green again, but when the default settings are applied it sets all of the inbound classes bandwidth back to none. I know that some people suggest that it makes no difference if these settings are set to the none value, but i have found that i get serious lag on the Playstation network if i don't allocate bandwidth specifically to my ps3 and am downloading at the same time. Basically i have my ps3 in the highest class with no bandwidth restrictions, and other internet based traffic set to high with 90% inbound allocation. This fixes the lag problem. So my question is, is there any way to prevent the STM script from wiping out the inbound class settings and setting them all to none when a router reboot occurs (the outbound settings remain uneffected and dont change. Thanks Rob
Hi Rob, This is a bug. When STM prevention kicks in, the script stores the user-specified inbound rates in a temporary nvram variable, while it temporarily sets the rate to 100 across the board to ensure that the inbound rate limiting works (this is done because if they're set to "None", as in the majority of cases, the rate won't be limited properly and STM prevention won't work). However, it doesn't do an nvram commit so the temporary variable is trashed if the router is power cycled while STM prevention is active (it should be preserved through a soft reboot - via the Tomato GUI - though). I've added a simple null check to the code that restores the rates from this variable, to ensure that it doesn't replace the inbound rates when the temporary variable is empty. This should prevent the behaviour you describe, and the rates you specified in the Tomato QOS GUI will be restored from nvram on reboot. Try with the latest version of the script (v1.05) and let me know if you still see this problem. Thanks for catching this, and I'm glad you're finding the script useful!
Hi Tievolu, Many thanks for this. I have just downloaded the latest version of the script, and will test it. Before i read your post i violated the STM on purpose to do some more testing. When i rebooted the router, and the STM script came back up, the 1.05 script remembered that the STM had been violated, and the sentence was still applied which is very cool! - Something it didn't do before, as it simply put the rates back to standard speed which kind of threw things off as far as QOS went until my 5 hour stint had passed. I wont know until later on when the script has re-applied my standard bandwidth settings if it changes my custom inbound settings, but i will post tomorrow and let you know the resuts. One other thing (not sure if its possible), but it would be really nice if the status monitor remembered the amount of bandwidth transferred within the STM period (in case of needing to reboot the router mid monitoring session), so that the bandwidth stats used would be accurate and STM still applied when needed - then this would be perfect :biggrin: Thanks so much for your hard work on this - it really is an awesome piece of work, and i am sure that everyone shares my gratitude. Regards Rob
This is actually the way it behaved before for STM mitigation. I haven't changed anything in 1.05 apart from adding the null check I talked about in my last post. However, the script only remembers the STM violation for soft reboots - i.e. reboots triggered via the Tomato web interface. If you power cycle the router the state is lost (because the nvram variables recording the state of the STM violation are not committed). The script can't "remember" anything about STM prevention because it relies on the live bandwidth data produced by rstats, which is reset after a reboot. I have considered this, and it is possible in principal, but to be completely robust the script would need to use persistent nvram variables, which would mean doing an nvram commit every time they are set/unset. This could put quite a lot of extra strain on the router's flash memory, so on balance I don't think it's a good idea. I could try to modify the script so that the state would at least survive a soft reboot, but it would require at least a constantly updated record of the Rx/Tx bandwidth totals, and I think it might be tricky to work out the logic to correctly decide when to use this stored data. Moreover, the stored bandwidth totals would have to be combined with the real live data from rstats in the correct way - working this logic out would be tricky too (the more I think about it the more difficult it seems). Unfortunately I don't have as much time to spend on this script as I did a few months ago. I have time to fix relatively simple bugs, but not to add features like this. So the bottom line is that this will probably never get implemented, at least not by me anyway!
Hmmm, interesting - i only ever perform soft reboots via the gui, and yesterday it wiped out the STM sentence but with the new script tonight it didn't. In fairness i was doing a lot of testing yesterday and a lot of soft reboots so that maybe explains why - either way it now works for me. This was just really a "nice to have" question - the script is great so no great hardship if it can't be easily done, and the way you have explained it - it doesn't sound so easy :biggrin: Also an update on the v1.05 script, instead of waiting for my sentence to expire, i manually changed the qos bandwidth settings to different values to see what happened when the script re-applied the STM. My inbound values stay to how i set them, so looks like your fix works. Awesome work, thanks so much :thumbup:
I can't seem to get this to work with tomato 1.27, not sure what I'm doing wrong, I can run the script manually through ssh fine, and it changes the QoS settings for that one time, but if i put it in the WAN up scripts section it does not appear to run. The web page for monitoring it is blank and the tomato log doesn't mention it loading Any ideas? ps - excellent work tievolu I have been looking for this for ages!
hey, thanks for the script, i tried the newest script on your site, it tried to set the stm of on purpose to test your script. im running it with the L option im not using preventition or anything like that. It looks like your script set the qos before i had hit the limit . i hit about 80% of the stm limit, and the stm monitor page says i have hit it, but i havnt yet :S. thanks for any pointers you can give, mark
i just had a problem, i tried to test the stm preventation ( i used the -i flag), and it failed, and i hit the limit at 22.59 , when the limits ended at 21.00 , so now im stuck with 5 hours at throttled speeds Code: Jun 30 01:00:45 unknown user.notice STM-Monitor: Downloaded Tomato STM Monitor v1.05 Jun 30 20:31:10 unknown user.notice STM-Monitor: STM (RX) will trigger in 00h23m42s. Potential RX over 05h23m42s with/without STM Prevention: 22163 MB / 7403 MB => STM Prevention is beneficial. Jun 30 20:31:10 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of 285 kbits/s Jun 30 20:33:09 unknown user.notice STM-Monitor: STM (RX) will trigger in 00h18m10s. Potential RX over 05h18m10s with/without STM Prevention: 21884 MB / 6988 MB => STM Prevention is beneficial. Jun 30 20:33:09 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of 237 kbits/s Jun 30 20:35:08 unknown user.notice STM-Monitor: STM (RX) will trigger in 00h15m41s. Potential RX over 05h15m41s with/without STM Prevention: 21842 MB / 6801 MB => STM Prevention is beneficial. Jun 30 20:35:08 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of 222 kbits/s Jun 30 20:37:08 unknown user.notice STM-Monitor: STM (RX) will trigger in 00h13m04s. Potential RX over 05h13m04s with/without STM Prevention: 21789 MB / 6605 MB => STM Prevention is beneficial. Jun 30 20:37:08 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of 201 kbits/s Jun 30 20:39:08 unknown user.notice STM-Monitor: STM (RX) will trigger in 00h11m14s. Potential RX over 05h11m14s with/without STM Prevention: 21797 MB / 6468 MB => STM Prevention is beneficial. Jun 30 20:39:08 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of 189 kbits/s Jun 30 20:41:08 unknown user.notice STM-Monitor: STM (RX) will trigger in 00h08m56s. Potential RX over 05h08m56s with/without STM Prevention: 21768 MB / 6295 MB => STM Prevention is beneficial. Jun 30 20:41:08 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of 166 kbits/s Jun 30 20:43:09 unknown user.notice STM-Monitor: STM (RX) will trigger in 00h07m12s. Potential RX over 05h07m12s with/without STM Prevention: 21784 MB / 6165 MB => STM Prevention is beneficial. Jun 30 20:43:09 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of 150 kbits/s Jun 30 20:45:08 unknown user.notice STM-Monitor: STM (RX) will trigger in 00h06m07s. Potential RX over 05h06m07s with/without STM Prevention: 21850 MB / 6084 MB => STM Prevention is beneficial. Jun 30 20:45:09 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of 144 kbits/s Jun 30 20:47:08 unknown user.notice STM-Monitor: STM (RX) will trigger in 00h04m06s. Potential RX over 05h04m06s with/without STM Prevention: 21843 MB / 5933 MB => STM Prevention is beneficial. Jun 30 20:47:08 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of 111 kbits/s Jun 30 20:49:09 unknown user.notice STM-Monitor: STM (RX) will trigger in 00h02m56s. Potential RX over 05h02m56s with/without STM Prevention: 21903 MB / 5845 MB => STM Prevention is beneficial. Jun 30 20:49:09 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of 94 kbits/s Jun 30 20:51:09 unknown user.notice STM-Monitor: STM (RX) will trigger in 00h01m50s. Potential RX over 05h01m50s with/without STM Prevention: 21967 MB / 5763 MB => STM Prevention is beneficial. Jun 30 20:51:09 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of 71 kbits/s Jun 30 20:53:09 unknown user.notice STM-Monitor: STM (RX) will trigger in 00h01m03s. Potential RX over 05h01m03s with/without STM Prevention: 22056 MB / 5704 MB => STM Prevention is beneficial. Jun 30 20:53:09 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of 52 kbits/s Jun 30 20:55:09 unknown user.notice STM-Monitor: STM (RX) will trigger in 00h00m21s. Potential RX over 05h00m21s with/without STM Prevention: 22152 MB / 5651 MB => STM Prevention is beneficial. Jun 30 20:55:09 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of 25 kbits/s Jun 30 20:57:09 unknown user.notice STM-Monitor: STM (RX) will trigger in 00h00m05s. Potential RX over 05h00m05s with/without STM Prevention: 22281 MB / 5631 MB => STM Prevention is beneficial. Jun 30 20:57:09 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of 9 kbits/s Jun 30 20:59:09 unknown user.notice STM-Monitor: STM (RX) will trigger in 23h59m55s. Potential RX over 04h59m55s with/without STM Prevention: 22419 MB / 5619 MB => STM Prevention is beneficial. Jun 30 20:59:09 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of -26 kbits/s Jun 30 20:59:13 unknown user.notice STM-Monitor: STM Triggered: Evening Downstream (16:00 -> 21:00, RX, 1500 MB) Jun 30 20:59:14 unknown user.notice STM-Monitor: STM sentence started: applying STM QOS configuration (until 01:59) edit: just disabled qos and done a speed test, my internet is the same speed as it always is, so right now, the script is limmiting my connections when it doesnt need to.
As with last few posters I'm having a strange problem with this script also when its set to L option. During the hours when evening upstream is active downloads are counted as upstream bandwidth being used. During the hours when only downstream STM applies only about 10% of downloads are counted. This results in either the script triggering unnecessarily because it thinks I've gone over the 800MB upload limit or not triggering when I go over my download limit but the script still thinks I haven't. Hoping someone familiar with Linux scripting could check this script for errors.
Following on from the above it's the part of the script below that doesn't look right to me. # Read the stats file and extract the parts we're interested in (vlan1 RX and TX) read_next_rx_tx=0 while read line do case $line in *vlan1*) read_next_rx_tx=1 ;; esac if [[ $read_next_rx_tx = 1 ]] then case $line in *rx:*) rx_data=$line ;; *tx:*) tx_data=$line ; read_next_rx_tx=0 ;; esac fi done < $stats_file rm $stats_file Many thanks for your help.
You mention that you hit the limit at 22:59, but the tomato log shows a time of 20:59. It looks like your router time was not set correctly.
That part of the code is a bit of a fudge - if the structure of the rstats file changes it may well break. I haven't upgraded my router for a long time - I'm still on 1.22. Unfortunately I simply don't have the time to work on this script any more. If you figure out what's causing your problem let me know!
I'm planning to modify the script to add the parameters for the upgraded upload speeds VM are planning to roll out over the next few months. The options will be as follows: M - current M offering M+ - M offering with upgraded upload (1Mb) L - current L offering L+ - L offering with upgraded upload (1Mb) XL - current XL offering XL+ - XL offering with upgraded upload (2Mb) XXL - XXL offering with upgraded upload (5Mb) and new STM policy Note that anyone currently using the M+ option will need to switch to M until their upload speed is upgraded (M+ was previously used to indicate M accounts that had been upgraded to 10Mb - I assume they have all been upgraded by now?).
Please note new management rules also. http://www.virginmedia.com/myvirginmedia/faster_upload_traffic_management_table.php Cheers.
Hi Tievolu, I think i may have found a bug with 1.07. I have a NAS on my home network, and i am currently in the process of migrating my media files over to an external hard disk which is attached to the USB port of my router (I plan to use my RT-N16 as a media server). The STM monitor appears to be treating this traffic as WAN traffic, and the STM sentence has been incorrectly applied. Is there any way to get round this? Cheers Rob
The script is 100% reliant on the data provided by rstats - if rstats is counting your USB traffic as WAN traffic (i.e. on vlan1) then there's nothing the script can do to separate it out from genuine WAN traffic. Have a look at the bandwidth stats in the Tomato GUI, which work off exactly the same data. I suspect they will also be incorrect.
Hello tievolu first of all thank you so much for the script. I've been looking for something like this for ages! I've just installed it on 2 routers in two different locations and I'm now testing them out. I did this literally 30 minutes ago so I guess I just need to wait to hit the peak hour to see something on the stats page, don't I? (please see image in attachment). I wanted to ask 3you questions if you don't mind: 1) In my understanding virgin-media (btw I used to work for them long time ago, lol!) considers weekend and bank holidays in a different way. For weekends they refer to "midday to midnight" period but they don't actually specify what they do. The helpdesk wouldn't give away details either. Do you have better information in regards? does your script already differentiate between weekday/weekend (if)? 2) How does your capping integrate with the built in tomato QoS? It is just a parent class for upload and download? I'm asking as I already do some class capping doing layer 7 matching. 3) Recalling the previous post about NAS traffic... before finding your script I was thinking to write one, and my initial idea was to use a QoS class and use the class counter rather than rstat. In theory: totals - USB traffic = what needs to be counted. Not sure how difficult this would it be though. Thanks you so much again! Stef
Actually (extending my prev post): 1) stat page is working 2) still open to discussion 3) I have the same problem as I have a NAS on my lan. As I run Victek mod on asus RT-N16 (like many of us on this forum I guess) wan interface is actually vlan2! vlan 1 is counting LAN traffic. So sould you either: a) add an option to manually specify the vlan interface linked to the WAN b) even better: let the script work out this by itself. e.g. filtering out IP adresses rfc1918 should leave you with 1 IP and 1 interface linked to it. From there picking up the right vlan should be trivial. In the meantime I've added this line after the script is downloaded and the chmod: sed 's/vlan1/vlan2/g' /tmp/stm-monitor.sh > /tmp/TMPFILE && mv /tmp/TMPFILE /tmp/stm-monitor.sh 4) Perhaps the stat page could be linked via .css 5) How about modifying the script behaviour like this: a) allow the script to use local sotrage e.g. /cifs1 /cif2/ mnt/usb .... a) at boot time verify the script version online (first line?) b) if version online is higher download and store locally, if equal run the local copy. c) in any case always run the script from local storage. This should help a lot in case for whatever reason your repository goes own. To be fair I'm mainly concern about point 3, but I'm happy to give ideas away if it can help anyhow, if not never mind Again thank you so much for the hard work! Stef
You can use "nvram get wan_ifname" instead of specifying the vlan manually. For example, used to reach pppoe bridged modem: ip addr add 192.168.0.13/24 dev $(nvram get wan_ifname) brd + #Assign address to vlan port iptables -I POSTROUTING -t nat -o $(nvram get wan_ifname) -d 192.168.0.0/24 -j MASQUERADE
Good stuff. I still think this should be included in the script itself but the following 2 lines between wget and chmod do the job nicely: wan=`nvram get wan_ifname` sed s/vlan1/$wan/g /tmp/stm-monitor.sh > /tmp/TMPFILE && mv /tmp/TMPFILE /tmp/stm-monitor.sh So my WANUP script now looks like: wget -O /tmp/stm-monitor.sh http://www.tievolu.co.uk/stm/stm-monitor.sh wan=`nvram get wan_ifname` sed s/vlan1/$wan/g /tmp/stm-monitor.sh > /tmp/TMPFILE && mv /tmp/TMPFILE /tmp/stm-monitor.sh chmod 755 /tmp/stm-monitor.sh cru a STM-Monitor "1-59/2 * * * * /tmp/stm-monitor.sh L -i" logger -t STM-Monitor "Downloaded `head /tmp/stm-monitor.sh | grep \"STM Monitor\" | sed 's/# //g'`" Thanks
Ok, I've been testing the script for 2 days now and I'm very impressed with the result. There's only one case I run into that started me mumble about.That goes back to my point #2 I happen to have an Internet TV and use BBC iplayer on a daily basis. Today it wan't usable at all (braking out every 10 seconds) as I left my laptop running bittorrent and eating up the quota made available. I've ended up adding a filter blocking out all the layer 7 matched IPP2P during critical hours, but this is of course not optimal. I'm not sure how many other users have this very same problem, but an option I thought about would have not just to use quota limitation but also an understanding of the traffic using it e.g. IPP2P can use max x% of what the quota available no matter what. This way delay sensitive data can be better used. Yes, you'll still have the same problem if quota is hit with non IPP2P traffic but I think that's rather unlikely as e.g. a BBC video is about 300 Mb/hour So in a nutshell e.g.: -- 100% cap all other traffic -- 30% cap IPP2P only -- 0% with an option to allow IPP2P to get to 100% of the quota e.g. 15 minute before the end of the monitored time. So yes, this would indeed need an interaction with bit count per QoS class. Option b) If all this sounds too complicated how about just allowing a timing cap for the manually defined P2P class during monitored time? Timed QoS settings is unfortunately not an option as per today and I guess it would be something rather trivial compared to the current script complexity. I'm not sure how helpful this input can be, but please consider this as my 2 cents Thanks again!
RS232's extra lines have fixed the problem top which was being caused by STM Monitor reading vlan1 instead of vlan2. However I've found another slight problem, bought some games in the Steam sale and the huge downloads revealed the following. Dec 31 12:55:04 ASUS-RT-N16-Tomato-Toastman user.notice STM-Monitor: STM Triggered: Daytime Downstream (10:00 -> 15:00, RX, 3000 MB) Dec 31 12:55:04 ASUS-RT-N16-Tomato-Toastman user.notice STM-Monitor: STM sentence started: applying STM QOS configuration (until 17:55) STM is triggered at 12:55 PM and the router applies the reduced Upload/Download limits, all good so far. But at 4 PM the STM Monitor starts to count download traffic even though STM has already been triggered. Dec 31 17:57:01 ASUS-RT-N16-Tomato-Toastman user.notice STM-Monitor: STM sentence served: applying normal QOS configuration Dec 31 18:05:04 ASUS-RT-N16-Tomato-Toastman user.notice STM-Monitor: STM Triggered: Evening Downstream (16:00 -> 21:00, RX, 1500 MB) Dec 31 18:05:05 ASUS-RT-N16-Tomato-Toastman user.notice STM-Monitor: STM sentence started: applying STM QOS configuration (until 23:05) Seven minutes after lifting STM it's reapplied due to the STM Monitor having already been counting downloads since 4PM Dec 31 23:07:01 ASUS-RT-N16-Tomato-Toastman user.notice STM-Monitor: STM sentence served: applying normal QOS configuration STM Monitor applies normal router speeds but STM is still in effect and wasn't lifted by ISP until 1:30am approx. Which means it wasn't actually triggered until about 8:30PM. During the 11:07pm to 1:30am period pings went through the roof as Steam saturated all the download bandwidth. STM Monitor should only start to count bandwidth once any previous STM sentences have been lifted. Sorry tievolu I only seem to pop on now and again to complain about something. It's a very useful tool for anyone on Virgin Media Broadband and I really do appreciate all the work you've put into it.
I've had to disable Prevention, it just doesn't seem to work in benefit of the users experience. Sure, maybe the user DOES get better possible download if it avoids the STM -- but it would do when its set to 1kbps and all traffic grinds to a crippling halt. I've even seen it apply -14kbps before now (yes, minus). So from now on, I'm just having mitigation.
I've been tweaking the script on a daily basis to get the best result. May I suggest to give it a go with the following setting ?: 1) use prevention 2) (before the cru command in the WAN UP script) Code: echo "STM_PREVENTION_THRESHOLD=10" >> /tmp/stm-monitor.cfg This way the real monitoring/tweaking is started after 10% of the bandwidth consumption, and avoid the "worst case scenario" where you eat up 80% of the bandwidth allowance at e.g. 3:30PM forcing the script to throttle the 20% left over until 9Pm. Actually this would just fail... 3) If you really want to improve user experience try also to reduce the STM_INBOUND_BANDWIDTH_WITH_STM. I have a L connection and use 1 Mbit with STM, but I may also try lower e.g. 800Kbit e.g. (before the cru command in the WAN UP script) Code: echo "STM_INBOUND_BANDWIDTH_WITH_STM=1000" >> /tmp/stm-monitor.cfg With these settings you'll cap the bandwidth much earlier and have a yes slower service but fast enough to browse internet and even watch videos during STM period. 4) use tomato access restriction to match and match (layer 7) and totally disable any IPP2P traffic between 10Am->3Pm and 4Pm->9PM 5) As sometime P2P traffic is not properly matched by the layer 7 filtering set the P2P client(s) to behave as per point 4 (e.g. utorrent has an internal scheduler where you can specify when stop/start the downloads) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Beside this I have a feature request: As VM applies a different cap for the two periods of the day I see a value in having different settings per different time of the day. e.g. 10 to 3 prevention at 90% (as nobody is home in any case) 4 to 9 prevention at 10% But any setting you may want really. HTH Stef
Thanks to everyone for the recent discussion! It's nice to see so much interest in the script. I've added the code to get the WAN interface name from nvram, rather than assuming it's vlan1, and I've also tried to address the problem highlighted by AsusTomato above. The script will now only count traffic from the end of the last STM sentence, if that was more recent than the beginning of the current STM period. A test version of the script, including these changes, is available here: [EDIT: The changes are now available in v1.08.] As for some of the other questions raised recently... the script is based on the traffic management policy that VM publish here: http://shop.virginmedia.com/help/traffic-management/traffic-management-faster-uploads.html This is the only information available to me. Currently the script mitigates/prevents STM by simply altering the total outbound/inbound max bandwidth values. The suggestions for tighter integration with QoS are interesting, but I'm not going to have enough time to look into that. If anyone else wants to give it a go, please feel free to modify my script any way you like. BTW don't apologise for raising problems - it's how we make things better
Thanks tievolu! I've started to modify the script to see how it reacts. It's well written and the topis itself is rather complex in nature so I guess it will take me sometime to get my head around. In the mean time one question on the script as it is now: I've set the threshold to 5% (for a test) and after some downloading my inbound cell eventually got amber (please see picture in attachment). Now, if the box is amber why the default QoS settings are still applied and not the STM ones (I have a L connection and currently running the script in prevention mode)? Thanks rs232
It started out as quite a simple script, but it's evolved into a bit of a mess and could do with being re-factored using functions. Unfortunately I don't have the time. I've tried to comment it as much as possible to keep it vaguely maintainable - with a bit of effort you should be able to work out what it's doing The QoS max inbound/outbound settings will only be changed if the rate exceeds the maximum allowed - i.e. if the cell turns red. Here's a description of exactly what happens: http://www.tievolu.co.uk/stm/stm-monitor.html#PREVENTION
Yes, I think so. I imagine that your bandwidth would still get counted on the stats page, so it might still be of some use, but the STM mitigation/prevention wouldn't work because they both use QoS. I'm not 100% sure though - it depends what the command "service qos restart" does when QoS is not already started. It might start QoS.....
It's already protected by the router admin password, because it's served by the same http server as the other admin pages.
The two changes discussed in post #69 are available in v1.08, which is now being served up from the normal URL (http://www.tievolu.co.uk/stm/stm-monitor.sh) Let me know if you have any problems.
i will have to test again, as i am sure it let me in with no password, will have to check when get home
If you're already logged into the admin interface you won't get prompted for the password when you access the stats page (just as you won't get prompted each time you access different pages in the admin interface). If you explicitly log out of the admin interface (bottom of the menu on the left hand side), then try to access the stats page, you should be prompted for your admin password.
Think it must have been Chrome playing up, I deleted all saved password but it still didn't prompt me, tested on different PC prompted for password have just installed the 1.08 script and will see how it performers
I think there's a problem with the mitigation logic. If I hit either limit then both download and upload are limited in the QoS settings, but virgin doesn't seem to do the same. After hitting the download limit, I can still reach my usual upload speed (after disabling the STM monitor script).
This definitely was not the case when I first wrote the script a couple of years ago - if you hit any STM limit your upload and download both got capped. However, it's quite possible that they've updated/improved their methods since then. It would clearly be much more sensible if they limited only the direction for which the STM limit was exceeded. Unfortunately I can't find any official Virgin Media documentation stating what they do in this regard, and I hardly ever trigger STM any more myself (the new 30Mb STM limits are pretty generous!), so I need some further confirmation of this before I modify the script to work this way - the changes would be fairly extensive and fundamental. I've posted the question on the Virgin Media support forum. Let's see what happens...
Tievolu, I'm getting an issue with v1.10 of this script. I have the following line appearing in my logs, and the STM Monitor does not start "Apr 16 11:41:01 RT-485B39E85566 user.notice STM-Monitor: Usage: stm-monitor.sh [S|M|M+|L|L+|XL|XL+|XL++|XXL] [-p|-i]" The line in my WAN-UP script reads "cru a STM-Monitor "1-59/2 * * * * /jffs/stm-monitor.sh [XL] [-i]" Any ideas? Thanks Rob
Hi Rob, You don't need the square brackets - i.e. the command in your WAN-UP script should read like this: Code: cru a STM-Monitor "1-59/2 * * * * /jffs/stm-monitor.sh XL -i" T
I think the progress bar looks very sexy http://greg.agiletortoise.com/2007/11/07/servoy-simple-html-progress-bar/ It should be very easy to be implemented in the tievolu script to report on bandwidth usage
no reboot required If like me, you like to keep your up-times going.. (its a Linux user thing). Then I realised you can have this script up and running without rebooting. In your routers status web-page. Goto Status>Overview then click the >WAN>status>release button telnet into your router and issue these commands:- Code: wget -O /jffs/stm-monitor.sh http://www.tievolu.co.uk/stm/stm-monitor.sh chmod 755 /jffs/stm-monitor.sh Go back to Status>Overview then click >WAN>Status>Renew You may need to click it twice and make sure your status is connected, with an IP address. back in your telnet session, issue:- Code: cru a stm-monitor.sh "1-59/2 * * * * /jffs/stm-monitor.sh [broadband type] [-p|-i]" Where [broadband type] = S, M, M+, L, L+, XL, XL+, XL++, or XXL, [-p] activates STM Prevention, and [-i] activates Intelligent STM Prevention/Mitigation. You will notice I uncapitalised the "STM" part of the file name, Linux is sensitive about this. if you now goto your monitor page http://192.168.1.1/ext/stm-monitor.htm you should see it up and running. However, if I any of this is incorrect, I must apologise. It "appears" to work for me though it is 4:4odd am. But im sure someone more intelligent will be able to tweak this a little, and I shall amend and give due credit.
Would this write the stats on jffs every time the script is called? If that's the case I seem to remember that using jffs for frequents writes is not suggested. However it's a good finding as tomatousb users can change jffs for usb. Cifs is another elegant possibility... Thanks for sharing!
Hi Tievolu, I'm sure you're already aware that VM is doubling the connections speeds sometime later on this year: So 10/1Mb --> 20/2 30/3Mb --> 60/6 50/5Mb --> 100/10 Not sure what will happen to the existing 100 though. It should be easy to modify the script accordingly. rs232
To answer the question from last year (!) - the script, stats and HTML are all held in the temporary RAM disk, so there are no issues with wearing out your flash. Regarding the recent STM policy changes and upgrades, I'm in the process of updating the script. If someone can tell me the exact download/upload speeds for each tier (from the modem settings) it would be helpful, because they don't always match the headline rate. For instance, from looking at my own modem config pages I know that the 30Mb tier is actually 33330000 b/s download, and 3333000 b/s download, (about 31.8Mb down, 3.18 Mb down). If no-one has this information, I'll just use the headline rates for the tiers that I don't know about. The state of VM's offerings is a bit of a mess at the moment, so I'll also be changing the option names to match the tiers described on VM's STM page to try and keep it as simple as possible: http://help.virginmedia.com/system/...E=Cable&CMD=VIEW_ARTICLE&ARTICLE_ID=2781#tier So the list of options will be: XXL100 XL60 XXL XL30 L30 ML20 XL L M S5 This means that existing installations will break when the new script is downloaded. Sorry about that, but keeping the old options plus the new ones would be really messy. I'm also going to base everything on the speeds after the upload speed upgrades. If I tried to handle all the speeds, before and after the upgrade, the script would have 17 options, which is a bit silly. Remember that you can always set your own speeds if necessary: http://www.tievolu.co.uk/stm/stm-monitor.html#CUSTOMISE EDIT: Updated script available here for testing: [EDIT: Changes are available in v1.11] Let me know if you spot any problems, otherwise I'll make this v1.11 next week:
I have been upgraded this morning from 10 to 20Mb, however my upload speed will stay at 1Mb until October due to a hub limitation . VM explained me that not all the customer can get any speed they want. e.g. with my postcode the hub serves 5/10/20/60/100 only, so 30, 50 and 120 are not an option. My contract is a "L". I guess in October when the hub will be upgraded my L will go directly from 20/1 to 30/3 as this is the basic contract they're currently selling. VM confirmed that on my 20/1 contract my data capacity is7GByte for the morning and 3.5GByte for the afternoon/evening. http://www.virginmedia.com/images/STM_20Mb_below_800pxA.jpg Considering the number of options available during this upgrade period perhaps it's a good idea to modify the script and allow the user to specify custom speed/capacity as an alternative to contract type. Thanks! rs232
I think the setting for my "L upgrade" contract should be:# Settings specific to LU (20Mb down, 1Mb up) if [[ $1 = "LU" ]] then # STM limits (values in MB) stm_period_bw_limit_mb_1=7000 stm_period_bw_limit_mb_2=3500 stm_period_bw_limit_mb_3=3000 # QOS settings (values in Kbits/s) inbound_bandwidth_without_stm=20240 inbound_bandwidth_with_stm=5120 outbound_bandwidth_without_stm=932 # 1024 * 0.91 outbound_bandwidth_with_stm=233 # 932 * 0.25P.S. according to you test script all the "inbound_bandwidth_with_stm" is set to 50% of the inbound_bandwidth_without_stm. Shouldn't this be 25% instead as the level of speed reduction is 75% ?Regardsrs232
Sorry I got the 50% level of reduction now... there are two different tables: one for <20Mb and one for >30Mb.
Ok, I fixed the reduction percentage for ML20 and below. I think everything else is ok. With regard to specifying options on the command line, there is already a way to customise the config so I'll think I'll leave it at that. The alternative is to force everyone to specify six command line options defining all the characteristics of their connection. I've made the STM period bandwidth limits customisable in the same was as the other options, as people without the upgraded upload speeds may have to tweak them. Let me know if you spot any other problems. If not, I'll make the current test script live as v1.11 next week.
v1.11 is now live as: http://www.tievolu.co.uk/stm/stm-monitor.sh v1.10 can still be downloaded from the following URL if, for some reason, you don't want to move up: http://www.tievolu.co.uk/stm/archive/stm-monitor-v1.10.sh Let me know if you have any problems.
About the script version 1.11: So far so good. Many thanks for the update! I have to admit that this script was a vital piece of software on my tomato when my afternoon cap was set to 1.5Gb, nowadays I get 3.5G for the same period of time and chances are that this will increase again in the future.I can now watch 3 iplayer videos in the afternoon without breaching the cap. BTW I've just noticed that on ML20 it's impossible to brake the upload cap as there's not enough bandwidth to make this happen. I *hope* this script will be needed less and less (If VM wants) in the future ;-) Great work Tievolu!
Agreed - I think this script will slowly become obsolete, which is a good thing. It's only really useful now for households with multiple users, where QOS is essential (whether STM'd or not) to make the link usable for web browsing while someone else is hammering the upstream with torrent traffic, for instance. However, even that scenario is probably not as big a problem as it used to be now that VM apply their own traffic shaping to P2P/Usenet traffic. My life is moving on as well. Since I created script I've had two kids, so I now have hardly any time at all to spend on things like this. To be honest, I quite like it that way Anyway, the net of this is that this may well be the last update from me. We'll see.
The STM rules have changed according to this post FAQ on the Virgin Media Forum about the Traffic Management Changes it says, If I go over the downstream threshold while downloading will my upstream be traffic managed? No you will only get traffic managed on the traffic that exceeds a threshold, upstream or downstream. Does version 1.11 now only apply STM to download or upload alone depend on which has gone over the limit? I'm unable to test it myself since I'm still using 1.10 until my area gets the speed upgrades at the end of this year.
This is a news to me, I haven't tried recently but I'm pretty sure that the limit was applied to both upstream and downstream as soon as either cap was overtaken. Worth testing! They may have change something...
No, v1.11 does not take this into account - this is definitely a recent change. I asked VM about the behaviour last year and was told that in general both upstream and downstream would be limited, regardless of which triggered STM. This will require some pretty fundamental changes to the script. As I mentioned before, I don't have much time to devote to this anymore so it might take a while, if I am able to do it at all.
I don't think this is as complicated as I first thought. I've had an initial stab at implementing the necessary changes, but I don't have time to test them out - I'm relying on you lot to do that The test version of the script is here: [EDIT: Changes are available in v1.12] Please give it a go and let me know if you have any problems.
