Virgin Media UK STM Monitoring Script

STM Monitor v1.12 is now live.

Let me know if you have any problems.

Hi tievolu,
thankyou so much to keep this updated knowing you are very busy with other things in your life. Can we just know what's the difference between v1.11 and v1.12?

Appreciated!

v1.12 takes into account the fact that VM now only cut the bandwidth in the direction that triggered STM, rather than limiting both. See posts 97-100.

cool! thanks.

Today I breeched my limit in downstream and I can confirm that the limit was applied only to my downloads. Cool!

Just an insignificant note:

STM Mitigation: Active - Downstream STM sentence will end at :56


The ":56" used to be "0:56" not sure this is wanted but just to let you know any ways.

Thanks again!!
rs232

Hi all,

Any chance of some help with this script I have tried for hours tryig to get it to work but get nothing. I am not the most technical person but can follow instructions .

I have tried the trusted and secure and after each attampe I receive no erros but the web page you goto is just a white page and thats it.

Hope somebody can help

Thanks

Paul

1) Do you have QoS enabled on your router?
2) What contract/speed do you have? -> you should be able to find this on your "my virgin media page"

Any ways:

Copy the following in Administrator/Scripts/WAN UP

NOTE this script above is based on ML20 contract (20Mb down 1Mb up) You should change this according to your VM contract. Possible options are:

Size: S (5Mb)
Size: M (10Mb)
Size: L (10Mb)
Size: XL (20Mb)
Size: ML 20 (20Mb)
Size: L30 (30Mb)
Size: XL 30 (30Mb)
Size: XXL 50 (50Mb)
Size: XL 60 (60Mb)
Size: XXL 100

Ref: http://help.virginmedia.com/system/...R_TYPE=Cable&CMD=VIEW_ARTICLE&ARTICLE_ID=2781

Once saved into WAN UP, reboot the router. After 5 minutes you'll find a page at your router address admin page + "/ext/stm-monitor.htm". e.g.

https://192.168.1.1:80/ext/stm-monitor.htm

Thanks it nows seems to work.

Quick question will changing the QOS settings to none effect this script ?

The only think you it needs from QoS (AFAIK) are bandwidth in and bandwidth out.

e.g. for my ML20 I've set

in: 18000
out: 840

Thanks got it working

the -i works a treat if I select -p it just says disabled, I'd rather not hit the cap and have speed reduced any ideas ?

I strongly advice not to use prevention. Breaching the cap is not a big drama especially with the new speeds out there.
Getting close to your cap limit and using prevention can seriously impact your internet connection and make it literally unusable until the time frame is finished.
When I say unusable I mean unable to load a any webpage.
Trust me I've tried them all ;-)

Hi,

I have been using this script with some sucess but since the 21st June I was getting stm'd a lot anyway after 5 days on the phone asking why I was constantly getting traffic managed apparenlty on the 21st of June the new STM was sent live. The operater gave me the 30mb details which I will list below could this be incorapated into the script ?

9am to 3pm 4GB speed reduced to 15mb for 1 hour if you carry on downloading and exceed another 5.5GB you are reduced to 9mb for 2 hours
3pm til 00:00 2.8GB reduced to 15mb for an 1 hour if you carry on downloading and exceed 3.6GB your speed is reduced to 6mb for 2 hours
00:00 to 9am Unlimited

Seems to me Virgin have given free upgrades and need users to cut back to stop network getting to busy.

I asked why this wasn't on the site he could not answer

To me this is CRAP should have changed to sky while I had chance.

Paul

Hi tievolu.
According to this page: http://www.virginmedia.com/images/STM_20Mb_below_800pxA.jpg

ML20 is now called M/L20 and the upload cap 3pm to 8pm has been halved from 3500Mb to 1750Mb,

Here's the other limits for connections above 30Mb: http://www.virginmedia.com/images/STM_30Mb_above_800pxA.jpg

Tievolu, I've tried to create v1.13beta myself, you can find it here below in attachment:

You migh or might not like it but things I've modified are:

1) updated the connections codes, speeds and caps as per VM webpages:
http://www.virginmedia.com/images/STM_20Mb_below_800pxA.jpg
http://www.virginmedia.com/images/STM_30Mb_above_800pxA.jpg

2) The script now displays only:

• Physical ML20 limit
• Current QoS values
• VM STMed QoS values

3) At the bottom of the page the customisable fields have been removed adding instead a link to your STM homepage where everything is well explained

4) Within the script the:
outbound_bandwidth_without_stm
inbound_bandwidth_without_stm
have been taken out of the loops and filled in with the NVRAM QoS settings.

5) The list of speed loops has been modified from a series of
IF -> THEN -> FI
into a single
IF -> THEN -> ELIF -> THEN -> ELIF -> THEN ....

6) Each loop has two new variables $downs and $ups with the contract speed. I though of this to simplify future management/modification. Everything else within the loop is calculated.

7) I didn't get the *0.91 in the loop calculation of v1.12. My understanding of QoS is to apply a 15-25% reduction to the nominal speed, but in any case I thought that comes from QoS now so this issue it's dealt with on the QoS page. The way it works in this beta is:
a) not STMed --> valued coming from the QoS config page
b) STMed --> Values calculated from the physical link speed and %restriction only.

The only point I wasn't very sure about about B) is whether apply simply
physicallinkspeed * %STMrestriction
or
physicallinkspeed * %STMrestriction * *somepercentage* (something in between 75-85%) to prevent QoS to use all the STM bandwidth) so I left it physicallinkspeed * %STMrestriction for the time being.

8) Page status redesign, including a progress bar. Not that there was something wrong with the old one at all. just personal preference.

After so many changes i would be surprise if there are no bugs, so whoever wants to test it please do.


NOTE: I couldn't find a way to solve the bug when the STM is triggered and the message displayed is doesn't display the end hour correctly:
STM Mitigation: Active - Downstream STM sentence will end at :02



I'm not sure what you think about this version, I've just modified what I thought was worthy. Do with this script whatever your like, but I'm quite happy to maintain it considering you've express various time your concerns on the amount of time you can put on it in the future.

Cheers
rs232

hi Tievolu, I'm still working on my personal version of the script (beta4 in attachment). This version solved the hour missing bug when STMed.

I wanted to ask you something quick.
Can you please help me to work out what is the 720 in the total_transferred_bytes variable? Perhaps the last 10 hours of data?

Code:

            total_transferred_bytes=`echo $data | awk '
                {
                    values_string = substr($0, index($0, "[") + 1, index($0, "]") - 6);
                    split(values_string, values_array, ",");
                    for (i=(721-number_of_values); i<=720; i++) {
                        total += values_array[i];
                    } printf("%0.f" total);
                }' number_of_values=$number_of_values total=0`

many thanks!

This is my personal version of the script. I'll call you STM-Monitor-rs232_v1 as I don't want to interfere with future Tievolu's versions if any Here the differences:

0) Based on Tievolou's v1.12. All the credit to him for his excellent script!
1) removed prevention, now the script accepts no parameters other then the mandatory contract speed
2) solved a bug with sentence end hour missing
3) redesign + progress bar
4) minor optimisation
5) Use of a new temporary file (/tmp/stm-monitor.var)
6) simplified the contract definition to speed up future changes if any
7) added reference to hh:mm left to the end of the sentence
8) added reference to last STM end time
9) changes to the crontab definition. As there's no prevention the script is updated every 5min within active periods and less frequently outside.
10) added reference to transferred data after the period if ended (reset daily after 1am)
11) added two new customisable variables: $QOS_IN_PERCENTAGE & $QOS_OUT_PERCENTAGE:
12) added reference to hostname/domain (in case you have multiple routers/connections)


Installation:
A) copy the script somewhere safe (e.g. cifs or usb). I'll use cifs1 in this example below, change accordingly.

B) Add the following lines to Admin/scripts/WAN UP

Code:

cp /cifs1/stm-monitor-rs232_v1.sh /tmp/stm-monitor.sh
chmod 755 /tmp/stm-monitor.sh
cru a STM-Monitor-day "1-59/5 10-20 * * *  /tmp/stm-monitor.sh [Broadband contract code]"
cru a STM-Monitor-evening "10,30,50 21-23 * * * /tmp/stm-monitor.sh [Broadband contract code]"
cru a STM-Monitor-night "55 0-9 * * * /tmp/stm-monitor.sh [Broadband contract code]"
logger -t STM-Monitor "Downloaded `head /tmp/stm-monitor.sh | grep \"STM Monitor\" | sed 's/# //g'`"

Replace "Broadband contract code" above with your/ Supported code are:
"S5", "M10", "L10", "XL20", "ML20", "L30", "XL30", "XXL50", "XL60", "XXL100" or "100"
If unsure double check these pages:
http://www.virginmedia.com/images/STM_20Mb_below_800pxA.jpg
http://www.virginmedia.com/images/STM_30Mb_above_800pxA.jpg

C) Point your browser to

Code:

http(s)://IP:(port)/ext/stm-monitor.htm

Now either wait until the script is run by the crontable (details above), or for the inpatients go to Tomato's admin interface tools/system and run:

Code:

/tmp/stm-monitor.sh [Broadband contract code]

NOTE: The XL60 speed has an upload of only 3Mb. Reading up in internet this will be put at 6Mb gradually on a area basis. So expect a new code to come out to differentiate between XL60/3 and XL60/6

Cheers
rs232

FYI, is you are on anything less than 30M call virgin media and they'll put you immediately on 30/2 (so called L30). They will not do this automatically unless you ask!
Mind you you need to have a superhub or they won't upgrade. If you don't have it they'll offer to send a free one for 20 GBP. Unless your old modem is broken and the delivery is free of course...

I have added support for saving data in case somebody restarts the router.

http://pastebin.com/Wj47ZCLb

It's based on the original script.
This might be still buggy, but the usage is: stm-monitor.sh [S5|M|L|XL|ML20|L30|XL30|XXL|XL60|XXL100] [-p|-i] [-f] BACK-UP_FILE
The back-up file has 3 lines:
timestamp
rx_data
dx_data

Each run the script pulls as much as possible relevant data from the back-up file to rx_data (tx_data) variable, adds the newest data from rstats and saves the updated rx_data (tx_data) to the back-up file.

I realised the above would exit with "file not writeable" error if the back-up file doesn't exist yet (on its first run), so here's a quick fix: http://pastebin.com/QQ5jvKnG

Hi, this sounds interesting.
Could you:
1) post a screenshot of what it looks like
2) post a diff to see what it was added to the original script? I wouldn't mind add this to my version of the script if you don't mind.

Regards

Hey. Screenshot would be identical to the original, so omitted.

I created a new Pastebin post, which highlights the difference from the original (v1.12), you should be able to add it easily and feel free to:

http://pastebin.com/diff.php?i=bacvb4XG

I changed the way we read the parameters, so examine the start of the script.

Note the main block regarding back-ups is

486

if [[ $use_backup -eq 1 ]]; then

..

fi

567

for some reason is not highlighted.

Important changes are lines

471-

481

where the data is read from rstats file (I decided to get of the brackets straight away), subsequently

633-

639

had to be slightly changed.

Look out for line 786, which caused me problems, since it had ${1}, which should be just changed to $bandwith_type. I simply replaced $1 with $bandwith_type everywhere else.

Hope it helps.

P.S. regarding your question what was 720, it's just 24 (hours) * 60 (minutes) / 2 (minutes); 2 - for time period of rstats

Thanks for this! I'll look into this when I get a spare hour or so.

I guess what I was asking with the screenshot is what is looks like when reporting historical data (if the scripts includes the reporting function at all of course).

Thanks again

2rs232: I streamlined the main block. The new version is edited in pastebin #119 (http://pastebin.com/QQ5jvKnG). (I decided to use egrep instead of awk, it seems faster).

I have also included a fix for people who put a script in a different directory than default:

1. # STM Monitor version
2. my_path="`dirname \"$0\"`" # relative
3. my_path="`( cd \"$my_path\" && pwd )`"
4. stm_monitor_version=`head $my_path/stm-monitor.sh | grep "STM Monitor" | sed 's/# //g'`

Happy merging!

Haven't been here for a while. Just a quick note to let you all know that I won't be updating the script any more - I need to upgrade my routers and for budget reasons I'm switching to a couple of TP-Link Atheros-based routers which won't run Tomato (I'm looking at the Gargoyle firmware as the next best thing).

Looks like the script is in safe hands though! Happy tinkering everyone...

Talk about overcomplicating it...

Code:

my_path=`dirname "$0"`
stm_monitor_version=`grep -m 1 "STM Monitor" "$my_path/stm-monitor.sh" | sed 's/# //g'`

1. There is no point to the use of double-quotes during the variable assignment; issues with spaces in paths happen during variable expansion, not assignment,
2. As such, there is no need for the excessive escaping/backslashing of double-quotes,
3. The cd && pwd is silly; if this is an attempt at error checking (i.e. if you can't cd to $my_path then bail out), then it's done wrong, because the script will continue on regardless if the cd failed or not; if you want error checking for this, add an if [ -f "$my_path/stm-monitor.sh" ]; then ... fi statement around the $stm_monitor_version assignment, then put your error handler in an else,
4. Get rid of the call to head; just use grep -m 1, which limits the match count to 1, i.e. stop reading from stdin once the first match is encountered (starting at the top of the file), exit, and thus issuing SIGPIPE,
5. Note that I'm using double-quotes around the variable and path in the grep (re: variable expansion), in case $my_path contains spaces,
6. I'm sure there's a way to do all this with just sed, but can't be bothered to remember sed syntax these days.

Code:

$ /home/jdc/blah\ blah/x
STM Monitor 1.0

$ ls -l /home/jdc/blah\ blah/*
-rw-------    1 jdc       users      59 Feb 13 06:16 /home/jdc/blah blah/stm-monitor.sh
-rwx------    1 jdc       users     149 Feb 13 06:16 /home/jdc/blah blah/x

$ cat /home/jdc/blah\ blah/stm-monitor.sh
#
# stuff we don't care about
#
#
#
# STM Monitor 1.0
#
#

VirginMedia UK has released today the new fair-usage policies.
http://www.virginmedia.com/images/STM20MblargeA.jpg
http://www.virginmedia.com/images/STM30MblargeA.jpg

I have a updated the mod script from v1 to v2.

The main change are:

1) the time-frame for the evening down period moved from 15-21 to 16-22
2) added reference to the P2P/newsgroup time based limitation (usage independent).
3) added the new 120 connection
4) updated percentage of limitation during STM
5) added link to VM fair usage policy tables
6) cosmetic adjustments

The installation procedure is also slightly different, I suggest you remove the old script and use either the online or offline version of the script here below in your wan up:

online autoupdating version: (will change when I change the source file)

Code:

wget -O /tmp/stm-monitor.sh http://tomatothemebase.eu/wp-content/uploads/stm-monitor-rs232.zip
unzip -o /tmp/stm-monitor.zip
rm /tmp/stm-monitor.zip
chmod 755 /tmp/stm-monitor.sh
cru a STM-Monitor-day "1-59/5 10-21 * * * /tmp/stm-monitor.sh [broadband type]"
cru a STM-Monitor-evening "10,30,50 22-23 * * * /tmp/stm-monitor.sh [broadband type]"
cru a STM-Monitor-night "55 0-9 * * * /tmp/stm-monitor.sh [broadband type]"
logger -t STM-Monitor "Downloaded `head /tmp/stm-monitor.sh | grep \"STM Monitor\" | sed 's/# //g'`"

online static version: (will always download v2 and never update)

Code:

wget -O /tmp/stm-monitor.sh http://tomatothemebase.eu/wp-content/uploads/stm-monitor-rs232_v2.zip
unzip -o /tmp/stm-monitor.zip
rm /tmp/stm-monitor.zip
chmod 755 /tmp/stm-monitor.sh
cru a STM-Monitor-day "1-59/5 10-21 * * * /tmp/stm-monitor.sh [broadband type]"
cru a STM-Monitor-evening "10,30,50 22-23 * * * /tmp/stm-monitor.sh [broadband type]"
cru a STM-Monitor-night "55 0-9 * * * /tmp/stm-monitor.sh [broadband type]"
logger -t STM-Monitor "Downloaded `head /tmp/stm-monitor.sh | grep \"STM Monitor\" | sed 's/# //g'`"

Offline version: (will never change)
Download the archive and extract the script on your local storage of preference (e.g. cifs1)

Code:

 
cp /cifs1/stm-monitor-rs232_v2.sh /tmp/stm-monitor.sh
 
chmod 755 /tmp/stm-monitor.sh
cru a STM-Monitor-day "1-59/5 10-21 * * * /tmp/stm-monitor.sh [broadband type]"
cru a STM-Monitor-evening "10,30,50 22-23 * * * /tmp/stm-monitor.sh [broadband type]"
cru a STM-Monitor-night "55 0-9 * * * /tmp/stm-monitor.sh [broadband type]"
logger -t STM-Monitor "Downloaded `head /tmp/stm-monitor.sh | grep \"STM Monitor\" | sed 's/# //g'`"

Where [broadband type] is the broadband type = "S5", "M10", "L10", "XL20", "ML20", "L30", "XL30", "XXL50", "XL60", "XXL100" , "100" or "120" (no quotes)

You can find below the screenshot of mod rs232_V2. A zip archive with the script is also in attachment



Let me know if any problems.

P.S. sorry to the people who gave suggestions lately I admit I haven't properly look into those yet, but I'm planning to sooner rather than later. I'm particularly interested in saving stats.

Regards
rs232

Just to let you know VM changed completely the fair usage policy introducing some new concept. Read to believe:
http://my.virginmedia.com/traffic-management/traffic-management-policy-30Mb-or-higher.html

This makes the old script useless and it will need some serious re-writing...