pixelserv compiled to run on router WRT54G

Discussion in 'Tomato Firmware' started by Jedis, Sep 5, 2009.

  1. HunterZ

    HunterZ Network Guru Member

    Update on HZ10:

    I finally got it building, and the binaries are each a little over 200 bytes larger (probably due to the fact that some extra work is performed to break the child processing into a separate function, and because the child processing no longer gets to inherit main()'s variables for free). That's not bad at all.

    The source is now split up as follows:
    • pixelserv.c contains the main() function, minus almost all of the stuff that happens inside the fork()'d child processes.
    • socket_handler.c contains a socket_handler() function that implements almost all of the logic (everything but closing a couple of file handles) that used to be inside the ford()'d child process. It also contains the static response strings as variables that are global to that file.
    • socket_handler.h contains the public interface to socket_handler().
    • util.c contains the volatile sig_atomic_t stats variables and almost all of the functions other than main() and socket_handler() - most notably signal_handler(), get_version(), get_stats(), etc. I may move all functions that are only used by socket_handler() to socket_handler.c so that they move out of scope of main().
    • util.h contains all of the #define's, system header #include's, response enum, public (extern) interface to the volatile sig_atomic_t stats variables, public interfaces to util.c functions, and changelog comments.

    The only part that still feels a bit yucky is that everything that can be configured via the command line has to be passed on to socket_handler(). There's really no way around this, though. I've thought about encapsulating it all in a struct so that I can just pass that around and so that I could also move command line processing out of main(), but that's probably overkill at this point.

    Still, I feel a lot better about the integrity of the code with it being split up a bit. This will make it harder for bugs to creep in as new features are added.

    Tomorrow I will do a final cleanup pass (probably just moving some stuff from util.c to socket_handler.c) and then start looking into non-blocking pipe I/O.
  2. jerrm

    jerrm Network Guru Member

    OpenVPN listens on all interfaces/addresses by default.

    Adding "local" (with the appropriate IP) to the VPN custom config will bind OpenVPN to a specific IP.

    This can be a little problematic if the WAN IP is dynamic. Tomato's "automatic" and "external" firewall rule options aren't very good if you want to use the LAN IP and will require tweaks.
    Last edited: Sep 9, 2014
  3. mstombs

    mstombs Network Guru Member

    To back up other replies, default is now ports 80 and 443 to avoid the need for a DNAT. If you already have a program listening on 443 this pixelserv will fail to start. To only listen on port 80 as per previous versions you must specify "-p 80" on the pixelserv command line, then use your DNAT as before.

    By the way - I think I saw an ad trying to use port 81 the other day, 8080 and 8090 are official alternate http ports, but it seems that 81, 82, are sometimes used. These ports can be added from the command line, but if become more popular (adservers avoiding blocks on port 80?) could change defaults!
    Last edited: Sep 10, 2014
  4. Almaz

    Almaz Serious Server Member

    You are absolutely right. That's what I was having problem with. Pixelserv would fail to start on 443. It would be great to be able to use other ports but using DNAT shouldn't make it any difference.
  5. HunterZ

    HunterZ Network Guru Member

    It would actually be nice if pixelserv could just listen to all ports on its IP, but from what little I've found it seems like this is not possible. However, you could probably use an iptables rule to direct all TCP connections on the pixelserv IP to port 80.

    On the other hand, this may make non-HTTP/SSL connections fail more slowly than the default reject-with tcp-reset behavior that I believe adblock sets up for unrecognized ports.
  6. leandroong

    leandroong LI Guru Member

    My stats update, in case you want some info
    /media/optware/adblock/pixelserv version: V35.HZ8 compiled: Sep 5 2014 20:05:40 from pixelserv.c
    904 req, 0 err, 0 tmo, 12 cls, 0 nou, 0 pth, 95 nfe, 701 ufe, 0 gif, 1 bad, 0 txt, 0 jpg, 0 png, 0 swf, 1 ico, 0 ssl, 3 sta, 0 stt, 91 rdr
  7. jerrm

    jerrm Network Guru Member

    It can effectively be done via iptables, redirecting all traffic to the desired port.
  8. HunterZ

    HunterZ Network Guru Member

    leandroong: Thanks, I was mostly interested in HZ10 reports to see if I could figure out what was causing the slowness for people, but I think it was just weirdness with blocking pipe reads. I'm going to implement non-blocking pipe reads in HZ10, which should hopefully clear up the issue.

    jerrm: That's what I said in the second sentence of your quote :p
  9. koitsu

    koitsu Network Guru Member

    ...or you could have just gone with mmap like I told ya. ;) (Sorry, I couldn't resist, haha)
  10. HunterZ

    HunterZ Network Guru Member

    That would be complicated because multiple child processes could try to write data at the same time, and the parent could try to read at the same time as the children are writing. The pipe works like a thread safe queue, which makes it comparatively simple.
  11. koitsu

    koitsu Network Guru Member

    Sort of -- that's what semaphores/mutexes are for. pipe(2) with O_NONBLOCK would be sufficient, sort of -- you need to watch out for edge cases like when write(2) only does a partial write due to the pipe buffer becoming full (errno=EAGAIN). With O_NONBLOCK and writes exceeding PIPE_BUF (I can't tell per pipe(7) if on Linux this is 65536, 4096, or 512 -- really I can't), you end up with interspersed/interleaved written data, at which point you might as well just implement a basic global mutex/semaphore. *grunts* Yay programming... Haha :)
  12. HunterZ

    HunterZ Network Guru Member

    lol well we'll see how it goes. I'm only writing and reading a single int over the pipe for each browser connection that is handled, so it would be surprising to see a pipe buffer overflow.

    On the other hand, a mutex would simplify things because I could store all the stats in shared memory and let the child processes/threads queue up to directly increment the stat counters, instead of using the current method of looking at the exit codes of the child processes in a signal handler to figure out which counter to increment. Edit: It would also eliminate the possible ugliness of the pipe read and stats update needing to occur in the main thread, which steals some time away from dispatching incoming socket connections to child processes.
  13. mstombs

    mstombs Network Guru Member

    Presumably the static version was compiled 1 second after the dynamic? Here's what mine currently reports

    /pixelserv version: V35.HZ8 compiled: Sep 5 2014 20:05:39 from pixelserv.c
    16408 req, 0 err, 1377 tmo, 34 cls, 0 nou, 0 pth, 937 nfe, 7877 ufe, 215 gif, 384 bad, 0 txt, 2 jpg, 3 png, 55 swf, 9 ico, 4269 ssl, 24 sta, 0 stt, 1222 rdr
    Must find out what device(s) generating most of the timeouts!
  14. HunterZ

    HunterZ Network Guru Member

    Yes, I modified build.sh to pop out host, standard, static, and tiny binaries.
  15. HunterZ

    HunterZ Network Guru Member

    Testing HZ10 now, with pipe I/O being monitored by the same select() call that waits for socket connections in multiport mode. Pipe reads are handled in main() at a lower priority than dispatching socket connections to handler processes. I also made a small optimization of building the select() file descriptor set only once instead of prior to every select() call.

    Before I release it (sometime tomorrow hopefully), does anyone have any objection with the redirect feature being enabled by default? I will probably make the -r parameter do nothing, and add -R for *disabling* redirect, so that I don't break people's launcher scripts (e.g. adblock.sh).

    I'm also thinking of adding an pixelserv uptime dump to the stats report. I'm not sure if calling clock() from time.h is the best way to do it, versus just using system time deltas (koitsu question I guess?), as clock() has some funny language about processor time consumed, which I'm worried may mean that it could get thrown off by idle-versus-busy time or by forking.
  16. mstombs

    mstombs Network Guru Member

    Not sure this works, the sets can be modified by the select call, and I just copied some example code from somewhere. But thinking about it, I am not sure we currently handle multiple sockets becoming readable at same time, so resetting the sets may lose connections? This is also the select that needs that retry macro, and I was also unsure about the first parameter, some examples use a big number, others +1 on the last file descriptor. I think I tried most combinations before...

    Fine by me, code seems robust (I didn't crash it!) but can't yet handle all Google ad links, so development to be encouraged!

    I'm sure many examples in the Tomato source, I'm sure I've seen gettimeofday used, shouldn't be an issue if only looking at the seconds part, but you will need to add code comment about fact that the 32 bit time is going to rollover in 2038..
  17. jerrm

    jerrm Network Guru Member

    I'd vote for it being default too. Greatly improves the adblock experience IMO.
  18. Goggy

    Goggy Network Guru Member

    I am too voting for -r as default ...
  19. HunterZ

    HunterZ Network Guru Member

    I found some other example code that generates the set once and then makes a copy at the start of the loop for select to operate on. It seems to work.

    I'm assuming that when multiple descriptors are ready but we only read from one of them, select will return immediately on the next call, as the remaining descriptors should still be ready for reading. The only potential danger is probably an increased chance of timeout, but that's probably a low risk in this application.

    Still, I will probably put in some logic to remove FDs from the working copy set as I handle them, and then skip the select call on subsequent loop iterations when the set is not empty. When the set becomes empty I will just re-copy the master set to the working set and make another call to select. This will have the advantage of avoiding the wasted time of unneeded select calls while also preventing newer connections from getting priority over old ones.
  20. Beast

    Beast Network Guru Member

    I vote yes to enable -r by default. HZ put a lot of work into getting it to work and as far as I can see, all of us want it to work.
  21. AndreDVJ

    AndreDVJ LI Guru Member

    I vote yes, to enable redirection by default.
  22. HunterZ

    HunterZ Network Guru Member

    So I had what appeared to be a temporary hang during a stats request, which lasted for a minute or two and manifested as a spinning page load icon in Firefox. Syslog showed that the socket write failed with a broken pipe status while servicing the stats request, which apparently indicates that the socket was closed before or during the write. Unfortunately I don't see a better way to handle this (fortunately rare) situation, especially since the code is already pretty paranoid and handles each socket connection in a fork process.

    I'm still hemming and hawing over the best way to get program uptime (i.e., elapsed time since program start), as there isn't an obvious facility for it (especially in the 2.4 kernel). It would be nice if it weren't affected by system clock time changes, in case pixelserv is started before the system time is set on boot.

    Edit: I got redirect enabled by default this morning in my local copy, with -r now ignored (but not generating errors, for backwards compatibility) and -R recognized as a request to disable redirect. I should probably move the command line handling to a separate function or two for readability purposes, but I want to get HZ10 out first.
    Last edited: Sep 14, 2014
  23. koitsu

    koitsu Network Guru Member

    If you're needing this within the program itself, the easiest way to do it is to use clock_gettime(), but it really depends on what you're trying to track/accomplish. I'm always wary of mentioning that function to people, because people end up doing things like using it while() loops without sleep() or other conditionals and with certain params it can cause a major performance hit on the kernel. (Here's an example of some mainstream nosql database crap that uses select() as a course timer instead of clock_gettime(), the result being the daemon chews up 7-10% CPU at all times. MongoDB, what a pile...)

    Another possible solution is to look at the mtime of /proc/{pid}/stat (meaning the "file" itself). That shouldn't change unless the process is actually restarted (while most other files in /proc/{pid} will show present time). I read one post somewhere from some dude that said the time is sometimes wrong of /proc/{pid}/stat, but I haven't seen any sign of that on my RT-N66U -- ls -l /proc/[0-9]*/stat looks correct to me. The rest would be simple math, but you'd probably need to call gettimeofday() (which can be expensive sometimes), and that just circles back around to discussing clock_gettime() again.

    The other solution is to read the 22nd field of /proc/{pid}/stat. See proc(5) for the format of that file and the format of that field (be aware that it's a 64-bit unsigned number). On Linux 2.6 you have to divide the number by sysconf(_SC_CLK_TCK), since the number is represented in clock ticks. On Linux 2.4 the number is represented in jiffies, and you may have to parse /proc/timer_list to figure that out (I'm really not familiar with jiffies... frickin' made-up time units, gee thanks...)

    Hope this helps in some manner!
  24. HunterZ

    HunterZ Network Guru Member


    How expensive is it to look at /proc stuff versus making time.h calls?

    Do I have to access /proc stuff by opening it as a file, or is there a C API for it that works well enough on K24?

    Edit: To clarify, I just want to be able to calculate the time in seconds since the program was started. I'm not interested in processor utilization times or any of that voodoo.
  25. koitsu

    koitsu Network Guru Member

    In True UNIX Fashion(tm), almost everything under /proc is a file, meaning you treat them just as you would a standard file on a filesystem. You can absolutely call stat(2) on something in /proc and get details about it just as you would a normal file outside of /proc. No need for an API -- that's the beauty of /proc on Linux.

    It's not expensive at all to look at /proc/{pid}/something. You know what your own process pid is (through getpid()), so:

    char mystatfile[PATH_MAX];
    struct stat sb;
    pid_t mypid = getpid();
    /* Maybe %d not %u; can't remember if pid_t is signed or unsigned */
    /* Also add code to make sure snprintf() copied enough data */
    snprintf(mystatfile, sizeof(mystatfile), "/proc/%u/stat", mypid);
    /* Make sure to handle error scenarios where stat() fails (ex. == -1). */
    /* It shouldn't, but never assume everything works.  :) */
    /* Brain isn't working right now, maybe this should be stat(&mystatfile, &sb); */
    stat(mystatfile, &sb);
    printf("process started: %s\n", ctime(&sb.st_mtime));
    (Now if only the BSD guys would follow suit and stop with this sysctl nonsense... ;) So sad to see Linux people jumping on that ship too)

    You can figure out the offset/how many seconds the thing has been running by doing simple math between a couple clock fields. If you're going to do time differential, however, make sure you use things that go off of localtime and not gmtime -- if I remember right the time stored is localtime, not gmtime.

    The only question I'd have is: how often do you need to know the uptime of the process? Is it something you need every second? Or is it only returned if, say, someone polls a status port the process listens on, etc.? If the latter, then doing the math only when queried isn't that bad. If it's something you need constantly, then calling clock-related function (ex. gettimeofday(2)) can be expensive, and you may want to look into using clock_gettime() instead (those return timespec structs with time_t tv_sec which is what you'd want)

    P.S. -- I hate time_t. It's one of the *IX typedefs that is utterly ridiculous because on some platforms it's an unsigned long, other it's a signed long, and on (many) it's a signed long long. Figuring out how to print this as a decimal/integer number is a PITA because you have to force cast -- yet you don't know what to cast because it varies per arch... *sigh* :)
  26. HunterZ

    HunterZ Network Guru Member

    So /proc/{pid}/stat is created when the program starts, and its modification timestamp is never changed?

    It's only going to be called when a TCP client requests statistics, and possibly when the program is terminated. This means that it should happen fairly rarely unless someone is doing a denial-of-service attack test, in which case they deserve whatever behavior they get.

    I should point out that the following quote from gettimeofday()'s man page scares me in regards to what I want to use it for:
    This is potentially problematic because the lack of RTC on routers means that it's conceivable that the time may jump from something like POSIX epoch to current time *after* pixelserv is started.

    clock_gettime(CLOCK_MONOTONIC, ...) sounds better, but can still me messed up by NTP. Unfortunately CLOCK_MONOTONIC_RAW is not available in Linux 2.4.

    I guess I'll run clock_gettime(CLOCK_MONOTONIC) early in main() to capture something close to a startup time, then call it again as needed and calculate the difference to get the elapsed uptime of the program.

    Edit: I wonder how the Linux 'uptime' command is implemented? I couldn't grok it from reading the source that I dug up because there were too many API layers. I suppose I could also just snapshot that at start of main() and then read it again as needed and take a difference.
    Last edited: Sep 14, 2014
  27. mstombs

    mstombs Network Guru Member

    miniupnpd uses time and uptime at start - but presumably is vulnerable to big clock changes after start, and only seems to use to schedule rule cleanings

    see for example
  28. koitsu

    koitsu Network Guru Member

    Correct. As mentioned I have seen only one post/comment from some random Internet dude saying his changed (while the program was running), and that is something I have never seen in my entire life. The mtime should not (will not) change. Forked children would get their own PIDs, so the parent PID's mtime (for /proc/{parentpid}/stat}) would remain the same. Pretty simple.

    Worrying about how Linux "uptime" is implemented I don't think is relevant -- that's all kept track of and managed in the kernel, where you have direct/easy access to timecounters (tied into interrupts; like on an x86 PC you'd probably have TSC, ACPI, or HPET available to keep track of time at a consistent interval). You're in userland, so you're limited to what libc and syscalls provide.
  29. HunterZ

    HunterZ Network Guru Member

    I implemented some code to take a clock_gettime(CLOCK_MONOTONIC, ...) reading at program start, and then it takes another whenever stats are generated and does a difftime().

    Also, I think you mentioned earlier not being sure what the pipe() buffer size is. On my RT-N66U, PIPE_BUF is 4096 bytes.
  30. HunterZ

    HunterZ Network Guru Member

    HZ10 released (finally!): https://github.com/HunterZ/pixelserv/releases/tag/V35.HZ10

    Tons of under-the-hood changes, but here is what may be noticeable to end-users:
    • Redirect is now enabled by default:
      • -r no longer does anything (but doesn't generate an error either).
      • -R has been added to *disable* redirect if desired.
    • Uptime (in seconds) of the program is now reported in the stats.
    • Binary sizes are a bit larger for various reasons.
      • If this is problematic for anyone, it's still possible to reduce size by building without some features.
    • A new binary flavor is included, pixelserv.fast, which is optimized for performance (-O3) instead of the standard size optimization (-Os).

    Functionally, the biggest under-the-hood change is that the same code used to watch for incoming socket connections also watches for reception size data from child processes. This means that it no longer has to *expect* pipe data from each child process, but can instead just process whatever comes in whenever it comes in. If this works better for people, I may eventually move all stats reporting to the pipe mechanism, as I think it may be a bit cleaner than the current system of catching child process exit status in a signal handler.

    Edit: Also, pipe reads/writes are non-blocking in an attempt to further avoid hangs.

    I've also broken up the code into multiple files (as I mentioned previously), which I believe makes the code more maintainable and easier to follow. There are probably other opportunities to break it down further, but it's probably not worth the bother at this time.

    Sorry for the long wait. I kept thinking of new things to add or tweak.
    HitheLightz and Goggy like this.
  31. leandroong

    leandroong LI Guru Member

    /media/optware/adblock/pixelserv version: V35.HZ10 compiled: Sep 14 2014 22:08:47
    22 uts, 1 req, 0 avg, 0 rmx, 0 err, 0 tmo, 0 cls, 0 nou, 0 pth, 0 nfe, 0 ufe, 0 gif, 0 bad, 0 txt, 0 jpg, 0 png, 0 swf, 0 ico, 0 ssl, 1 sta, 0 stt, 0 rdr

    Still same loading of youtube main page under win7, https://www.youtube.com/

    I did some stupid experimentation of cgi script and adblock:
    1. No more sources downloading, I did it manually
    2. combine all sources into blocklist format (manually also)
    3. modify adblock.sh, almost remove all, it just need to monitor action for stop,restart.
    3. CGI script only has actions for "restart", "stop" and "log erase"
    All this changes result in faster loading, less than 2 secs.
    Last edited: Sep 15, 2014
  32. leandroong

    leandroong LI Guru Member

    anyway, I consider it perfect enough, I don't mind waiting for youtube loading for 1 min. Thanks again
  33. leandroong

    leandroong LI Guru Member

    @HunterZ, my slow youtube main page downloads due to ads "ad.doubleclick.net" has been solved by my router. Padavan FW has firewall->URL Filter, after adding that, opening of main youtube become spontaneous.
    Now, I'm so happy.
  34. HunterZ

    HunterZ Network Guru Member

    That's good to hear!

    I'm also curious to know whether anyone on Tomato has good or bad experiences with HZ10, compared to HZ8 and HZ9.
  35. leandroong

    leandroong LI Guru Member

    ad.doubleclick.net is handle by padavan FW using iptable, if i'm not mistaken

    Chain urllist (1 references)
    target prot opt source destination
    REJECT tcp -- webstr: url ad.doubleclick.net reject-with tcp-reset

    In terms of speed HZ10 seems to be faster. Here is my stats
    /media/optware/adblock/pixelserv version: V35.HZ10 compiled: Sep 14 2014 22:08:47
    19865 uts, 482 req, 317 avg, 1096 rmx, 0 err, 0 tmo, 3 cls, 0 nou, 0 pth, 17 nfe, 311 ufe, 1 gif, 0 bad, 0 txt, 0 jpg, 0 png, 0 swf, 1 ico, 137 ssl, 9 sta, 0 stt, 3 rdr
  36. HunterZ

    HunterZ Network Guru Member

    Hmm, looks like your pixelserv is serving mostly JavaScript (ufe) and SSL (ssl) responses.

    It looks like Youtube uses SSL (HTTPS) connections for ad.doubleclick.net stuff. I wonder if that's somehow related to your slowdowns, and whether it's also related to something in Padavan versus Tomato.

    You may want to try the old way of having iptables reject *all* port 443 traffic on the pixelserv IP with a tcp-reset, instead of letting it go to pixelserv. It may be faster for your setup.
  37. leandroong

    leandroong LI Guru Member

    ad.doubleclick.net is the reason of slow page loading, waiting for 1 min before proceed loading the rest of photos.
    How do I enter that in iptables? I can test

    edit2: with the added iptables, waiting time is reduce to almost 0 sec
    Last edited: Sep 15, 2014
  38. mstombs

    mstombs Network Guru Member

    No slowdowns here with HZ8 or previous pixleserv with Chrome+adblock or IE on Win7. I wouldn't have thought iptables alone would work - see the big thread on trying block https with access restrictions! I can only see one link to ad.doubleclick.net a long url including a big random number. If I click on that on its own in Chrome immediately get "Certificate-based authentication failed" as expected.

    To check if a host (or domain) is blocked from a win7 cmd window :-

    C:\Users\>nslookup ad.doubleclick.net
    Server:  rtn66u.lan
    Name:    ad.doubleclick.net
    if that doesn't resolve to the pixelserv IP its not a problem with this thread!
    jerrm likes this.
  39. HunterZ

    HunterZ Network Guru Member

    What about HZ10?
  40. mstombs

    mstombs Network Guru Member

    Not yet downloaded...

    have now...

    off with the old, on with the new

    Sep 6 11:26:45 rtn66u daemon.info pixelserv[31742]: /mnt/usb4gb/pixelserv version: V35.HZ8 compiled: Sep 5 2014 20:05:39 from pixelserv.c
    Sep 6 11:26:45 rtn66u daemon.notice pixelserv[31744]: Listening on :
    Sep 6 11:26:45 rtn66u daemon.notice pixelserv[31744]: Listening on :
    Sep 15 21:19:23 rtn66u daemon.info pixelserv[31744]: 24455 req, 0 err, 1985 tmo, 58 cls, 0 nou, 0 pth, 1217 nfe, 11570 ufe, 283 gif, 500 bad, 0 txt, 2 jpg, 4 png, 58 swf, 10 ico, 7191 ssl, 26 sta, 0 stt, 1551 rdr
    Sep 15 21:19:23 rtn66u daemon.notice pixelserv[31744]: exit on SIGTERM
    Sep 15 21:20:10 rtn66u daemon.info pixelserv[2455]: /mnt/usb4gb/pixelserv.fast version: V35.HZ10 compiled: Sep 14 2014 22:08:48
    Sep 15 21:20:10 rtn66u daemon.notice pixelserv[2457]: Listening on :
    Sep 15 21:20:10 rtn66u daemon.notice pixelserv[2457]: Listening on :
    working fine, no slowdowns, also noticed my process IDs have looped, uptime 28 days.

    /mnt/usb4gb/pixelserv.fast version: V35.HZ10 compiled: Sep 14 2014 22:08:48
    700 uts, 146 req, 282 avg, 1626 rmx, 0 err, 2 tmo, 0 cls, 0 nou, 0 pth, 9 nfe, 19 ufe, 4 gif, 0 bad, 0 txt, 0 jpg, 0 png, 0 swf, 0 ico, 105 ssl, 5 sta, 0 stt, 2 rdr
    playing pixelserv bingo, I can't generate err or txt, can anyone?

    /mnt/usb4gb/pixelserv.fast version: V35.HZ10 compiled: Sep 14 2014 22:08:48
    3333 uts, 452 req, 364 avg, 2191 rmx, 0 err, 41 tmo, 1 cls, 1 nou, 1 pth, 22 nfe, 134 ufe, 12 gif, 9 bad, 0 txt, 1 jpg, 1 png, 1 swf, 1 ico, 190 ssl, 16 sta, 4 stt, 15 rdr
    Last edited: Sep 15, 2014
  41. strictlyfocused

    strictlyfocused Networkin' Nut Member

    Ive been running HZ10 since it was released this morning and I think it's easily the best version yet. Everything I've tested has worked great (youtube, redirects, etc). This is on an e3200 running the latest (121) build from Shibby.
    /mnt/sda1/adblock/pixelserv version: V35.HZ10 compiled: Sep 14 2014 22:08:47
    26702 uts, 1375 req, 343 avg, 1823 rmx, 0 err, 32 tmo, 0 cls, 0 nou, 0 pth, 32 nfe, 129 ufe, 1 gif, 3 bad, 0 txt, 0 jpg, 1 png, 0 swf, 2 ico, 899 ssl, 3 sta, 0 stt, 273 rdr
  42. vincom

    vincom LI Guru Member

    HunterZ you should start a new thread for this
    question, newb howto instructions on getting this installed for the ac66u
    i do have adblock w/piexlserv v34 running now from the script-clean-lean-and-mean-adblocking thread
  43. HunterZ

    HunterZ Network Guru Member

    vincom: My pixelserv V35.HZ10 should work as a drop-in replacement for pixelserv v34.
  44. vincom

    vincom LI Guru Member

    so i just copy pixelserv, pixelserv.fast, pixelserv.static, and pixelserv.tiny to my adblock folder where adblock.sh and my current pixelserv are located
    Last edited: Sep 16, 2014
  45. AndreDVJ

    AndreDVJ LI Guru Member

    Honestly I did not experience such issues with slowdowns...

    About the build, you can pick one of them. I compiled the "fast" version and I am using it now. You should not have space issues.
  46. HunterZ

    HunterZ Network Guru Member

    Just use pixelserv, or rename pixelserv.fast to pixelserv if you want to go for performance over size.
  47. AndreDVJ

    AndreDVJ LI Guru Member


    Happened with me the same issue as leandroong. Thumbnails take ages to load on YouTube.

    I got this from syslog
    Sep 16 20:07:07 WNR3500L daemon.err pixelserv[7247]: attempt to send response for status=22 resulted in send() error: Connection reset by peer
    Sep 16 20:07:07 WNR3500L daemon.warn pixelserv[7247]: connection handler exiting with EXIT_FAILURE status
    Sep 16 20:07:07 WNR3500L daemon.err pixelserv[7248]: attempt to send response for status=20 resulted in send() error: Connection reset by peer
    Sep 16 20:07:07 WNR3500L daemon.warn pixelserv[7248]: connection handler exiting with EXIT_FAILURE status
    Sep 16 20:07:07 WNR3500L daemon.err pixelserv[7249]: attempt to send response for status=20 resulted in send() error: Connection reset by peer
    Sep 16 20:07:07 WNR3500L daemon.warn pixelserv[7249]: connection handler exiting with EXIT_FAILURE status
    Sep 16 20:07:07 WNR3500L daemon.err pixelserv[7250]: attempt to send response for status=22 resulted in send() error: Connection reset by peer
    Sep 16 20:07:07 WNR3500L daemon.warn pixelserv[7250]: connection handler exiting with EXIT_FAILURE status
    Sep 16 20:07:07 WNR3500L daemon.err pixelserv[7251]: attempt to send response for status=22 resulted in send() error: Connection reset by peer
    Sep 16 20:07:07 WNR3500L daemon.warn pixelserv[7251]: connection handler exiting with EXIT_FAILURE status
    The following have been blocked around that time...
    Sep 16 20:07:07 www.googletagservices.com
    Sep 16 20:07:07 www.google-analytics.com
    Sep 16 20:07:07 cdn.mxpnl.com
    Sep 16 20:07:05 gg.google.com
    Sep 16 20:07:05 ad.doubleclick.net
    I also noticed that pixelservip/servstats ( wasn't working. The page with the stats refused to load. I restarted pixelserv and cleared the issue.
    EDIT: Found on the syslog:
    Sep 16 20:20:17 WNR3500L daemon.info pixelserv[1818]: 10167 uts, 186 req, 383 avg, 1050 rmx, 10 err, 1 tmo, 0 cls, 0 nou, 0 pth, 1 nfe, 82 ufe, 0 gif, 30 bad, 0 txt, 0 jpg, 0 png, 0 swf, 0 ico, 61 ssl, 0 sta, 0 stt, 1 rdr
    Sep 16 20:20:17 WNR3500L daemon.notice pixelserv[1818]: exit on SIGTERM
    If you need I can provide my dnsmasq/syslog logs (I save all kinds of logs on a USB flash drive).
    Last edited: Sep 17, 2014
  48. HunterZ

    HunterZ Network Guru Member

    Looks like the browser slammed the phone down before pixelserv got a chance to send the response, which in this case was of type SEND_UNK_EXT (which usually means javascript). This caused the send() to fail with errno=EPIPE.

    Do any TCP socket programming gurus know if there's a way to detect whether a socket connection has been closed by the peer before attempting a send()?

    Alternatively, could someone answer why send() appears to hang the entire program (not just the child process) in this case?

    Maybe I should be using non-blocking sends?
  49. koitsu

    koitsu Network Guru Member

    Sounds like a job for select(). Otherwise send() returning EPIPE is supposed to indicate the socket is closed or is in the process of being closed (regardless of what state, you can't send any data any longer). I'm going off of BSD man page documentation; Linux may behave differently.

    Regarding blocking behaviour on send(): sure, that can happen I would imagine -- it's going to depend on what's going on between the client and server at an even deeper layer (the TCP layer in this case). This is what timeouts are usually used for.

    I'm not too surprised the parent blocks as well as the child -- fork() results in the children getting copies of all the same fds that the parent has access to, so parent or child they'd both be affected. Only fds that you allocate *after* fork() are independent (child vs. parent, rather than shared).

    I think you're going to be forced to use O_NONBLOCK and watch out for EAGAIN. Otherwise you might have to consider using real threads and not a fork model.

    My general view is the same as this fellow's here (and the UNIX Network Programming link he gives you might help you out): http://stackoverflow.com/questions/14427898/socket-from-parent-to-child
  50. HunterZ

    HunterZ Network Guru Member

    I've considered using threads. It should be easier to do now that I've broken the program apart a bit into multiple files/functions, as there is now a better defined separation and interface between the connection listener parent process and individual connection handler child process. Threads may also be a performance gain because they are lighter weight than forks.

    I'll probably look into non-blocking sends first though.
  51. leandroong

    leandroong LI Guru Member

    loading time for main youtube is 1min 20secs maximum. Sometimes 40secs.

    /media/optware/adblock/pixelserv version: V35.HZ10 compiled: Sep 14 2014 22:08:47
    41345 uts, 143 req, 385 avg, 1229 rmx, 0 err, 0 tmo, 0 cls, 0 nou, 0 pth, 12 nfe, 128 ufe, 0 gif, 0 bad, 0 txt, 0 jpg, 0 png, 0 swf, 0 ico, 0 ssl, 1 sta, 0 stt, 2 rdr
  52. M0g13r

    M0g13r LI Guru Member

    still running HZ6 .... for me it feels like that it is the fastest version for now
    has everything i need ..... think everything after that is overkill ... just my 2 cents :)

    @mstombs i have much txt in all pixelserv versions .... here it's the browser (K-meleon) with an extra internal blocklist
    461 req, 120 err, 0 gif, 10 bad, 264 txt, 0 jpg, 0 png, 0 swf, 0 ico, 38 ssl, 1 sta, 28 rdr
  53. mstombs

    mstombs Network Guru Member

    I haven't yet looked at code, but I guess recent versions don't collect txt which is a default reply for nfe, ufe etc.
    i haven't yet seen the 'block'. but I am quite surprised at the rmx (biggest packed received), I guess genuine some web page attaching big cookie to a request? Collecting size info is from HZ9 on, yes just 'for info'!

    /mnt/usb4gb/pixelserv.fast version: V35.HZ10 compiled: Sep 14 2014 22:08:48
    138635 uts, 2874 req, 733 avg, 33480 rmx, 0 err, 213 tmo, 4 cls, 1 nou, 1 pth, 216 nfe, 1134 ufe, 65 gif, 56 bad, 0 txt, 1 jpg, 1 png, 1 swf, 1 ico, 971 ssl, 24 sta, 4 stt, 179 rdr
    youtube loads in 1.47s (Chrome page load timer)
  54. mstombs

    mstombs Network Guru Member

    But aren't you running pixelserv on ?

    This is normal exit from pixelserv, presumably when you you restarted it.
  55. HunterZ

    HunterZ Network Guru Member

    I didn't realize it until now, but there are no longer any cases that map to the txt counter. Most of them end up on nfe or ufe. I may turn on .js as a recognized txt response extension, as I think that's where the majority of them come from.
  56. mstombs

    mstombs Network Guru Member

    I don't see evidence that send is blocking - the syslog entry correctly reports the send() failed - I am sure this is quite common previously didn't bother with a message. I see you have code commented the send as potentially blocking, could use flag "MSG_DONTWAIT (since Linux 2.2) Enables nonblocking operation", but since all the responses are small I am not sure if it is a real possibility - all send() has to do is pass on the message to the kernel? http://linux.die.net/man/2/send
  57. HunterZ

    HunterZ Network Guru Member

    No, not with blocking a send(). I think what's happening is that the send() tries to put some data onto the socket and then waits around (i.e., blocks) for an acknowledgement until it either gets one or until there is some internal timeout. In the case that the client terminates the connection ungracefully ("connection reset by peer"), send() won't get an acknowledgement and will block for the full timeout period and then return EPIPE.

    I'll look into MSG_DONTWAIT as it may be easier than setting the whole socket connection to non-blocking.
  58. koitsu

    koitsu Network Guru Member

    MSG_DONTWAIT is a Linux-ism for just using fcntl() with O_NONBLOCK from what I can tell. Might as well just use fcntl() + F_SETFL.
  59. mstombs

    mstombs Network Guru Member

    But the same socket is used for read and write, we need the reads to block with a short timeout. If you try to close the comms channel while there is still data around the kernel sends an aggressive RST (red message in wireshark) BTDTGTTTS! It would be a waste of router CPU to continuously loop outside the read!
  60. koitsu

    koitsu Network Guru Member

    Possibly a job for epoll(). I'm used to things like BSD kqueue though. The premise/model is that the kernel tells userland when something happens I/O-wise, so there's no "spinning".
  61. HunterZ

    HunterZ Network Guru Member

    I'd really like to do things with Boost.Asio, but I'm not sure if getting the lib to build on mips is feasible, and it would almost certainly result in huge binaries because of the need for static linking.

    I think the kqueue alternative on Linux may be libevent/libev. I'm not sure if they're included with Tomato or not.

    pixelserv currently uses select() to avoid spinning on most I/O.
  62. koitsu

    koitsu Network Guru Member

    libevent/libev is just a library that acts as a convenience wrapper around whatever the OS supports (and it determines that at compile-time, not run-time). So for BSD you'd get select/poll/kqueue, for Linux you'd get select/poll/epoll, for Solaris you'd get select/poll/someothersyscallicantrememberthenameof. libevent does not come with TomatoUSB, although some of the TomatoUSB third-party tools/daemons/whatever may build it + statically include it (I'm not sure though, I'd have to grep through everything). If you were to use it, you'd need to compile it in statically to the pixelserv binary (the binary would be larger, but that's just how it goes).

    Boost is enormous and absolutely not a framework that should even be involved in what this daemon is (and even more so because it's mainly intended for embedded environments). That's my $0.02 though.

    Before even remotely considering C++ or frameworks, be aware of the fact that libc on TomatoUSB is uClibc, not GNU libc (glibc) (not to be confused with glib!) There is a surprisingly large number of libraries and frameworks which do not work with uClibc and instead make assumptions about semantics and behaviour that is tied directly to glibc. Here's a great resource comparing all the libcs (pros and cons). uClibc I do not particularly like but EVERY SINGLE PART of TomatoUSB is tied into this heavily, and changing it would be a pain (musl is overall a better choice). Anyway that's a whole separate topic I'd rather not get into...

    I'm sorry that I don't know more about what this pixelserv thing actually is/does/is implemented/the overall topology of what's trying to be accomplished. I'm giving *IX and programming advice where applicable without knowing that, and I just don't have the time (and also not much interest, but that's just me) to dedicate to figuring it out.
  63. HunterZ

    HunterZ Network Guru Member

    No worries. All of your posts have been significantly helpful in directing my efforts/research.

    If it helps, here's a thumbnail of what pixelserv does:
    • main():
      • Sets up socket connection listener file descriptors for TCP ports 80 and 443.
      • Sets up a signal handler to catch SIGCHLD so that it knows when a child process exit()s.
      • Goes into a loop:
        • Calls select() with no timeout to wait/block for connection requests on one or more ports.
        • accept()'s first pending connection request, which gives a new file descriptor for that specific connection.
        • fork()'s a child process to handle that connection fd.
        • close()'s the connection fd, as the parent process doesn't need it.
    • child process:
      • Performs a select() with timeout to wait for the socket connection to have something ready for read.
      • If something is available for read, it recv()'s a chunk of data.
      • If the data is an TLS/SSL handshake, it chooses a TLS access denied alert response string.
      • If the data is an HTTP GET request, it chooses an appropriate response string based on the URL specified by the GET request.
      • The selected response send()'s to the socket connection with the MSG_NOSIGNAL option.
      • The write direction of the socket connection is shutdown().
      • If that shutdown() succeeded, it goes into a loop:
        • Calls select() with timeout to check whether more data is waiting on the socket.
        • If data is found, it recv()'s a chunk to a throwaway buffer.
        • Loop terminates when select() returns an error/timeout or when recv() returns an error or no data received.
        • I believe the ideal case here is that select() will always return true until recv() returns zero, indicating EOF.
      • shutdown()'s the read direction of the socket connection.
      • close()'s the socket connection.
      • exit()'s with an exit code indicating which response type was sent (or which failure state was encountered).
    • signal handler:
      • In the case of SIGCHLD, does a while(waitpid(-1,,WNOHANG) > 0) loop:
        • If waitpid() returned WIFEXITED status, one of many sig_atomic_t statistics counters are incremented depending on the child's exit status.
  64. Tolocdn

    Tolocdn Serious Server Member

    I get the following error trying to run pixelserv (have tried .fast .static as well)
    ./pixelserv: line 1: syntax error: unexpected "("

    No luck with the googles in reference to this error :S
  65. HunterZ

    HunterZ Network Guru Member

    What router, kernel (if you know - usually K24 or K26), and firmware version are you running?

    I've had that error before when trying to run stuff compiled for the wrong kernel version, so it's possible that you're running K24 and my binary only works on K26?
  66. mstombs

    mstombs Network Guru Member

    Binaries should work on both K24 or K26, or at least give sensible error if missing library link. That error is usually caused by a corrupt transfer, must use 'binary mode' in Filezilla/winscp, not text/ascii mode. If file not identified as a ELF binary, the shell tries to run it as a script.
  67. leandroong

    leandroong LI Guru Member

    Finally, after reading backward and decided to apply the addl iptables rules given by darkingh93, yahoo main page loading disappear. Here is the iptables changes needed:
    stop() {
    elog "Stopping"
    rm "$CONF" &>/dev/null
    killall pixelserv &>/dev/null
    ifconfig $BRIDGE:1 down &>/dev/null
    iptables -D INPUT -p all -d $redirip -j REJECT &>/dev/null
    iptables -D INPUT -i $BRIDGE -p tcp -d $redirip --dport 80 -j ACCEPT &>/dev/null
    iptables -D INPUT -s $redirip -p all -j logdrop &>/dev/null
    iptables -D FORWARD -s $redirip -p all -j logdrop &>/dev/null
    elog "Done, restarting dnsmasq"
    service dnsmasq restart
    if [ "$PIXEL_IP" != "0" ]; then
    if ps | grep -v grep | grep -q "$prefix/pixelserv $redirip"; then
    elog "pixelserv already running, skipping"
    elog "Setting up pixelserv on $redirip"
    iptables -vL INPUT | grep -q "$BRIDGE.*$redirip *tcp dpt:www" || {
    iptables -I INPUT -p all -d $redirip -j REJECT
    iptables -I INPUT -i $BRIDGE -p tcp -d $redirip --dport 80 -j ACCEPT
    iptables -I INPUT -s $redirip -p all -j logdrop
    iptables -I FORWARD -s $redirip -p all -j logdrop
    ifconfig $BRIDGE:1 $redirip up
    "$prefix/pixelserv" $redirip $PIXEL_OPTS
    Note: I think this should be applied on page 1 of Haarp adblock. I think Haarp adblock is applicable to all router FWs and more simplified.

    edit2: code updated, remove forwading of port 443 to port 80.
    Last edited: Sep 21, 2014
  68. Tolocdn

    Tolocdn Serious Server Member

    I'm running: Tomato Firmware 1.28.0000 -122 K26ARM USB AIO-64K
    The files sit on a 8GB FAT formatted USB Key attached to the router (a R7000 nighthawk).
    This is with HZ10 btw.

    I have tried a Windblows transfer (usb stick is shared), a WinSCP ftp transfer with binary checked off for transfer as well.
  69. leandroong

    leandroong LI Guru Member

    @Tolocdn, HZ-pixelserv will not work, it is not compiled 100% static, it requires entware lib "libgcc_s.so.1".
    Remedy, ask andreiDVJ to compile the latest to produce static.
  70. HunterZ

    HunterZ Network Guru Member

    Tolocdn: You're on ARM rather than MIPS. The machine code of the binary is probably not compatible due to the different CPU architecture. You'll probably need to build it yourself or have someone with a Tomato ARM toolchain compile for you.

    leandroong: you shouldn't need to have iptables forward port 443 to port 80 unless you've got something weird going on, as pixelserv listens for connections on both ports by default.
  71. leandroong

    leandroong LI Guru Member

    [QUOTE="leandroong: you shouldn't need to have iptables forward port 443 to port 80 unless you've got something weird going on, as pixelserv listens for connections on both ports by default."[/QUOTE]

    Thanks. Just modified and tested.
  72. Tolocdn

    Tolocdn Serious Server Member

    I'm all up for compiling my own :) Now I just need to find a "how-to" for semi-noobs :)
  73. leandroong

    leandroong LI Guru Member

  74. leandroong

    leandroong LI Guru Member

    @HunterZ, You may want to replace your pixelserv.static binary with my tomatoware compiled in your site with this:

    Attached Files:

  75. HunterZ

    HunterZ Network Guru Member

    Thanks I'll add it as a second download attached to the release. I really need to get Tomatoware set up.

    Edit: Added, thanks.
    Last edited: Sep 21, 2014
  76. HunterZ

    HunterZ Network Guru Member

    Installing Tomatoware now. It's a huge thing, so it's taking a long time.

    Here are some ideas for HZ11 and beyond. The biggest thing I want to accomplish is to minimize hanging from attempting to send() to a connection that was abruptly closed by the client.

    • generate HTTP 204 response for google /generate_204 URLs.
      • This will prevent Android from thinking there's a walled garden / splash page on the LAN when clientX.google.com is blocked.
      • Firebug shows that the official response looks like this:
        HTTP/1.1 204 No Content
        Content-Length: 0
        Content-Type: text/html; charset=UTF-8
        Date: Sun, 21 Sep 2014 17:42:25 GMT
        Server: GFE/2.0
        X-Firefox-Spdy: 3.1
    • deprecate stats counter code in SIGCHLD handler with pipe I/O
      • use struct to contain response type and receive size
      • will need to keep zombie reaping code, as automatic reaping via ignoring SIGCHLD may not work in K24
    • be sure to close socket without further I/O if send() returns EPIPE or recv() returns 0
      • may already work this way, but need to double check
    • switch to building with Tomatoware so that I can produce true static binaries
    • recognize .js (javascript) extension as an explicit case of a txt (SEND_TXT) response
      • .js accounts for the majority of ufe (unrecognized file extension) responses
      • if I can whittle down the ufe responses enough, I can syslog() them to make sure there aren't any other file types we should be handling with non-text responses

    • non-blocking send()?
      • not sure how this would work without polling, as we don't want to tear down the socket while it's still in progress.
    • use threads instead of fork()s?
      • could be a performance gain because it eliminates cloning of parent process memory, need to reap zombies, need to use IPC pipe (can be replaced with mutex access to shared data), etc.
    • try finish recv()ing before send()ing response?
      • currently done the other way around, but it sounds like sockets may be less cranky if the whole request is read before attempting to send the response
      address long SSL response time issue?
      • firebug reports that SSL responses seem to often take a long time (up to 100ms, when HTTP almost never takes over 10ms)
      • may want to try other alert types or tightening the socket close code for SSL responses to see if they cause things to close down faster?
    • track average and max connection handler run time statistics?
  77. mstombs

    mstombs Network Guru Member

    I did look at some of these with wireshark, the browser may try 2 or 3 times with lower levels of security on each request, still need to compare with REJECT or maybe redirect to NGINX to handle properly?
    ISTR different alert codes worked better with different browsers.

    Still quite surprised quite how big requests my install has captured

    /mnt/usb4gb/pixelserv.fast version: V35.HZ10 compiled: Sep 14 2014 22:08:48
    520726 uts, 11454 req, 1034 avg, 42283 rmx, 0 err, 1086 tmo, 22 cls, 1 nou, 1 pth, 506 nfe, 4360 ufe, 180 gif, 154 bad, 0 txt, 1 jpg, 1 png, 3 swf, 2 ico, 4325 ssl, 50 sta, 8 stt, 752 rdr
    /mnt/usb4gb/pixelserv.fast version: V35.HZ10 compiled: Sep 14 2014 22:08:48
    521732 uts, 11507 req, 1050 avg, 42969 rmx, 0 err, 1086 tmo, 22 cls, 1 nou, 1 pth, 507 nfe, 4371 ufe, 195 gif, 164 bad, 0 txt, 1 jpg, 1 png, 3 swf, 2 ico, 4339 ssl, 52 sta, 8 stt, 752 rdr
    I have read that GET requests normally have 8kB max, current code would surely truncate redirects if not read in the first CHAR_BUF_SIZE recv.
    Last edited: Sep 21, 2014
  78. HunterZ

    HunterZ Network Guru Member

    Yeah I've read that you don't generally want to try for more than 4KB at a time, and concatenating that many reads would be a pain. It's possible that the large requests are SSL stuff or something though; I probably ought to track request size per response type as an experiment.
  79. mstombs

    mstombs Network Guru Member

    I know this or similar has been in many scripts, but it is a really expensive way of finding out if the pixelserv process exists, I think you can use

    if pidof pixelserv; then
    which just uses one busybox function call. NB pidof will return multiple answers if transiently there are multiple child process alive

    root@rtn66u:/tmp/home/root# pidof pixelserv.fast
    10755 10754 2457
    [I didn't rename pixelserv.fast to pixelserv ....]
    Last edited: Sep 21, 2014
  80. HunterZ

    HunterZ Network Guru Member

    See here for response: http://www.linksysinfo.org/index.ph...and-mean-adblocking.68464/page-10#post-251078
  81. pharma

    pharma Network Guru Member

    Small observation using pixelserv.fast ... some Web pages were hanging with HZ10. No hangs occurring with HZ8.

    Big thanks for your efforts!
    Last edited: Sep 22, 2014
  82. HunterZ

    HunterZ Network Guru Member

    What firmware are you using? Can you post stats and syslog from a pixelserv session that has exhibited the issue?
  83. HunterZ

    HunterZ Network Guru Member

    Got Tomatoware working. The normal and fast builds are 2K larger, while the tiny build is about 400 bytes smaller.

    Tomatoware uses gcc 4.6.4 dated 2011, while the Tomato cross-compiler toolchain I was using is gcc 4.2.4 dated 2007.

    The fully static build sometimes generates the following warnings:
    /cifs1/tomatoware/bin/../../tomatoware/lib/libc.so.0: warning: gethostbyname_r is obsolescent, use getnameinfo() instead.
    /cifs1/tomatoware/bin/../../tomatoware/lib/libc.so.0: warning: gethostbyaddr_r is obsolescent, use getaddrinfo() instead.
  84. mstombs

    mstombs Network Guru Member

    The toolchain is a bit broken then, last time I looked we do use getaddrinfo, and not the older gethostby... functions directly. The newer versions are all more IPV6 friendly, which may explain the size increase? IIRC the socket options are IPV4 specific, and "AF_INET" had to be added to prevent pixelserv.c trying to connect to IPV6 hostname on firmwares with IPV6 enabled.

    I wonder if musl, a new alternative to uclibc being introduced in dd-wrt would be better for small static compiles?

    Last edited: Mar 21, 2015
  85. pharma

    pharma Network Guru Member

    I'm using Tomato Firmware v1.28.0506 MIPSR2Toastman-RT-N K26 USB VPN on a RT-N66U router. OS is Windows 7 64bit and browser is IE 11, changing compatibility on browser has no effect. I've already reverted to HZ8 since others are dependent on having access, so will try to switch back at a later time to get stats and syslog info. One web page where hangs were consistent is http://www.bbc.com/news/, normally occurring after a few minutes of using HZ10. Does not happen with HZ8 ...

    Config file setting are below:

  86. leandroong

    leandroong LI Guru Member

    @pharma, I'm on HZ10-pixelserv. I don't have any hung issue on your given site, http://www.bbc.com/news/
    edit2: Did you have same firewall as mine, posted above.
  87. pharma

    pharma Network Guru Member

    My setup would be the default (normal), I don't have any special rules for redirecting ports. I'll do some more investigating but seems weird it does not happen with HZ8. When HZ10 was running I tried the default pixelserv, pixelserv.fast and pixelserv.static but same result. I'm also using Jerrm's latest AdBlock script.
  88. mstombs

    mstombs Network Guru Member

    Shame on the BBC, they do use a 3rd party tracking site which is in blocklists
    the http GET has lots of query string parameters which may be personal, so different users may get different results. If not blocked it receives a 1x1 43 byte gif that is strangely familiar!

    Do you use redirect "-r" with HZ8, this is default in HZ10 could be the important difference?
    Last edited: Sep 22, 2014
  89. pharma

    pharma Network Guru Member

    Yes, I used the same config file (posted above) which has "-r" for both HZ8 and HZ10. It does happen with a few other sites but BBC is my homepage so noticed it more often. If this is the default in HZ10 but not with HZ8 should I be using this parameter setting in HZ10? Or perhaps another?

    Interesting! I figured as much when I had to disable IE's "Do Not Track" setting a few month ago. :)

  90. HunterZ

    HunterZ Network Guru Member

    The redirect behavior is enabled by default in HZ10, and the -r parameter is simply ignored. I also added a -R parameter in case people want to disable it for some reason.
  91. pharma

    pharma Network Guru Member

    I think my initial choice is to enable the redirection behavior since this is what I currently use in HZ8 (based on my config file). Once I get home I'll do some more testing.

    Keep up the great work!
  92. mstombs

    mstombs Network Guru Member

    So its not the redirection code, BBC also seem to be using scorecardresearch.com, and they seem to be continuously pinging these sites to get both a page view count and measure of how long you stay on pages, so it may be you notice slowdowns on these pages but probably maybe caused elsewhere with a blocked pipe!
  93. HunterZ

    HunterZ Network Guru Member

    I'm thinking about writing some timing code to put around all of the blocking I/O calls and in all of the loops to generate a syslog if pixelserv gets hung up anywhere for too long. This may help us narrow down these temporary hangs.

    I'm going to also try to do everything I can to avoid the program getting hung up in the first place, which will probably involve switching to non-blocking operations (if only so that I can control the timeout times).

    Note to self: Use FD_COPY() instead of the assignment operator to copy the fd_set, just in case.
  94. mstombs

    mstombs Network Guru Member

    Last edited: Sep 22, 2014
  95. HunterZ

    HunterZ Network Guru Member

    From what I've seen it's more often a struct containing (at minimum) an array of integral types (probably file descriptors).

    Other people suggested using memcpy as a fallback, so maybe I'll do that.

    Edit: Or maybe I don't even have to worry:
  96. mstombs

    mstombs Network Guru Member

    Well I still haven't got HZ10 stuck

    /mnt/usb4gb/pixelserv.fast version: V35.HZ10 compiled: Sep 14 2014 22:08:48
    611206 uts, 13689 req, 1098 avg, 44285 rmx, 0 err, 1343 tmo, 24 cls, 1 nou, 1 pth, 606 nfe, 5294 ufe, 275 gif, 215 bad, 0 txt, 1 jpg, 1 png, 4 swf, 2 ico, 4954 ssl, 70 sta, 8 stt, 886 rdr
    And I have been monitoring what's going on by running tcpdump on the router, logging ALL relevant traffic

    root@rtn66u:/tmp/mnt/usb4gb# tcpdump -i br0 host -nnvvXS
    I have captured a series of 40k+ messages, all http POST from an old Cyanogenmod HTC Desire

    0x0030:  128a f941 504f 5354 202f 6161 702e 646f  ...APOST./aap.do
            0x0040:  2048 5454 502f 312e 310d 0a43 6f6e 7465  .HTTP/1.1..Conte
            0x0050:  6e74 2d4c 656e 6774 683a 2034 3430 3438  nt-Length:.44048
            0x0060:  0d0a 436f 6e74 656e 742d 5479 7065 3a20  ..Content-Type:.
            0x0070:  6170 706c 6963 6174 696f 6e2f 6f63 7465  application/octe
            0x0080:  742d 7374 7265 616d 0d0a 486f 7374 3a20  t-stream..Host:.
            0x0090:  6461 7461 2e66 6c75 7272 792e 636f 6d0d  data.flurry.com.
    which got the correct 501 not implemented response. Wonder what malware that is - seems to be a gps track log or phone backup...!
    Last edited: Sep 22, 2014
  97. HunterZ

    HunterZ Network Guru Member

  98. mstombs

    mstombs Network Guru Member

    Well flurry.com is in the great pgl.yoyo dnsmasq format domain block list. Seems it could be an app crash and huge memory dump being uploaded, so possibly genuine, but not nice if on mobile broadband!
  99. buggage

    buggage Network Guru Member

    I've been experiencing some 'hangs' with the newer versions, and have a couple test cases and some info - hopefully it helps.

    I have adblock running on a Linksys E4200, with latest Tomato RAF (Tomato RAF Firmware v1.28.9014 MIPSR2-RAF-v1.3g K26 USB)

    Using Jerrm's latest script(s). Only changed to host block instead of optimize.

    Config file is just standard, no changes using default block lists:


    Tried various newer Pixelserv versions below:

    Started with HZ10: Only (so far) seem to get websites hung up at cnn.com and weather.com. Not consistently, but semi-often the site freezes before anything on screen loads, and I can tell in the statusbar (Chrome) it's waiting for cdn.optimizely.com. CNN or Weather.com "freeze" indefinitely. As soon as I stop/restart adblock, they load right up. Same happens when using IE.

    Dropped back to HZ9: CNN and Weather.com consistently initially stop loading each time, again they appear to stop when waiting for cdn.optimizely.com, but sites seem to self-recover and continue loading after about 15 seconds each time.

    Dropped back to HZ8: So far, I cannot (or have not) been able to reproduce that error with either of those sites, or any others.

    Tried different block lists for kicks, and switched between hosts and optimize for the block list setup, but didn't seem to change things, only the different pixelserv versions look like they had any effect.

    Anyway not sure if this helps any or not, but just reporting what I've been seeing.
  100. HunterZ

    HunterZ Network Guru Member

    Thanks, I'll see if I can reproduce using those sites. Any chance you could post stats and/or syslog data from a pixelserv session that has encountered hangs?

    Update: Wow, weather.com in desktop Firefox really does a number on pixelserv. What's really interesting is that I don't see any error, timeout, or closed connection counts, nor any syslog complains from pixelserv.
    Last edited: Sep 23, 2014
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice