Frameworking realtime communications

While looking into flexible ways of writing an IRC bot(the nature of which will probably be in a "Tech Tuesday" post for tomorrow; It's a comm routing bot, so don't complain about IRC != IM quite yet.), I tried using IM libraries. I started with libpurple, because I use Pidgin all the time. That was utter failure, as I couldn't find enough organized documentation to even write a stub quickly.

Then I tried playing with telepathy. I wasn't able to get it to work in the few minutes I had left that evening, but I did learn a few things.

First, it's awesomely flexible, and there are tools that exist that allow a GUI user to essentially interact with it the same way your program might, which let me immediately dive in and try testing some capabilities.

Second, it's missing a decent IRC backend. The closest was "haze", which I couldn't get to connect to more than one IRC network at a time. Turns out Haze is just a glue module between Telepathy and libpurple, and the one-IRC-connection-at-a-time thing is a libpurple limitation. (I'm glad I didn't waste my time thoroughly studying the libpurple header files; It wouldn't have done what I needed, anyway.) I might be able to use multiple libpurple instances, but I don't know how safe that would be; libpurple uses a common filesystem location, I don't have control over it through Telepathy, and I can't trust it to be multi-instance safe in its access patterns.

Telepathy is interesting because it allows multiple connections, any of which routed through any number of connection modules. One could conceivably create a program that talks to fifteen different networks using one or as many protocols.

Darn cool. I wish it had better support for IRC as a client (and maybe as a server...who knows what kinds of options that opens up?). Support for things like identi.ca and Twitter would be pretty cool as well.

Console and Virtual machines

I know I may occasionally come off a bit hard-core in favor of the the text-mode and console environments, but, at the moment, there's something impacted beyond my preference of interface.

I've been working a lot with virtual machine setups over the last couple years. Rosetta Code has been running on VPS accounts under Xen (first at Slicehost, now at Linode) for quite some time, and then there's the various things I've done at work using various versions of VMWare.

VMWare is interesting, though. In a graphical OS such as Windows or Linux, it supports forcing the guests's virtual display to conform to VMWare's available view area; That's rather nice, as it means you're not stuck looking at a postage stamp on a huge monitor, or scrolling around a huge virtual display when you've got to look at it through a postage stamp. (Unity and Fusion are awesome and something else, but that's another tangent.)

Unfortunately, there are times and setups where you cannot have the overhead of a full GUI system. Linux being what it is, having the history it has, and the community origins it did, supports text-only environments. Additionally, Linux/UNIX console tools, coming from the roots they do, support resizing the display of text-only applications. Try resizing an xterm or PuTTY window, for example; The number of rows and columns change with the size of the window--you're not stuck in 80x24 or 132x24 land. (Yes, this sort of thing depends on termcap support, but that's not important at the moment.)

Unfortunately, unlike the with GUI goodness of X, VMWare can't resize the guest OS console to scale the same way. Sure, I can set up OpenSSH, connect in with PuTTY and get that same behavior, but what if I'm tweaking iptables chains? Fat lot of fun it is disconnecting oneself.

Oddly enough, Xen seems to have a similar problem. With both Linode and Slicehost, you get an AJAXy "console" much like you get if you run Linux without a GUI on an actual PC. Unfortunately, while my browser window might have a client area of 1680x1050, and the console font might only take 8x16 pixels per character, I'm not going to get a console environment of 210x65, no matter how nice such a large console might be.

Yes, I understand there's a variety of obstacles to the kernel framebuffer supporting dynamically changing display sizes. At the top of the list: Kernel programming is non-trivial. Yeah, I understand that. I occasionally watch lkml and read up on various subsystems for amusement, need and curiosity. Sure, there are framebuffer apps that assume that once the framebuffer size is set, it 's fixed. Oddly enough, GUI environments (even under X) did the same thing at one time, and some still do. That guest resizing deal only really works well with specific window managers; Some of my personal favorites are left out in the cold.

I'm not demanding anything; I can live with annoyances as I always have. But I'd like to put the bug in folks' ears and have them think about this sort of things...

Not done *yet*, but...

I sprung for a Linode 540 account. I copied RC over, and have been tuning everything all evening. The new VPS server is serving up 5-7Mb/s to my stress testing. Of dynamically-generated pages.

Points of interest:

• 540MB of RAM.
• memcached -k -m128M
• php5-xcache
• Tuned MySQL parameters
• The site's code is on a tmpfs mount
• Not using Squid
• My stress testing was done from a connection capped at 6Mb/s
• My stress testing was eight concurrent runs of wget -m --spider http://hostname

Kinda pleased.

Still have to set up all the subdomains, but the site should run a lot faster once I'm ready to switch DNS over. I'm also spending most of my CPU time in system, and that's coming from the apache forked processes (post-fork, mind you). I haven't figured out why.

You might have noticed the mention of tmpfs...I had a lot of spare RAM. I have a hard upper limit on the number of running forked PHP processes, an upper limit on memcached, and MySQL doesn't seem to be able to be coaxed into keeping its M_DRS memory resident. So I had about 300MB of RAM used for file cache.

Since I don't need to change the PHP files often, losing the contents of the tmpfs mount in an outage isn't a major concern, and I can cron an rsync to disk.

A Linux Success Story (What Ubuntu is good for)

First, some background.

A few months ago, we got my grandmother an old laptop so that she could get online and check her email without bringing her old mid-tower PC up from the basement. My parents set her up with Windows XP and a few basic apps, and I set up Thunderbird to tie in to her GMail account via IMAP.

That worked fine for a month or two, until she tried to go to a website she'd seen in a flier she'd gotten in the mail. She always types URLs into the Google Search Engine, and, in this case, the top result for the URL wasn't the site itself, but a page that popped up a fancy in-page Javascript widget that looked very much like Windows XP, with a warning that a virus had been detected on her computer, and asked her if she wanted it removed.

In her panicked state, she clicked "Yes" or "Run, whichever looked to her like it might resolve the problem presented to her by the dialog she was seeing.

Symantec didn't catch it. In fact, near as I could tell, Symantec was no longer on the system; The updater program was there, but there weren't any antivirus binaries to be found. Yet the system was getting unusually flaky. I spent two days trying to clean it up with antimalware like Avast, Spybot and AVG, and none of them detected anything when I scanned.

It was time to wipe and reinstall, but I didn't have the software keys, my parents did. I booted into an Ubuntu 9.04 live CD, installed Thunderbird and configured it to connect to my grandmother's email via IMAP. As she tried the liveCD, I tweaked this or that to make the system more usable and convenient for her. I increased the system font sizes, increased the display DPI, made the mouse cursor bigger (which was tricky; Turns out one has to switch away from the default mouse theme to get control over the cursor size), made the panels 48px tall instead of their defaults, put her two most used apps in launchers (firefox and thunderbird, now), hid the pager, disabled the touch pad click and scroll functionality, and moved the GNOME menu applet to the bottom-left corner of the screen.

By this point, the system was working very well for her. I consulted with my parents, and explained to them that even with the keys for the Windows software, Ubuntu was going to be easier for me to configure and maintain for her, and she'd been using and enjoying the liveCD configuration I'd set up for her thus far. After the phone call, I imaged the laptop's hard disk (for backup purposes; It's entirely plausible that the hard drive disk image will need to be restored to the trojan-laden state at some point.), and installed Ubuntu to the hard disk itself, blowing away the Windows install that was currently on it.

So far, so good. I've even installed Pidgin, which gives her quick and easy access to family and friends, as well as a DVD player, which makes it easier for her to see movies and such.

Ubuntu isn't for a guy like me who wants to choose his window manager or frequently compile development versions of media codecs, but it's great for someone like my grandmother who gets good usage out of the usability work that's gone into GNOME. She doesn't configure it, I do. But she uses it once I've configured it, and it works well for her that way.

Complete site backups in under five minutes

I've changed how site backups for Rosetta Code work.

Previously, a site backup was a rather manual affair of mysqldump, tar and scp. I've got a fair number of large tarballs that only contain the contents of the httpd root as well as database SQL dumps. Consumed space and bandwidth grows fairly quickly; This is part of why incremental backup strategies get devised.

Now, I have a set of nested makefiles on an offsite system with a few different targets. The root level makefile has 'backup', 'recurse' and 'git' targets. The 'backup' target depends on the other two. The recurse target drops into a few different subdirectories, one each for databases, webroot and logs. Each subdirectory has its own make file with a 'backup' target.

The databases subdirectory connects to the server, has the server do a mysqldump of the databases to a server-local file, and then uses rsync to copy the SQL dump file to the local system.

The webroot subdirectory uses rsync to copy the webroot to the local system.

The logs subdirectory uses rsync to copy system log files to the local system.

After running the recurse target, the root makefile runs the git target, which updates a local git repository with the modifications since the last time a backup was done. This is relatively cheap; Since the data has already been copied to the local system, the server isn't loaded down with the subsequent processing.

Once the whole thing has been primed (the first backup takes quite a while, as all of the data has to be copied), a full backup run takes less than five minutes to save off the changes from an hours' subsequent site traffic.

The biggest problem with the system is that serverside CPU usage is fairly heavy with the mysqldump and the rsync work. Serverside work currently takes four of the five minutes, while the git processing takes the rest. Hopefully, I'll be able to offset this by moving some batch processing typically done on the server to offsite, on better, faster hardware.

In the near future, I plan to add chunks of /etc to the backup process.

A comparison of compressors on SQL

I was cleaning and reorganizing data on my computers, and taking the opportunity to compress anything large. that I wouldn't need to see inside as part of, e.g. indexing.

I looked at one of my snapshots of Rosetta Code's database; Uncompressed, it occupied over 600MB. Compressed with bzip2, it occupied about one tenth of that. I decompressed and recompressed it with rzip, and was sufficiently surprised at the results that I tried to do a fairly thorough comparison of bzip2, rzip and gzip. Based on my use case, I collected data on compression ratio and speed. I did not collect data on RAM usage. (Though I do know that rzip at max compression exceeds the amount of RAM available on my basic Slice.)

It took (bzip2,rzip,gzip) (5m29s, 3m1s, 1m3s) to achieve compression ratios of (11.2, 605, 7.48).

Here's the raw data:

shortcircuit@dodo~/comprcompa
04:54:56 $ ./comprcompa.sh rcode_20090704_2029.sql
The primary purpose here is to compare compression ratios for database SQL dump backups.
Running environment is a Gentoo system running an AMD Phenom 9650.
As most things in Gentoo are typically compiled from source, these are the CFLAGS used:
CFLAGS="-march=amdfam10 -O2 -pipe"
our source database dump
-rw------- 1 shortcircuit shortcircuit 633986350 2009-08-02 02:58 rcode_20090704_2029.sql
Starting memory conditions; If there's a great deal of room for cache, we won't hit disk as frequently
             total       used       free     shared    buffers     cached
Mem:       7936588    7730932     205656          0     207452    6504216
-/+ buffers/cache:    1019264    6917324
Swap:            0          0          0
streaming original file into /dev/null via dd, to pull it into cache
1238254+1 records in
1238254+1 records out
633986350 bytes (634 MB) copied, 1.07369 s, 590 MB/s
starting uptime:  04:55:01 up 13 days, 17:13,  3 users,  load average: 0.67, 0.85, 0.71
Starting bzip2 -fk9

real	5m28.838s
user	5m28.279s
sys	0m0.537s
Post-bzip2 uptime:  05:00:29 up 13 days, 17:18,  3 users,  load average: 1.05, 1.03, 0.83
Pull original back into cache, for fair comparison
1238254+1 records in
1238254+1 records out
633986350 bytes (634 MB) copied, 1.097 s, 578 MB/s
Starting rzip -k9

real	3m0.765s
user	3m0.445s
sys	0m0.307s
Post-rzip uptime:  05:03:31 up 13 days, 17:21,  3 users,  load average: 1.21, 1.08, 0.87
Pull original back into cache, for fair comparison
1238254+1 records in
1238254+1 records out
633986350 bytes (634 MB) copied, 1.0498 s, 604 MB/s
Starting gzip -9

real	1m33.356s
user	1m32.011s
sys	0m0.890s
Post-gzip uptime:  05:05:06 up 13 days, 17:23,  3 users,  load average: 1.11, 1.07, 0.89
Final file sizes:
-rw------- 1 shortcircuit shortcircuit 56789389 2009-08-02 02:58 rcode_20090704_2029.sql.bz2
-rw------- 1 shortcircuit shortcircuit 84747906 2009-08-02 02:58 rcode_20090704_2029.sql.gz
-rw------- 1 shortcircuit shortcircuit 10472407 2009-08-02 05:03 rcode_20090704_2029.sql.rz
shortcircuit@dodo~/comprcompa
05:05:06 $ 

Harvesting of SourceForge projects and spamming SF users

I got an email from "Apparition " telling me my ActivityRank value was 0, and that I should add photo and blog entires to increase it. The email included a link to "http://group.ps/apparition"

"Apparition" is the name of a project I created years ago when I was in college and was convinced that I could do a better job writing computer lab imaging software than the "Ghost" software that was being used in the lab I worked in at the time. (Heck, I suspect that's even more true, now. Imagine imaging a lab, but having the imaging software using bittorrent on the local switch to distribute the drive images. Certainly would have worked better than imaging one machine in the lab to get past the building-building bottleneck, then having that machine serve up to the other 63 PCs in the lab...)

I never went anywhere with it. Haven't even really thought about it in the last few years. The website that looks like it sent me the email appears to have found my old project on SourceForge and sent an email to my SourceForge account, which was forwarded to my personal email. My first impression was a targeted malware campaign. I've grabbed grou.ps and grou.ps/apparition with wget and examined them with less and links, and neither *appears* to contain malicious code to my untrained eye, mostly jQuery code to control the interface to the social networking site. That's not to say it's safe; I wouldn't open it in a full browser outside of a clean VM, for the sake of being paranoid.

I'm fairly confident it's a programmatic attack, as Apparition is probably the least interesting of the SF projects I started and never went anywhere with. Even if it's not an attempt at spreading malware or collecting personal info of technical users and people with access to source code repos, it bothers me a bit that someone appears to be using programmatic means to harvest SF accounts, create places for them on a social networking site, and it bothers me that that email somehow got through SourceForge's filters.

Here are the headers:

Delivered-To: mikemol@gmail.com
Received: by 10.150.123.8 with SMTP id v8cs281312ybc;
        Fri, 31 Jul 2009 23:44:12 -0700 (PDT)
Received: by 10.100.216.7 with SMTP id o7mr4434688ang.120.1249109052582;
        Fri, 31 Jul 2009 23:44:12 -0700 (PDT)
Return-Path: 
Received: from mx.sourceforge.net (mx.sourceforge.net [216.34.181.68])
        by mx.google.com with ESMTP id 13si10290449yxe.76.2009.07.31.23.44.10;
        Fri, 31 Jul 2009 23:44:11 -0700 (PDT)
Received-SPF: fail (google.com: domain of bounce@grou.ps does not designate 216.34.181.68 as permitted sender) client-ip=216.34.181.68;
Authentication-Results: mx.google.com; spf=hardfail (google.com: domain of bounce@grou.ps does not designate 216.34.181.68 as permitted sender) smtp.mail=bounce@grou.ps; dkim=pass (test mode) header.i=@grou.ps
Received-SPF: pass (3b2kzd1.ch3.sourceforge.com: domain of grou.ps designates 67.228.206.32 as permitted sender) client-ip=67.228.206.32; envelope-from=bounce@grou.ps; helo=mail01.grou.ps;
Received: from mail01.grou.ps ([67.228.206.32])
	by 3b2kzd1.ch3.sourceforge.com with esmtp 
	(Exim 4.69)
	id 1MX8K6-0001oT-B0
	for shortcircuit@users.sourceforge.net; Sat, 01 Aug 2009 06:44:10 +0000
Received: from mail01.grou.ps (localhost [127.0.0.1])
	by mail01.grou.ps (Postfix) with ESMTP id 354613106EB
	for ; Thu, 30 Jul 2009 19:41:38 -0500 (CDT)
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=grou.ps; h=date:to:from
	:subject:message-id:list-unsubscribe:mime-version:content-type;
	 s=s1; bh=1s8xCFXOMjsEryNQjM/Jgn/L6VI=; b=D1qzcHRVWOs5w7fXza79KX
	QN5oOAE19VQ2tLJZsXbuSYJ22ZUBqdp5RoA4cXBbxta4f+9VOc8QSaPmytOFcURt
	0gQ2k9LeWahR63fxVLPDqkLpBmtRl59VKZN7TF4f9IfJ19/RdfhYqvnV/GbCcoE1
	XNNHwnMdiKiDfxSmZTxAI=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=grou.ps; h=date:to:from:subject
	:message-id:list-unsubscribe:mime-version:content-type; q=dns;
	 s=s1; b=HLOKH9ucB8bSXrWREFZ47U5qEHfgyCo2LN/5MvP+rrc6A1ZrDmF8zdL
	cpZzq1P4n43XFjssW18HRk/076lQHYxvi8XlcMuOk9hleImk22W366VZo+mnID+V
	5JYJlMNR1nMrB0x76i9RJ9fiCcSGTivRoDi6vrOOVmyj/FIIhqM0=
Received: from localhost.localdomain (unknown [67.228.115.98])
	by mail01.grou.ps (Postfix) with ESMTP id 2A2103106EA
	for ; Thu, 30 Jul 2009 19:41:38 -0500 (CDT)
Date: Thu, 30 Jul 2009 19:41:38 -0500
To: shortcircuit 
From: Apparition 
Subject: Apparition: Weekly Newsletter
Message-ID: <8507431ef5f59bdc6fecbb3f67dfa0e1@localhost.localdomain>
X-Priority: 3
X-Mailer: GROU.PS Mailer
List-Unsubscribe: http://grou.ps/noemail.php?x1=%25qCGbT5-%3B%5C9Wr%2BK8Asq4%27%3FWmJIX6%24%272%23xR&x2=%251H3qV%7B%23O%5Bd%5Bb%27%7E1%27%27%3A1%7CV%5C6vC%2FA%7ByN%21lU
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="b1_8507431ef5f59bdc6fecbb3f67dfa0e1"
X-Spam-Score: -0.5 (/)
X-Spam-Report: Spam Filtering performed by mx.sourceforge.net.
	See http://spamassassin.org/tag/ for more details.
	-1.5 SPF_CHECK_PASS         SPF reports sender host as permitted sender for
	sender-domain
	-0.0 SPF_PASS               SPF: sender matches SPF record
	-0.0 DKIM_VERIFIED          Domain Keys Identified Mail: signature passes
	verification
	0.0 DKIM_SIGNED            Domain Keys Identified Mail: message has a signature
	1.0 HTML_MESSAGE           BODY: HTML included in message
	0.0 AWL                    AWL: From: address is in the auto white-list
X-Headers-End: 1MX8K6-0001oT-B0

On syntax highlighting and artificial intelligence

So on Rosetta Code, we use GeSHi for syntax highlighting. The relationship between Rosetta Code, GeSHi, a programming language and the code written in that language is fairly simple. (The exact order of events inside GeSHi might be slightly different; I haven't delved deeply into its core)

Rosetta Code (by way of a MediaWiki parser extension) gives GeSHi a few pointers about how it wants the code formatted, the language the code sample will be in, and, finally, the code sample itself.

GeSHi takes the code example, and loads the language file named after the language in question. Each language file defines a PHP associative array that contains(among a couple other things) simple rules for how GeSHi can apply formatting to the code in a way that will clarify it to the viewer. These rules include lists of known keywords of various classifications, symbols used for normal commenting conventions and optional regex matching rules for each, among other things.

It's a perfectly reasonable, fairly static approach that allows syntax highlighting to cover a broad variety of languages without knowing how to parse that language's actual syntax, and so avoiding having a syntax error break the whole process.

Unfortunately, it requires Rosetta Code to be able to tell GeSHi what language a code sample is written in. It also leads to odd scenarios where a supported language and an unsupported language are so closely related that examples written for the unsupported language can be comfortably highlighted using the the rules for the supported language.

And I have yet to learn of a good way to do syntax highlighting for Forth. (The Forth developers appear to pretty much keep to their own community, and don't seem to do much in the way of outreach, which makes finding a solution relatively difficult, but I digress...)

So what does this have to do with artificial intelligence? Well, in identifying a language without being told what it is, of course!

A few solutions have been discussed. One approach that has been attempted had something to do with Markov Chains. The code is in the GeSHi repos, and I haven't looked at it.

One solution I suggested was to run the code example through all the supported languages (Yes, I know, that's expensive. Not something to be done in real time.), and select the ruleset based on how many rules(X) were matched for a language and how much of the code sample was identified(Y). Using a simple heuristic of (a*X)/(b*Y), you can account for a number of matched rules while hopefully accounting for an overly-greedy regex rule.

How can we take this a step farther? How about formatting languages we don't know about?

Well, many, many languages have rules in common. Common keywords, common code block identifiers, common symbols for comments, common symbols for quotation, etc. This tends to result from their being derived or inspired in some way by another language. For the sake of avoiding pedantry, I'll just say that C, C++, Perl, Python, PHP, Pascal and Java all have a few common ancestors.

One way would be to note the best N language matches, take an intersection of their common rules, and apply that intersection as its own ruleset. This would certainly work for many of the variants of BASIC out there, as well as specialized variants of common languages like C and low-level ISAs.

Localnet attempt

At some point in the next few weeks, I'm going to take one of my ASUS WL-330gEs and set it up as an AP without an Internet Connection. I'll use WPA2 encryption to keep the wireless communications encrypted (even client-client communications pass through the AP first, and are protected by a pairwise cypher), make the key evident in the SSID, and have all DNS requests resolve to a box I'll be running a web server on with a greeting page, an explanation, and phpBB.

I'm curious to see what happens; There are a couple tech-savvy neighbors around here, including one I suspect is running a honeypot. I'm wondering if I might be able to spawn a by-wireless neighborhood BBS.

I broached the idea of a localnet to grlug once before, but I emphasized its potential as a wireless mesh cloud with gated for-pay access to the open Internet. There are a number of problems with using a mesh for any latency-sensitive activities, so many folks would be disappointed when they find that not all Internet connections are created equal.

So I'm going to drop the Internet from the equation, and focus on community-centric services like a bulletin board and, maybe, a local email and XMPP server.

Let's see if they notice...

Well, this is neat

I've got a working Gentoo system. I feel like I've gotten the hang of Portage, the USE system, logging and configuration.

Is it faster than other Linux distros? I couldn't tell you; I didn't arrange for any meaningful benchmarks. Not that there is such a thing.

Is it faster than my other box? Well, sure, but it has 2.5x the bogomips and 4x the RAM.

And all that compiling? Well, I can rebuild my entire OS overnight, so that doesn't even really get in my way.

Why Gentoo? I got sick of running an OS that with every upgrade, and where those In The Know reinstalled the new version rather than upgrade. And I'm not talking about Windows. Also, I have a processor (AMD Phenom X4 9650) that has a ton of features that aren't being used by most distributions--even 64-bit ones--that I wanted to see what it felt like to take advantage of them.

No, I wouldn't recommend it to most people. Heck, I don't know if I know anyone I'd recommend it to in the Linux world, just because it requires a significant amount of low-level knowledge to get working. Sure, there's a nice tutorial to walk you through chroot, mounting proc and friends, configuring your kernel and installing grub--but if you don't know what you're doing, screwing up in that installation phase is easy to do and difficult to recover from. And the first couple times you start X and discover it ignoring keyboard and mouse input are sure to be interesting. (hint: configure portage to log out those elog notices and then read them.)

My observations on the ASUS WL-520GU

The Wireless:

I have to think that when someone at ASUS submitted a design proposal for this device, what they had in mind was what a wireless router would do if it were designed by Tim Taylor, and then trimmed it back so that it'd be legal.

To start with, here are a couple paragraphs taken from its quicksetup guide:

"This device and its antenna(s) must not be co-located or operating in conjunction with any other antenna or transmitter."

"To maintain compliance with FCC's RF exposure guidelines, this equipment should be installed and operated with minimum distance 20cm between the radiator and your body. Use on[ly] the supplied antenna."

Don't panic, the wifi transceiver in your laptop isn't going to sterilize you.

I stopped by #hamradio on Freenode to discuss it, and wound up looking up the FCC ID on the FCC's website.[1] I pulled a copy of the correspondance regarding the RF exposure, to figure out what's going on.

First, according to that correspondance, the maximum safe exposure limit for RF in the 2.4GHz range for uncontrolled exposure (i.e. consumer setup) is 1mW/cm^2. Taking the inverse square law backward 20cm, that tells me that the antenna is radiating at 4000mW, or 4W.

Taking this back to #hamradio, it was explained to me that 4W is the maximum legal[2] ERP[3] for a one-to-many link[4] on the 2.4GHz unlicensed band. The prohibition on using a different antenna is so that the consumer doesn't replace it with an antenna that has a higher gain than the one that came with the router.

The RF exposure documentation in the correspondance between the engineers in Taiwan and the FCC also contains the design and specs of the antenna, which indicate the antenna is omnidirectional and has a gain of 1.9dBm. If the consumer uses an antenna with a gain higher than that, they risk going over that 4W ERP limit, which in turn brings us back to that 20cm safe distance warning.

4W is more than enough, though; My laptop used to be able to pick up seven or so wireless networks from the neighborhood sitting right here in the house. Now I can only see me, most of the time. I imagine it's possible to see me from Ottawa county.

[1] If you're curious, the FCC ID is MSQWL520GUGC.

[2] Possibly only so for unlicensed user; A licensed ham gets more privileges, but there are additional

strings attached. Unlicensed equipment and configurations are targeted for the least-common-denominator.

[3] Effective Radiating Power; Essentially how much energy you'll see coming out of an antenna.

[4] This is something of an odd requirement that I may not understand properly. The assumption, apparently, is that if you're going to have a higher effective power output, you're only going to use it for point-to-point links. There's also a rule saying you should only put out as much power as needed for the application, but that would seem to leave things a little loose, still.

The Wired

The WL-520GU has four 10/100 LAN ports built in, as well as a WAN port presumably for attaching to your DSL or cable modem. Fine for access from a wired device to a wireless, or from a wired device to the Internet by way of today's broadband, but it's not going to grant impressive degrees of performance going from one wired device to another. If you have need of high-speed data transfer from two wired devices, it's probably best to chain the WL-520GU with a gigabit switch, and accept that you'll be stuck sharing a 100Mb pipe between wired-to-Internet and wired-to-wireless connections.

The Firmware

This is the part that Tim Taylor overlooked. The firmware has a great array of features, even going so far as to provide an AJAX-augmented web interface that gives you a live overview of your network in your browser, complete with problem analysis and diagnosis.

But they missed some of the more basic points of having functional software. So far, I've hit three nasty bugs.

The first bug I encountered was related to the first configuration option I changed: I turned off the wireless radio until I had everything configured the way I wanted it. I got everything configured, and then I turned the radio back on. When I turned the radio back on, the firmware lost all of the configuration settings I'd put in, defaulting to a completely open (no WEP, no WPA, no nuthin') 802.11bg network with an SSID of "default" and a broadcast power of 4W[1]. Not fun.

The second bug (well, there are two here) I encountered was in the configuration of the router's IP address. At first, I accidentally told it to change the IP address outside of the network's IP range and netmask. It kindly warned me that I was being stupid, so I clicked Cancel. And it kindly went ahead and applied the changes anyway. The second half of the bug was when I got in for another go, and discovered that it wasn't possible to set them at the same time; The interface lost the configuration as soon as you switched to the other property page; You had to click Apply in order to save the changes. At which point you were hosed. It's this second half of the bug that may well have caused me to discover the first half.

The third bug was the most nefarious, and cost me several hours and a second call to Comcast tech support. When you click Apply? It applies changes to its configurations, and implies it's going to reboot, shows a pretty "progress meter", and so on. When you click Reboot? Well, it tells you it's going to reboot, has the browser show a pretty "progress meter" of the percentage of the bootup, and then lets you go on and do your thing. Or try to; The router never really reboots, and you can see that by watching the lights on its front. That, in turn, helped confuse Comcast's modem[3], which in turn made things difficult for me.

[1] I add this for a bit of perspective. There's no configuration option for controlling the radio power output, but for about twenty minutes[2], I had a completely open wireless network visible to much of the suburb.

[2] It took me that long to make the connection between "I'm not able to attach to my network" and "wait...default...that's ME?!" I assumed I'd screwed up a setting on my laptop, or that it was looking for the old router's MAC address, or something along those lines.

[3] Whose firmware is pretty shoddy in itself. There's a reason they tell you to power off the router and modem, bring up the modem, and then bring up the router. I used to think Comcast was logging MAC addresses and hostnames, but that was apparently based on a coincidence of "try this theory and see if it works" and "it works!" combined with poor process control. At this point, I think the modem simply drops any DHCP packets once there's a lease in place, until the lease expires. But I'm definitely not interested in poking it to test this.

The Conclusion

I'm not going to make any conclusions. I've drawn up all my observations and the like above. Look them over and figure it out if it works for you. For my purposes, it Works For Now. But I'm back in the position where I don't even want to breath on the configuration for fear of spending horrendous amounts of time getting the connection to the Internet working again.

In my opinion, the WL-330gE was a much more solid product. I used one as my home router for over a year and a half, and liked it enough to buy a second. I won't be buying a second WL-520gE. If this guy dies, I'll probably buy a wired-only router, and use one of the WL-330gEs as an access point to the wired network.

Caveat Emptor.

But what about (custom firmware)

I don't feel like messing with it. Might solve problems, but I really don't feel like messing with it. There's even a firmware community built around the WL-500 series. But I really don't feel like messing with it.

Finally got around to using sshfs

So I finally got around to using sshfs.

It's a heck of a lot nicer to use than sftp.

For giggles, mount a system's log directory, and run tail -f on syslog or some such.

Tell me the Why and Why Not

I often have questions, and they're often very detailed.



I don't just want the answer to the question.

I don't just want the answer to the question you think I should be asking.



I want the answer to the question, the answer to the question you think I should be asking, why you think I should be looking for your answer, and why you think I shouldn't be looking for the answer to the question I asked.



You see, I realize I don't know everything, and I know I will never know the best way to do things. However, I am perfectly willing to read any and every piece of reference material I can get my hands on if it will lead me to understand the why and the why not of the choices that I need to make. With the understanding of the underlying reasons, I can apply that understanding inductively to other problems with superficially similar structure. I can learn more not just about how the thing it self is alike and different from something else, but I can learn more about how its structure and basis for existence is alike and different from that of something else.



Understanding where things stand is one thing, understanding why things stand is another, but taking those and getting things closer to where they can be is everything.

Anyone notice...

...what ESR has been up to lately?

Getting Started

This book is the book that first got me interested in technology in general, an electronics specifically. My interest in computers and programming grew largely from an early recognition that they were large and sophisticated implementations of the concepts in that book.

And a tip of the hat to my grandfather who bought it for me all those years ago. And bought me a breadboard. And lots of battery holders, LEDs, resisters, little DC motors, etc.

Get your kid started on a hobby early on; It could grow into a passion that breaks down the boundaries of work and play.

802.11 b/g channel wavelengths

(Writing this so that I can refer back to it, rather than keep having to look it up. Channels are 1-14, with the first row being channel 1. Consider these rough numbers; I didn't pay much attention to sigfigs, aside from using nine digits of c and four of the channel frequency as documented here. I also didn't use a bignum library, so IEEE754 floating point errors might be an issue; Depends on Perl's default floating-point implementation.)

full wave,half wave,quarter wave (centimeters)
12.429206384743,6.21460319237148,3.10730159618574
12.4034943318163,6.20174716590815,3.10087358295408
12.3778884393064,6.18894421965318,3.09447210982659
12.3523880510919,6.17619402554594,3.08809701277297
12.3269925164474,6.16349625822368,3.08174812911184
12.3017011899877,6.15085059499384,3.07542529749692
12.2765134316134,6.13825671580672,3.06912835790336
12.2514286064569,6.12571430322844,3.06285715161422
12.2264460848287,6.11322304241436,3.05661152120718
12.2015652421652,6.10078262108262,3.05039131054131
12.1767854589764,6.08839272948822,3.04419636474411
12.1521061207945,6.07605306039724,3.03802653019862
12.127526618123,6.06376330906149,3.03188165453074
12.0689395330113,6.03446976650564,3.01723488325282

A Twist on the standard PC repair shop

It occurs to me that there's one thing a brick-and-morter PC repair shop could do to set itself apart from others, get more customers, and increase customer satisfaction even in the face of utterly unrepairable systems.

If the system is deemed to be too expensive to fix, have a firing range on site where the customer can "take it out back and put it out of their misery." (With all appropriate and necessary supervision, of course...)

I'm annoyed at SourceForge.

I've been trying for several days now to get ImplSearchBot accepted as a SourceForge project. Both times, it's been rejected for revision as "We're sorry, but due to the thousands of project requests we get per day, we don't have time to look at all the links." (Though I didn't get the reply to the first attempt until Monday, when I submitted it on Saturday.)

I was mildly annoyed the first time, as the links I'd included pointed to the bot's existing page which contained its description of behavior and its source code, and the description wasn't that long. Still, I understood where they were coming from, and replaced all the URLs in my request with several paragraphs detailing the behavior, theory and benefit behind how ImplSearchBot works. Then I resubmitted.

The next day, Tuesday, I got another rejection notice, for the same reason. Scratching my head, I looked over the request I'd sent, thinking I'd forgotten to delete the URLs in addition to the technical description I'd provided. No dice; I'd correctly removed the URLs.

So I scratched my head some more, trying to figure out why they were complaining about links. Then I saw it. I'd included as part of its description "I created ImplSearchBot as part of administering rosettacode.org" -- I often refer to Rosetta Code as rosettacode.org, for the simple reason that the phrase "Rosetta Code" sometimes gets confused with either spoken/written language software or the software used to run old Mac software on newer Macs. Someone or something saw the term "rosettacode.org" and decided that they would have to click on it before making a host/no-host decision, and clicked the "reject because of URLs" quick-reject button.

I'll try one more time, removing mention of Rosetta Code's domain name. If it gets rejected again, I'll find someone else to host it. Google Code has a nice system going, but I know a fair number of folks who don't want to touch it because they don't like the privacy implications of having that much mineable data under the control of a single company. SourceForge was a close second choice because they've got a long and established history hosting Open Source projects, and I've participated and run projects there before. I'm not crazy about GitHub, partially because I haven't yet learned Git, partially because they're a new player, and partially because their free accounts are limited in size.

The Web is shrinking.

From 2007 to 2008, the top resolution of visitors to Rosetta Code was 1280x1024 at 27% of visits. Then, in the 2008 to 2009 range, 1280x1024, while stop the top resolution, reduced in share to 23% of visitors.

Meanwhile, 1280x800, in third for both years, rose from 11% in the 2007-2008 range to 17% in the 2008-2009 range.

(For the curious, 2nd place is held for both time periods by 1024x768, which fell from 24% of visits to 19% of visits.)

So while the width of the average viewer's browser window is staying roughly the same, the height of the browser window is shrinking.

What this means for web developers, in a practical sense, is that they have to shrink any header bar they have on their website in order to increase content exposure to viewers in that crucial first few seconds; The more uninteresting branding crap a viewer sees, the less likely he is to stick around.

Another possible approach to deal with the shrinking vertical screen space is to use Javascript to immediately center the top of the browser window on the first content div. Conveniently, if any given div has an id in Firefox, you can tread that ID the way you might have treated a named anchor in ancient versions of HTML. So if you give your content div and id of "content", having Javascript send your user to (PAGE)#content will level the top of your content with the top of their browser's viewable area--cut the fat, give them the meat. (Direct links work, too, but search engines aren't likely to point a user to a named div.)

The one tricky part is to make sure that you don't replace any anchor they might already have been intending to navigate to.

So I'm running Gentoo.

I was going to go with Arch, but when it came down to it, a guy I'm constantly running into on IRC offered to walk me through the Gentoo side of things. Which is essentially how I wound up getting started in Linux ten years ago, albeit with Debian.*

And "running" might be a strong word. I'm still getting things set up. But, honestly, it's not significantly harder than jumping head-first into a GUI-less Debian install was ten years ago. At least I'm not coming from a DOS/Win16/Win95 background this time...

No, I'm not saying Gentoo is less advanced that Debian was ten years ago. Well, maybe with some of the package management. But that's just a case of helping the bullet get to where you pointed the gun.

I'm sure that after I've run this for six or twelve months, I'll be so accustomed to it that it won't be an issue for me. And I can give it try for at least that long...

* Technically, I started with Red Hat 5.4, but since everyone I knew IRL ran Debian, I switched so that I could have some human support.