Brablc.com

I needed to find activity for certain user in the postfix log and did not find anything useful, so here is my attempt: https://github.com/brablc/postfix-tools

Watch the gist for comments with examples. It creates nice stats when combined with pflogsumm.pl (see http://jimsun.linxnet.com/postfix_contrib.html).

In this case Group Office database name is groupofficecom and RoundCube is roundcube, adapt script if necessary. Script is written for MySQL. RoundCube stores contacts details as vCard, the script might be interesting even for this case. You may want to add the commented unique index, to be able to call the script multiple times without creating duplicates.

We host several thousands e-commerce entities at Shoptet.cz. Each on its own domain. A good half of them is using freemail as their main e-mail address, some are using our mailboxes and some use their own. We have to transmit emails generated by their customers and order processing for all of them and we offer basic newsletter solution too (along with advanced integration with MailChimp).

As anyone else we have been forced by Seznam.cz to introduce DKIM for mass emails. We have learned that Seznam.cz is going to build sender’s reputation on the combination of DKIM selector and domain – this means we must use different combination for every e-commerce entity. And we do not want to allow users to change this combination – if they behave like spammers, they should not have bad influence on other our users.

Solution

We use postfix and naturally we started with opendkim. The documentation is not the strongest part of the project, so after many hours we came with this setup:

/etc/opendkim.conf
...
SigningTable lua:/etc/opendkim/SigningTable.lua
KeyTable lua:/etc/opendkim/KeyTable.lua
...


SigningTable.lua was tough, giving us endless Segmentation fault until we discovered, we cannot return nil.

The global variable query passed to SigningTable contains content of the “From” header. This does not allow us to distinguish e-commerce entities. People could use their freemail address. Until recently there existed directive SenderHeaders Sender,From where different headers could have been used, but this had gone in opendkim 2.10.0 release. We have decided to put it back, but not that versatile – we just hardcoded support for header called: X-DKIM-Sender. Because of internal guts it must have @ in the content, so we pass information about our entity as selector=domain@dkim. You can see in the script below, that we will strip @dkim, and separate selector and domain using the = sign.

You can see we sign everything coming from this mail server with the same private key. Our public key is a TXT record on dkim.shoptet.cz. In order to create a unique selector for every entity we created wildcard CNAME: *._domainkey.news.shoptet.cz CNAME dkim.shoptet.cz. The final step is to modify the sending applications to include the X-DKIM-Sender header. If the user has verified their domain, we would pass this:

X-DKIM-Sender: shoptet=brablc.com@dkim

If they did not we will use this:

X-DKIM-Sender: myecommerce.cz=news.shoptet.cz

Resume

We have combined this with Postfix Sender Rewriting Scheme daemon (to deliver all bouncing messages and auto-replies to the real sender) and hardened SPF record for news.shoptet.cz.

SpamAssassin now shows in the first case DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU. In the later DKIM_VALID_AU is missing obviously.

This article was written to help others seeking help with opendkim, perhaps they will not have to read so much C code as I had to to figure this all out 😉

Customer usage of SMTP

Out customers send their email from their email clients using our SMTP server. In this case we have to use email address from their MAIL FROM SMTP command, otherwise they would be able to forge From: header. This requires changes in opendkim: configure --enable-sender_macro

/etc/opendkim.conf
...
SenderMacro {mail_addr}

Finally on this SMTP server we must use special /etc/opendkim/KeyTable.lua:

Almost 7 years since the last release, finally SiteBar 3.4 has been released. The source code has been moved from publicly accessible but privately hosted subversion repository to GitHub – https://github.com/brablc/sitebar .

Actually I do not expect any rush in getting contributions, but it is kind of nice to see, that something with a codebase from 2005 can still be useful for someone.

I will use this post and move here comments, that landed on other pages of this blog.

When importing files from family iPhones on MacBook Air, Google Picasa does not process rotation properly. I have to use iPhoto. However iPhoto does not export properly date of videos. In order to keep my archive on disk properly organized, I wrote myself two scripts.

I export images and videos from iPhoto using:

This will place them into subfolder on the disk. Now in each directory I run script iphoto-touch-mov-by-prev-image, which will touch the videos to have the same date/time as the closest image.

Afterwards I run all files through a script iphoto-copy-export-to-archive-dir which will put them into YEAR/YEAR-MONTH-DAY subfolder in my archive.

In order to process multiple subdirectories I use:

#!/bin/bash
for DIR in `find . -type d`; do
    iphoto-touch-mov-by-prev-image $DIR
    iphoto-copy-export-to-archive-dir /Volumes/Archive $DIR/*.*
done

Probably with firmware 4.0.2 or 4.0.5 I have lost possibility to run music station and QAirPlay and other applications which run on secure port 8081. I have tracked the problem down to the fact, that /etc/config/apache/extra/apache-ssl.conf is not included. Looking at /etc/init.d/Qthttpd.sh I have discovered, that it is because /sbin/getcfg QWEB Enable_SSL -u -d NULL returns 0 instead of expected 1. I have fixed the problem by running:

/sbin/setcfg QWEB Enable_SSL 1
/etc/init.d/Qthttpd.sh restart


Now apache listens on por 8081 again and I can run QNAP applications. About two hours of lost time by Googling a solution. Proper answer was missing. There have not been any nonstandard changes on my side, I just wanted to run everything on secure ports.

I have configured a something unrelated on my router and after restart it came with strange message:

Settings have been updated. Web page will now refresh.
Changes have been made to the IP address or port number. You will now be disconnected from RT-N66U.
To access the settings of RT-N66U, reconnect to the wireless network and use the updated IP address and port number.

What a wonderful message, it must have been created by some excellent creative mind!

However, this page would not go away. Deleting HTTP authentication would not help, deleting cookies either, certificates nop. Finally I remembered that there was some strange request to allow saving offline data. And here it is:

Just delete the data. It was my first touch with offline data and it did not work! Great experience, now I will never let any application to store its data.

UPDATE: Firefox has preference called “Tell me when a website asks to store data for offline use”. However it uses this only if application wants to store more than 52MB of data. Check your about:config and probably change following to be more restrictive:

offline-apps.allow_by_default;true
offline-apps.quota.warn;51200



echo "show databases" \
| mysql --skip-column-names \
| grep -v -P '^(performance|information)_schema$' \
| awk '{ print "mysqldump --events --routines --triggers " $1 " | bzip2 > /var/backups/" $1 ".sql.bz2" }' \
| sh


This is something every admin must need, but I have not found such script. So here you are.

#!/bin/bash

echo "\
10 aspmx.l.google.com
20 alt1.aspmx.l.google.com
20 alt2.aspmx.l.google.com
30 aspmx2.googlemail.com
30 aspmx3.googlemail.com
30 aspmx4.googlemail.com
30 aspmx5.googlemail.com\
" | awk '{print "/usr/local/psa/bin/dns -a '$1' -mx \"\" -mailexchanger " $1 " -priority " $2 }' | sh


I have a brand new machine with configuration stated above, moreover I’m using Nginx as proxy server to Apache. And it comes loaded with a lot of problems.

• Machine cannot be shutdown – with last message in the console:
  

  init: psa-monit main process ended, respawning

• After restart the machine reports following problems to console:
  

  Starting nginx: nginx: [emerg] bind() to [2001:8d8:8ae:e500::ae:4abd]:80 failed (99: Cannot assign requested address)[FAILED]

• httpd process gets loaded with strange settins – it does not see VirtualHosts and serves all requests from default host, although it has opened all virtual host log files

Root Cause

Is unclear to me, but probably it is caused by incorrect change of starting scripts from Plesk. The scripts moved between Centos 5 and Centos 6 from /etc/inittab to /etc/init/psa-monit.conf and /etc/init/psa-wdcollect.conf . And I believe the scripts are not correct. They execute too early and respawn does not correctly handle situation when stopping machine.

Premature execution seems to cause that Apache gets started on port 80, Nginx refuses start consequently. Restarting Nginx seems to fix the port binding (strange) but Apache still does not serve proper pages. As if the started instance did not have NameVirtualHost setup properly and is serving default VirtualHost.

Workaround

I’m not familiar with Centos init procedure good enough, and I believe the problem is inside Plesk code, not just in the init configuration. Changing the init script as shown below would solve all the problems. It will however disable automatic restart when the watchdog fails itself. But this seems acceptable for me – it is bettern than machine which cannot be restarted.

Yes I admit, the sleep is funny 😉