Message Board

http:BL Use/Development

DNS Server performance issue

Author: A.Sleeis (4 Dec 13 12:04pm)

Hi,

Around November 17 or so, I started getting reports that my forum website was being REALLY slow (but inconsistently so). Requests were taking 5, 10, sometimes 30-60 seconds to come back (each http request, so you can imagine what a full page takes).

It took me a while to figure out the cause, but narrowed it down to the RBL use via ModSecurity. At first, I thought it was my local hosting providers DNS servers having an issue, but now think otherwise. I setup a script to query the httpbl.org dns servers directly and time it. I just picked a single IP to lookup over an over. I had it sleep 1 second between queries to not flood anything... results came back with what looks a lot like the httpbl.org DNS servers were my issue.

It looks like periodically the DNS server is taking 5 or 10 (or sometimes 15) seconds to respond (not sure why such clear multiples of 5). Also, it seems that dns1 takes 3 times longer than dns2 (still tiny time diff, tho). dns3 was word than that (as test output below shows).

I ran the same script against my local hosting providers DNS lookups for other hostnames just to compare against "normal" DNS lookups locally. No such issues. The only periodic delay I would see from them was maybe 1 second. The 5 and 10 second delays is what I was seeing in my own web server response times.

As such, I had to pull out the DNS BL lookup for anything but POSTs. POSTs are still painful, but at least much less so than all traffic to the site.

Can something be done to look into this? It started very clearly on November 17th from what I can tell in my logs and from what my forum users were saying. Is anyone else seeing this issue? I would like to be able to apply the IP checks for more than POSTs again, if possible.

Details about my test to show the times, are below. I checked dns1, dns2, and dns3, with 50 requests each. The average time across all requests was 0.717 seconds.

Thanks,
-Alex

Simple script:
perl -e 'for($x=1; $x<4; $x++) { for (my $i=0; $i<50; $i++) { my $output = join("", `time dig \@ns$x.httpbl.org __MYKEY__.25.192.191.88.dnsbl.httpbl.org 2>&1`); my ($tm) = ($output=~/(\d+\:\d+\.\d+)elapsed/); my ($sec)=($tm=~/:(\d+\.\d+)/); $total_time+=$sec; $count++; print localtime() . "\tDNS query time: $tm\t\@dns$x\n"; sleep(2); }} my $avg_time=$total_time/$count; print "\nAverage time:\t$avg_time\nRequests:\t$count\n"; '

Output:
Wed Dec 4 11:50:08 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:50:10 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:50:12 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:50:14 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:50:16 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:50:18 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:50:20 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:50:27 2013 DNS query time: 0:05.06 @dns1
Wed Dec 4 11:50:29 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:50:31 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:50:33 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:50:35 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:50:37 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:50:40 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:50:42 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:50:44 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:50:46 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:50:48 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:50:50 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:50:52 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:50:54 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:50:56 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:50:58 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:51:00 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:51:02 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:51:04 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:51:06 2013 DNS query time: 0:00.07 @dns1
Wed Dec 4 11:51:08 2013 DNS query time: 0:00.07 @dns1
Wed Dec 4 11:51:11 2013 DNS query time: 0:00.07 @dns1
Wed Dec 4 11:51:13 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:51:15 2013 DNS query time: 0:00.07 @dns1
Wed Dec 4 11:51:17 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:51:19 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:51:21 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:51:23 2013 DNS query time: 0:00.07 @dns1
Wed Dec 4 11:51:35 2013 DNS query time: 0:10.07 @dns1
Wed Dec 4 11:51:42 2013 DNS query time: 0:05.06 @dns1
Wed Dec 4 11:51:49 2013 DNS query time: 0:05.06 @dns1
Wed Dec 4 11:51:51 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:51:53 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:51:55 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:52:03 2013 DNS query time: 0:05.07 @dns1
Wed Dec 4 11:52:05 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:52:07 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:52:19 2013 DNS query time: 0:10.07 @dns1
Wed Dec 4 11:52:21 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:52:23 2013 DNS query time: 0:00.07 @dns1
Wed Dec 4 11:52:25 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:52:27 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:52:29 2013 DNS query time: 0:00.06 @dns1
Wed Dec 4 11:52:31 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:52:33 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:52:35 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:52:42 2013 DNS query time: 0:05.02 @dns2
Wed Dec 4 11:52:44 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:52:51 2013 DNS query time: 0:05.02 @dns2
Wed Dec 4 11:52:53 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:52:55 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:52:57 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:52:59 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:53:01 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:53:03 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:53:05 2013 DNS query time: 0:00.03 @dns2
Wed Dec 4 11:53:07 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:53:19 2013 DNS query time: 0:10.02 @dns2
Wed Dec 4 11:53:22 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:53:24 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:53:26 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:53:33 2013 DNS query time: 0:05.02 @dns2
Wed Dec 4 11:53:35 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:53:37 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:53:39 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:53:41 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:53:43 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:53:45 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:53:47 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:53:49 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:53:51 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:53:53 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:53:55 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:53:57 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:53:59 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:54:01 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:54:03 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:54:05 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:54:07 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:54:09 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:54:11 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:54:13 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:54:15 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:54:22 2013 DNS query time: 0:05.02 @dns2
Wed Dec 4 11:54:24 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:54:26 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:54:28 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:54:30 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:54:32 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:54:34 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:54:36 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:54:38 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:54:40 2013 DNS query time: 0:00.02 @dns2
Wed Dec 4 11:54:43 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:54:45 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:54:47 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:54:54 2013 DNS query time: 0:05.17 @dns3
Wed Dec 4 11:54:56 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:54:58 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:55:01 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:55:03 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:55:05 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:55:12 2013 DNS query time: 0:05.17 @dns3
Wed Dec 4 11:55:14 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:55:17 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:55:19 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:55:21 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:55:23 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:55:25 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:55:27 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:55:30 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:55:37 2013 DNS query time: 0:05.17 @dns3
Wed Dec 4 11:55:39 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:55:41 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:55:43 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:55:45 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:55:48 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:55:50 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:55:52 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:55:54 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:55:56 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:55:59 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:56:01 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:56:03 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:56:05 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:56:07 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:56:09 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:56:12 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:56:14 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:56:16 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:56:18 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:56:20 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:56:22 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:56:25 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:56:27 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:56:29 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:56:31 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:56:43 2013 DNS query time: 0:10.17 @dns3
Wed Dec 4 11:56:46 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:56:48 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:56:50 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:56:52 2013 DNS query time: 0:00.17 @dns3
Wed Dec 4 11:56:54 2013 DNS query time: 0:00.17 @dns3

Average time: 0.717333333333333
Requests: 150

Post Edited (4 Dec 13 12:11pm)

Re: DNS Server performance issue

Author: H.User1325 (4 Dec 13 4:14pm)

I would suggest that the delay is a host issue not a http:BL problem. You said
"I ran the same script against my local hosting providers DNS lookups for other hostnames just to compare against "normal" DNS lookups locally. No such issues."

The difference in times could be due to the number of links needed to find a DNS server that can respond to your request. Researching a problem I am having with http:BL I can across a long discussion on the possible delays due to the DNS lookup procedures. It was a discussion about the php gethostbyname() function, but would have the same underlying functionality no matter which language is being used.

To directly answer you question, no I have not noticed such a delay on any of my implementations, 2 forums and a website "leave a comment" form and 2 "join our emailing list" forms.

We do seem to use the BL differently. I use it for my information, not to block access. In the case of my forums, I let the comment spammers register and submit a post which must be reviewed before being displayed. I want the evidence to submit to stopformspam.com, then I block them.

Re: DNS Server performance issue

Author: A.Sleeis (10 Mar 14 4:43pm)

Thanks. Your response prompted me to try a few more tests to identify true root cause. It seems my hosting provider changed some of their DNS servers, but not my resolv.conf file (Linux). The failure in my test that masked this was that I was performing an explicit lookup against the httpbl.org DNS servers, but I was doing so via "hostname" (duh). So the added delay was when the local (provider) DNS servers were looking up the httpbl.org hostname.

On my forums, it would be too much work to have to review posts. And I don't have any modules to make it so the first few posts would need review for a new user. That might help, but I'm not about to write my own forum MOD. My forums are trafficked enough, though, with regular valid new users, that I think it would still be too much work to manually review for my moderators and I. I just hook it into the filtering with ModSecurity, and direct users to a page saying that if they think this is in error, to contact me. So far, only one "false positive" has contacted me, and users of my site are pretty vocal when having issues.

Anyway... thanks again for the response. It was the added push I needed to dig deeper and fix the issue.