New Deadwood snapshot: Resolve speeds comparable to Unbound

It looks like I have solved the issues people have reported with Deadwood being sluggish. The two bugs I fixed were one where Deadwood would delay timeout_seconds (normally 2 seconds) if there was a problem getting an upstream IP from its cache, and another where Deadwood would delay timeout_seconds if there were upper case letters in the name of a DNS reply.

I just ran Juergen’s test with Deadwood and Unbound. Deadwood is able to resolve the list of domain names on my network in 2:42 (two minutes forty-two seconds) and 2:54 (the 2:42 figure was done starting Deadwood with no cache); Unbound needed 2:57 to resolve the same list. In addition, namebench shows Deadwood doing a lot better than Unbound (far fewer lost queries and better average response time to the queries Deadwood does resolve) on my machine.

Steve Gibson’s DNS benchmark tool is buggy and incorrectly reports that Deadwood drops most DNS packets; I have set up Deadwood to report every reply it sends as well as every incoming query, and have verified that Deadwood replies to most, if not all, of its incoming queries when Gibson’s DNS benchmark is run against it.

In addition, I have revised Deadwood’s internal hash algorithm to be more secure: The algorithm, in addition to using a multiplication constant that randomly changes every time Deadwood is compiled in CentOS Linux or a new Deadwood release is made (which Deadwood has always done), now uses an additive constant that is different every time Deadwood is started. Deadwood is hard-coded to use /dev/urandom (secret.txt in Windows systems) for entropy to generate this additive constant from (we also get a bit of entropy from the timestamp); if this file is not found, Deadwood will fallback to using a hard-coded additive constant for its core hash.

I would like to thank Juergen Daubert, Wayne, and Marlon for their bug reports, feedback, and suggestions getting programs to help debug Deadwood’s performance.

The snapshot can be downloaded here:

http://maradns.org/deadwood/snap/deadwood-H-20100908-3.tar.bz2

Windows users can use this snapshot that has the improved performance (but not the improvements to Deadwood’s internal hashing function):

http://maradns.org/deadwood/snap/Deadwood-H-20100908-win32.zip

Please report any bugs to the mailing list [1].

- Sam

[1] Crash reports are useless unless they are accompanied by a stack trace or a recipe for reproducing the crash; Valgrind errors are invalid unless Deadwood is compiled with -DVALGRIND_NOERRORS

Note: Yeah, I’m still looking for work. Send me an email if you have a job for me.

Update: Anyone who downloaded deadwood-H-20100908-3 should download deadwood-H-20100908-4; this update is a hotfix that fixes dictionary variables which I broke in deadwood-H-20100908-3. Note also that dictionary variables work with the most recent Windows binary; it was the improved hash compression security which temporarily broke dictionary variables.

Yes, I do perform regressions to catch things like this, but only for numbered release (e.g. I’ll do the regressions before releasing Deadwood 2.9.07) and not for snapshot releases.

New Deadwood snapshot: Queries nearly three times faster

There has been some discussion today about the fact Deadwood, while very good at resolving names in general, is a good deal slower than other DNS servers.

Looking at the code, I already found the #1 offender that was responsible for about two thirds of the slowdown, and have patched this bug in today’s Deadwood snapshot. Anyone who feels Deadwood is a bit sluggish is encouraged to download and use today’s snapshot.

In addition, one issue that I have been thinking about for a while is that the hash compression function is not quite random enough. So, I’ve revised the hash compression core to add a random 16-bit number for each iteration of the hash compression. The number is randomly generated with Deadwood’s secure PRNG (Read: RadioGatún[32]) and is different every time Deadwood is started.

Deadwood snapshots can be looked at here:

http://maradns.org/deadwood/snap/

New Deadwood snapshot: MX record lookups disabled by default

Over the years, people have reported issues using MaraDNS as part of a mail hub, processing MX queries. This is a manner of using MaraDNS which I plain simply do not perform, and therefore have not adequately tested. The fact that releases of Deadwood with broken MX lookup support have been out for over a year, with no one letting me know about this bug on the MaraDNS mailing list (or me becoming aware of it myself with my own “dogfood” testing; I only became aware of it when I decided to make a MX lookup with Deadwood on a lark) indicate that this is not something widely used, much less tested.

My general feeling is that, if an average client machine is making a MX query, most likely it has been taken over by a spambot sending out spam. I have been using Microsoft Outlook and other MUAs (mail user agents) with Deadwood for over a year while its MX processing was broken and never saw a problem; a MUA does not perform MX lookups to send mail, but instead connects to the SMTP server, which does the actual MX lookups.

That in mind, in today’s snapshot of Deadwood, I have explicitly disabled MX lookups; if an MX lookup is made, the query is rejected and the IP is logged as being a possible spam zombie. There is a parameter which I have documented that will enable MX lookups again.

The fact of the matter is this: Using Deadwood with a mail hub making a large number of MX requests is completely untested. If one wants to use Deadwood in this manner, please subscribe to the MaraDNS mailing list and express the intention to help beta-test Deadwood for this use case which I do not have.

In the meantime, by disabling and logging MX lookups in Deadwood, we can make networks a little more secure and resistant to spam zombies commandeering machines on our networks.

New Deadwood snapshot: Non-A records now resolve

I guess not that many people are testing Deadwood.

Let me be honest about the type of testing I’m doing with Deadwood: I am testing Deadwood’s as a DNS server used for casual web surfing on today’s internet. That means that most of the queries done are A queries; I don’t do much testing with AAAA (IPv6) queries because, quite bluntly, my ISP doesn’t offer IPv6 service and I can think of only one site on the Internet with an AAAA record: ipv6.google.com.

Likewise, I haven’t done any testing with MX records. I haven’t touched MX records at all because I’m not using Deadwood with a mail hub. So, it came as an unpleasant shock to me when I discovered last night that Deadwood hasn’t been properly resolving MX records for over a year.

In Deadwood 2.4.05 (released August 9, 2009), I added the ability to rotate resource records. Unknown to me, this feature broke Deadwood with any variable-length record, such as a MX record pair where the hostnames are of different lengths.

The advantage of using Deadwood’s robust string library with almost all of Deadwood’s DNS processing is that this bug did not result in any memory corruption or cause any other problems; the only issue has been that Deadwood would not resolve MX or other variable length records.

The idea that some freetards advocate that open source software is magically tested and that all bugs become shallow is nothing more than so much—how should I say this—mental masturbation. The author of Mailman, John Viega, points out how wrong this is in his excellent essay “The Myth of Open Source Security”.

Excuse me for the blunt wording, but I am a little frustrated right now: I have devoted a decade of my life to MaraDNS, and I haven’t even been able to get a job because of my hard work; while I have had a couple of interviews I would not have had because of MaraDNS, so far I have gotten no offers. Yes, those are interviews I would not had have if it weren’t for MaraDNS, and, yes, MaraDNS kept my skills from getting completely rusty during the ’00s when I was concentrating on learning Spanish and living in Mexico. But, looking back, I really wish I had spent more time learning C++ and object oriented programming and less time editing the Wikipedia or posting to /..

It looks like Deadwood isn’t getting much external testing; not one person saw Deadwood’s issue with MX records in the year Deadwood has had this bug. So, yeah, like everything else with MaraDNS, I’m responsible for just about all of the testing. This isn’t a complete loss; people have asked me in interviews about my testing methodology for MaraDNS.

I have just uploaded a snapshot which fixes the issue with rotating records like MX records:

http://maradns.org/deadwood/snap/

As a side benefit, since I have fixed this issue, TTL aging and RR rotation now work with ANY records.

Next: Set up a test for this issue, as well as DNS compression tests for all of the resource record types MaraDNS supports. If I get time, I will also set up tests to make sure Deadwood correctly handles things like SRV records with compression pointers, and that Deadwood doesn’t compress SRV records.

Deadwood 2.9.06 released

I have just released Deadwood 2.9.06. This is Deadwood 2.9.05 with its handling of empty DNS packets tweaked to handle the corner case of a DNS server giving out an empty packet better. It’s been two weeks since the last Deadwood testing release and it looks like I’m getting down to the nitty-gritty corner cases that only affect a tiny percentage of DNS resolutions (properly configured DNS servers do not give out empty DNS packets).

My plan is to release Deadwood 3.0.01 in a little over two weeks.

Meanwhile, Deadwood can be downloaded here:

http://maradns.org/deadwood/testing/

New Deadwood snapshot: Better heuristics for empty DNS packets

When a remote server sends us an empty DNS packet, what does that really mean? Well, it usually means that the hostname does not always exist. But, not always.

Let’s take the hostname ibnlive.in.com. It has three DNS servers: 123.108.40.31, 123.108.40.32, and 123.108.40.81. Two of the servers (123.108.40.32 and 123.108.40.81) do the right thing: They redirect the name to Akamai’s horrendous mess of DNS resolution. However, 123.108.40.31 is broken and gives us a blank DNS packet with a SERVER FAIL message.

The DNS administrators for in.com aren’t trying to tell us that ibnlive.in.com doesn’t resolve. No, they’re trying to tell us that their DNS administrator is asleep and misconfigured one of their three DNS servers.

So, I’ve tweaked the heuristics of Deadwood to treat a DNS packet with a “SERVER FAIL” RCODE as being one where the DNS server is asleep at the wheel, so instead of treating the packet like “this name does not exist” (like what we normally do with a blank DNS reply), we ignore the packet and bug the next DNS server on the list. The code isn’t perfect: We wait timeout_seconds before contacting the next server, but it’s good enough to have domains mismanaged like this correctly resolve.

I should note that the bugs I’m finding in Deadwood right now are not bugs caused by Deadwood not being RFC-compliant—the last bug I fixed like that was on August 9. No, at this point, I’m fine-tuning Deadwood to more or less do the right thing with mismanaged domains and broken DNS servers.

As always, the snapshot can be looked at here:

http://maradns.org/deadwood/snap/

New Deadwood snapshot: Better handling of empty packets

I have uploaded a new snapshot of Deadwood which improves Deadwood’s handling of empty packets. A number of broken DNS configurations and servers out on the internet, such as the DNS servers answering the IPv6 (AAAA) query for wwwchase.gslb.bankone.com, send out an empty DNS packet instead of a correctly made SOA-in-NS-section DNS reply when a host name can not be found.

Another situation where a DNS server gives out a reply like this is when someone stops paying for their domain, so the ISP suspends their account by removing the zone for their domain. However, this information has not been changed upstream. As a result, when we contact the DNS servers for a given zone, the DNS servers give us a blank QUERY REFUSED reply.

Since these do happen, Deadwood now pretends that a blank DNS reply is a low-TTL “SOA-in-NS-section” reply. This should speed up some queries, especially on IPv6-capable computers that perform AAAA queries before doing A queries.

It can be looked at here:

http://maradns.org/deadwood/snap/

I should point out that I wrote the code to do this in a way to minimize the amount of code added to Deadwood and to maximize code reuse by having the relevant code in DwRecurse.c call the already created make_synth_not_there_answer() function made a little-more general purpose.