When configured as a forwarder, to allow stale answers, and an answer to a lookup contains several layers of cnames (3 in this case before the A record) if the query upstream ever returns a 0 TTL for the last record (the A record) then future queries (for some amount of time) will only return that last record. This causes responses that don't answer the queries and is interpreted by the query software differently depending on the software (NODATA or other interesting errors).

Relevent settings :-

stale-answer-enable: yes
stale-cache-enable: yes 
max-stale-ttl 300
stale-answer-client-timeout 0
stale-answer-ttl 30
stale-refresh-time 30

Interesting subpoint, when in this state it looks like BIND starts doing lookups on that actual A record at a predicable rate so its presumed that the 0 TTL lookup is somehow messing up the state on the above cached 3 cnames without allowing them to be requeried.

Workaround :-

min-cache-ttl: 5
stale-answer-client-timeout: 1800

----------------------------------------------------------

Steps to reproduce -

Build 2 DNS servers once acting as Master and the second acting as forwarder -

node-0 - Master

node-1 - forwarder (to node-0)

On master DNS in named.conf create a zone -

~~~

zone "redhat.com" IN {
        type master;
        file "redhat.com.zone";
        allow-update \{ none; };
};

~~~

Then create the zone file, initially the TTL value is set to 10

~~~

$TTL   10
@               IN SOA  redhat.com. root (
                                        42              ; serial (d. adams)
                                        3H              ; refresh
                                        15M             ; retry
                                        1W              ; expiry
                                        1D )            ; minimum
                IN NS           localhost
localhost       IN A            127.0.0.1
foo.redhat.com.             IN CNAME        bar.redhat.com.
bar.redhat.com.             IN CNAME        baz.redhat.com.
baz.redhat.com.             IN CNAME        station1.redhat.com.
station1.redhat.com.        IN A            192.168.0.101

~~~

Restart named service.

-------------------------

Next, on slave configure named as below -

In named.conf also add below parameters in global section -

~~~

stale-answer-enable: yes;
stale-cache-enable: yes;
max-stale-ttl 300;
stale-answer-client-timeout 0;
stale-answer-ttl 30;
stale-refresh-time 30;

~~~

Also configure a forwarder as below -

~~~

forwarders \{ node-0; };

~~~

Restart named.

---------------------------------------

To test, run below dig command from forwarder (node-0)

dig  @localhost foo.redhat.com +short

We will get below output -

bar.redhat.com.
baz.redhat.com.
station1.redhat.com.
192.168.0.101

Now, if we change TTL to 0 on master DNS and restart named there and again try to execute dig from caching/forwarding DNS we get below - (No cname records)

192.168.0.101

The workaround is to add/modify below parameters -

min-cache-ttl: 5
stale-answer-client-timeout: 1800