Just a short post to help others out should they be relying on Apple OS X Server for DNS name resolution in their network. If you happen to also be running IPv6 in your network (and at this point WHY wouldn't you be... seriously!) then you will notice that the OS X server won't respond to DNS queries on any of it's IPv6 interface addresses.
Here's what a packet trace looks like. Notice the OS X server responding to the client DNS query over IPv6 with a "port unreachable" error:
Apple OS X Server DNS Resolution Fails on IPv6 Interfaces
Once this occurs, most clients will attempt name resolution over IPv6 to additional configured IPv6 DNS servers. If all of those servers are Apple servers, then all name resolution transported over IPv6 will subsequently fail as well. Should that happen, then the client falls back to name resolution over IPv4 (which you see in the packet trace with frame numbers 3-6). The OS X server will actually provide both A (IPv4) and AAAA (IPv6) records to the client when queried over an IPv4 interface.
Note: remember that is NEVER a good idea to configure a primary internal DNS server and public backup DNS servers (such as Google, OpenDNS, or your ISP). In this scenario, when your client attempts to resolve internal domain names with your OS X server using IPv6 and fails it will fallback to the public DNS servers using IPv6 which will respond that "no such name" exists. At this point, the client has received a DNS response and will NOT fallback to name resolution over IPv4 to your OS X server. Internal name resolution will break for both A and AAAA records. Not good! Instead, configure your internal DNS server with Forwarders to the public DNS servers.
But, we really should have our DNS servers responding to queries on IPv6 interfaces to provide efficient name resolution, clean up our network traffic, and provide native end-to-end IPv6 services without relying on IPv4 sandwiched in the middle.
Taking a quick gander at the open sockets on the OS X server reveals the issue: it isn't listening for DNS queries on any of it's IPv6 interfaces.
Apple OS X Server Only Listens for DNS Queries on IPv4 Interfaces by Default
We can fix this by simply stopping the DNS service, manually editing the BIND named.conf configuration file, then starting the DNS service again.
First, stop the DNS service through Server App, or through the Terminal by entering this command:
sudo serveradmin stop dns
Next, you'll need to edit the BIND named.conf file through the Terminal:
sudo vi /etc/named.conf
Update: in Yosemite this file was relocated to /Library/Server/named/named.conf
Press 'i' to insert text into the file. Navigate to the "options" section and insert the following anywhere in the option list:
Press 'Esc' to stop editing the file, then type ':wq' to write the changes to the file and quit the VI editor. The option section should look similar to this when finished:
BIND named.conf Support for Queries on IPv6 Interfaces
Finally, start the DNS service again through the Server App, or through the Terminal by entering this command:
sudo serveradmin start dns
Name resolution on IPv6 interfaces should now work!
We can see that the OS X server is listing for DNS queries on IPv6 interfaces:
Apple OS X Server Listening for DNS Queries on IPv6 Interfaces
And a quick test by a client shows that the server is indeed responding properly now:
Apple OS X Server DNS Resolution Now Works on IPv6 Interfaces
As a final note, you can continue to create, modify, and delete name records with the Server App and the BIND configuration file will retain the option to listen on any IPv6 interfaces. So you won't be relegated to manual DNS entry updates through the command line in order to retain full IPv6 name resolution.
Why Apple hasn't enabled DNS resolution on IPv6 interfaces by default is beyond me. But at least you now can fully support your IPv6 network deployment with a DNS service that actually works :)