This is reproducable on the BeagleBone (white) with WL1835 cape which was used to verify WiLink8 functionality. That is running the Angstrom distribution.
From the BeagleBone:
Linux am335x-evm 3.12.10-ti2013.12.01 #22 Thu May 8 14:33:47 IDT 2014 armv7l GNU/Linux
wlcore: wl18xx HW: 183x or 180x, PG 2.2 (ROM 0x11)
wlcore: loaded
wlcore: driver version: ti2013.12.01-5-g0d378db
wlcore: compilation time: Thu May 8 14:39:03 2014
wlcore: PHY firmware version: Rev 8.2.0.0.195
wlcore: firmware booted (Rev 8.8.0.0.13)
wlan0 Link encap:Ethernet HWaddr 34:B1:F7:E3:04:8D
inet addr:172.16.10.108 Bcast:0.0.0.0 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:27691 errors:0 dropped:0 overruns:0 frame:0
TX packets:16393 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:15271650 (14.5 MiB) TX bytes:4613878 (4.3 MiB)
wlan1 Link encap:Ethernet HWaddr 34:B1:F7:E3:04:8E
inet addr:172.30.1.1 Bcast:172.30.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:6549 errors:0 dropped:7 overruns:0 frame:0
TX packets:4388 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:478837 (467.6 KiB) TX bytes:787831 (769.3 KiB)
Driver versions from our device:
wlcore: wl18xx HW: 183x or 180x, PG 2.2 (ROM 0x11)
wlcore: loaded
wlcore: driver version: ol_r8.a9.14
wlcore: PHY firmware version: Rev 8.2.0.0.224
wlcore: firmware booted (Rev 8.9.0.0.31)
mDNS traffic over AP mode (hostapd) works fine with a USB stick which we used on our product prior to integrating the WiLink device. I have not attempted to wedge the drivers for that into the BeagleBone, as we are confident that given multicast traffic is being treated the same on the Beaglebone using the WiLink8 as it does in our own environment, it appears to be in the driver.
iptables has been cleared - there are no iptables rules in place which would interfere with any traffic.
To test on the BeagleBone, I installed the avahi-daemon using opkg and set up a simple configuration file for it (mimic'ing the config used on our device, which has been functioning for some time, excepting this WiLink8 AP issue). In STA connections, it works fine, and when avahi-daemon first starts up (including when it is killed and re-run), the data is multicast on the AP interface (wlan1), implying Tx of multicast is functional. No mDNS traffic is seen on the wlan1 interface while in AP mode from any of the clients.
This behaviour is not seen when the same hostapd and avahi components and configurations are used with a USB WiFi device.
If I can provide any additional information, including tcpdump captures, let me know. This is holding up a product release because the mDNS functionality is significant to device discovery.
Note that I have a tcpdump capture from a workstation of a multicast from the Beagle when the avahi initializes, so I know it is Tx'ing the multicast:
Ethernet II, Src: 34:b1:f7:e3:04:8e (34:b1:f7:e3:04:8e), Dst: IPv4mcast_00:00:fb (01:00:5e:00:00:fb)
Destination: IPv4mcast_00:00:fb (01:00:5e:00:00:fb)
Address: IPv4mcast_00:00:fb (01:00:5e:00:00:fb)