Nick please review "edit" sample of data table locally. (not at GAE)
when you will click edit and try to enter letters in numerical fields - nothing will apears in rich:messages. So I added a4j:outputPanel and messages are appears there fine.
Note: on the page with single form and out of any popups rich:message works fine. E.g. - CSV sample.