Note that "colorface" is his term for the collective practice of "blackface," "brownface," "yellowface," and "redface." In other words, of whites playing minorities on the stage and screen.
Casting & Race Part 2: Defacing Color
By J. Chang
One of the significant problems of colorface at the time, beyond just keeping actors of color out of work, and being a tool of widespread proliferation of racism, was that, because of racism, it also impeded the actor's craft. Due to the segregated society and the limited meaningful interaction of people between races, people of color were likely mysterious to the white actors, and believing what racism would be telling these actors, they consequently restrained themselves from actually performing anything more than a series of stereotypes. In that sense, people of color watching these films would immediately be able to point out that, "that black person is nothing like an actual black person!" (using the vernacular of the time, of course). Unfortunately, also because of this racism, I'm pretty certain that the vast majority of the audience (likely white), also would not be fazed by these ridiculous portrayals of people of color.
Finally, I think that, for the large part, the mainstream audience has largely bought the Mighty Whitey myth. Part of that also includes this concept that white equals neutral, as opposed to a distinct race, and can consequently fill any role. Which is why I think the public response to characters getting actors of the wrong race cast can often be so minimal. Well, that and the cynic in me screaming that the mainstream audience (as well as the majority of people) tend towards apathy when it comes to more "invisible" issues like systemic racism that don't obviously impact their daily lives in a tangible way.