I've experienced the same thing and haven't been able to figure it out.

My circumstances are quite astonishingly similar as well...no black hat techniques and only manual search engine submissions and submissions to directories with categories of relevancy.

It got me....and I would love to hear the theories on it.

turtlwaX