Wendell Berry, adultery, and epistemological modesty

Gearing up for a post on why I object to Wendell Berry has meant reading him—always a mixed bag. On one hand, he writes CSB-friendly sentences like this one from "The Problem of Tobacco":

Though it was often said, when I was a boy, that smoking would "stunt your growth," we did not know any smokers who had been stunted—unless, perhaps, they had been intending to be giants.

On the other hand, his charming stories always end in claims like this one:

A public issue, properly speaking, can only be an issue about which the public can confidently know. Because most sexual conduct is private, occurring only between two people, there are typically no witnesses. Apart from the possibility of a confession, the public can know about it only as a probably unjudgeable contest of stories. (In those rare instances when a sexual offense occurs before reliable witnesses, then, of course, it is a legitimate public issue.)

Leaving aside the disturbing parenthetical, Berry's understanding of sex scandal shaming seems backwards to me. Why not simply say "Adultery is a private matter," or, if you want to be more nuanced, "In order to figure out what kind of adulterer this guy is and how upset I should be about it, I would have to become familiar with aspects of his life (and his wife's and mistress's) that I would prefer remain protected by privacy?"

Grounding your objection to sex scandals on privacy rather than unknowability needs conditions and clarifications, of course — "The rules of privacy are different for public figures" is an obvious one. "It would be inappropriate to fire you for having an affair, but I'm still going to shame you in other ways" is a less obvious caveat. Still, these fuzzy rules seem to me better guides for determining one's reaction to someone else's sexual disgrace than epistemological modesty. If Berry were certain that a well-qualified Supreme Court nominee was slippin' around and he was the only one who knew about it, would he really blab? If a moral leader (i.e. preacher) were uncertainly suspected of being an adulterer, would Berry really rule the issue out of bounds?

Goldwater's response to the Walter Jenkins affair was "It was a sad time for Jenkins' wife and children, and I was not about to add to their private sorrow," not "I wasn't there so I couldn't say for sure." "The private realm" means something more than "Things that happen with fewer than three people in the room."