YOU know you intentions, and they are honorable. But others don't. Now, if you were retired or pastoring in city 200 miles away, where there was virtually no danger of you attempting to poach members, it might not be a thing. But with you an active minister, perhaps close enough to persuade someone to make the drive, AND WITH THE INSECURITY THAT SOME PASTORS FEEL...and with the bad guys who HAVE abused other pastors by indeed trying to poach...It's no wonder that pastors can feel threatened. IDEALLY, with you in such a situation, it would be thoughtful of you, I think, to give the pastor a head's up, at least. Or ask to join with him in visitation, etc. You might make a new friend, as well as avoid later drama.