What you have described is a great struggle. You have a duty to protect your young children from unwholesome influences. Yet you also have a duty to be charitable to your neighbor. I would like to address two sides of this issue.
On the one hand, you are fortunate to have so much in common with another home schooling family. Many families pray for such a situation. Based upon my own experience, it is rare to have a close fit between home schoolers because each family is so unique. You are meeting your children's needs, and other families are doing the same for their children to the best of their ability. That means that there will naturally be different levels of expectations and standards of behavior.
On the other hand, this friendship is creating negative effects. Unless you have determined that continuing this friendship in any degree whatsoever is harmful to your children, you can say "yes" but with reservations and some guidelines. You do not have to have a close friendship with this family in the same way that you would not necessarily have to have a close friendship with every work colleague or every family relation. Sometimes we keep a safe distance from each other for various reasons, but we may still establish a working (civil) relationship with each other. So you have a third possibility beyond having no relationship and a close relationship.
Regarding the bad habits your children bring home, these episodes are teachable moments. Children are going to imitate behaviors, both good and bad. You are in a wonderful position to counsel and instruct your children as to the correct behavior. Habits take a while to take hold of a child. Because a child suddenly tries out a new and different behavior doesn't mean that it is going to become an ingrained habit. Your children will be picking up behaviors from a variety of sources, including relatives or parishioners or acquaintances that you simply cannot avoid. I recommend choosing a method to deal with these behaviors and sticking with it. This method will be invaluable as your children grow and widen their social contacts.
I would like to share how my husband and I dealt with behavioral situations that were harmful and yet actively encouraged by relatives. Two of our relatives who lived in town encouraged our youngsters to jump on the furniture and to run about in restaurants. We felt awful for disciplining our children because they had been told to misbehave, and then we were scolded for disciplining them by these relatives. This was quite a struggle for several years, but we didn't give up. We consistently reminded our children of the proper behavior and asked our relatives to cease their encouragement. When they found that we were consistent no matter the time or location, they finally gave up. Being in public or celebrating the child's birthday was no excuse. I remember getting so frustrated at one point that I threatened to stand one of these adult relatives in a corner if he did not stop his incitement of my children. Yes, there was a long moment of stunned silence and some furious sulking on the part of the relatives.
You may wish to bring up your expectations and standards of behavior with the other homeschooling family when the situation arises. There are times when we must establish our limits immediately. For example, if you hear cursing, then you should stop your conversation with the other parent and explain calmly that you do not allow your children to curse. Give her an opportunity to address the situation, but if she does not, then go ahead and speak with all the children. Tell them that their language is inappropriate, and you will have to leave if this behavior continues. You will have to be consistent and follow through with your standards in private and in public. This will be an excellent model for your children because they will have your example to follow with their own children in the future.
I understand your offering up these visits as a penance. I had the same penitential experience with visits to the two relatives I mentioned in my example. But please focus on the good coming from these visits as well. I had the tendency to discount any good, but I knew in my heart that these visits (and the resulting trials) were important for my children. However, you do not have to continue the friendship if you decide that more harm than good is resulting. Make that decision after looking at the facts and searching your heart. I hung onto a friendship with another home schooling family for too long because we did have much in common. In several ways, I thought this woman was a better Catholic than I was. There was much to admire about her. But the matter of physical safety kept coming up. At one point, I actually thought we might come to blows because I refused to allow her to leave her child alone in her car which was parked in my driveway on a hot summer's day. She wanted to compromise by leaving the car running with the air conditioner on, and I had to block her way out of my front door and tell her that I would never allow that. She grew angry, but I was not giving way even if it wasn't my child. This was against the law and against common sense, but she was used to doing this. I addressed this and other concerns with her and her husband and was brushed off. We finally ended the friendship abruptly but with a clear explanation of the cause.
So do not prolong a harmful friendship because of common interests. Your primary responsibility is for the spiritual, emotional, intellectual, and physical well-being of your own children, not the well-being or emotional comfort of another adult.
I know that this is a difficult situation and decision. I will be praying for you and your children.
May God grant you peace,