It is worth noting that in Ephesians 4 Paul focuses on unity within the body before he focuses on our relationship with those outside the body. Ephesians 4:1 serves as the introduction to the rest of the letter. Everything Paul says after this statement flows out of “living lives worthy of the calling by which we have been called.” The call of Ephesians 4:1 is to live in such a way that you accurately represent what God has done in your life. What he has done in principle we are to live in practice. 4:2-6:20 is what that statement looks like in practice.
In 4:2-16 Paul focuses on walking in unity. In 4:17-24 he focuses on walking differently than we once walked (or even in the present to walk differently than those outside the body). I do not think it is an accident that Paul focuses on relationships within before he focuses on our relationship with the outside world.
In other words, if you look different from the world but you aren’t in unity with your brothers and sisters in Christ then you aren’t really walking in a manner worthy of the calling. Jesus died to redeem our relationship with him but also to redeem our relationship with one another. Relationships and unity within the body is not an option.