Commenting on James 4:4 Ray Ortlund notes:
Spiritual adultery consists in the lingering wish to retain the world’s favor even as one wishes to enjoy the benefits of redemption. (Ortlund, God’s Unfaithful Wife, 143).
We can’t have our hearts in both worlds. Matthew 6:24 tells us this much about money (but it’s really about idolatry isn’t it?). “No man can serve two masters…”.