John Mackay is very helpful in his commentary on Jeremiah (volume 2) in his attempt to explain Jeremiah's prophecy of the new covenant (31:31-34).
Mackay's concern is to be true to Jeremiah's message. This requires recognising that Jeremiah spoke a message to a particular group of people in a particular setting (even though chapter 31 is one of the more debated chapters in Jeremiah when it comes to working out what that setting is). But (no less important) it also recognises that Jeremiah's message contributed to the final canon of Scripture and so has something far wider to say. The challenge is to hear the message as a single message that has both a historical context and a canonical one.
Enough preamble. Without further ado, here is Mackay's paragraph:
“In considering these verses in their total scriptural context we must strive to do justice to two aspects of the message. In the first instance the new covenant spoke to the people of Jeremiah's day, on the verge of the Exile, of a time of renewal beyond that Exile. But equally, though the historical reality of the return from Babylon constituted an astounding fulfilment of God's promise to restore, it did not exhaust what is here promised and foretold (Heb. 8:8-13;9:15-22; 10:16-17). There is no doubt that the ultimate realisation of this prophecy is found not in the earthly reality of the New Testament church but in its glorified heavenly consummation. At present, however, there has been divinely brought into existence in the Christian church a greater fulfilment of this promise. Just as those believing Jews who entered into the covenant newly set before them by God were enabled to experience in measure the wonder of his gracious provision here clearly detailed, so to an even greater extent those who are covered by the blood of the new covenant inaugurated by Jesus Christ enter into that same provision, which will be completely consummated at the Lord's return. This is not to impose on the Old Testament text an alien message, but to see in the successive fulfilments of the prophecy that which Scripture assures us is there. In Jeremiah's day the future prospects of the people of God were presented in terms of the national destiny of the people who were then the embodiment of the church of God. With the realisation of the universal aspect of the covenant promise (Gen. 12:3; Acts 3:25) the way Jeremiah and his contemporaries understood these promises is not negated or reversed, but broadened and internationalised. The new covenant does not exclude Israel after the flesh, but includes it on the same basis as it does those from all nations who receive the mercy of God and have faith in Jesus Christ.” (Pages 238-239)