Augustine says that authorship of the New Testament books is determined in just the same way as authorship is determined for secular books of antiquity. Here is what he wrote on the subject from Contra Faustum (33:6).
You are so hardened in your errors against the testimonies of Scripture, that nothing can be made of you; for whenever anything is quoted against you, you have the boldness to say that it is written not by the apostle, but by some pretender under his name. The doctrine of demons which you preach is so opposed to Christian doctrine, that you could not continue, as professing Christians, to maintain it, unless you denied the truth of the apostolic writings. How can you thus do injury to your own souls? Where will you find any authority, if not in the Gospel and apostolic writings? How can we be sure of the authorship of any book, if we doubt the apostolic origin of those books which are attributed to the apostles by the Church which the apostles themselves founded, and which occupies so conspicuous a place in all lands, and if at the same time we acknowledge as the undoubted production of the apostles what is brought forward by heretics in opposition to the Church, whose authors, from whom they derive their name, lived long after the apostles? And do we not see in profane literature that there are well-known authors under whose names many things have been published after their time which have been rejected, either from inconsistency with their ascertained writings, or from their not having been known in the lifetime of the authors, so as to be banded down with the confirmatory statement of the authors themselves, or of their friends? To give a single example, were not some books published lately under the name of the distinguished physician Hippocrates, which were not received as authoritative by physicians? And this decision remained unaltered in spite of some similarity in style and matter: for, when compared to the genuine writings of Hippocrates, these books were found to be inferior; besides that they were not recognized as his at the time when his authorship of his genuine productions was ascertained. Those books, again, from a comparison with which the productions of questionable origin were rejected, are with certainty attributed to Hippocrates; and any one who denies their authorship is answered only by ridicule, simply because there is a succession of testimonies to the books from the time of Hippocrates to the present day, which makes it unreasonable either now or hereafter to have any doubt on the subject. How do we know the authorship of the works of Plato, Aristotle, Cicero, Varro, and other similar writers, but by the unbroken chain of evidence? So also with the numerous commentaries on the ecclesiastical books, which have no canonical authority, and yet show a desire of usefulness and a spirit of inquiry. How is the authorship ascertained in each case, except by the author’s having brought his work into public notice as much as possible in his own lifetime, and, by the transmission of the information from one to another in continuous order, the belief becoming more certain as it becomes more general, up to our own day; so that, when we are questioned as to the authorship of any book, we have no difficulty in answering? But why speak of old books? Take the books now before us: should anyone, after some years, deny that this book was written by me, or that Faustus’ was written by him, where is evidence for the fact to be found but in the information possessed by some at the present time, and transmitted by them through successive generations even to distant times? From all this it follows, that no one who has not yielded to the malicious and deceitful suggestions of lying devils, can be so blinded by passion as to deny the ability of the Church of the apostles— a community of brethren as numerous as they were faithful— to transmit their writings unaltered to posterity, as the original seats of the apostles have been occupied by a continuous succession of bishops to the present day, especially when we are accustomed to see this happen in the case of ordinary writings both in the Church and out of it.
Source: NewAdvent.org (Contains the full text of Contra Faustum 33.)
Here is another translation of a portion of the passage, quoted by Bart Ehrman in Forgery and Counterforgery. Ehrman begins at the point where Augustine gives the example of Hippocrates.
Were not certain books that were produced under the name Hippocrates, the highly renowned physician, rejected as authoritative by physicians? Nor did a certain similarity of topics and language offer them any help. For, compared to the books that it was clear were really Hippocrates‘ books, they were judged inferior, and they were not known at the same time at which the rest of his writings were recognized as truly his. But how is it proven that these books are really his when, compared to them, the books brought forth out of the blue are rejected? How is it proven so that, if anyone rejects this, he is not even refuted but laughed at, except because a series of physicians, from the time of Hippocrates down to the present time and thereafter, has commended them so that to have any doubt about them is the mark of a madman?
How do people know that the books of Plato, Aristotle, Varro, Cicero, and other such authors are their works except by the same unbroken testimony of the ages following one upon another?
Many authors have written extensively on the Church’s writings, not, of course, with canonical authority but with some desire to be helpful or to learn. How is it determined who wrote what except by the fact that, in the times in which each author wrote them, he made them known and published them for those for whom he could, and from them they were passed on to future generations, one after another, with unbroken knowledge that was quite widely accepted, down to our times, so that, when asked whose book is whose, we do not hesitate what we ought to reply?
…Since that is the case, who, then, is blinded by such great madness—unless he has been corrupted by agreeing with the wickedness and fallacies of lying demons—as to say that the Church of the apostles, so faithful and so numerous a harmony of brothers, could not have merited faithfully to transmit their writings to future generations, though the sees of the apostles have been preserved down to the present bishops in an utterly certain line of succession, especially since this is so much the case with any people’s writings, whether outside the Church or even in the Church?
Source: Ehrman, Bart D. Forgery and Counterforgery: The Use of Literary Deceit in Early Christian Polemics. Oxford University Press, 2012, page 144 in the Kindle edition.
This passage from Augustine is also referenced by Samuel P. Tregelles in A Lecture on the Historic Evidence of the Authorship and Transmission of the Books of the New Testament. Samuel Bagster and Sons, 1852, pages 5-8 and re-capped in pages 65-66.
For other posts on this subject, see Posts on the Means of Determining Authorship of Ancient Texts.