The triennial Torah reading cycle, in which the Torah is read in shorter sedarim and takes approximately three and a half years to complete, predates our current annual cycle. Dating back to antiquity, it was in use in Eretz Yisrael until fairly late. It was not until the annual reading cycle, popular in the large diaspora community of Bavel (Iraq), took hold across world Jewry, that Simchat Torah became a distinct occasion.
Already in the (Babylonian) Talmud, which reflects the practices of that community, the last day of Sukkot is marked as the day on which the Torah is completed. Note, however, that the haftarah for Ve-Zot ha-Bracha is different from the one we read today:
In other words, we see a link between diaspora (in this case, Bavel) and the institution of Simchat Torah. As well, in the diaspora a second day of yom tov was observed for Shmini Atzeret, creating a "site" for liturgical creativity.
The haftarah mentioned in the Gemara is different from the one we read today, Yehoshua 1:1-18. The first time we find "our" haftarah mentioned is in Seder Amram Gaon:
Tosafot attribute this change to the period of the Geonim, specifically to Rav Hai Gaon:
Gaonic literature also relates Simchat Torah customs we are familiar with today:
Also around this time, piyyutim were composed for the various parts of the special Torah service of Simchat Torah.
The Tur (14th century) describes a practice of Simchat Torah that we readily recognize as our own:
The Beit Yosef, the extensive commentary on the Tur that would be abridged into Shulchan Aruch, elaborates further:
A full account of the day is then offered by the (Ashkenazi) Rema in his glosses to Shulchan Aruch:
The custom of children parading with flags during the hakkafot of the Sifrei Torah is first mentioned in 1672 in the Takkanot (enactments or rules) of the the Polish kehilla of early modern Amsterdam. However, from the text there it appears that already then it was an old custom. Another early source is found in a book by a German Christian Hebraist (one who studied Hebrew, which was not uncommon among the well-educated in early modern Europe):
Johann Bodenschatz, Kirchliche Verfassung der heutigen Juden (Erlang, 1748): They [the children] hold onto their flags upon which is inscribed 'standard of the camp' and the names of the tribes. They march as if they were soldiers.
Chaim Vital, Sha'ar Ha-Kavvanot (trans. Morris Faierstein, Jewish Customs of Kabbalistic Origin).The custom to take the scrolls out of the Ark and to circumambulate the Ark in the morning service, the afternoon service, and the evening service at the end of the festival, is a true custom. It is already written in the Zohar, Parshat Pinhas, page 256b, in the Raya Mehemna, and this is what it says: “Israel is accustomed to rejoice in it and it is called Simhat Torah. They crown the Torah scroll with its crown, etc.” I saw that my teacher (R. Isaac Luria), of blessed memory, was very punctilious in this matter, to walk around after the Torah scroll, either before it or after it, to dance and sing after it with all of his ability, on the night at the end of the festival after the evening prayer. He was very punctilious to do seven complete hakkafot, aside from the complete hakkafot on the day of Simhat Torah. However, I never found the matter of the hakkafot during the day and I did not see it. On the night at the end of the festival, I did see him go to another synagogue and do seven hakkafot. He continued on his way and found another synagogue where they did the hakkafot later and did the hakkafot with them.
In the Soviet Union
Prof. Shalom Sabar writes: Among Soviet Jewish youth seeking forms of expressing their Jewish identification, Simchat Torah gradually became, during the 1960s, the occasion of mass gatherings in and around the synagogues, mainly in the great cities Moscow, Leningrad, Riga, and others.
According to recollections of Jews from the FSU, this was the case even among secular Russian Jews, including those who chose to remain secular once they emigrated to free democracies. However, the Soviet claim that Simchat Torah gatherings were entirely secular affairs is clearly belabored:
"Soviet Agency Says Simchat Torah Observance Was ‘folk Custom’, Not Religious," Jewish Telegraphic Agency, November 1, 1967.
The official Soviet overseas propaganda news agency conceded today that great numbers of young Moscow Jews had participated in the dancing and singing that marked the end of Simchat Torah but asserted that they came out, not in observance of religious practices, but as participants in a folk custom.
The Novosti Press Agency, in a dispatch signed by Samuel Rosin, a Novosti correspondent, distributed here today by the Soviet Embassy, said that large crowds had danced in the Moscow streets last Saturday night, but “nobody prayed” and the Jewish community was looking forward to the celebration next Sunday of the 50th anniversary of the Russian Revolution.
In Modern Israel
Modern Israel uses the universally accepted annual cycle of Torah reading Simchat Torah, though in a form special to Eretz Yisrael. Simchat Torah in Israel is marked as a special occasion but on the 22nd of Tishrei, concurrent with Shmini Atzeret. Outside of Israel, Simchat Torah continues to be celebrated on the 23rd of Tishrei, the yom tov sheni of galuyot of Shmini Atzeret.