The triennial Torah reading cycle, in which the Torah is read in shorter sedarim and takes approximately three and a half years to complete, predates our current annual cycle. Dating back to antiquity, it was in use in Eretz Yisrael until fairly late. It was not until the annual reading cycle, popular in the large diaspora community of Bavel (Iraq), took hold across world Jewry, that Simchat Torah became a distinct occasion.

Diaspora

Already in the (Babylonian) Talmud, which reflects the practices of that community, the last day of Sukkot is marked as the day on which the Torah is completed. Note, however, that the haftarah for Ve-Zot ha-Bracha is different from the one we read today:
יוֹם טוֹב הָאַחֲרוֹן קוֹרִין ״כׇּל הַבְּכוֹר״, מִצְוֹת וְחוּקִּים וּבְכוֹר, וּמַפְטִירִין ״וַיְהִי כְּכַלּוֹת שְׁלֹמֹה״. לְמָחָר קוֹרִין ״וְזֹאת הַבְּרָכָה״, וּמַפְטִירִין ״וַיַּעֲמֹד שְׁלֹמֹה״.
The baraita continues: ...On the last Festival day of Sukkot, i.e., the Eighth Day of Assembly, they read the portion of “All the firstborns,” starting with the portion of “You shall tithe,” since it includes many mitzvot and statutes relating to gifts for the poor, who should be helped during this period of rejoicing, and it concludes with the halakhot governing firstborns (Deuteronomy 14:22–16:17). And they read as the haftara the portion of “And it was so, that when Solomon had made an end of praying” (I Kings 8:54–9:1), which occurred on that day. On the next day, the second day of the Eighth Day of Assembly in the Diaspora, they read the portion of “And this is the blessing” (Deuteronomy, chapters 33–34) until the end of the Torah, and they read as the haftara “And Solomon stood” (I Kings 8:22–53).
In other words, we see a link between diaspora (in this case, Bavel) and the institution of Simchat Torah. As well, in the diaspora a second day of yom tov was observed for Shmini Atzeret, creating a "site" for liturgical creativity.

Geonim

The haftarah mentioned in the Gemara is different from the one we read today, Yehoshua 1:1-18. The first time we find "our" haftarah mentioned is in Seder Amram Gaon:
Tosafot attribute this change to the period of the Geonim, specifically to Rav Hai Gaon:
למחר קרינן וזאת הברכה ומפטירין ויעמוד שלמה - ויש מקומות שנהגו להפטיר בויהי אחרי מות משה ושיבוש הוא שהרי הש"ס אין אומר כן ויש אומרים שרב האי גאון תקן לומר ויהי אחרי מות משה אבל אינן יודעין הסברא אמאי שנה סדר הש"ס:
The next day we read Ve-Zot ha-Bracha with Ve-Yaamod Shlomo as the Maftir. There are places where the custom was to read Va-Yehi Acharei Mot Moshe [Joshua 1] as the Maftir, but they should be ashamed of this because does the Talmud not say explicitly what [the Maftir should be]? There are those who say that it was Rav Hai Gaon who made this decree to say Va-Yehi Acharei Mot Moshe, but this is because they are ignorant. Why change the order in the Talmud?
Gaonic literature also relates Simchat Torah customs we are familiar with today:
ובי”ט [=וביום טוב] שני זה שהוא אחרון, רגילין אצלנו שמרקדין אפילו כמה זקנים בשעה שאומרים קלוסין לתורה – אלא שזו משום שבות הוא ונהגו בה היתר ביום זה בלבד לכבוד התורה.
On this second day of Yom Tov, which is the last, we are accustomed that even some of the elders dance when praises are said to the Torah. But this is because [the prohibition against dancing] is shevut [rabbinic] and they acted leniently on this day only, in honor of the Torah.
Also around this time, piyyutim were composed for the various parts of the special Torah service of Simchat Torah.

Rishonim

The Tur (14th century) describes a practice of Simchat Torah that we readily recognize as our own:
ליל תשיעי מתפללין כמו בליל שלפניו ומקדשין ואומרים זמן ולמחר מתפללין כמו ביום שלפניו ומוציאין ג' ספרים וקורין בא' וזאת הברכה והמפטיר קורא בשני כמו אתמול ומפטיר במלכים ויעמוד שלמה לפני ה' ובירושלים יש שמפטירין ויהי אחרי מות משה והכי מנהגינו ובשלישי מתחיל בראשית וקורין אותו ש"ת לפי שמסיימין בו התורה וראוי לשמוח בסיומה ורגילין להתחיל מיד בראשית כדי שלא יהא פתחון פה לשטן לקטרג לומר כבר סיימו אותה ואינם רוצים לקרותה עוד. ומרבין בפיוטים ...ויש מקומות שמוציאין כל הספרים ואומרים על כל אחד ואחד פיוט כל מקום לפי מנהגו ונוהגין באשכנז שהמסיים והמתחיל נודרין נדבות וקוראין לכל מרעיהן ועושין משתה ושמחה וי"ט לסיומה של תורה ולהתחלתה.
The night of the ninth [of Sukkot] we pray like on the night before [of the eighth, i.e. Shmini Atzeret] and say the kiddush on the holiday. Then the next day we pray as on the previous day. After that, we remove three Torah scrolls. From the first one we read Ve-Zot ha-Bracha and the one saying the Maftir reads from the second scroll just like on the day before and concludes with the section in Melachim [Alef], Ve-Yaamod Shlomo lifnei Hashem. There are those in Jerusalem who say as the Maftir Va-Yehi acharei mot Moshe, and so is our custom. From the third Torah scroll, he begins Bereshit. We call this Simchat Torah since we complete the Torah. It is appropriate to rejoice in its conclusion but we are used to beginning Bereshit immediately so that there won't provide Satan with a reason to defame us by saying, "See, they've just finished it and they don't want to read it anymore!" We also say many poems...and there are places where they remove all the Torah scrolls [from the ark] and say on each and every one a poem, each locale according to its own custom. In Ashkenaz it is the custom that the one who reads the end [of the Torah] and the one who reads the beginning take up a collection and call upon all their friends and make a feast and rejoice at the conclusion of the Torah and its beginning.

Acharonim

The Beit Yosef, the extensive commentary on the Tur that would be abridged into Shulchan Aruch, elaborates further:
ונוהגים באשכנז שהמסיים והמתחיל נודרין נדרים [צ"ל נדבים] וקוראים לכל מרעיהם ועושין משתה ושמחה וי"ט לסיומה של תורה... ומצאתי עיקרו של מנהג במדרש בתחלת שיר השירים וַיַּעֲמֹד לִפְנֵי אֲרוֹן בְּרִית ה' וכו' וַיַּעַשׂ מִשְׁתֶּה לְכָל עֲבָדָיו אמר רבי יצחק מכאן שעושין סעודה לגמרה של תורה:
In Ashkenaz they are accustomed that the one who concludes [the Torah] and the one who begins [it] take up a collection and call upon all their friends and make a feast and rejoice in the conclusion of the Torah... I found the essence of this custom in the Midrash at the beginning of Shir ha-Shirim [on the verse from Melachim Alef 3:15], "He [Shlomo] stood before the Ark of the Covenant of G-d" which continues, "and then he made a feast for all his servants" - Rabbi Yitzhak said that from here [we derive that] we make a feast upon the completion of the Torah.
A full account of the day is then offered by the (Ashkenazi) Rema in his glosses to Shulchan Aruch:
סדר יום שמחת תורה ובו סעיף אחד במקום שעושין שני ימים טובים ליל תשיעי מקדשין ואומרים זמן ולמחר מוציאין שלשה ספרים וקורין באחד וזאת הברכה עד סוף התורה ובשניה בראשית עד אשר ברא אלהים לעשות ובשלישית קורא המפטיר כמו אתמול ומפטיר ויהי אחרי מות משה: הגה וקורין י"ט האחרון שמחת תורה לפי ששמחין ועושין בו סעודת משתה לגמרה של תורה ונוהגין שהמסיים התורה והמתחיל בראשית נודרים נדבות וקוראים לאחרים לעשות משתה (טור) ועוד נוהגין במדינות אלו להוציא בשמחת תורה ערבית ושחרית כל ספרי תורות שבהיכל ואומרים זמירות ותשבחות וכל מקום לפי מנהגו ועוד נהגו להקיף עם ספרי התורה הבימה שבבית הכנסת כמו שמקיפים עם הלולב והכל משום שמחה ונהגו עוד להרבות הקרואים לספר תורה וקוראים פרשה אחת הרבה פעמים ואין איסור בדבר [מנהגים ורי"ב סימן פ"ד] עוד נהגו לקרות כל הנערים לספר תורה וקורים להם פרשת המלאך הגואל וגו' ובלילה קורים בספר תורה הנדרים שבתורה וכל מקום לפי מנהגו. עוד נהגו לסיים התורה אף על קטן העולה אע"ג דיש אומרים דדוקא תלמיד חכם צריך לסיים [מרדכי הגהות קטנות] בזמן הזה שהחזן קורא אין לחוש [ד"ע] במקום שאין להם רק שני ספרי תורות קורין בראשונה וזאת הברכה ובשניה בראשית וחוזרין ולוקחין הראשונה לעניינו של יום וכן עושין כל מקום דבעינן שלשה ספרי תורות ואין להם רק שתים (מצא כתוב):
The Order of Simchat Torah: In places where they do two days of Yom Tov, Kiddush on the ninth night includes Shehecheyanu. The next day, we take out three Torah scrolls. From the first, we read "And this is the blessing" [Deuteronomy 33:1] until the end of the Torah. From the second, we read from "In the beginning" [Genesis 1:1] until "that God made" [2:3]. From the third, the maftir reads the same as the previous day. The haftarah is "After Moses died" [Joshua 1:1]. Rem"a: The last day of Yom Tov is called "Simchat Torah" because we rejoice on it, making a festive meal in honor of the finishing of the Torah. It is customary for the person who finishes the Torah and the one who starts Genesis to make a donation and invite everybody to a party (Tur). It is customary in these countries to take all of the Torahs out of the Ark on Simchat Torah at night and in the morning and to sing songs and praises. Every place should follow its customs. It is also the custom to circle the synagogue's Bimah with the Torah scrolls just like we circle with the lulav. This is all done out of joy. It is also the custom to have many readers from the Torah. We read the same section many times, and this is not forbidden (Minhagim, Rivas"h 84). It is also the custom to call up all of the children to the Torah and to read "the angel who has redeemed me..." [Genesis 48:6]. In the evening, we read the special sections from the Torah that are normally auctioned off, every place according to its custom. It is also the custom that even a child can finish off the Torah, even though there are those who say that specifically a scholar should finish it (Mordechai's small notes). Nowadays when the chazzan does the actual reading, there is no issue (his own opinion). In a place with only two Torah scrolls, we read "And this is the blessing" from the first, "In the beginning" from the second, and then we go back and reuse the first for the section associated with the day. This is done whenever three Torahs are needed but there are only two (found written somewhere).
The custom of children parading with flags during the hakkafot of the Sifrei Torah is first mentioned in 1672 in the Takkanot (enactments or rules) of the the Polish kehilla of early modern Amsterdam. However, from the text there it appears that already then it was an old custom. Another early source is found in a book by a German Christian Hebraist (one who studied Hebrew, which was not uncommon among the well-educated in early modern Europe):
Johann Bodenschatz, Kirchliche Verfassung der heutigen Juden (Erlang, 1748): They [the children] hold onto their flags upon which is inscribed 'standard of the camp' and the names of the tribes. They march as if they were soldiers.

Lurianic Kabbalah

Chaim Vital, Sha'ar Ha-Kavvanot (trans. Morris Faierstein, Jewish Customs of Kabbalistic Origin).The custom to take the scrolls out of the Ark and to circumambulate the Ark in the morning service, the afternoon service, and the evening service at the end of the festival, is a true custom. It is already written in the Zohar, Parshat Pinhas, page 256b, in the Raya Mehemna, and this is what it says: “Israel is accustomed to rejoice in it and it is called Simhat Torah. They crown the Torah scroll with its crown, etc.” I saw that my teacher (R. Isaac Luria), of blessed memory, was very punctilious in this matter, to walk around after the Torah scroll, either before it or after it, to dance and sing after it with all of his ability, on the night at the end of the festival after the evening prayer. He was very punctilious to do seven complete hakkafot, aside from the complete hakkafot on the day of Simhat Torah. However, I never found the matter of the hakkafot during the day and I did not see it. On the night at the end of the festival, I did see him go to another synagogue and do seven hakkafot. He continued on his way and found another synagogue where they did the hakkafot later and did the hakkafot with them.

In the Soviet Union

Prof. Shalom Sabar writes: Among Soviet Jewish youth seeking forms of expressing their Jewish identification, Simchat Torah gradually became, during the 1960s, the occasion of mass gatherings in and around the synagogues, mainly in the great cities Moscow, Leningrad, Riga, and others.
According to recollections of Jews from the FSU, this was the case even among secular Russian Jews, including those who chose to remain secular once they emigrated to free democracies. However, the Soviet claim that Simchat Torah gatherings were entirely secular affairs is clearly belabored:
The official Soviet overseas propaganda news agency conceded today that great numbers of young Moscow Jews had participated in the dancing and singing that marked the end of Simchat Torah but asserted that they came out, not in observance of religious practices, but as participants in a folk custom.
The Novosti Press Agency, in a dispatch signed by Samuel Rosin, a Novosti correspondent, distributed here today by the Soviet Embassy, said that large crowds had danced in the Moscow streets last Saturday night, but “nobody prayed” and the Jewish community was looking forward to the celebration next Sunday of the 50th anniversary of the Russian Revolution.

In Modern Israel

Modern Israel uses the universally accepted annual cycle of Torah reading Simchat Torah, though in a form special to Eretz Yisrael. Simchat Torah in Israel is marked as a special occasion but on the 22nd of Tishrei, concurrent with Shmini Atzeret. Outside of Israel, Simchat Torah continues to be celebrated on the 23rd of Tishrei, the yom tov sheni of galuyot of Shmini Atzeret.