Photo by Ray Aucott on Unsplash
As part of the special avodah of Yom Kippur, the Kohen Gadol would select lots over two goats to determine which would be a Korban and which would be led out of the city to a cliff and pushed off to its death. This act would effect atonement for Klal Yisrael and for the Mikdash, signifies by a strip of red wool turning white.
This unfortunate goat is commonly known in English as a "scapegoat," as in "escaped goat," from the early 16th-century English (Protestant Christian) translation of the William Tyndale Bible. Its vernacular meaning, as one who has does nothing wrong but must be sacrificed for the greater good, is already an interpretation and a theology (as is Tyndale's translation, like all translation).
This source sheet offers several traditional ways of understanding the scapegoat ritual.

Description of the Ritual in the Torah

וְלָקַ֖ח אֶת־שְׁנֵ֣י הַשְּׂעִירִ֑ם וְהֶעֱמִ֤יד אֹתָם֙ לִפְנֵ֣י יהוה פֶּ֖תַח אֹ֥הֶל מוֹעֵֽד׃ וְנָתַ֧ן אַהֲרֹ֛ן עַל־שְׁנֵ֥י הַשְּׂעִירִ֖ם גֹּרָל֑וֹת גּוֹרָ֤ל אֶחָד֙ לַיהוה וְגוֹרָ֥ל אֶחָ֖ד לַעֲזָאזֵֽל׃ וְהִקְרִ֤יב אַהֲרֹן֙ אֶת־הַשָּׂעִ֔יר אֲשֶׁ֨ר עָלָ֥ה עָלָ֛יו הַגּוֹרָ֖ל לַיהוה וְעָשָׂ֖הוּ חַטָּֽאת׃ וְהַשָּׂעִ֗יר אֲשֶׁר֩ עָלָ֨ה עָלָ֤יו הַגּוֹרָל֙ לַעֲזָאזֵ֔ל יׇֽעֳמַד־חַ֛י לִפְנֵ֥י יהוה לְכַפֵּ֣ר עָלָ֑יו לְשַׁלַּ֥ח אֹת֛וֹ לַעֲזָאזֵ֖ל הַמִּדְבָּֽרָה׃ וְהִקְרִ֨יב אַהֲרֹ֜ן אֶת־פַּ֤ר הַֽחַטָּאת֙ אֲשֶׁר־ל֔וֹ וְכִפֶּ֥ר בַּֽעֲד֖וֹ וּבְעַ֣ד בֵּית֑וֹ וְשָׁחַ֛ט אֶת־פַּ֥ר הַֽחַטָּ֖את אֲשֶׁר־לֽוֹ׃
Aaron shall take the two he-goats and let them stand before יהוה at the entrance of the Tent of Meeting; and he shall place lots upon the two goats, one marked for יהוה and the other marked for Azazel. Aaron shall bring forward the goat designated by lot for יהוה, which he is to offer as a sin offering; while the goat designated by lot for Azazel shall be left standing alive before יהוה, to make expiation with it and to send it off to the wilderness for Azazel. Aaron shall then offer his bull of sin offering, to make expiation for himself and his household. He shall slaughter his bull of sin offering,

Theory 1: It's the Fault of the Generation of the Flood

תָּנָא דְּבֵי רַבִּי יִשְׁמָעֵאל: ״עֲזָאזֵל״ — שֶׁמְּכַפֵּר עַל מַעֲשֵׂה עוּזָּא וְעַזָּאֵל. תָּנוּ רַבָּנַן, ״אֶת מִשְׁפָּטַי תַּעֲשׂוּ״ — דְּבָרִים שֶׁאִלְמָלֵא (לֹא) נִכְתְּבוּ דִּין הוּא שֶׁיִּכָּתְבוּ, וְאֵלּוּ הֵן: עֲבוֹדָה זָרָה, וְגִלּוּי עֲרָיוֹת, וּשְׁפִיכוּת דָּמִים, וְגָזֵל, וּבִרְכַּת הַשֵּׁם.
§ The school of Rabbi Yishmael taught: Azazel is so called because it atones for the actions of Uzza and Azael. These are the names of “sons of God” who sinned with “daughters of men” (Genesis 6:2) and thereby caused the world to sin during the generation of the Flood.

Theory 2: We Can't Understand It

״אֶת חוּקּוֹתַי תִּשְׁמְרוּ״ — דְּבָרִים שֶׁהַשָּׂטָן מֵשִׁיב עֲלֵיהֶן, וְאֵלּוּ הֵן: אֲכִילַת חֲזִיר, וּלְבִישַׁת שַׁעַטְנֵז, וַחֲלִיצַת יְבָמָה, וְטׇהֳרַת מְצוֹרָע, וְשָׂעִיר הַמִּשְׁתַּלֵּחַ. וְשֶׁמָּא תֹּאמַר מַעֲשֵׂה תוֹהוּ הֵם, תַּלְמוּד לוֹמַר: ״אֲנִי ה׳״, אֲנִי ה׳ חֲקַקְתִּיו, וְאֵין לְךָ רְשׁוּת לְהַרְהֵר בָּהֶן.
The phrase: And you shall keep my statutes, is a reference to matters that Satan and the nations of the world challenge because the reason for these mitzvot are not known. They are: The prohibitions against eating pork; wearing garments that are made from diverse kinds of material, i.e., wool and linen; performing the ḥalitza ceremony with a yevama, a widow who must participate in a levirate marriage or ḥalitza; the purification ceremony of the leper; and the scapegoat. And lest you say these have no reason and are meaningless acts, therefore the verse states: “I am the Lord” (Leviticus 18:4), to indicate: I am the Lord, I decreed these statutes and you have no right to doubt them.
גרלות. ידועים מדברי קבלת אבותינו... ואם יכולת להבין הסוד שהוא אחר מלת עזאזל תדע סודו וסוד שמו כי יש לו חברים במקרא ואני אגלה לך קצת הסוד ברמז בהיותך בן שלשים ושלש תדענו:
LOTS. We know what they are from the words of our sages... If you are able to understand the secret that follows Azazel then you will know its secret and the secret of its name, for it has comrades in Scripture. I will reveal a bit of its secret to you in a hint. When you are thirty-three, you will understand it.
But see also Ramban (in the long comment on Vayikra 16:8, excerpted below but without the part that references Ibn Ezra), who says that Ibn Ezra's secret is that Azazel is a demon...

Theory 3: It's Demonic

וגורל אחד לעזאזל ...וזה סוד הענין כי היו עובדים לאלהים אחרים הם המלאכים עושים להם קרבנות והם להם לריח ניחוח כענין... אבל צוה הקב"ה ביום הכפורים שנשלח שעיר במדבר לשר המושל במקומות החרבן והוא הראוי לו מפני שהוא בעליו ומאצילות כחו יבא חורב ושממון כי הוא העילה לכוכבי החרב והדמים והמלחמות והמריבות והפצעים והמכות והפירוד והחרבן
והכלל נפש לגלגל מאדים וחלקו מן האומות הוא עשו שהוא עם היורש החרב והמלחמות ומן הבהמות השעירים והעזים ובחלקו עוד השדים הנקראים מזיקין בלשון רבותינו ובלשון הכתוב (ויקרא י״ז:ז׳) שעירים כי כן יקרא הוא ואומתו שעיר
...וזה טעם הגורלות כי אילו היה הכהן מקדיש אותם בפה ליהוה ולעזאזל היה כעובד אליו ונודר לשמו...
AND THE OTHER LOT FOR AZAZEL. ...Now this is the secret of the matter. They used to worship “other gods,” namely, the angels, bringing offerings of a sweet savor to them... Now the Torah has absolutely forbidden to accept them as deities, or to worship them in any manner. However, the Holy One, blessed be He, commanded us that on the Day of Atonement we should let loose a goat in the wilderness, to that “prince” [power] which rules over wastelands, and this [goat] is fitting for it because he is its master, and destruction and waste emanate from that power, which in turn is the cause of the stars of the sword, wars, quarrels, wounds, plagues, division and destruction.
In short, it is the spirit of the sphere of Mars, and its portion among the nations is Esau [Rome], the people that inherited the sword and the wars, and among animals [its portion consists of] the se’irim (demons) and the goats. Also in its portion are the devils called “destroyers” in the language of our Rabbis, and in the language of Scripture: se’irim (satyrs, demons), for thus he [i.e., Esau] and his nation were called sa’ir.
...This then is the reason [for having someone] who casts the lots [on the two goats]. If the priest were to dedicate them merely, by word of mouth [without casting the lots], saying, “one for the Eternal” and “one for Azazel,” that would be like worshipping [Azazel] or taking a vow in its name...
Rav Saadia Gaon reports this opinion (which he does not endorse):
ספר אמונות ודעות ג י: הקרבן אשר היו מקריבים לעזאזל ביום הכיפורים, כי כבר נדמה לבני אדם שהוא שם שד.
Sefer Emunot ve-Deot, 3:10: The offering that they would sacrifice to Azazael on Yom Kippur, there are people to whom it appears to be the name of a demon.

Theory 4: It's Metaphysical

Many commentators and thinkers take this approach as a starting assumption: that throwing the goat off the cliff effects real change in the world, actually changing the nature of Klal Yisrael. A characteristic articulation is made by Yehuda ha-Levi:
יהודה הלוי, ספר הכוזרי ג נג: וְכִפּוּר הָעֲוֹנוֹת בְּיוֹם הַכִּפּוּרִים וְטָהֳרַת מִקְדָּשׁ מֵהַטֻּמְאוֹת בִּשְׂעִיר עֲזָאזֵל עִם הַמַּעֲשִׂים הַתְּלוּיִם בּוֹ.
Yehuda ha-Levi, Sefer ha-Kuzari 3:53: The forgiving of [intentional] sins on Yom Kippur and the purification of the Beit ha-Mikdash from impurities by means of the goat of Azazel with all the actions that pertain to it.

Theory 5: It's Symbolic

Rambam says that the scapegoat ritual is symbolic and intended to act upon our emotions, inducing us to do teshuvah.
(כ) וכאשר היה ׳שעיר המשתלח׳ – לכפרת חטאים גדולים כולם, עד שאין ׳חטאת ציבור׳ שיכפר מה שהוא מכפר, וכאילו הוא נושא כל החטאים – מפני זה לא נרצה לזביחה ולא לשריפה ולא להקרבה כלל, אלא הרחיקו תכלית ההרחקה ויושלך ל׳ארץ גזרה׳ – רצוני לומר: שאין בה ישוב. ואין ספק לאדם שהחטאים אינם משאות שיעתקו מגב איש אחד לגב איש אחר, אבל אלו המעשים כולם משלים להביא מורא בנפש עד שתתפעל לתשובה – כלומר: שכל מה שקדם ממעשינו נקינו מהם והשלכנום אחרי גוינו והרחקנום תכלית ההרחקה.
(20) The goat [of the Day of Atonement] that was sent [into the wilderness] (Lev. 16:20, seq.) served as an atonement for all serious transgressions more than any other sin-offering of the congregation. As it thus seemed to carry off all sins, it was not accepted as an ordinary sacrifice to be slaughtered, burnt, or even brought near the Sanctuary; it was removed as far as possible, and sent forth into a waste, uncultivated, uninhabited land. There is no doubt that sins cannot be carried like a burden, and taken off the shoulder of one being to be laid on that of another being. But these ceremonies are of a symbolic character, and serve to impress men with a certain idea, and to induce them to repent; as if to say, we have freed ourselves of our previous deeds, have cast them behind our backs, and removed them from us as far as possible.