What is bowing?
אמר רבי תנחום אמר רבי יהושע בן לוי כנגד שמונה עשרה חוליות שבשדרה ואמר רבי תנחום אמר רבי יהושע בן לוי המתפלל צריך שיכרע עד שיתפקקו כל חוליות שבשדרה עולא אמר עד כדי שיראה איסר כנגד לבו רבי חנינא אמר כיון שנענע ראשו שוב אינו צריך אמר רבא והוא דמצער נפשיה ומחזי כמאן דכרע
Rabbi Tanḥum said that Rabbi Yehoshua ben Levi said: In those blessings where one is required to bow, one who prays must bow until all the vertebrae in the spine protrude. Establishing a different indicator to determine when he has bowed sufficiently, Ulla said: Until he can see a small coin [issar], on the ground before him opposite his heart (Rav Hai Gaon). Rabbi Ḥanina said: There is room for leniency; once he moves his head forward, he need not bow any further. Rava said: But that applies only if he is exerting himself when doing so, and he appears like one who is bowing.
ת"ר קידה על אפים וכן הוא אומר (מלכים א א, לא) ותקד בת שבע אפים ארץ כריעה על ברכים וכן הוא אומר (מלכים א ח, נד) מכרוע על ברכיו השתחואה זו פישוט ידים ורגלים וכן אומר (בראשית לז, י) הבוא נבוא אני ואמך ואחיך להשתחות לך ארצה:
The Sages taught: The term kidda indicates falling upon one’s face...Keria means descending upon one’s knees...Hishtaḥava’a, this is prostrating oneself while spreading one’s arms and legs in total submission.
When do we have to bow?
כְּרִיעָה כֵּיצַד. הַמִּתְפַּלֵּל כּוֹרֵעַ חָמֵשׁ כְּרִיעוֹת בְּכָל תְּפִלָּה וּתְפִלָּה. בִּבְרָכָה רִאשׁוֹנָה בַּתְּחִלָּה וּבַסּוֹף. וּבְהוֹדָיָה בַּתְּחִלָּה וּבַסּוֹף. וּכְשֶׁגּוֹמֵר הַתְּפִלָּה כּוֹרֵעַ וּפוֹסֵעַ שָׁלֹשׁ פְּסִיעוֹת לַאֲחוֹרָיו. וּכְשֶׁהוּא כּוֹרֵעַ נוֹתֵן שָׁלוֹם מִשְּׂמֹאל עַצְמוֹ וְאַחַר כָּךְ מִימִין עַצְמוֹ וְאַחַר כָּךְ מַגְבִּיהַּ רֹאשׁוֹ מִן הַכְּרִיעָה. וּכְשֶׁהוּא כּוֹרֵעַ בְּאַרְבַּע הַכְּרִיעוֹת כּוֹרֵעַ בְּבָרוּךְ וּכְשֶׁהוּא זוֹקֵף זוֹקֵף בַּשֵּׁם. בַּמֶּה דְּבָרִים אֲמוּרִים בְּהֶדְיוֹט. אֲבָל כֹּהֵן גָּדוֹל כּוֹרֵעַ בִּתְחִלַּת כָּל בְּרָכָה וּבְסוֹף כָּל בְּרָכָה. וְהַמֶּלֶךְ כֵּיוָן שֶׁשָּׁחָה בָּרִאשׁוֹנָה אֵינוֹ מַגְבִּיהַּ רֹאשׁוֹ עַד שֶׁגּוֹמֵר כָּל תְּפִלָּתוֹ:
Rambam, Hilkhot Tefillah 5:10
Bowing: What is implied? One praying bows five times in each and every Amidah: In the first blessing, at the beginning and at the end; in the blessing of thanks, at the beginning and at the end; and upon completing the Amidah, one bows and takes three steps backwards while bowing. She takes leave from her left and afterwards, from her right. Then, she lifts her head up from the bowed position. When she bows the [other] four times, she does so at [the utterance of the word] "Blessed" and straightens up when [reciting] God's name. To whom does the above apply? To an average person. However, the High Priest bows at the beginning and end of each and every blessing. A queen bows at the beginning [of the Amidah] and does not lift her head until she completes her whole Amidah.
Bowing on Rosh Hashanah
() עָלֵינוּ לְשַׁבֵּחַ לַאֲדון הַכּל. לָתֵת גְּדֻלָּה לְיוצֵר בְּרֵאשִׁית. שֶׁלּא עָשנוּ כְּגויֵי הָאֲרָצות. וְלא שמָנוּ כְּמִשְׁפְּחות הָאֲדָמָה. שֶׁלּא שם חֶלְקֵנוּ כָּהֶם וְגורָלֵנוּ כְּכָל הֲמונָם: שֶׁהֵם מִשְׁתַּחֲוִים לְהֶבֶל וְרִיק וּמִתְפַּלְלִים אֶל אֵל לא יושִׁיעַ: וַאֲנַחְנוּ כּורְעִים וּמִשְׁתַּחֲוִים וּמודִים לִפְנֵי מֶלֶךְ מַלְכֵי הַמְּלָכִים הַקָּדושׁ בָּרוּךְ הוּא:
It is our duty to praise the Master of all, to acclaim the greatness of the One who forms all creation. For God did not make us like the nations of other
lands, and did not make us the same as other families of the Earth. For they prostrate themselves to vanity and nothingness, and pray to a god who cannot deliver. And we bend our knees, and bow down, and give thanks, before the Ruler, the Ruler of Rulers, the Holy One, Blessed is God.
Rabbi Yoel Sirkes (1561–1640), Bayith Chadash, Orach Chaim 133
We conclude the service with Alenu in order to reinforce our faith in the unity of God's kingdom before we return from the synagogue. We thus strengthen our faith that, "God will wipe out false gods from the earth, and all idols will be cut off, so that the world will be rectified in the Kingdom of the Almighty."
Even though we must then go and deal with people of other faiths, seeing that they are dominant, our hearts will not be drawn to their gods, and we will not have any sinful thoughts.
Full prostration on Yom Kippur during the Avodah service
וְהַכֹּהֲנִים וְהָעָם הָעוֹמְדִים בָּעֲזָרָה. כְּשֶׁהָיוּ שׁוֹמְעִים אֶת הַשֵּׁם הַנִּכְבָּד וְהַנּוֹרָא מְפֹרָשׁ יוֹצֵא מִפִּי כֹהֵן גָּדוֹל בִּקְדֻשָּׁה וּבְטָהֳרָה. הָיוּ כּוֹרְעִים וּמִשְׁתַּחֲוִים וּמוֹדִים וְנוֹפְלִים עַל פְּנֵיהֶם. וְאוֹמְרִים בָּרוּךְ שֵׁם כְּבוֹד מַלְכוּתוֹ לְעוֹלָם וָעֶד:
And the priests and the people who were standing in the Temple Courtyard; when they heard the glorious and awesome Name [of God] explicitly pronounced from the mouth of the High Priest, with holiness and with purity, they bowed, prostrated themselves, offered thanks, and fell upon their faces, and said: “Blessed [is His] Name, His glorious kingdom is forever and ever.”
The Koren Yom Kippur Mahzor, Seder HaAvodah, Chief Rabbi Lord Jonathan Sacks, 21st century UK
Prostration was performed daily in the Temple by officiating priests but was restricted in the synagogue to two occasions: 1) during the first paragraph of Aleinu in the Musaf Amida on Rosh HaShana and on Yom Kippur; and 2) here, retelling the story of the High Priest's service on Yom Kippur. At all other times, we merely bow rather than kneel and fall on our faces. The reason is to mark a clear distinction between the synagogue and the Temple. The synagogue may be a "miniature Temple" but a symbol is not the same as what it symbolizes. Only on Yom Kippur do we deliberately re-create the atmosphere of the Temple: in two ways: by this threefold (in some rites, fourfold) prostration and by saying aloud throughout the day "Blessed be the name of God's glorious kingdom forever and all time," normally said silently.