אִישׁ חַרְבּוֹ, רַבִּי אֶלְעָזָר אוֹמֵר בֶּן שְׁלשׁ עֶשְׂרֵה שָׁנָה הָיוּ.
[What is the meaning of the words,] "Each man took his sword"?
Rabbi Elazar says: [They are called "man" because] they were 13 years old.
איש הוי מבן י"ג שנה ולא בפחות שלא מצינו בכל התורה שיהא קרוי איש בפחות מבן י"ג אבל בבן י"ג מצינו שקראו הכתוב איש כדכתיב (בראשית ל״ד:כ״ה) ויקחו שני בני יעקב שמעון ולוי איש חרבו וגמירי שמעון ולוי בההיא שעתא בני י"ג שנה הוו והרוצה לחשוב יצא ויחשוב:
The term "ish" (man) means 13-years-old, and not younger. We don't find anywhere in the Torah that someone is called "ish" if they are less than 13. But we do find someone who is 13 called "ish," as in the verse, "And the two sons of Jacob, Shimon & Levi, each man took their sword;" and we have the teaching that at the time of that incident, they were 13 years old. If you want to make the calculation, go ahead and calculate it.
(יא) בַּת שְׁתֵּים עֶשְׂרֵה שָׁנָה וְיוֹם אֶחָד וּבֶן שְׁלֹשׁ עֶשְׂרֵה שָׁנָה וְיוֹם אֶחָד שֶׁהֵבִיאוּ שְׁתֵּי שְׂעָרוֹת הֲרֵי הֵן כִּגְדוֹלִים לְכָל הַמִּצְוֹת וּמַשְׁלִימִין מִן הַתּוֹרָה. אֲבָל אִם לֹא הֵבִיאוּ שְׁתֵּי שְׂעָרוֹת עֲדַיִן קְטַנִּים הֵן וְאֵינָם מַשְׁלִימִין אֶלָּא מִדִּבְרֵי סוֹפְרִים. קָטָן שֶׁהוּא פָּחוֹת מִבֶּן תֵּשַׁע אֵין מְעַנִּין אוֹתוֹ בְּיוֹם הַכִּפּוּרִים כְּדֵי שֶׁלֹּא יָבֹא לִידֵי סַכָּנָה:
(11) A female who is twelve years old and one day and a male who is thirteen years old and one day, who manifest [signs of physical maturity - i.e.,] two [pubic] hairs, are considered to be adults with regard to [the observance of] all the mitzvot, and are obligated to complete their fast according to the Torah. If, however, they did not manifest [signs of physical maturity], they are still considered to be minors, and are obligated to complete their fast only by virtue of Rabbinic decree.
By the age of 13 one has acquired the intellectual characteristics and attitudes of an adult — maturity of intellect and discernment. It is for this reason that a person is then obligated to perform all the commandments.
וששאלת מאין לנו דבן י"ג שנה ויום אחד הוא בר עונשין אבל פחות מכן לא דע כי הלכה למשה מסיני הוא והוא בכלל שיעורין חציצין ומחיצין שהן הלכה למשה מסיני
Regarding your question as to the source for considering 13 to be the age of punishment (i.e., of being an adult): Know that this is Halacha l'Moshe m'Sinai [a law given to Moses at Sinai], and is considered one of the shiurim [measurements].
According the the Rosh (and others), the source for the obligation to perform mitzvos at age 13 is a dictate handed down by G‑d to Moshe at Sinai.
As such, it follows along the lines of other supra-rational edicts regarding measurements and amounts. According to this opinion, the obligation to perform mitzvos at 13 has nothing to do with maturity or discernment; it is a supra-rational law.
שֶׁלֹּא נִתְּנוּ הַשִּׁעוּרִין אֶלָּא לְיִשְׂרָאֵל בִּלְבַד.
The specification of shiurim (minimum amounts) only applies to Jews.
What is a practical difference between reason #1 (that age of bar mitzvah is dependent on the age at which (most) people reach maturity) and #2 (that it is halacha l'Moshe m'Sinai)?
SPIRITUAL TAKEAWAY
In terms of spiritual service, the difference between these two opinions relates to the manner in which a Jew is to approach the performance of Torah and mitzvos :
According to the first opinion, the approach is one of serving G-d logically; if the age at which one becomes obligated to perform mitzvos depends on one’s intellectual maturity, it is understandable that the service commences with logic and comprehension.
According to the second opinion, however, the obligation to begin performing mitzvos at 13 is supra-rational — because G‑d has so commanded. It therefore follows that the approach to the performance of mitzvos involves the supra-rational acceptance of the Divine Yoke.
Nevertheless, even those who hold the first opinion — that the age for beginning one’s service is gleaned from the verse “each man took his sword” — also agree that the performance of mitzvos is bound up with mesirus nefesh, i.e., serving G‑d in a self-sacrificial manner that transcends the bounds of intellect.
That this is indeed so is amply demonstrated by the fact that those who hold this opinion derive it from the verse “each man took his sword ” — an action that demands self-sacrifice.
This in no way contradicts the earlier statement that this manner of service demands comprehension and intellect, for though they maintain that the action should be performed with understanding and discernment, they agree that the foundation of Divine service lies in acceptance of the Divine Yoke. Then, and only then, can a person be assured that he will not be blinded by his own logic, and that his performance of mitzvos will be done in an entirely proper manner.