(9) This is the line of Noah.—Noah was a righteous man; he was blameless in his age; Noah walked with God.— (10) Noah begot three sons: Shem, Ham, and Japheth. (11) The earth became corrupt before God; the earth was filled with lawlessness. (12) When God saw how corrupt the earth was, for all flesh had corrupted its ways on earth, (13) God said to Noah, “I have decided to put an end to all flesh, for the earth is filled with lawlessness because of them: I am about to destroy them with the earth.
(א) ותשחת. לְשׁוֹן עֶרְוָה וַעֲ"זָ (סנה' נ"ז).... (ב) ותמלא הארץ חמס. גֶּזֶל:
(1) ותשחת [THE EARTH] WAS CORRUPT — It means lewdness and idolatry.... (in next verse of this chapter where Rashi states that this has reference to lewdness) (Sanhedrin 57a). (2) ותמלא הארץ חמס AND THE EARTH WAS FILLED WITH VIOLENCE — means robbery.
(ב) כי מלאה הארץ חמס. לֹא נֶחְתַּם גְּזַר דִּינָם אֶלָּא עַל הַגֶּזֶל (סנה' ק"ח):
(2) כי מלאה הארץ חמס FOR THE EARTH IS FILLED WITH VIOLENCE — Their fate was sealed only on account of their sin of robbery (Sanhedrin 108a).
So Rashi identified sexual lewdness, idolatry and robbery as the causes of the destruction of humanity, possibly identifying them as hamas.
What is missing in his trilogy of human misdeeds?
But see the earliest commentary of Humash, Onkelos, who offers a different understanding.
(יא) וְאִתְחַבָּלַת אַרְעָא קֳדָם יְיָ
וְאִתְמְלִיאַת אַרְעָא חֲטוֹפִין:
Other occurrances of the word hamas in Tanakh
(ה) שִׁמְע֥וֹן וְלֵוִ֖י אַחִ֑ים
כְּלֵ֥י חָמָ֖ס מְכֵרֹתֵיהֶֽם׃
Their weapons are tools of lawlessness.
(1) You must not carry false rumors; you shall not join hands with the guilty to act as a malicious witness:
9x in Psalms; 5x in Proverbs
Isaiah 59: the entire chapter deals with human misdeeds which serve as barriers between humanity and God.
Verse 6 states:
....מַֽעֲשֵׂיהֶם֙ מַֽעֲשֵׂי־אָ֔וֶן
וּפֹ֥עַל חָמָ֖ס בְּכַפֵּיהֶֽם׃
Their deeds are deeds of mischief,
Their hands commit lawless acts,
Prior to the Flood, the human misdeed cited frequently in Genesis is murder.
(ח) וַיֹּ֥אמֶר קַ֖יִן אֶל־הֶ֣בֶל אָחִ֑יו וַֽיְהִי֙ בִּהְיוֹתָ֣ם בַּשָּׂדֶ֔ה וַיָּ֥קׇם קַ֛יִן אֶל־הֶ֥בֶל אָחִ֖יו וַיַּהַרְגֵֽהוּ׃
(כג) וַיֹּ֨אמֶר לֶ֜מֶךְ לְנָשָׁ֗יו עָדָ֤ה וְצִלָּה֙ שְׁמַ֣עַן קוֹלִ֔י נְשֵׁ֣י לֶ֔מֶךְ הַאְזֵ֖נָּה אִמְרָתִ֑י כִּ֣י אִ֤ישׁ הָרַ֙גְתִּי֙ לְפִצְעִ֔י וְיֶ֖לֶד לְחַבֻּרָתִֽי׃
(8) Cain said to his brother Abel …*… Ancient versions, including the Targum, read “Come, let us go out into the field.” and when they were in the field, Cain set upon his brother Abel and killed him.
(23) And Lamech said to his wives,
“Adah and Zillah, hear my voice;
O wives of Lamech, give ear to my speech.
I have slain a person for wounding me,
And a lad for bruising me.