רַבִּי יוֹסִי אוֹמֵר, יוֹסֵף הָיָה אוֹמֵר לוֹ: שֶׁהָיוּ נוֹתְנִין עֵינֵיהֶן בִּבְנוֹת הָאָרֶץ. אָמַר לוֹ הַקָּדוֹשׁ בָּרוּךְ הוּא, חַיֶּיךָ, בּוֹ בַדָּבָר תִּקְלֶה, הֲרֵי אֲדוֹנָתְךָ נוֹתֶנֶת עֵינֶיהָ בָּךְ, שֶׁנֶּאֱמַר: וַתִּשָּׂא אֵשֶׁת אֲדֹנָיו.
R. Yose said: Joseph told him [Jacob]: that they gave their eyes into the daughters of the land. Said to him The Holy One, blessed be He: Be assured, in this very thing is [your] snare (תִּקְלֶה). Your mistress gives her eyes in you, as is said: His master’s wife cast her eyes upon Joseph.
(my translation)
Note that here the comparison is between (1) young men who desire non-Jewish women, and (2) a non-Jewish woman who desires a Jewish man.
Here, the kashe is about what Joseph told Jacob about his brothers, and the Midrash's logic holds that people are punished according to their very word. Hence, just right before, Joseph is sold as a slave because he accused his brothers to be treating Jacob's servants' sons as slaves. In fact, the Midrash is gathered around the lemma Gen 38:7 (וַתִּשָּׂא אֵשֶׁת אֲדֹנָיו), and seeks to interpret why this happened. The Midrash's answer is: that Joseph's "temptation/seduction" served as a punishment for lying to Jacob about his brothers (7:1), along with his enslavement and the slaughtering of a goat to cover Joseph's garment in blood.
This compares to Mishna Sotah 1:8. Actually, we'll find in the comparison to that text, that it's not just about the sinful action that will be repeated in the punishment, but rather that the body part executing the sinful conduct will be involved in its punishment. We learn that the source of Asalom's misconduct is his hair, hence he is hung from his own hair. Samson desired with his eyes - therefore he is blinded. Interestingly, the context of Sotah is punishment for sexual deviance. The Mishna also discusses, that this logic holds true for good, like the reward for Joseph burying his father is that Moses took Joseph's bones with them when they left Egypt.
In the measure with which a person metes out to others, they mete out to him. She adorned herself for a sin; the Omnipresent made her repulsive. She exposed herself for a sin; the Omnipresent exposed her. She began the transgression with the thigh and afterwards with the womb; therefore she is punished first in the thigh and afterwards in the womb, nor does all the body escape.
Also later midrash relates this interpretive formula or principle, that G-d punishes and rewards in the same way that the transgression was. The following section from Midrash Tanhuma seems a bit out of context, as the general section is about the question of why two additional years were added to Joseph's imprisonment.
Later on in Midrash Tanhum, we learn that in fact that Joseph's brothers deduced from him being יפה תאר ומראה יפה that they should be looking for Joseph in a red-light district - which actually also Joseph suspected as he sent his guards to be looking there.
(ב) אָמַר רַבִּי יְהוּדָה בַּר סִימוֹן, אַף יוֹסֵף יוֹדֵעַ הָיָה שֶׁאֶחָיו יוֹרְדִין לְמִצְרַיִם לִשְׁבָּר אֹכֶל. מֶה עָשָׂה הוֹשִׁיב שׁוֹמְרִים עַל כָּל הַפְּתָחִים וְאָמַר לָהֶן, רְאוּ, כָּל מִי שֶׁיִּכָּנֵס לִשְׁבָּר אֹכֶל, כִּתְבוּ שְׁמוֹ וְשֵׁם אָבִיו וְלָעֶרֶב הָבִיאוּ לִי פִתְקְכֶם. עָשׂוּ כָּךְ. כֵּיוָן שֶׁבָּאוּ בְּנֵי יַעֲקֹב, כָּל אֶחָד וְאֶחָד נִכְנַס בַּשַּׁעַר שֶׁלּוֹ, וְכָתְבוּ אֶת שְׁמוֹתָם וְשֶׁל אֲבִיהֶם. לָעֶרֶב הֵבִיאוּ פִתְקָם לִפְנֵי יוֹסֵף. זֶה קוֹרֵא רְאוּבֵן בֶּן יַעֲקֹב, וְאֶחָד קוֹרֵא שִׁמְעוֹן בֶּן יַעֲקֹב, וְאֶחָד לֵוִי, וְכֵן הַשּׁוֹעֲרִים כָּל אֶחָד וְאֶחָד אֶת שֶׁלּוֹ. מִיָּד אָמַר לָהֶן יוֹסֵף, סִתְמוּ אֶת הָאוֹצָרוֹת וּפִתְחוּ אוֹצָר אֶחָד, וְנָתַן שְׁמוֹתָם לְבַעַל הָאוֹצָר וְאָמַר לוֹ: רְאֵה כְּשֶׁיָּבֹאוּ אֲנָשִׁים אֵלּוּ לְיָדְךָ, תְּפֹשׂ אוֹתָם וְשַׁגֵּר אוֹתָם לְפָנָיו. עָבְרוּ שְׁלֹשָה יָמִים וְלֹא בָאוּ. מִיָּד נָטַל יוֹסֵף שִׁבְעִים גִּבּוֹרִים מִבֵּית הַמֶּלֶךְ וְשִׁגֵּר בִּשְׁבִילָם לְבַקֵּר אוֹתָם בַּשּׁוּק. הָלְכוּ מָצְאוּ אוֹתָם בְּשׁוּק שֶׁל זוֹנוֹת. וּמַה טִּיבָן בְּשׁוּק שֶׁל זוֹנוֹת. אֶלָּא אָמְרוּ, יוֹסֵף אָחִינוּ יְפֵה תֹאַר וִיפֵה מַרְאֶה, שֶׁמָּא הוּא בְּקֻבָּה שֶׁל זוֹנוֹת. מִיָּד תָּפְשׂוּ אוֹתָם וֶהֱבִיאוּם לִפְנֵי יוֹסֵף.
(2) R. Judah the son of Simeon said that Joseph was aware that his brothers were descending to Egypt to obtain corn. What did he do? He stationed guards at each gate and ordered them to examine every person who entered to purchase food. They were to record his name, and the name of his father, and then to bring the list to him. They did as directed. When Jacob’s sons entered through different gates, their names and the name of their father were written down. At night, they (the guards) brought their lists to Joseph. One of the guards read the name of Reuben the son of Jacob, another read Simeon the son of Jacob, and still another read Levi; and so each gatekeeper read the name that was on his list. Thereupon Joseph ordered them to close all but one of the storehouses. He gave their names to the official in charge of the open storehouse, and commanded: “When these men come to you, seize them and bring them to me.” Three days passed by but still they did not appear. Then Joseph selected seventy powerful men from the king’s guard and sent them to the marketplace to seek them out. They searched for them until they found them in the district in which the harlots resided. Why did the brothers go to the district in which the harlots lived? They had said to each other: “Our brother Joseph is extremely well built and handsome, perhaps he is in a brothel.”8Not as a patron but as an inmate of a homosexual brothel. Then the guards seized the brothers and brought them to Joseph.
Bavli Sotah 13b has it that Potiphar indeed bought Joseph as a sex slave for himself.
“And Potiphar, an officer [seris] of Pharaoh’s, bought him" Rav says: that he bought him for himself. (Gabriel came and castrated him.) Gabriel came and mutilated him. Originally is is written: Potiphar, and in the end: Potiphera.
(my translation)
He said to her: No.
She said to him: I will incarcerate you in the prison. He said to her: I do not fear you, as it is stated: “God releases prisoners” (Psalms 146:7).
She said to him: I will cause you to be bent over with suffering.
He said: “God straightens those who are bent over” (Psalms 146:8).
She said I will blind your eyes.
He said to her “God opens the eyes of the blind” (Psalms 146:8).
She gave him a thousand talents of silver to submit to her, “to lie with her and be with her” (Genesis 39:10), and he refused.
When cross-dressing in halacha is discussed, this machloket is often cited. However, what seems suspicious here is how central the specific locality is to the discussion. Already from the beginning the place of restriction is reduced to the market place. Rabbi Yohanan, quite a femme Rabbi himself, limits or extracts this further to a place where the halachic category of sodomy might be suspected to be close.
A man should not go out to the marketplace perfumed. Rabbi Aba, the son of Rabbi Hiyya said in the name of Rabbi Yohanan, [this only applies] in a place where there is fear of “lying with a man.”
Here we have another source that could be related to lineage, the evil eye. This is also the source text for Rabbi Yohanan elsewhere to identify himself as a descendant of Joseph.
אמר רבי יוחנן אנא אישתיירי משפירי ירושלים האי מאן דבעי מחזי שופריה דרבי יוחנן נייתי כסא דכספא מבי סלקי ונמלייה פרצידיא דרומנא סומקא ונהדר ליה כלילא דוורדא סומקא לפומיה ונותביה בין שמשא לטולא ההוא זהרורי מעין שופריה דר' יוחנן איני והאמר מר שופריה דרב כהנא מעין שופריה דרבי אבהו שופריה דר' אבהו מעין שופריה דיעקב אבינו שופריה דיעקב אבינו מעין שופריה דאדם הראשון ואילו ר' יוחנן לא קא חשיב ליה שאני ר' יוחנן דהדרת פנים לא הויא ליה... אמרו ליה רבנן לא מסתפי מר מעינא בישא אמר להו אנא מזרעא דיוסף קאתינא דלא שלטא ביה עינא בישא דכתיב (בראשית מט, כב) בן פורת יוסף בן פורת עלי עין...
יומא חד הוה קא סחי ר' יוחנן בירדנא חזייה ריש לקיש ושוור לירדנא אבתריה אמר ליה חילך לאורייתא אמר ליה שופרך לנשי א"ל אי הדרת בך יהיבנא לך אחותי דשפירא מינאי קביל עליה בעי למיהדר לאתויי מאניה ולא מצי הדר אקרייה ואתנייה ושוייה גברא רבא... חלש דעתיה דרבי יוחנן חלש ריש לקיש אתאי אחתיה קא בכיא אמרה ליה עשה בשביל בני אמר לה (ירמיהו מט, יא) עזבה יתומיך אני אחיה עשה בשביל אלמנותי אמר לה (ירמיהו מט, יא) ואלמנותיך עלי תבטחו נח נפשיה דר' שמעון בן לקיש והוה קא מצטער ר' יוחנן בתריה טובא אמרו רבנן מאן ליזיל ליתביה לדעתיה ניזיל רבי אלעזר בן פדת דמחדדין שמעתתיה אזל יתיב קמיה כל מילתא דהוה אמר רבי יוחנן אמר ליה תניא דמסייעא לך אמר את כבר לקישא בר לקישא כי הוה אמינא מילתא הוה מקשי לי עשרין וארבע קושייתא ומפריקנא ליה עשרין וארבעה פרוקי וממילא רווחא שמעתא ואת אמרת תניא דמסייע לך אטו לא ידענא דשפיר קאמינא הוה קא אזיל וקרע מאניה וקא בכי ואמר היכא את בר לקישא היכא את בר לקישא והוה קא צוח עד דשף דעתיה [מיניה] בעו רבנן רחמי עליה ונח נפשיה
Rabbi Yoḥanan said: "I alone remain of the beautiful of Jerusalem." One who wishes to see the beauty of Rabbi Yoḥanan should bring a new silver goblet from the smithy and fill it with red pomegranate seeds and place a diadem of red roses upon the lip, and position it between the sunlight and shade. That luster is a semblance of Rabbi Yoḥanan’s beauty. Is that so? [Was Rabbi Yoḥanan so beautiful?] But doesn’t the Master say: The beauty of Rav Kahana is a semblance of the beauty of Rabbi Abbahu; the beauty of Rabbi Abbahu is a semblance of the beauty of Jacob, our forefather; and the beauty of Jacob, our forefather, is a semblance of the beauty of Adam haRishon [who was created in the image of God]? And yet Rabbi Yoḥanan is not included in this list. Rabbi Yoḥanan is different from these other men, as he did not have a beard... The Rabbis said to him: Isn’t the Master worried about the evil eye [since you are so beautiful]? Rabbi Yoḥanan said to them: I come from the offspring of Joseph, over whom the evil eye does not have dominion, as it is written: “Joseph is a fruitful vine, a fruitful vine by a fountain [alei ayin]” (Genesis 49:22)...
One day, Rabbi Yoḥanan was bathing in the Jordan River. Reish Lakish saw him and jumped into the Jordan, pursuing him. [At that time, Reish Lakish was the leader of a band of marauders]. Rabbi Yoḥanan said to him: Your strength is fit for Torah study. Reish Lakish said to him: Your beauty is fit for women. Rabbi Yoḥanan said to him: If you return and study Torah, I will give you my sister, who is more beautiful than I am. He accepted upon himself [to study Torah]. He wanted to jump back out of the river to bring back his clothes, but he was unable to return. Rabbi Yoḥanan taught Reish Lakish Bible, and taught him Mishna, and turned him into a great man... [Years past and Reish Lakish and R. Yochanan became chevrutot very close friends. One day, they got in a serious quarrel] Rabbi Yoḥanan was offended, [and in turn] Reish Lakish fell ill. Rabbi Yoḥanan’s sister, came crying to him [begging that he pray for Reish Lakish’s recovery'. She said to him: Do this for the sake of my children. He said to her, “Leave your fatherless children, I will rear them” (Jeremiah 49:11). She said to him: Do so for the sake of my widowhood. He said to her [the rest of the verse], “And let your widows trust in Me.” Ultimately, Rabbi Shimon ben Lakish died. Rabbi Yoḥanan was sorely pained over losing him. The Rabbis said: Who will go to calm [Rabbi Yoḥanan’s] mind? Let Rabbi Elazar ben Pedat go, as his statements are sharp. He went and sat before [Rabbi Yoḥanan.] Regarding every matter that Rabbi Yoḥanan would say, he say to him: [There is a ruling which] is taught that supports you. Rabbi Yoḥanan said to him: Are you comparable to the son of Lakish? The son of Lakish, when I would state a matter, he would raise twenty-four difficulties against me [to disprove my claim] and I would answer him with twenty-four answers, and the halakha by itself would become broadened. And you say: There is a ruling which is taught that supports you. Do I not know that what I say is good?! Rabbi Yoḥanan went around, rending his clothing, weeping and saying: Where are you, son of Lakish? Where are you, son of Lakish? He screamed until his mind was taken from him. The Rabbis prayed and requested mercy on him and his soul rested.
(יח) בשעה שיצא יוסף למלוך על מצרים, היו בנות מלכים מציצות עליו דרך החרכין, והיו משליכות עליו שירין, וקטלין, ונזמים, וטבעות, כדי שיתלה עיניו ויביט בהן, אעפ"כ לא היה מביט בהן.
When Joseph went forth to rule over Egypt, daughters of kings used to look at him through the lattices and throw bracelets, neccklets, earrings, and finger rings to him, so that he might lift up his eyes and look at them; yet he did not look at them...
Here too, the case of Tamar (illegitimate sex to ensure genealogical lineage) is directly compared to the "seduction" of Joseph. Interestingly, both cases are related to differently (Tamar or eshet Potiphar).
כדי לסמוך מעשה תמר, למעשה אשתו של פוטיפר. מה זו לשם שמים, אף זו לשם שמים. דאמר רבי יהושע בן לוי: רואה היתה באסטרולוגין שלה, שהיא עתידה להעמיד ממנו בן, ולא היתה יודעת אם ממנה אם מבתה
To tell you that just as that one [the incident of Tamar] was meant for the sake of heaven, so too this one [the incident of Potiphar’s wife] was meant for the sake of heaven. For she saw through her astrology that she was destined to raise children from him (Joseph), but she did not know whether [they would be] from her or from her daughter. ...
In this text, Potiphar's wife is compared to the savage animal that Jacob suspects for having killed Joseph. Interestingly also, here again a link is drawn to the Tamar narrative.
This view is taken up by Rashi: