1 Introduction
In the context of the digital age, social media has become an important platform for political communication, allowing political leaders to directly interact with the public and disseminate their views and ideas widely through platforms like Twitter. However, this widespread dissemination of information is not always positive. In particular, polarizing discourse, a speech aimed at undermining democratic institutions, encouraging political polarization, and social division, has garnered widespread attention from academia and the public worldwide. For example, Chambers, et al.
[1] explored the relationship between different countries through the analysis of political sentiment on social media, while Jiang, et al.
[2] studied information dissemination and political orientation detection using the Retweet-BERT model. Solovev, et al.
[3] showed that political discourse on social media can exhibit significant disparities based on personal attributes, contributing to our understanding of the dynamics of hate speech directed at politicians. These studies highlight the importance and complexity of social media in current political communication, pointing out that speech poses a potential threat to social and political stability, emphasizing the importance of monitoring and analyzing such speech.
The emergence of models like Generative Pre-trained Transformer (GPT) and Bidirectional Encoder Representations from Transformers (BERT), brings possibilities to automatically identify and analyze polarizing discourse in large-scale text data. Sudheesh, et al.
[4] emphasized the potential of BERT in analyzing sentiment on social media, providing insights into the complex dynamics of online political discourse. Kawintiranon, et al.
[5] introduced PoliBERTweet, a pre-trained language model specifically designed for analyzing political content on Twitter. Keating, et al.
[6] highlighted the role of social media in enhancing democratic participation, yet also notes its contribution to political polarization. These technologies provide researchers with powerful tools to more effectively identify and analyze congressional members' discourse on social media, helping researchers gain a deeper understanding of the nature and impact of political discourse on social media, thereby promoting a better understanding of political polarization and democratic health.
However, as research progresses, some issues arise, such as: How to efficiently obtain and process large amounts of text data when studying political discourse on social media; understanding and analyzing political discourse not only requires dealing with the complexity of language but also considering the subjectivity and ambiguity of speech; researching political discourse on social media also involves ethical and privacy issues, such as how to collect and analyze data without infringing on individual privacy, which is an important issue that researchers must address.
This study seeks to investigate how advanced natural language processing (NLP) models — Specifically large language models (LLMs) such as GPT and BERT — Can be utilized to analyze polarizing discourse among U.S. congressional members on Twitter. By collecting and examining this data, the research aims to uncover the key characteristics, dissemination patterns, and potential impacts of such rhetoric, thereby enhancing our understanding of the relationship between political communication dynamics and social stability.
The structure of the paper is as follows. In Section 2, we review previous research in related fields, including studies on polarizing discourse, the application of natural language processing technology, and the potential of these technologies in political discourse analysis. In Section 3, we introduce the data collection and preprocessing methods, as well as elaborate on the GPT and BERT models used in this study, including model selection, feature extraction, and experimental design. In Section 4, we present the experimental results and discuss the findings to explore the trends and patterns of polarizing discourse among congressional members. In Section 5, we summarize the main findings of this study, discuss the limitations of the research, and explore future research directions, as well as the potential contributions of this study to political discourse oversight.
2 Literatures
As social media becomes an indispensable part of modern political communication, analyzing political discourse on platforms and its potential impact on democracy has become a hot topic in academia. In this regard, Sombatpoonsiri
[7] showed that political leaders' discourse on social media significantly influences voters' political views, especially in environments of political polarization. Johann
[8] pointed out that polarizing discourse is not only found in fringe parties or groups but also frequently occurs in the discourse of mainstream political figures, exacerbating social divisions and instability to some extent. Choi
[9] studied delved further into the construction of polarizing discourses and their impact on practices and public perception, demonstrating how polarizing discourse forms in society and is disseminated through the media. Alizadeh, et al.
[10] examined the political implications of content moderation, reflecting the polarizing nature of such discussions. In examining polarizing discourse, Davis
[11] outlined the online 'anti-public sphere' where radical and unethical democratic discourse norms are routinely flouted. Thompson
[12] found that Republicans are significantly more supportive of polarizing norms and practices, such as violence against political opposition, primarily motivated by racial threat. Limerick
[13] analyzed how populist and anti-populist discourse concerning African-Americans is manifested in US political and media discourses, particularly focusing on the Central Park Five case.
In terms of understanding and analyzing human language, natural language processing techniques, especially GPT and BERT, have shown great potential. GPT relies on large amounts of text data for pretraining and can generate coherent text sequences, while BERT significantly enhances the understanding ability of language models by understanding the complex relationships between vocabulary and their contexts. These technologies have enriched the toolbox for mining and analyzing political discourse on social media in fields such as text classification, sentiment analysis, and question answering
[14, 15]. Kang, et al.
[16] provided a comprehensive review of NLP applications in management research, highlighting advances in addressing NLP tasks and the challenges and opportunities. Lundberg, et al.
[17] investigated the linguistic profile of Twitter trolls, revealing significant differences in the use of language between trolls and genuine personal messages. These findings are crucial for improving automated detection systems for identifying troll accounts. Bowler, et al.
[18] conducted a survey experiment to assess the impact of polarizing rhetoric from President Trump on citizens' democratic attitudes. Their findings suggest significant pushback against polarizing messages, particularly among Democrats.
The application of NLP techniques such as GPT and BERT to analyze political discourse on social media, especially polarizing discourse, provides new methods and perspectives for research. van Vliet, et al.
[19] demonstrated how to use big data analysis and NLP techniques to reflect issues in international relations in real-time through the establishment of a multi-national parliamentary dataset and analysis platform, providing powerful tools for parliamentary discourse analysis. By analyzing congressional members' Twitter discourse, researchers can identify the characteristics of polarizing discourse and further explore its impact on the stability of democratic institutions and citizen political participation. Guo, et al.
[20] analyzed the accident reports from the artificial intelligence generated contents (AIGC) and discussed the regulatory mechanisms and implementation paths after the generation of inappropriate speech, providing new perspectives for political discourse analysis.
In summary, the application of natural language processing technology not only opens new avenues for analyzing political discourse but also poses new challenges and opportunities for understanding and addressing polarizing discourse on social media. By conducting in-depth research on political discourse on social media, especially congressional members' Twitter discourse, researchers can more accurately reveal the dissemination mechanisms of polarizing discourse and its impact on social and political attitudes, thereby promoting a better understanding of political polarization and democratic health.
3 The Model
3.1 Data
To analyze polarizing discourse among U.S. Congressional members on Twitter, we collected tweets from January 1, 2020, to June 30, 2022, using the Twitter API. Our dataset includes tweets from 436 congressional members, including both senators and representatives. We excluded tweets containing non-English text or those consisting solely of URLs or images, resulting in a final dataset of approximately one million tweets.
To ensure data quality and reliability, we performed several preprocessing steps prior to analysis. First, we removed duplicate tweets and retweets to avoid biases due to redundant data. Next, we tokenized the text, removed punctuation, and converted all text to lowercase for consistency. Additionally, we eliminated stop words and applied lemmatization to further standardize the text. Finally, we filtered out subjects unrelated to politics, such as those concerning personal life or entertainment, ensuring the dataset focused solely on political issues.
After data cleaning, the next step was to label the tweets to identify polarizing content. Given the volume of over one million tweets, manual annotation was impractical. Therefore, we utilized a large language model — ChatGPT — for annotation. ChatGPT has demonstrated excellent performance in various NLP tasks, making it well-suited for analyzing large-scale text and identifying patterns in political discourse. However, using ChatGPT for full-scale tagging would have incurred high economic costs (e.g., approximately fanxiexian_myfh50 for labeling 20, 000 samples). To address this, we used the data labeled by ChatGPT as training samples to fine-tune BERT, another high-performance pre-trained language model, and used BERT to label the remaining data. This approach proved to be both effective and cost-efficient.
3.2 Pre-Label with ChatGPT
We utilized the ChatGPT online API to automatically label the sampled dataset of 20, 000 tweets to determine whether each tweet was polarizing. Through natural language processing techniques, the GPT model can comprehend the context of tweets, thereby effectively predicting and labeling their political tendencies. Considering the inherent imbalance in the data, where polarizing remarks are far fewer than democratic ones, we imposed a constraint that the proportion of tweets labeled with polarizing tags should be within 5% of the total dataset to ensure that only obvious polarizing remarks are labeled. Additionally, we introduced four categories for polarizing remarks (ethnic identity, conspiracy theories, public denigration, support for autocracy) and required the GPT to identify them, thereby assisting in improving the accuracy of GPT labeling.
The prompts for labeling Tweeter contents with ChatGPT is shown below.
| text = f"""{content_list[count*40:count*40+40]}""" |
| prompt = f""" |
| Please thoroughly analyze the provided public statements, each one is delimited by triple backticks, and identify any instances of polarizing rhetoric. |
| polarizing rhetoric encompasses public discourse that not only rejects civil discourse but also undermines the principles of democracy while seeking to delegitimize political opponents. It can be classified into four distinct themes: |
| 1. Ethno nationalist identities: emphasizing the superiority of a particular ethnic group and delegitimizing others. |
| 2. Conspiracy theories: spreading unverified information or theories to create political instability. |
| 3. Overt delegitimization: openly rejecting the validity of democratic institutions or processes. |
| 4. Support for autocracy: advocating for concentrated power and absence of political pluralism. |
| Examine each statement closely, determining if any examples of antidemocratic rhetoric exist. If an instance is identified, please specify the corresponding theme from the four listed above. If no such instance is found, label it as 'no theme'. Note, aim to keep the sample size of the marked polarizing statements within 5% |
| Unless a statement has obvious characteristics of polarizing rhetoric, try not to label it as such. |
| Provide your analysis for each statement into newline-delimited |
| JSON format with the following keys: |
| - 'id': a unique identifier for each statement - 'is_anti': a 0-1 value indicating if the statement is anti-democratic or not |
| - 'theme': if the statement is judged as polarizing, provide the corresponding theme. If not, return 'no theme'. |
| For example, the output format should be like this: |
| (*@\{@"id": "0", "is_anti": 0, "theme": "no theme"@*)(*@\}*) |
| (*@\{@"id": "1", "is_anti": 0, "theme": "no theme"@*)(*@\}*) |
| (*@\{@"id": "2", "is_anti": 1, "theme": "ethno nationalist"@*)(*@\}*) |
| ... |
| The text for analysis: |
| '''{text}''' |
| """ |
Table 1 provides an example of these labeled data.
Table 1 Polarizing rhetoric examples |
| polarizing | Example |
| Ethnonationalism identity | Our history, culture reflect the values of a specific heritage. |
| Conspiracy theories | I am concerned about the future of our country because there are many hostile forces around us trying to undermine our stability. |
| Overt delegitimization | Our electoral system has been manipulated and undermined, leading to distortion of election results. |
| Support for autocracy | We need a leader to centralize power and achieve unity and stability in the nation. |
After GPT annotation, we initially randomly sampled 2, 000 instances, of which 53 were genuinely polarizing. GPT successfully identified 47 of these instances, resulting in a recall rate of 88.7%. Subsequently, we manually reviewed 300 instances labeled as polarizing by GPT, of which 284 were accurately identified as polarizing, yielding an accuracy rate of 94.7%. The following are examples of true positives (TP), false positives (FP), and false negatives (FN) in the dataset, as shown in
Table 2.
Table 2 Examples of model predictions |
| Category | Example |
| TP | Its interesting that a certain COVID lab relocated on the 2nd of December 2019... We know that when you move a lab, it disturbs everything... We need a full amp; thorough investigation into the this Lab NOW! RT if you agree. |
| FP | After a year... Schiff still can't find a reliable source to back his ridiculous investigations. |
| FN | Florida lawmakers just passed a blatantly discriminatory anti-LGBTQ+ bill. Legislation designed to dehumanize vulnerable LGBTQ+ kids. And its just one piece of a larger campaign of hatred led by GOP politicians. |
3.3 Label the Whole Dataset with BERT
Regarding the selection of the BERT model, we initially tested and compared Distil-BERT, BERT-base, and BERT-large. After splitting the 20, 000 generated data points from ChatGPT into a training set of 15, 000 data points and a test set of 5, 000 data points, we trained each model and evaluated their performance on the test set using accuracy, recall, and F1 score metrics, as shown in
Table 3.
Table 3 Models performance metrics |
| Model | Distil-BERT | BERT-Base | BERT-Large |
| Accuracy | 0.934 | 0.946 | 0.969 |
| Recall | 0.5 | 0.54 | 0.7 |
| F1 | 0.490 | 0.564 | 0.695 |
Based on the comparison, we chose BERT-large, which exhibited the best performance, to automatically label the remaining approximately 980, 000 unlabeled data points and identify polarizing remarks among them. Through this series of steps, we were able to effectively identify and analyze polarizing discourse within large-scale Twitter data, providing a robust tool and method for studying political discourse.
The annotated data can be found by clicking this link.
The complete dataset is stored in CSV format with a total size of about 3 gigabytes (Gb). It contains 1, 048, 515 samples, each including lawmakers' personal information, the content of their tweets, and an indicator of whether the tweet was polarizing.
4 Results
Using the annotation results from BERT-large, we analyze polarizing rhetoric among congressional members across various factors, uncovering complex interactions between political, experiential, and geographical influences.
4.1 Chi-Square Test of Polarizing Rhetoric
Most notably, there is an evident partisan divide. As shown in
Figure 1, Republican members exhibit the highest proportion of polarizing rhetoric at 1.74%, with the most pronounced emotional resonance compared to Independents at 1.46% and Democrats at 0.94%. This trend suggests that such rhetoric is significantly more entrenched within the Republican party, potentially encouraged, or at least more prominently featured in the party's political discourse. Additionally, we conducted a chi-square test to determine whether there are significant differences in polarizing rhetoric across the parties. The results indicate that
, confirming that significant differences do indeed exist among the parties.
Figure 1 Polarizing ratios by party |
Full size|PPT slide
Delving deeper into the House of Representatives, where a total of 784, 549 statements were analyzed, polarizing rhetoric accounts for 1.30%. In comparison, the Senate, with 263, 966 statements analyzed, exhibits an polarizing rhetoric rate of 1.10%. This suggests that the House is more actively engaged in polarizing discourse. Several factors may underlie this disparity, including the larger representation in the House, which statistically increases the likelihood of such rhetoric, and the presence of potentially more extreme or radical factions. Conversely, the nature of the Senate — With fewer members, longer terms, and higher public profiles — May foster more cautious public discourse.
Figure 2 shows the polarizing rhetoric ratios across different states, indicating that the prevalence of such rhetoric is not uniform across the country. New Jersey has the highest proportion, suggesting that regional political climates and local issues play a major role in shaping congressional discourse. In contrast, states like Delaware and New Hampshire show minimal polarizing rhetoric, highlighting the diverse political cultures across the United States. Notably, North Carolina, the birthplace of Trump's gunman, also ranks high in polarizing rhetoric. The chi-square test result (
) confirms significant variation in polarizing rhetoric among states.
Figure 2 Polarizing ratios by state |
Full size|PPT slide
The temporal trends in the data highlight how external events can significantly influence the prevalence of polarizing rhetoric. As shown in
Figure 3, peaks in such discourse at specific times indicate the impact of real-world events, which may prompt congressional members to engage in polarizing rhetoric. For instance, January sees a notable peak, coinciding with the start of legislative sessions, discussions on the new fiscal year budget, and Martin Luther King Jr. Day. These peaks may reflect discussions on new agendas, political posturing by newly elected members, or controversies surrounding budget allocations and democratic principles. The chi-square test result (
) further confirms the significant temporal variation in polarizing rhetoric.
Figure 3 Polarizing ratios by month |
Full size|PPT slide
In addition, the political careers of congressional members also influence their use of polarizing rhetoric.
Figure 4 shows an inverse relationship between members' tenure and their likelihood of engaging in such rhetoric. Newly elected members are more inclined to make these statements, suggesting that new representatives are more likely to challenge the status quo or use extreme language to solidify their positions and appeal to specific voter bases (
).
Figure 4 Polarizing ratios by days in office |
Full size|PPT slide
Finally, we analyzed the thematic distribution of polarizing rhetoric among Democrats and Republicans, as shown in
Figures 5 and
6. An interesting pattern emerges: While both parties engage in overt delegitimization, Democrats do so more frequently (79.9%) compared to Republicans (66.1%). Conversely, conspiracy theories constitute 19.2% of polarizing rhetoric among Republicans, whereas among Democrats, this figure is only 10.8%.
Figure 5 Proportion of different themes in Democratic polarizing rhetoric |
Full size|PPT slide
Figure 6 Proportion of different themes in Republican polarizing rhetoric |
Full size|PPT slide
In summary, polarizing rhetoric in Congress is shaped by various factors, including partisan divisions, chamber dynamics, regional cultures, external events, and members' tenure. These influences provide key insights into the state of democratic discourse within the legislature. Understanding these patterns is essential for identifying which groups are more prone to polarizing rhetoric and predicting its occurrence. It also emphasizes the importance of supporting new members to foster a more democratic approach.
4.2 Regression Analysis of Polarizing Rhetoric
We constructed a logistic regression model to examine the factors influencing polarizing rhetoric. The model includes several independent variables, such as party affiliation, days in office, state, and month. To process categorical variables, including state and party, we avoided the traditional one-hot encoding approach, which converts these variables into dummy variables. Using one-hot encoding would generate a large, sparse matrix (if all class variables are used in one-hot encoding, the resulting data contains many zeros) that could negatively impact the accuracy of regression results, especially given the asymmetric nature of the sample (polarizing instances comprise only 1.5% of the total sample). Instead, we processed the categorical variables by calculating the proportion of polarizing rhetoric within each state and party, and used these proportions to represent the characteristics of the variables. We then conducted logistic regression analysis using a statistical package in Python. The results of the logistic regression are presented in
Table 4.
Table 4 Logistic regression for polarizing rhetoric |
| X | Coef. (Std. Error) |
| party | 68.2875*** (2.239) |
| state | 64.1050*** (1.181) |
| month | 0.0174*** (0.003) |
| daysInOffice | 7.076 10** (3.35 10) |
| 1048515 |
| 0.02818 |
Table 4 reveals several important findings. First, the F-test indicates that the overall regression equation is significant at the 0.001 level. Additionally, the beta test shows that each coefficient is statistically significant. Thus, despite the low goodness of fit (
), the regression equation remains explanatory. Note that the low
is due to the many factors influencing polarizing rhetoric that are not fully captured. However, the factors identified in our model do have a significant impact.
The analysis uncovers complex interactions among party affiliation, geographic region, time, and tenure. First, party affiliation plays a dominant role, with a regression coefficient of 68.2875, suggesting that it is the strongest predictor of polarizing rhetoric. As previous findings indicate, Republican members show a higher rate of polarizing rhetoric (1.74%) compared to Independents and Democrats, underscoring its deeper entrenchment in Republican discourse.
Second, geographic location is also a significant factor. The coefficient for state-level polarizing rhetoric is 64.1050, indicating that regional political climates and local issues substantially influence congressional discourse, which is consistent with earlier observations of considerable variation across states.
Third, the month variable has a negative coefficient of 0.0174, implying a slight decrease in polarizing rhetoric over time. This aligns with previous findings that show peaks in January, suggesting that members are more likely to engage in such rhetoric at the beginning of legislative sessions or during budget discussions, potentially as a form of political posturing.
Lastly, the coefficient for days in office, though small, is statistically significant, suggesting a decline in the likelihood of polarizing rhetoric as members' tenure increases. Newly elected members appear more likely to use such rhetoric, possibly to challenge the status quo or appeal to their voter base.
In summary, the regression analysis results align with those of the earlier chi-square test. Given the higher prevalence of polarizing rhetoric and conspiracy theories within the Republican Party compared to the Democratic Party, along with the high rate of polarizing speech in North Carolina, it is plausible to hypothesize that increased polarizing rhetoric, particularly involving conspiracy theories, may have contributed to incidents of violence, such as the assassination attempt on President Trump.
5 Conclusion
We constructed a corpus of annotated tweets from U.S. congressional members and trained a BERT-based prediction model using a subset of this data annotated with ChatGPT. To our knowledge, this study is among the first to employ LLMs for analyzing polarizing patterns in political tweets, introducing a novel approach to studying political communication. The model was applied to one million congressional tweets from January 2020 to June 2022, providing a detailed analysis of sentiment dynamics and rhetorical patterns among U.S. lawmakers on Twitter. By examining the distribution of rhetoric across party lines, time periods, and regions, we explore potential factors contributing to heightened political tension and division.
Our findings indicate that party affiliation, geographic region, timing of political events, and lawmakers' tenure in office all influence the likelihood of employing rhetoric that may contribute to polarization. Party and regional differences emerged as the most significant factors. Polarizing rhetoric was observed more frequently during key periods, such as the annual budget discussions in January or transitions of newly elected members into office. Additionally, the types of rhetoric varied between parties; for example, certain discourse patterns, such as skepticism of institutional frameworks, were more prevalent among specific groups.
These findings contribute to a broader understanding of political communication in the social media era. To foster a more constructive discourse environment, it is essential for lawmakers to promote transparency and encourage respectful engagement. Social media platforms also play a critical role by improving their content monitoring systems to identify potentially harmful rhetoric while upholding balanced moderation standards. Furthermore, empowering citizens to critically evaluate discourse by considering factors such as lawmakers' party affiliation, regional context, and tenure may help mitigate the influence of divisive narratives.
Despite its contributions, this study has certain limitations. For instance, the analysis relies on publicly available social media data, which may not fully capture all dimensions of political discourse, including private communications or less visible platforms. Additionally, while natural language processing techniques are powerful, they still face challenges in handling linguistic nuances, such as subjectivity and ambiguity.
Future research could expand on these findings by exploring discourse across multiple social media platforms and analyzing additional formats, such as video and audio content. Further investigations into the relationships between lawmakers' backgrounds, party dynamics, and regional characteristics may uncover more nuanced motivations and communication strategies. Advancing LLMs to better understand and predict the societal impacts of divisive rhetoric will also be a valuable direction for future research.
{{custom_sec.title}}
{{custom_sec.title}}
{{custom_sec.content}}
PDF(184 KB)
Figure 1 Polarizing ratios by party
