---
title: "Tutorial 5: Introduction to Text Analysis"
subtitle: "assemblykor - Korean National Assembly Data for Political Science Education"
output: learnr::tutorial
runtime: shiny_prerendered
---

```{r setup, include=FALSE}
library(learnr)
library(assemblykor)
library(dplyr)
library(ggplot2)
library(tidyr)
library(stringr)
knitr::opts_chunk$set(warning = FALSE, message = FALSE, fig.width = 7, fig.height = 5)
data(bills)
data(speeches)
set_ko_font()
tutorial_options(exercise.eval = FALSE)
```

```{r theme, echo=FALSE}
htmltools::includeCSS(
  system.file("tutorials/css/assemblykor.css", package = "assemblykor")
)
```

## Learning Objectives

**Research Question**: How has the National Assembly's agenda changed over time?

So far we have worked with numeric data (bill counts, wealth, votes). But one of the most important data types in political science is **text**: bill rationales, floor speeches, press releases. Discovering patterns in text is what text analysis does.

You will learn:

- Text preprocessing (tokenization, stopword removal)
- Word frequency analysis and visualization
- TF-IDF: finding distinctive words for each group
- Keyword tracking: agenda change over time
- Analyzing committee discourse with `speeches` data

## 1. Turning Text into Data

To a computer, text is just a string. To analyze it, we need to convert text into **structured data**. The most basic approach is the **bag of words** model: treat text as a set of words and count each word's frequency.

### Preprocessing Flow

```
Raw text -> Tokenization (split words) -> Stopword removal -> Frequency count -> Analysis
```

## 2. Downloading Bill Texts

`get_bill_texts()` downloads rationale text for 60,925 bills (~25MB on first run).

```{r download, exercise=TRUE, exercise.eval=FALSE}
texts <- get_bill_texts()
dim(texts)
head(texts$propose_reason, 2)
```

### Merging with Bill Metadata

```{r merge, exercise=TRUE, exercise.eval=FALSE}
texts <- get_bill_texts()

bill_text <- bills %>%
  inner_join(texts, by = "bill_id") %>%
  filter(scrape_status == "ok") %>%
  mutate(text_length = nchar(propose_reason))

cat("Bills with text:", nrow(bill_text), "\n")
summary(bill_text$text_length)
```

### Text Length Distribution

The length of the rationale text itself is a meaningful variable.

```{r length-dist, exercise=TRUE, exercise.eval=FALSE}
texts <- get_bill_texts()

bill_text <- bills %>%
  inner_join(texts, by = "bill_id") %>%
  filter(scrape_status == "ok") %>%
  mutate(text_length = nchar(propose_reason))

ggplot(bill_text, aes(x = text_length)) +
  geom_histogram(bins = 80, fill = "steelblue", color = "white") +
  scale_x_log10(labels = scales::comma) +
  labs(x = "Rationale Length (characters, log scale)", y = "Count",
       title = "Bill Rationale Text Length Distribution") +
  theme_minimal()
```

```{r quiz-text, echo=FALSE}
quiz(
  question("Given the skewed text length distribution, which summary statistic better represents a 'typical' bill rationale length?",
    answer("Mean, because it uses all data points"),
    answer("Median, because it is robust to extreme values", correct = TRUE),
    answer("Maximum, because it captures the full range"),
    answer("Standard deviation, because it measures spread"),
    correct = "Correct! With right-skewed data, the mean is pulled up by very long texts. The median gives a better sense of the typical case."
  )
)
```

## 3. Keyword Tracking: Agenda Change Over Time

The simplest yet powerful text analysis is tracking **how often specific keywords are mentioned** over time.

```{r keyword-trend, exercise=TRUE, exercise.eval=FALSE}
texts <- get_bill_texts()

bill_text <- bills %>%
  inner_join(texts, by = "bill_id") %>%
  filter(scrape_status == "ok")

keywords <- c("\uc778\uacf5\uc9c0\ub2a5", "\ub514\uc9c0\ud138", "\ud50c\ub7ab\ud3fc", "\uae30\ud6c4", "\ud0c4\uc18c", "\ubc18\ub3c4\uccb4")

keyword_df <- do.call(rbind, lapply(keywords, function(kw) {
  bill_text %>%
    mutate(year = as.integer(format(propose_date, "%Y"))) %>%
    group_by(year) %>%
    summarise(
      mentions = sum(str_detect(propose_reason, kw), na.rm = TRUE),
      total = n(),
      .groups = "drop"
    ) %>%
    mutate(keyword = kw, pct = mentions / total * 100)
}))

ggplot(keyword_df, aes(x = year, y = pct, color = keyword)) +
  geom_line(linewidth = 1) +
  geom_point(size = 1.5) +
  labs(x = "Year", y = "Mention Rate (%)", color = "Keyword",
       title = "Keyword Trends in Bill Rationales (20th-22nd)") +
  theme_minimal()
```

Which keywords surged? Think about what social issues were happening at that time.

> **Caution**: Keyword frequency reflects policy attention, but the same word can mean different things in different contexts. Keyword analysis is an exploratory tool, not a conclusion by itself.

## 4. Tokenization and Stopword Removal

Splitting text into individual words is called **tokenization**.

Korean specificity: Korean attaches particles to words ("law-object", "law-subject", "law-at"), so the same word appears in different forms. Perfect processing requires a morphological analyzer, but for simple analysis, **whitespace-based tokenization** is still useful.

```{r tokenize, exercise=TRUE, exercise.eval=FALSE}
library(tidytext)
texts <- get_bill_texts()

bill_text <- bills %>%
  inner_join(texts, by = "bill_id") %>%
  filter(scrape_status == "ok", assembly == 22)

# Whitespace tokenization (sample up to 5,000 bills)
sample_bills <- bill_text %>%
  slice_sample(n = 5000)

tokens <- sample_bills %>%
  select(bill_id, committee, propose_reason) %>%
  unnest_tokens(word, propose_reason, token = "words")

cat("Token count:", nrow(tokens), "\n")
tokens %>% count(word, sort = TRUE) %>% head(20)
```

### Stopword Removal

The top words after tokenization are particles and function words like "of", "at", "to". These are called **stopwords** and should be removed for analysis.

```{r stopwords, exercise=TRUE, exercise.eval=FALSE}
library(tidytext)
texts <- get_bill_texts()

bill_text <- bills %>%
  inner_join(texts, by = "bill_id") %>%
  filter(scrape_status == "ok", assembly == 22) %>%
  slice_sample(n = 5000)

tokens <- bill_text %>%
  select(bill_id, committee, propose_reason) %>%
  unnest_tokens(word, propose_reason, token = "words")

ko_stopwords <- c(
  "\uc758", "\uc5d0", "\ub97c", "\uc744", "\uc774", "\uac00", "\uc740", "\ub294", "\ubc0f", "\ub4f1",
  "\ud55c", "\ud558\ub294", "\uc5d0\uc11c", "\uc73c\ub85c", "\ub85c", "\uc704\ud55c", "\ub300\ud55c", "\ub530\ub978",
  "\uadf8", "\uc218", "\uac83", "\uc788\ub294", "\ud558\uc5ec", "\ub610\ub294", "\uad00\ud55c", "\uc704\ud558\uc5ec",
  "\uacbd\uc6b0", "\uc788\ub2e4", "\ud574\ub2f9", "\ud544\uc694", "\ud604\ud589", "\ub300\ud558\uc5ec", "\ub54c\ubb38\uc5d0",
  "\ud558\uace0", "\ub300\ud574", "\uc774\ub97c", "\ub530\ub77c", "\ud558\uae30", "\ud558\uba70", "\uc788\uc73c\uba70",
  "\uaddc\uc815", "\uad00\ub828", "\uc815\ud558\uace0", "\ub9c8\ub828", "\uc788\ub3c4\ub85d", "\uc704\ud574"
)

tokens_clean <- tokens %>%
  filter(!word %in% ko_stopwords, nchar(word) >= 2)

tokens_clean %>%
  count(word, sort = TRUE) %>%
  head(20) %>%
  ggplot(aes(x = reorder(word, n), y = n)) +
  geom_col(fill = "coral") +
  coord_flip() +
  labs(x = NULL, y = "Frequency",
       title = "Most Frequent Words (22nd Assembly Bills)") +
  theme_minimal()
```

## 5. TF-IDF: Distinctive Words by Committee

Simple frequency alone cannot distinguish "common words used in all bills" from "words used specifically in one committee."

**TF-IDF (Term Frequency - Inverse Document Frequency)** solves this:

- **TF**: How often does this word appear in a specific group (committee)?
- **IDF**: How **rare** is this word across all groups?
- **TF-IDF = TF x IDF**: Words that are frequent in one group but rare overall = **distinctive words**

```{r tfidf, exercise=TRUE, exercise.eval=FALSE, fig.height=7}
library(tidytext)
texts <- get_bill_texts()

bill_text <- bills %>%
  inner_join(texts, by = "bill_id") %>%
  filter(scrape_status == "ok", assembly == 22) %>%
  slice_sample(n = 5000)

ko_stopwords <- c(
  "\uc758", "\uc5d0", "\ub97c", "\uc744", "\uc774", "\uac00", "\uc740", "\ub294", "\ubc0f", "\ub4f1",
  "\ud55c", "\ud558\ub294", "\uc5d0\uc11c", "\uc73c\ub85c", "\ub85c", "\uc704\ud55c", "\ub300\ud55c", "\ub530\ub978",
  "\uadf8", "\uc218", "\uac83", "\uc788\ub294", "\ud558\uc5ec", "\ub610\ub294", "\uad00\ud55c", "\uc704\ud558\uc5ec",
  "\uacbd\uc6b0", "\uc788\ub2e4", "\ud574\ub2f9", "\ud544\uc694", "\ud604\ud589", "\ub300\ud558\uc5ec", "\ub54c\ubb38\uc5d0",
  "\ud558\uace0", "\ub300\ud574", "\uc774\ub97c", "\ub530\ub77c", "\ud558\uae30", "\ud558\uba70", "\uc788\uc73c\uba70",
  "\uaddc\uc815", "\uad00\ub828", "\uc815\ud558\uace0", "\ub9c8\ub828", "\uc788\ub3c4\ub85d", "\uc704\ud574"
)

tokens_clean <- bill_text %>%
  select(bill_id, committee, propose_reason) %>%
  unnest_tokens(word, propose_reason, token = "words") %>%
  filter(!word %in% ko_stopwords, nchar(word) >= 2)

committee_tfidf <- tokens_clean %>%
  count(committee, word, sort = TRUE) %>%
  bind_tf_idf(word, committee, n)

top_committees <- c("\ubcf4\uac74\ubcf5\uc9c0\uc704\uc6d0\ud68c", "\uad6d\ud1a0\uad50\ud1b5\uc704\uc6d0\ud68c",
                    "\uacfc\ud559\uae30\uc220\uc815\ubcf4\ubc29\uc1a1\ud1b5\uc2e0\uc704\uc6d0\ud68c", "\ud658\uacbd\ub178\ub3d9\uc704\uc6d0\ud68c")

committee_tfidf %>%
  filter(committee %in% top_committees) %>%
  group_by(committee) %>%
  slice_max(tf_idf, n = 8) %>%
  ggplot(aes(x = reorder_within(word, tf_idf, committee), y = tf_idf)) +
  geom_col(fill = "steelblue") +
  facet_wrap(~ committee, scales = "free_y", ncol = 2) +
  coord_flip() +
  scale_x_reordered() +
  labs(x = NULL, y = "TF-IDF",
       title = "Distinctive Words by Committee") +
  theme_minimal() +
  theme(strip.text = element_text(size = 10))
```

Check whether each committee's distinctive words reflect its policy jurisdiction.

```{r quiz-tfidf, echo=FALSE}
quiz(
  question("The word 'patient' appears 500 times in Health Committee bills and 50 times across all other committees. The word 'law' appears 500 times everywhere. Which has a higher TF-IDF for the Health Committee?",
    answer("'law', because it appears 500 times too"),
    answer("'patient', because it is concentrated in the Health Committee", correct = TRUE),
    answer("They have equal TF-IDF since both appear 500 times"),
    answer("Cannot determine without knowing document lengths"),
    correct = "Correct! 'patient' is distinctive to Health, while 'law' is common everywhere. TF-IDF rewards rarity across groups."
  )
)
```

## 6. Discourse Analysis with Speeches Data

The `speeches` data contains full transcripts from the 22nd Assembly's Science & ICT Committee. This enables a different kind of analysis from bill text.

```{r speeches-overview, exercise=TRUE}
cat("Speeches:", nrow(speeches), "\n")
cat("Speaker roles:\n")
table(speeches$role) %>% sort(decreasing = TRUE) %>% head(8)
```

### Legislator vs Bureaucrat Speech Comparison

```{r role-compare, exercise=TRUE, exercise.eval=FALSE, fig.height=5}
library(tidytext)

ko_stopwords <- c(
  "\uc758", "\uc5d0", "\ub97c", "\uc744", "\uc774", "\uac00", "\uc740", "\ub294", "\ubc0f", "\ub4f1",
  "\ud55c", "\ud558\ub294", "\uc5d0\uc11c", "\uc73c\ub85c", "\ub85c", "\uc704\ud55c", "\ub300\ud55c", "\ub530\ub978",
  "\uadf8", "\uc218", "\uac83", "\uc788\ub294", "\ud558\uc5ec", "\ub610\ub294", "\uad00\ud55c", "\uc704\ud558\uc5ec",
  "\uacbd\uc6b0", "\uc788\ub2e4", "\ud574\ub2f9", "\ud544\uc694", "\ud604\ud589", "\ub300\ud558\uc5ec", "\ub54c\ubb38\uc5d0",
  "\ud558\uace0", "\ub300\ud574", "\uc774\ub97c", "\ub530\ub77c", "\ud558\uae30", "\ud558\uba70", "\uc788\uc73c\uba70",
  "\uaddc\uc815", "\uad00\ub828", "\uc815\ud558\uace0", "\ub9c8\ub828", "\uc788\ub3c4\ub85d", "\uc704\ud574"
)

role_tokens <- speeches %>%
  filter(role %in% c("legislator", "minister", "vice_minister")) %>%
  mutate(role_label = case_match(role,
    "legislator" ~ "Legislator",
    "minister" ~ "Minister",
    "vice_minister" ~ "Vice Minister"
  )) %>%
  select(role_label, speech) %>%
  unnest_tokens(word, speech, token = "words") %>%
  filter(!word %in% ko_stopwords, nchar(word) >= 2)

role_tfidf <- role_tokens %>%
  count(role_label, word) %>%
  bind_tf_idf(word, role_label, n)

role_tfidf %>%
  group_by(role_label) %>%
  slice_max(tf_idf, n = 10) %>%
  ggplot(aes(x = reorder_within(word, tf_idf, role_label), y = tf_idf)) +
  geom_col(fill = "steelblue") +
  facet_wrap(~ role_label, scales = "free") +
  coord_flip() +
  scale_x_reordered() +
  labs(x = NULL, y = "TF-IDF",
       title = "Distinctive Words by Speaker Role (22nd Sci & ICT Committee)") +
  theme_minimal()
```

Compare what different language legislators and bureaucrats use in the same committee.

### AI Keyword Speech Patterns

How have AI-related speeches changed over time?

```{r ai-speeches, exercise=TRUE, fig.height=4}
ai_speeches <- speeches %>%
  filter(str_detect(speech, "\uc778\uacf5\uc9c0\ub2a5|AI|\uc5d0\uc774\uc544\uc774")) %>%
  mutate(month = format(date, "%Y-%m"))

ai_trend <- ai_speeches %>%
  count(month) %>%
  mutate(month = as.Date(paste0(month, "-01")))

ggplot(ai_trend, aes(x = month, y = n)) +
  geom_col(fill = "coral") +
  labs(x = NULL, y = "AI-related Speeches",
       title = "AI Discourse Trend in Sci & ICT Committee (22nd)") +
  theme_minimal()
```

## 7. Morphological Analysis: Beyond Whitespace Tokens

Korean is an agglutinative language: particles attach directly to words, so
whitespace tokenization counts "국민이", "국민을", and "국민은" as three
different tokens. **Morphological analysis** splits words into meaningful
units and separates particles and endings.

Installing a morphological analyzer is hard for beginners, so `assemblykor`
ships pre-tokenized morphemes for all 15,843 speeches, produced with the
Kiwi analyzer. Only content morphemes are included: nouns (NNG/NNP), verbs
(VV), adjectives (VA), adverbs (MAG), and foreign words (SL, e.g., "AI").
Verbs and adjectives are lemmatized to dictionary form ("-다").

```{r morph-overview, exercise=TRUE}
tokens_morph <- get_speech_tokens()
head(tokens_morph)
table(tokens_morph$pos)
```

### Top Nouns Without a Stopword List

POS tags replace the hand-made stopword list: filtering to nouns already
removes particles and function words.

```{r morph-nouns, exercise=TRUE, exercise.eval=FALSE, fig.height=5}
tokens_morph <- get_speech_tokens()

# Join speaker metadata; date + speech_order identifies one speech
morph_leg <- tokens_morph %>%
  inner_join(speeches %>% select(date, speech_order, role),
             by = c("date", "speech_order")) %>%
  filter(role == "legislator")

morph_leg %>%
  filter(pos %in% c("NNG", "NNP")) %>%
  count(token, sort = TRUE) %>%
  head(15) %>%
  ggplot(aes(x = reorder(token, n), y = n)) +
  geom_col(fill = "steelblue") +
  coord_flip() +
  labs(x = NULL, y = "Frequency",
       title = "Top Nouns in Legislator Speeches (Morpheme-based)") +
  theme_minimal()
```

```{r quiz-morph, echo=FALSE}
quiz(
  question("With whitespace tokenization, \"말했습니다\", \"말하고\", and \"말하면\" are three different tokens. After lemmatization in get_speech_tokens(), they become:",
    answer("Three different tokens, unchanged"),
    answer("One token: 말하다", correct = TRUE),
    answer("One token: 말"),
    answer("They are removed as stopwords"),
    correct = "Correct! Verbs and adjectives are reduced to their dictionary form (stem + 다), so all inflected forms are counted together."
  )
)
```

## 8. Temporal Comparison: Text Changes Across Assemblies

```{r temporal, exercise=TRUE, exercise.eval=FALSE, fig.height=5}
library(tidytext)
texts <- get_bill_texts()

bill_text <- bills %>%
  inner_join(texts, by = "bill_id") %>%
  filter(scrape_status == "ok", assembly %in% c(20, 22))

ko_stopwords <- c(
  "\uc758", "\uc5d0", "\ub97c", "\uc744", "\uc774", "\uac00", "\uc740", "\ub294", "\ubc0f", "\ub4f1",
  "\ud55c", "\ud558\ub294", "\uc5d0\uc11c", "\uc73c\ub85c", "\ub85c", "\uc704\ud55c", "\ub300\ud55c", "\ub530\ub978",
  "\uadf8", "\uc218", "\uac83", "\uc788\ub294", "\ud558\uc5ec", "\ub610\ub294", "\uad00\ud55c", "\uc704\ud558\uc5ec",
  "\uacbd\uc6b0", "\uc788\ub2e4", "\ud574\ub2f9", "\ud544\uc694", "\ud604\ud589", "\ub300\ud558\uc5ec", "\ub54c\ubb38\uc5d0",
  "\ud558\uace0", "\ub300\ud574", "\uc774\ub97c", "\ub530\ub77c", "\ud558\uae30", "\ud558\uba70", "\uc788\uc73c\uba70",
  "\uaddc\uc815", "\uad00\ub828", "\uc815\ud558\uace0", "\ub9c8\ub828", "\uc788\ub3c4\ub85d", "\uc704\ud574"
)

# Sample from each assembly
sample_both <- bill_text %>%
  group_by(assembly) %>%
  slice_sample(n = 3000) %>%
  ungroup()

era_tokens <- sample_both %>%
  select(bill_id, assembly, propose_reason) %>%
  unnest_tokens(word, propose_reason, token = "words") %>%
  filter(!word %in% ko_stopwords, nchar(word) >= 2) %>%
  mutate(era = ifelse(assembly == 22, "22nd", paste0(assembly, "th")))

era_tfidf <- era_tokens %>%
  count(era, word, sort = TRUE) %>%
  bind_tf_idf(word, era, n)

era_tfidf %>%
  group_by(era) %>%
  slice_max(tf_idf, n = 10) %>%
  ggplot(aes(x = reorder_within(word, tf_idf, era), y = tf_idf, fill = era)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~ era, scales = "free") +
  coord_flip() +
  scale_x_reordered() +
  scale_fill_manual(values = c("20th" = "steelblue", "22nd" = "coral")) +
  labs(x = NULL, y = "TF-IDF",
       title = "20th vs 22nd Assembly: Distinctive Words Comparison") +
  theme_minimal()
```

You can see how the Assembly's priorities have changed at the word level over six years.

## Summary

| Technique | Purpose | Key Function |
|-----------|---------|-------------|
| Keyword tracking | Temporal trend of specific words | `str_detect()` |
| Tokenization | Break text into word units | `unnest_tokens()` |
| Stopword removal | Filter out function words | `filter(!word %in% ...)` |
| Word frequency | Most frequently used words | `count(word)` |
| TF-IDF | Find distinctive words per group | `bind_tf_idf()` |
| Morphological analysis | Separate particles/endings, lemmatize, filter by POS | `get_speech_tokens()` |

> **Note**: Whitespace-based tokenization ignores Korean's agglutinative nature and is best for quick exploration. For serious analysis, use the morpheme tokens from Section 7.

## Exercises

### Exercise 1: Housing Keywords

Calculate the proportion of bills mentioning "housing" keywords by assembly.

```{r ex1, exercise=TRUE, exercise.eval=FALSE}

```

```{r ex1-hint}
# Use str_detect() with keywords joined by |
# Group by assembly and calculate proportions
```

```{r ex1-solution}
texts <- get_bill_texts()

bill_text <- bills %>%
  inner_join(texts, by = "bill_id") %>%
  filter(scrape_status == "ok")

bill_text %>%
  mutate(housing = str_detect(propose_reason, "\ubd80\ub3d9\uc0b0|\uc8fc\ud0dd|\uc784\ub300")) %>%
  group_by(assembly) %>%
  summarise(
    total = n(),
    n_housing = sum(housing, na.rm = TRUE),
    pct = round(n_housing / total * 100, 1)
  )
```

### Exercise 2: Top Speakers

Find the top 10 most frequent speakers among legislators in the `speeches` data, and create a bar chart.

```{r ex2, exercise=TRUE, fig.height=4}

```

```{r ex2-hint}
# Filter role == "legislator", count by speaker_name, take top 10
```

```{r ex2-solution}
speeches %>%
  filter(role == "legislator") %>%
  count(speaker_name, sort = TRUE) %>%
  head(10) %>%
  ggplot(aes(x = reorder(speaker_name, n), y = n)) +
  geom_col(fill = "steelblue") +
  coord_flip() +
  labs(x = NULL, y = "Speech Count",
       title = "Top 10 Most Active Legislators (22nd Sci & ICT Committee)") +
  theme_minimal()
```

### Exercise 3: Speech Length by Role

Compare average speech length between legislators and government officials. Why do they differ?

```{r ex3, exercise=TRUE}

```

```{r ex3-solution}
speeches %>%
  filter(role %in% c("legislator", "minister", "vice_minister")) %>%
  mutate(
    speech_length = nchar(speech),
    role_label = case_match(role,
      "legislator" ~ "Legislator",
      "minister" ~ "Minister",
      "vice_minister" ~ "Vice Minister"
    )
  ) %>%
  group_by(role_label) %>%
  summarise(
    n = n(),
    mean_length = round(mean(speech_length)),
    median_length = median(speech_length)
  )
# Legislators ask short questions; officials give long answers.
# This reflects the committee Q&A structure.
```
