Experiment 5

Objective: To understand text preprocessing techniques including tokenization, stop word removal, and script validation using NLTK.

Unofficial Journal

View the unofficial journal for reference

Reference Outputs

View the reference outputs for this experiment

Prerequisites

Install Python

Download Python for Windows

Install NLTK

Open your terminal or command prompt and run: pip install nltk

Perform

Open your text editor or IDE (IDLE, VS Code, etc.).
Create a new file named exp2.py.
Paste the code below.
Run the script.

Code

import nltk
from nltk.tokenize import word_tokenize
from nltk import pos_tag

text = input("Enter your text: ")

tokenized = word_tokenize(text)

tagged = pos_tag(tokenized)

print(tokenized)
print(tagged)

Open Colab