Back to Engineering Projects
NLP Research📚 AI Research
GMSAC — Gujarati Sentiment Analysis Corpus & Benchmark
A manually curated Gujarati sentiment analysis corpus with benchmark evaluations across multiple transformer models to support reproducible NLP research for low-resource languages.
Benchmark Results & Metrics
Corpus Size
15k+ Sentences
Fleiss Kappa
0.86 Agreement
Baseline Models
5 Benchmarked
Best F1-Score
0.82 (MuRIL)
01 // Overview
GMSAC provides a benchmark and resource corpus for Gujarati NLP, containing thousands of manually labeled entries. The study evaluates transformer backbones like MuRIL and IndicBERT.
02 // The Problem
Low-resource languages like Gujarati suffer from a lack of labeled datasets, forcing developers to rely on suboptimal machine translation models that miss cultural nuances.
03 // System Architecture
NLP Dataset Benchmarking Pipeline
Data was scraped and annotated by native speakers. Standardized pre-processing scripts clean Gujarati script, feeding into fine-tuned Indic transformer models.
ARCHITECTURAL DATA FLOW:Raw Gujarati Text -> Data Preprocessor -> MuRIL/IndicBERT Encoder -> Sentiment Classifier -> F1 Metrics
Architecture Components:
Scraping & Cleaning Scripts
Human Annotation Guidelines
IndicBERT/MuRIL Fine-tuning Hub
Evaluation Suite
04 // Implementation
Built annotation pipelines using Python and custom tooling. Fine-tuned multilingual BERT variants (mBERT, XLM-RoBERTa, MuRIL) using PyTorch and Hugging Face.
05 // Execution Workflow
11. Scrape text from news portals and social media.
22. Clean noise and format text.
33. Conduct double-blind human annotation.
44. Resolve label conflicts via curator review.
55. Train and evaluate transformer models on splits.
06 // Technology Stack
NLP & Transformers
Hugging FacePyTorchMuRILIndicBERT
Dataset & Scraping
BeautifulSoupPandasScikit-Learn
07 // Technical Challenges
⚠️Standardizing spellings and regional dialects in informal Gujarati social comments.
⚠️Ensuring high inter-annotator agreement among human labelers.
Lessons Learned
- •Pre-training language models on local scripts performs much better than using English-translated inputs.
- •Rigorous cleaning of non-Gujarati characters is vital for training stability.
Future Improvements
- •Expanding dataset size to include speech audio transcripts.
- •Releasing a fine-tuned, lightweight Gujarati model on Hugging Face hub.