Fine-tune IndoBERT
for Financial Sentiment

Indonesian financial text is its own language. Generic models don't know that.

A stock investor writing "saham BBRI nyangkut terus" and a food reviewer writing "makanannya enak" are both using Bahasa Indonesia. Only one is expressing financial distress. Generic sentiment models can't tell the difference.

Financial social media in Indonesia mixes formal Bahasa Indonesia, financial jargon ("bullish", "cut loss", "koreksi"), slang ("nyangkut" = stuck at a losing price, "cuan" = profit), and English ticker symbols — all in the same sentence. Pre-trained models trained on general corpora fail at this domain boundary.

The question isn't "which model is best for sentiment?" — it's "which model is best for this specific domain?" and "does fine-tuning actually close the gap?" This project answers both.

The fine-tuning pipeline, step by step.

Every text goes through preprocessing (NLTK + spaCy) before hitting the IndoBERT tokenizer. The model is fine-tuned end-to-end — no feature extraction, no frozen layers — with a classification head on top of the [CLS] token.

# Training config
model = AutoModelForSequenceClassification.from_pretrained(
    "indobenchmark/indobert-base-p1",
    num_labels=3,
    id2label={0:"negative", 1:"neutral", 2:"positive"},
)

training_args = TrainingArguments(
    num_train_epochs=3,
    per_device_train_batch_size=16,
    learning_rate=2e-5,
    warmup_ratio=0.1,            # 10% warmup prevents early instability
    weight_decay=0.01,           # AdamW regularization
    metric_for_best_model="f1_macro",
    load_best_model_at_end=True,
)

What four years actually changed — and what it didn't.

My 2021 thesis built a KNN + LSA chatbot. This project fine-tunes a transformer. The architecture is completely different. The fundamental engineering questions are identical.

Three epochs. Each one mattered differently.

Fine-tuning a pre-trained transformer is not "just run more epochs." Learning rate schedule, warmup, and early stopping on validation F1 (not accuracy) are what prevent catastrophic forgetting of pre-trained representations.

Why F1 macro, not accuracy, as the stopping criterion

Accuracy can improve even when the model gets lazier — just predict the majority class more often. Macro F1 penalizes this: every class must be predicted well, not just the most common one. Monitoring F1 on the validation set caught one epoch where accuracy kept rising while neutral-class F1 was already declining.

This is not theoretical. At epoch 4 in an early experiment, accuracy reached 89.1% — but macro F1 dropped to 0.8721. The model had started over-predicting "positive" at the expense of the balanced F1. Early stopping on F1 prevented this regression from becoming the final model.

Papers & resources.

1. [1]Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL 2019.
2. [2]Wilie, B., et al. (2020). IndoNLU: Benchmark and resources for evaluating Indonesian natural language understanding. AACL-IJCNLP 2020. Source of indobert-base-p1.
3. [3]Wolf, T., et al. (2020). Transformers: State-of-the-art natural language processing. EMNLP 2020 (System Demo). HuggingFace Transformers library.
4. [4]Vaswani, A., et al. (2017). Attention is all you need. NeurIPS 2017. Foundational transformer architecture.
5. [5]Sun, C., et al. (2019). How to fine-tune BERT for text classification? CCL 2019. Hyperparameter guidance for downstream task fine-tuning.
6. [6]Loshchilov, I., & Hutter, F. (2019). Decoupled weight decay regularization. ICLR 2019. AdamW optimizer.
7. [7]Honnibal, M., & Montani, I. (2017). spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. Source of id_core_news_sm.
8. [8]Bird, S., Klein, E., & Loper, E. (2009). Natural Language Processing with Python. O'Reilly Media. NLTK Indonesian stopwords corpus.