Unlock Every Story
Hidden in Your Chats

File Parsing & Format Detection

The first task is reading the exported file and understanding its structure. Different platforms export in completely different formats.

• WhatsApp (.txt): Four regex patterns covering Android, iOS bracket format, and US AM/PM locale. Multi-line messages are handled by checking whether each new line starts with a valid timestamp — if not, it is appended to the previous message.
• Telegram (.json): The messages array is traversed. Each entry's text field may be a plain string OR an array of entity objects — both are flattened by joining entity text values.
• Facebook (.json): Timestamps are Unix milliseconds (timestamp_ms). Facebook double-encodes Unicode as Latin-1 — ChatLens applies decodeURIComponent(escape(str)) to fix garbled text automatically.

Data Cleaning & Feature Engineering

Raw parsed data needs cleaning and enrichment. Derived features are extracted from each timestamp: hour, dayOfWeek, month, year — equivalent to pandas' df['hour'] = df['timestamp'].dt.hour. System messages are tagged and excluded. Messages are classified as text, media, link, or emoji_only.

Exploratory Data Analysis (EDA)

EDA summarises dataset characteristics before modelling. Messages are grouped by hour, weekday, month, and sender using hash-map accumulation — equivalent to df.groupby('hour')['msg'].count(). Rolling averages use a 7-day centred window. Surge detection flags days where count ≥ 2× daily average.

N-Gram Language Analysis

N-grams are contiguous sequences of N words. Bigrams (N=2): "good morning", "thank you". Trigrams (N=3): "I don't know", "happy new year". For each message, adjacent word pairs/triplets are extracted and counted in frequency maps. Bigrams and trigrams that consist only of stopwords are filtered out.

Vocabulary Richness — Type-Token Ratio

TTR = Unique Words ÷ Total Words. Measures linguistic diversity. Group chats typically score 0.1–0.4 because of repeated phrases. Higher per-sender TTR suggests more varied, expressive language. Computed independently for each top-10 sender.

Sentiment Analysis — VADER-Inspired Lexicon

~400 signal words embedded in analytics.js. Each message is tokenised. Positive and negative hits are counted. Score > 0 → Positive; Score < 0 → Negative; Score = 0 → Neutral. A separate emoji sentiment module classifies emojis using a curated positive/negative emoji dictionary — providing a second, independent sentiment signal.

Question & Conversation Starter Analysis

Question detection: Messages containing "?" or starting with question words (who, what, when, where, why, how, is, are, can, could, etc.) are flagged. Ranked per sender to reveal who drives discussions.

Conversation starters: A session begins when the gap since the last message is ≥ 60 minutes. The first sender in each new session is the "starter" — distinguishing initiators from responders.

Burst Analysis & Conversation Pace

Bursts: Sequences of 3+ consecutive messages where each gap < 3 minutes. Total bursts, average length, and max length are computed. Distribution across 4 size buckets (3–5, 6–10, 11–20, 20+) is shown.

Pace score: Total messages ÷ unique hours with activity. Measures intensity independent of date range. A group chatting 500 messages in 10 active hours (pace=50) is more intense than one chatting 1000 messages over 500 active hours (pace=2).

Keyword Co-occurrence & Topic Clusters

Co-occurrence: For each message containing a keyword, all other non-stopword tokens are collected and counted — revealing semantic context.

Topic clusters: Seed word dictionaries define topics (Work, Faith, Family, Money, Tech, etc.). If 2+ seed words appear in the top 30 most-used words, the topic is detected. Score = total frequency of matched seed words.

Conversation Health Score & Gini Coefficient

Composite 0–100 from 4 factors: Participation Balance (Gini-based, 25pts), Response Speed (% < 5 min, 25pts), Sentiment Ratio (pos/total, 25pts), Engagement Consistency (% days with messages, 25pts).