Skip to main content
← Back to blog
•by Engineering
researchdata-exportguide

Export Twitter Data for Research: Academic and Market Analysis Guide

Building a Research Dataset From Scratch

Most academic and market-analysis projects don't need a firehose — they need a repeatable pipeline that pulls tweets and profiles for a topic, cleans them up, and exports a dataset you can hand to pandas, R, or a lab partner. This guide walks through building exactly that: search collection, batch user lookup, network data, and CSV/JSON export, all in Python. (The reason a from-scratch pipeline is necessary at all traces back to Twitter shutting down the free Academic Research API in early 2023, pushing the official replacement to $42,000/month — out of reach for nearly every academic budget.)

XCROP fills this gap. With credit-based pricing starting at $0 (5,000 free credits) and scaling to $9.9/month for 2M credits on the Pro plan, researchers can collect meaningful datasets without grant-sized budgets.

What You Can Collect

XCROP provides access to the same data researchers need:

• Tweet content — Full text, media, metrics (likes, retweets, replies, views)
• User profiles — Bio, follower counts, account creation date, verification status
• Conversations — Full thread context with parent and reply chains
• Search results — Query-based collection with date and engagement filters
• Follower/following networks — Social graph data for network analysis
• Trending topics — Real-time trend tracking across regions

Old Academic API vs XCROP: A Comparison

FeatureTwitter Academic API (2022)XCROP (2026)
PriceFree (was) / $42k/mo (now)Free tier + $9.9/mo Pro
Monthly tweets10M tweets/mo~100k results on Pro
Full archive searchYesVia search endpoint
User lookupYesSingle + batch (100/call)
StreamingYes (filtered)Polling-based
Rate limits300 req/15min10-60 req/min by plan
Application process2-4 weeks reviewInstant signup
Data formatTwitter API v2 JSONNormalized JSON

XCROP does not offer full-archive search going back to 2006 like the old Academic API did, but for most research use cases — tracking narratives, analyzing engagement patterns, studying communities — the available data window is more than sufficient.

Setting Up Your Research Environment

BASH
pip install requests pandas
PYTHON
import requests
import pandas as pd
import json
import os
import time

API_KEY = os.environ["XCROP_API_KEY"]
HEADERS = {"Authorization": "Bearer " + API_KEY}
BASE = "https://xcrop.io/api/v2"

Collecting Tweets via Search

The search endpoint is your primary tool for building datasets. It supports keyword queries, hashtags, and user filters.

Basic Search with Pagination

PYTHON
def collect_search_results(query, max_results=500):
    """Collect tweets matching a search query with automatic pagination."""
    all_tweets = []
    cursor = None

    while len(all_tweets) < max_results:
        params = {
            "query": query,
            "count": 20,
            "sort": "latest"
        }
        if cursor:
            params["cursor"] = cursor

        response = requests.post(
            BASE + "/search",
            headers={**HEADERS, "Content-Type": "application/json"},
            json=params
        )

        if response.status_code != 200:
            print("Error: " + str(response.status_code))
            break

        result = response.json()
        tweets = result.get("data", [])
        all_tweets.extend(tweets)

        cursor = result.get("meta", {}).get("next_cursor")
        if not cursor:
            break

        time.sleep(1)  # Respect rate limits

    return all_tweets[:max_results]

# Collect tweets about Bitcoin ETF
tweets = collect_search_results("bitcoin ETF", max_results=200)
print("Collected " + str(len(tweets)) + " tweets")

Filtering by Engagement

PYTHON
# Search with minimum engagement thresholds
response = requests.post(
    BASE + "/search",
    headers={**HEADERS, "Content-Type": "application/json"},
    json={
        "query": "ethereum merge",
        "sort": "engagement",
        "count": 20
    }
)

Batch User Lookup

When studying communities or networks, you often need profile data for many users at once. The batch endpoint handles up to 100 users per call:

PYTHON
def batch_user_lookup(usernames):
    """Look up multiple user profiles in a single API call."""
    all_users = []

    # Process in chunks of 100
    for i in range(0, len(usernames), 100):
        chunk = usernames[i:i+100]
        response = requests.post(
            BASE + "/users/batch",
            headers={**HEADERS, "Content-Type": "application/json"},
            json={"usernames": chunk}
        )

        if response.status_code == 200:
            users = response.json().get("data", [])
            all_users.extend(users)

        time.sleep(1)

    return all_users

# Look up crypto KOL profiles
kols = ["CryptoHayes", "DefiIgnas", "Pentosh1", "CryptoCobain", "inversebrah"]
profiles = batch_user_lookup(kols)

for user in profiles:
    print("@" + user["username"] + " — " + str(user["followers"]) + " followers")

Building a Complete Dataset Collector

Here is a production-ready script that collects tweets around a research topic and exports to both CSV and JSON:

PYTHON
import requests
import pandas as pd
import json
import os
import time
from datetime import datetime

API_KEY = os.environ["XCROP_API_KEY"]
HEADERS = {"Authorization": "Bearer " + API_KEY, "Content-Type": "application/json"}
BASE = "https://xcrop.io/api/v2"

class DatasetCollector:
    def __init__(self, output_dir="./dataset"):
        self.output_dir = output_dir
        os.makedirs(output_dir, exist_ok=True)
        self.tweets = []
        self.users = set()

    def search_tweets(self, query, max_results=1000):
        """Collect tweets matching query with full pagination."""
        cursor = None
        collected = 0

        while collected < max_results:
            payload = {"query": query, "count": 20, "sort": "latest"}
            if cursor:
                payload["cursor"] = cursor

            response = requests.post(BASE + "/search", headers=HEADERS, json=payload)

            if response.status_code == 429:
                print("Rate limited. Waiting 60 seconds...")
                time.sleep(60)
                continue

            if response.status_code != 200:
                print("Error " + str(response.status_code) + ": stopping collection")
                break

            result = response.json()
            tweets = result.get("data", [])

            for tweet in tweets:
                self.tweets.append({
                    "tweet_id": tweet.get("tweet_id"),
                    "text": tweet.get("text"),
                    "created_at": tweet.get("created_at"),
                    "author_username": tweet.get("author", {}).get("username"),
                    "author_name": tweet.get("author", {}).get("name"),
                    "likes": tweet.get("metrics", {}).get("likes", 0),
                    "retweets": tweet.get("metrics", {}).get("retweets", 0),
                    "replies": tweet.get("metrics", {}).get("replies", 0),
                    "views": tweet.get("metrics", {}).get("views", 0),
                    "is_retweet": tweet.get("is_retweet", False),
                    "is_quote": tweet.get("is_quote", False),
                    "query": query
                })
                self.users.add(tweet.get("author", {}).get("username"))

            collected += len(tweets)
            cursor = result.get("meta", {}).get("next_cursor")

            if not cursor:
                break

            time.sleep(1)
            print("Collected " + str(collected) + " tweets...")

        return self

    def collect_user_tweets(self, username, max_results=200):
        """Collect a specific user's tweets."""
        cursor = None
        collected = 0

        while collected < max_results:
            params = {"count": 20}
            if cursor:
                params["cursor"] = cursor

            url = BASE + "/users/" + username + "/tweets"
            response = requests.get(url, headers=HEADERS, params=params)

            if response.status_code != 200:
                break

            result = response.json()
            tweets = result.get("data", [])

            for tweet in tweets:
                self.tweets.append({
                    "tweet_id": tweet.get("tweet_id"),
                    "text": tweet.get("text"),
                    "created_at": tweet.get("created_at"),
                    "author_username": username,
                    "author_name": tweet.get("author", {}).get("name"),
                    "likes": tweet.get("metrics", {}).get("likes", 0),
                    "retweets": tweet.get("metrics", {}).get("retweets", 0),
                    "replies": tweet.get("metrics", {}).get("replies", 0),
                    "views": tweet.get("metrics", {}).get("views", 0),
                    "is_retweet": tweet.get("is_retweet", False),
                    "is_quote": tweet.get("is_quote", False),
                    "query": "user:" + username
                })

            collected += len(tweets)
            cursor = result.get("meta", {}).get("next_cursor")

            if not cursor:
                break

            time.sleep(1)

        return self

    def export_csv(self, filename="tweets.csv"):
        """Export collected tweets to CSV."""
        df = pd.DataFrame(self.tweets)
        df = df.drop_duplicates(subset=["tweet_id"])
        filepath = os.path.join(self.output_dir, filename)
        df.to_csv(filepath, index=False, encoding="utf-8")
        print("Exported " + str(len(df)) + " tweets to " + filepath)
        return filepath

    def export_json(self, filename="tweets.json"):
        """Export collected tweets to JSON."""
        unique = {t["tweet_id"]: t for t in self.tweets}
        filepath = os.path.join(self.output_dir, filename)
        with open(filepath, "w", encoding="utf-8") as f:
            json.dump(list(unique.values()), f, indent=2, ensure_ascii=False)
        print("Exported " + str(len(unique)) + " tweets to " + filepath)
        return filepath

    def summary(self):
        """Print dataset summary statistics."""
        df = pd.DataFrame(self.tweets).drop_duplicates(subset=["tweet_id"])
        print("\n--- Dataset Summary ---")
        print("Total tweets: " + str(len(df)))
        print("Unique authors: " + str(df["author_username"].nunique()))
        print("Date range: " + str(df["created_at"].min()) + " to " + str(df["created_at"].max()))
        print("Avg likes: " + str(round(df["likes"].mean(), 1)))
        print("Avg retweets: " + str(round(df["retweets"].mean(), 1)))
        print("Avg views: " + str(round(df["views"].mean(), 1)))


# Usage example
collector = DatasetCollector(output_dir="./bitcoin_etf_dataset")

# Collect from multiple queries
collector.search_tweets("bitcoin ETF approval", max_results=500)
collector.search_tweets("BTC ETF inflow", max_results=300)

# Collect tweets from specific analysts
for analyst in ["EricBalchunas", "JSeyff", "Nate_Geraci"]:
    collector.collect_user_tweets(analyst, max_results=100)

# Export
collector.export_csv("btc_etf_tweets.csv")
collector.export_json("btc_etf_tweets.json")
collector.summary()

Collecting Network Data

For social network analysis, collect follower/following relationships:

PYTHON
def collect_followers(username, max_results=500):
    """Collect follower list for network analysis."""
    followers = []
    cursor = None

    while len(followers) < max_results:
        params = {"count": 20}
        if cursor:
            params["cursor"] = cursor

        response = requests.get(
            BASE + "/users/" + username + "/followers",
            headers=HEADERS,
            params=params
        )

        if response.status_code != 200:
            break

        result = response.json()
        data = result.get("data", [])
        followers.extend(data)

        cursor = result.get("meta", {}).get("next_cursor")
        if not cursor:
            break

        time.sleep(1)

    return followers[:max_results]

# Build a follower overlap matrix
target_users = ["CryptoHayes", "DefiIgnas", "Pentosh1"]
follower_sets = {}

for user in target_users:
    followers = collect_followers(user, max_results=200)
    follower_sets[user] = set(f["username"] for f in followers)
    print("@" + user + ": " + str(len(follower_sets[user])) + " followers collected")

# Calculate overlap
for i, u1 in enumerate(target_users):
    for u2 in target_users[i+1:]:
        overlap = follower_sets[u1] & follower_sets[u2]
        print(u1 + " <> " + u2 + " overlap: " + str(len(overlap)) + " users")

Compliance Considerations

When using Twitter data for research, keep these guidelines in mind:

1. Respect user privacy — Anonymize usernames in published research when studying individuals rather than public figures
2. No redistribution of raw data — Share tweet IDs instead of full tweet objects in published datasets (standard practice in computational social science)
3. Rate limit awareness — Build delays into your collection scripts to avoid hitting limits and wasting credits
4. Data retention — Store only what you need for your research question; delete datasets when the project concludes
5. IRB compliance — If your institution requires IRB review for social media research, XCROP data collection falls under the same frameworks as any API-based collection

This is a quick checklist, not a legal or ethics review — for a full breakdown of legal frameworks (ToS, GDPR, CCPA, copyright), ethical guidelines, and IRB expectations, see [Twitter/X Data for Academic Research](/blog/twitter-data-academic-research).

Credit Budget Planning

Plan your research budget based on the data you need:

Data TypeCredits per ResultPro Plan (2M credits)
Search results15 per tweet~133,000 tweets
User profiles18 per profile~111,000 profiles
Followers/following1 per result~2,000,000 entries
Batch user lookup18 per user~111,000 users
Trending topics100 flat~20,000 calls

For most research projects — a semester-long study collecting 10,000-50,000 tweets — the Pro plan at $9.9/month provides more than enough capacity. Larger longitudinal studies can purchase top-up credit packs for sustained high-volume collection.

What is Next?

• Conversation analysis — Use /tweets/{id}/conversation to study discussion threads and reply dynamics
• Trend tracking — Combine /trending with search to build longitudinal trend datasets
• Sentiment analysis — Feed collected tweets into NLP pipelines for sentiment and topic modeling
• Network visualization — Export follower graphs to Gephi or NetworkX for visualization