January 30, 2026•by Engineering

guidestreaming

The Problem with Twitter's Native Streaming

Twitter's Streaming API was once the gold standard for real-time social data. Then came the API v2 pricing changes — the Basic tier at $100/month gives you a single filtered stream connection with harsh rate limits, while Enterprise access starts at $42,000/month. For most developers and crypto teams, that's a non-starter.

XCROP offers a practical alternative: use the search endpoint with intelligent polling to build a streaming-like experience at a fraction of the cost. You won't get sub-second latency, but for most use cases — breaking news, token launch monitoring, event tracking — polling every 10-30 seconds is more than enough.

Architecture Overview

The polling-based stream works like this:

1. Search for tweets matching your query every N seconds

2. Deduplicate results using tweet IDs you've already seen

3. Process new tweets through your handler (alerts, database, webhook)

4. Adjust polling interval based on volume and rate limits

This approach gives you the flexibility of real-time monitoring without maintaining a persistent WebSocket connection.

Step 1: Basic Polling Stream

Here's a minimal Node.js implementation:

JAVASCRIPT

const API_KEY = process.env.XCROP_API_KEY;
const BASE_URL = "https://xcrop.io/api/v2";

class TwitterStream {
  constructor(query, options = {}) {
    this.query = query;
    this.interval = options.interval || 15000; // 15 seconds default
    this.seenIds = new Set();
    this.handler = options.onTweet || console.log;
    this.running = false;
    this.maxSeen = options.maxSeen || 10000;
  }

  async search() {
    try {
      const res = await fetch(BASE_URL + "/search", {
        method: "POST",
        headers: {
          "Authorization": "Bearer " + API_KEY,
          "Content-Type": "application/json",
        },
        body: JSON.stringify({
          query: this.query,
          sort: "latest",
          count: 20,
        }),
      });

      if (res.status === 429) {
        console.warn("Rate limited, backing off...");
        await this.sleep(60000);
        return [];
      }

      const json = await res.json();
      return json.data || [];
    } catch (err) {
      console.error("Search error:", err.message);
      return [];
    }
  }

  async poll() {
    const tweets = await this.search();
    const newTweets = [];

    for (const tweet of tweets) {
      if (!this.seenIds.has(tweet.id)) {
        this.seenIds.add(tweet.id);
        newTweets.push(tweet);
      }
    }

    // Prevent memory leak — trim old IDs
    if (this.seenIds.size > this.maxSeen) {
      const arr = Array.from(this.seenIds);
      this.seenIds = new Set(arr.slice(arr.length - this.maxSeen / 2));
    }

    for (const tweet of newTweets) {
      await this.handler(tweet);
    }

    return newTweets.length;
  }

  async start() {
    this.running = true;
    console.log("Stream started: " + this.query);
    console.log("Polling every " + (this.interval / 1000) + "s");

    while (this.running) {
      const count = await this.poll();
      if (count > 0) {
        console.log("Found " + count + " new tweets");
      }
      await this.sleep(this.interval);
    }
  }

  stop() {
    this.running = false;
    console.log("Stream stopped");
  }

  sleep(ms) {
    return new Promise(resolve => setTimeout(resolve, ms));
  }
}

Step 2: Monitor a Keyword

Let's put it to use — monitor mentions of a specific token:

JAVASCRIPT

const stream = new TwitterStream("$SOL pump", {
  interval: 10000, // poll every 10 seconds
  onTweet: async (tweet) => {
    const time = new Date(tweet.created_at).toLocaleTimeString();
    console.log("[" + time + "] @" + tweet.author.username + ": "
      + tweet.text.substring(0, 120));

    // Check for high-engagement tweets
    if (tweet.likes > 100 || tweet.retweets > 50) {
      console.log("  >> HIGH ENGAGEMENT: "
        + tweet.likes + " likes, " + tweet.retweets + " RTs");
    }
  },
});

stream.start();

// Stop after 1 hour
setTimeout(() => stream.stop(), 3600000);

Step 3: Multi-Keyword Monitoring

For tracking multiple topics simultaneously, run parallel streams with staggered intervals to stay within rate limits:

JAVASCRIPT

async function multiStream(queries) {
  const streams = queries.map((q, i) => {
    return new TwitterStream(q.query, {
      interval: q.interval || 15000,
      onTweet: async (tweet) => {
        console.log("[" + q.label + "] @" + tweet.author.username
          + ": " + tweet.text.substring(0, 100));
        // Send to webhook, database, Telegram, etc.
      },
    });
  });

  // Stagger start times to spread rate limit usage
  for (let i = 0; i < streams.length; i++) {
    setTimeout(() => streams[i].start(), i * 3000);
  }

  return streams;
}

// Example: monitor multiple crypto events
const streams = await multiStream([
  { query: "token launch", label: "LAUNCH", interval: 10000 },
  { query: "airdrop announcement", label: "AIRDROP", interval: 20000 },
  { query: "$BTC breaking", label: "BTC-NEWS", interval: 15000 },
]);

Step 4: Smart Polling with Adaptive Intervals

A fixed polling interval wastes credits during quiet periods and misses tweets during spikes. Here's an adaptive version:

JAVASCRIPT

class AdaptiveStream extends TwitterStream {
  constructor(query, options = {}) {
    super(query, options);
    this.minInterval = options.minInterval || 5000;   // 5s minimum
    this.maxInterval = options.maxInterval || 60000;   // 60s maximum
    this.baseInterval = options.interval || 15000;
  }

  async poll() {
    const count = await super.poll();

    // Adjust interval based on activity
    if (count > 10) {
      // High activity — poll faster
      this.interval = Math.max(this.minInterval,
        this.interval * 0.7);
    } else if (count === 0) {
      // No activity — slow down
      this.interval = Math.min(this.maxInterval,
        this.interval * 1.3);
    } else {
      // Normal activity — drift toward base
      this.interval = this.interval * 0.9 + this.baseInterval * 0.1;
    }

    return count;
  }
}

This keeps costs down during quiet hours while ramping up responsiveness when activity spikes.

Python Implementation

For Python users, here's a clean implementation using requests:

PYTHON

import requests
import time
import os

API_KEY = os.environ["XCROP_API_KEY"]
BASE = "https://xcrop.io/api/v2"

def stream_search(query, interval=15, max_iterations=None):
    seen = set()
    headers = {
        "Authorization": "Bearer " + API_KEY,
        "Content-Type": "application/json",
    }
    iteration = 0

    print("Streaming: " + query)
    while max_iterations is None or iteration < max_iterations:
        try:
            r = requests.post(BASE + "/search",
                headers=headers,
                json={"query": query, "sort": "latest", "count": 20})

            if r.status_code == 429:
                print("Rate limited, waiting 60s...")
                time.sleep(60)
                continue

            tweets = r.json().get("data", [])
            new_count = 0

            for tweet in tweets:
                if tweet["id"] not in seen:
                    seen.add(tweet["id"])
                    new_count += 1
                    author = tweet.get("author", {}).get("username", "unknown")
                    text = tweet.get("text", "")[:120]
                    print("  @" + author + ": " + text)

            if new_count > 0:
                print("Found " + str(new_count) + " new tweets")

            # Trim seen set to prevent memory growth
            if len(seen) > 10000:
                seen = set(list(seen)[-5000:])

        except Exception as e:
            print("Error: " + str(e))

        time.sleep(interval)
        iteration += 1

# Monitor SOL mentions for 100 iterations
stream_search("$SOL", interval=10, max_iterations=100)

Use Cases

Breaking News Detection

Monitor keywords like "breaking", "just in", or "ALERT" combined with crypto terms. When a tweet exceeds an engagement threshold within minutes of posting, trigger an alert to your team via Telegram or Discord webhook.

Token Launch Monitoring

Track mentions of new token tickers or contract addresses. Combine with the trending endpoint to cross-reference whether a token is gaining organic traction or being artificially pumped by bots.

Event Tracking

During conferences, AMAs, or protocol launches, monitor the event hashtag in real-time. Aggregate sentiment and key announcements into a live dashboard for your team.

Competitor Intelligence

Track mentions of competitor products or protocols. Set up streams for multiple competitor names and aggregate weekly reports on share-of-voice and sentiment shifts.

Credit Cost Planning

Each search poll costs approximately 300 credits (20 results x 15 credits each). Here's what different polling intervals cost per hour:

IntervalPolls/HourCredits/Hour

10s360108,000

15s24072,000

30s12036,000

60s6018,000

For cost efficiency, we recommend 15-30 second intervals for most use cases. The adaptive polling approach helps further — during quiet periods you might average 45-second intervals, cutting costs significantly.

The Pro plan (2,000,000 credits/month) supports continuous polling on multiple keywords simultaneously. For even higher-frequency monitoring, purchase top-up credit packs for additional capacity.

Comparison: XCROP Polling vs Twitter Streaming API

FeatureTwitter Streaming APIXCROP Polling

Cost$100-$42,000/moFree tier / from $4.9/mo

LatencySub-second5-60 seconds

Concurrent streams1 (Basic)Unlimited

Historical searchLimitedFull search

Setup complexityOAuth, webhooksSingle API key

Uptime dependencyTwitter serversYour polling loop

For crypto intelligence where 10-30 second latency is acceptable, XCROP polling delivers 90% of the value at a fraction of the cost.

Next Steps

• Combine streaming with the KOL timeline endpoint for influencer-specific monitoring

• Add batch user lookup to enrich tweet authors with full profile data

• Use the trending endpoint as a discovery layer to find new keywords to stream

• Build alerting pipelines with webhooks for high-priority events

Real-Time Twitter Data Streaming: Monitor Events as They Happen

The Problem with Twitter's Native Streaming

Architecture Overview

Step 1: Basic Polling Stream

Step 2: Monitor a Keyword

Step 3: Multi-Keyword Monitoring

Step 4: Smart Polling with Adaptive Intervals

Python Implementation

Use Cases

Breaking News Detection

Token Launch Monitoring

Event Tracking

Competitor Intelligence

Credit Cost Planning

Comparison: XCROP Polling vs Twitter Streaming API

Next Steps