Tips and Tricks - Using Web APIs

Sam Erickson
Updated 22 hours ago
0

Key Takeaways

Define the data you need, its required freshness, and whether you'll display it or retain it before choosing an API.
Evaluate rate limits, total quotas, authentication requirements, storage permissions, and pricing together.
Bound pagination and retries; use caching, queues, and webhooks where the provider supports them.
Keep server credentials out of frontend code and plan for expired tokens, outages, and response changes.

To use third-party APIs effectively, start with your application's data requirements, then check whether the provider's access rules, limits, authentication, and costs support them. Those constraints should shape your architecture—not become surprises after you've built the integration.

Getting a successful response is only the first milestone. The useful question is whether you can retrieve, use, and maintain that data reliably at your expected scale.

What Are You Doing With the Data?

Do you want to display a user's latest five posts, or store their posts in a database for later analysis? The first feature might need a short-lived cache and a clear freshness target. For the second, plan a background synchronization process, persistent storage, and a deletion workflow.

Write down these requirements before comparing providers:

Coverage: Which records, fields, users, and historical periods do you need?
Freshness: Must updates appear immediately, or is an hourly refresh acceptable?
Volume: How much history must you import, and how much changes afterward?
Use: Will you display, analyze, combine, redistribute, or retain the data?

If you're creating custom objects from social posts, define your own application model rather than passing the provider's entire response through your stack. Keep the fields you need and document their meaning.

Interface style is a separate decision. Our REST vs SOAP comparison explains protocol differences, but neither approach answers whether a provider permits or economically supports your use case.

Check Storage and Reuse Permissions First

Successful access isn't permission to retain, resell, or repurpose data. Read the specific API's terms and license before designing an archive; Google's API terms, for example, include restrictions on permanent copies and caching.

Record applicable retention deadlines, deletion obligations, attribution requirements, and restrictions on derived uses. Minimize personal data, including copies in logs and backups, and have your privacy or legal reviewer confirm consent requirements and regional or contractual constraints. Don't assume a user's authorization overrides the provider's license.

Understand API Rate Limits, Quotas, and Billing Units

Keep three different constraints separate:

Rate limit: How quickly you may consume a resource, often measured over a time window.
Quota: A total allowance, such as records retrieved or requests allowed during a billing period.
Billing unit: What you pay for, which might be requests, connected accounts, records, or subscriptions.

Check limits for the exact endpoint, account, application, user, and plan you intend to use. Don't assume every provider exposes the same headers: GitHub's rate-limit documentation describes its own response headers and multiple limiting mechanisms.

HTTP 429 Too Many Requests indicates rate limiting, but providers may also use other documented responses. Inspect the response headers and error payload rather than treating every access failure as an expired token.

When supplied, honor Retry-After. The HTTP specification allows its value to be either a number of seconds or an HTTP date. Also inspect provider-specific remaining-quota and reset information, including the documented units.

Coordinate requests through a bounded queue, including across application instances when they share a quota. Use bounded exponential backoff with jitter—a randomized delay—to avoid synchronized retry storms. AWS's retry guidance explains why retries need limits and why stacked retry layers can amplify load.

Fetch Multiple Pages Without Unbounded Scans

Pagination divides a result set into smaller responses. Follow the provider's documented continuation mechanism; for example, GitHub documents pagination links in response headers.

Set maximum pages, records, elapsed time, and concurrency for each job.
Stop when you've collected enough data or the provider indicates there are no more results.
Checkpoint completed work so a restart doesn't repeat the entire import.
Use incremental retrieval where supported, and deduplicate records by a documented stable identifier.

Treat continuation tokens as opaque values. If the API returns a next-page URL, validate its destination before attaching credentials.

Reduce Requests With Caching and Webhooks

Cache only when the provider permits it, and choose freshness based on the feature. Respect applicable cache directives. Where supported, ETag and If-None-Match enable conditional retrieval; an unchanged resource can produce a 304 Not Modified response without a response body. MDN's HTTP caching guide explains these mechanisms.

Don't assume conditional requests are free or quota-exempt. Check the provider's accounting rules, and scope private cache entries to the authorized user or tenant.

For changing data, consider webhooks instead of frequent polling. A webhook lets the provider notify your backend about an event, but introduces an inbound endpoint and delivery handling. Follow the provider's signature-verification procedure, durably enqueue accepted events, and make processing idempotent so duplicate delivery doesn't duplicate effects. GitHub's webhook guidance covers secrets, asynchronous processing, and delivery identifiers.

Plan reconciliation for missed events and use bounded polling when webhooks aren't available.

Choose the Right API Authentication and Authorization Flow

Authentication establishes identity; authorization determines permitted access. An API key, an OAuth access token, and a signed request are not interchangeable integration choices. Use the scheme the provider documents.

OAuth 2.0 is an authorization framework, not a standalone user-authentication protocol. Scopes describe requested access. For user-delegated access, follow the provider's authorization-code flow with PKCE, which helps protect the code exchange; current OAuth security guidance requires PKCE for public clients and recommends it for confidential clients.

For server-to-server access without user delegation, consider the client-credentials grant only when the provider supports it. If you're implementing user login, use the provider's OpenID Connect guidance instead of interpreting an API access token as proof of identity.

Use a maintained library that supports the documented flow, review its security status, request minimum necessary scopes, and plan token expiration, refresh, and revocation. Our explanation of how OAuth works provides a conceptual walkthrough.

Keep Server Credentials Out of Frontend Code

Keep server API keys and client secrets out of browser bundles, source control, URLs, and logs. Put privileged API calls behind an authenticated backend and load credentials through your deployment's secret-management mechanism. Authorize each backend operation; don't create an unrestricted proxy.

Browser-executed code cannot keep a client secret confidential. Provider-designated publishable identifiers are a different category, not permission to expose server credentials. Follow OWASP's secret-management guidance for access controls, auditing, rotation, and revocation. If a secret leaks, revoke or rotate it; deleting the latest source-code copy isn't enough.

Handle API Failures Deliberately

Set a timeout for each attempt and an overall operation deadline. Classify failures before retrying:

Failure	Recommended handling
Invalid request, such as HTTP 400	Fix the request; don't repeatedly send the same invalid input.
HTTP 401	Check credentials. Refresh only through the documented flow; stop or request reconnection if recovery fails.
HTTP 403	Inspect permissions, policy restrictions, and provider-specific throttling details.
HTTP 429	Pause the affected work and honor the provider's retry and reset instructions.
Timeout, network error, or transient server failure	Retry only when safe, within attempt and elapsed-time budgets.

A timeout doesn't prove the provider rejected a write. Retry writes only with documented idempotency guarantees—protection against repeating an operation's effects—or reconcile the outcome first.

Choose a user-facing fallback: show a permitted cached result with its update time, offer retry, or explain temporary unavailability. Don't silently represent an upstream failure as an empty dataset. In an Express backend, route failures through consistent error-handling middleware.

A Bounded Server-Side Request Example

This Node.js example uses built-in fetch and AbortSignal.timeout. Save it as api-read.mjs, configure API_URL as a trusted HTTPS endpoint returning JSON, inject API_TOKEN through your server environment, and run node api-read.mjs. It assumes the endpoint accepts bearer tokens.

The example policy allows three attempts, a five-second timeout per attempt, and a 15-second overall budget. It retries only 429 responses; add the provider's documented minimum delays and reset-header handling before adapting it to a specific API.

import process from 'node:process';
import { setTimeout as sleep } from 'node:timers/promises';

function retryAfterMs(value) {
  if (value === null) return 0;
  const text = value.trim();
  if (/^\d+$/.test(text)) return Number(text) * 1000;
  const timestamp = Date.parse(text);
  return Number.isNaN(timestamp)
    ? 0
    : Math.max(0, timestamp - Date.now());
}

async function readJson(url, token) {
  const deadline = Date.now() + 15_000;

  for (let attempt = 0; attempt < 3; attempt++) {
    const remaining = deadline - Date.now();
    if (remaining <= 0) throw new Error('API deadline exceeded');

    const response = await fetch(url, {
      method: 'GET',
      redirect: 'error',
      headers: {
        Accept: 'application/json',
        Authorization: `Bearer ${token}`,
      },
      signal: AbortSignal.timeout(Math.min(5000, remaining)),
    });

    if (response.ok) return await response.json();

    const status = response.status;
    const minimumWait = retryAfterMs(response.headers.get('retry-after'));
    await response.body?.cancel();

    if (status !== 429 || attempt === 2) {
      throw new Error(`API request failed: HTTP ${status}`);
    }

    const jitter = Math.random() * Math.min(4000, 500 * 2 ** attempt);
    const wait = Math.ceil(minimumWait + jitter);
    if (wait >= deadline - Date.now()) {
      throw new Error('Retry must be deferred beyond this operation');
    }
    await sleep(wait);
  }
}

try {
  const url = new URL(process.env.API_URL ?? '');
  if (url.protocol !== 'https:' || url.username || url.password) {
    throw new Error('API_URL must be a credential-free HTTPS URL');
  }
  const token = process.env.API_TOKEN;
  if (!token) throw new Error('API_TOKEN is required');

  await readJson(url, token);
  console.log('API read succeeded');
} catch (error) {
  console.error(error instanceof Error ? error.message : 'API read failed');
  process.exitCode = 1;
}

If the required delay exceeds your deadline, defer work to a queue or return a fallback; don't shorten the provider's delay. This example doesn't retry network failures, validate an application-specific schema, or coordinate shared quotas. Add those policies at the integration boundary, and avoid duplicating retries already performed by an SDK.

Estimate API Pricing and Other Costs

A free allowance isn't a capacity plan. Check hard quotas, feature restrictions, overage behavior, minimum commitments, and whether a budget alert actually stops spending. Verify whether failed calls, pagination, refreshes, and retries are billable.

Financial-data integrations illustrate why coverage and billing units both matter. Review the data you need in documentation such as Yodlee's account-aggregation guide, then price the required products. Plaid documents one-time, subscription, and per-request billing models; actual pricing depends on your agreement.

For a request-billed API, start with this simplified estimate:

Monthly requests = active users
                 x refreshes per user per day
                 x pages per refresh
                 x days
                 + initial imports, retries, and reconciliation

For illustration, 2,000 users refreshed four times daily, with three pages per refresh over 30 days, produce 720,000 requests before additional work. At a hypothetical $2 per 1,000 requests, that baseline costs $1,440. These are example assumptions, not a provider quote.

Model normal usage and a busy month separately. Include engineering, storage, queues, monitoring, support, and migration costs, and measure actual billable units during your pilot.

Plan for Response Changes and Deprecations

Pin an explicitly supported API version where available and track its retirement notices; GitHub's versioning documentation illustrates why version selection needs an upgrade plan.

Validate required fields and types at the boundary, tolerate unknown fields, and handle optional values deliberately. Test empty pages, malformed responses, expired credentials, throttling, and schema changes. Keep provider-specific parsing behind an adapter so a response change doesn't spread throughout your application.

API Selection and Production Checklist

Before committing, build a small end-to-end pilot and answer these questions:

Fit: Does the API cover the required fields, history, regions, and freshness?
Permission: Are display, analysis, retention, deletion, and downstream uses permitted?
Capacity and cost: Can the plan support initial imports, routine traffic, and peak usage?
Operations: Are documentation, sandbox access, support, status history, and any service-level agreement adequate?
Failure recovery: Have you tested timeouts, credential expiry, throttling, restarts, and missed events?
Observability: Can you track latency, failures, retries, queue age, quota consumption, and billable usage without logging secrets or unnecessary personal data?
Exit: Can you export permitted data and replace the provider without rewriting your application?

Approve the integration only when the pilot demonstrates the required data access within your permission, reliability, and budget constraints. That's a stronger decision than choosing the API with the easiest first request.

FAQ

How do you handle API rate limits?

Coordinate requests through a bounded queue, inspect provider-specific quota and reset information, and honor Retry-After when supplied. Use bounded exponential backoff with jitter, and defer work when the required wait exceeds the operation's deadline.

What is the difference between API authentication and OAuth authorization?

Authentication establishes identity; OAuth 2.0 authorizes limited access to protected resources through access tokens and scopes. For user login, follow an authentication protocol such as OpenID Connect rather than treating an API access token as proof of identity.

How do you keep API keys out of frontend code?

Keep server API keys in a backend secret-management mechanism and expose only authenticated, authorized application operations to the frontend. Don't embed secrets in browser bundles or create an unrestricted proxy to the provider.

Can you store data returned by a third-party API?

Only when the provider's terms, license, and applicable privacy requirements permit your intended use. Check retention periods, deletion duties, caching restrictions, and reuse permissions before storing anything.

How can you estimate third-party API costs?

Identify the provider's billing unit, then estimate usage including pagination, imports, refreshes, retries, and reconciliation. Model normal and peak demand, and include infrastructure, support, and maintenance rather than only the advertised request price.