Methodology
Every source, its terms, its cadence and its limits
This terminal reads 36 primary sources directly from their publishers. Each card names what the source is, what is known about redistributing it, how often the publisher updates it, how often this app re-reads it, the module in this repository that issues the request, and what it cannot tell you. Nothing here is a summary of a vendor datasheet — the 'Issued by' line is a path you can open.
How a figure reaches the screen
traceable to a file- Your browser asks this server
Every request goes to the FastAPI process on your own machine. No request from the page reaches a data publisher directly.
- The server asks the publisher
One outbound HTTPS call per source, from your machine, using the module named on each card below. There is no vendor, no aggregator and no proxy in between.
- The response is cached, sometimes
21 routes in app/main.py go through `cached_route`. Those can replay a previous response when the live call fails, and when they do the payload carries a `cache_state` block saying so.
- The payload names its own condition
A replayed payload sets stale, cache_status and cache_state.age_label. The screen renders the age rather than hiding it. A route outside `cached_route` carries no such stamp at all — this page calls that 'not stamped', not 'fresh'.
- An absent value stays absent
Number(null) is 0 in JavaScript, and this project has put '100% equity financed' on screen for a company that could not be priced. Every formatter in frontend/page_kit.js returns an em dash for null, empty string and NaN, and never a zero.
What this page does not claim
Two things a methodology page is normally expected to carry are absent, on purpose.
not measuredA per-screen dependency map
Item 71 asks which screens depend on each source. This app has no machine-readable screen-to-source registry, and several modules named below (phase5.py through phase8.py) each serve many screens, so a screen-level mapping could only be guessed. What is real is module-level: the 'Issued by' line on every card. A generated screen-to-source registry would close this properly; inventing the mapping would not.
not measuredA single freshness number for the whole terminal
Freshness is per route, not per product, and only the 21 routes that go through `cached_route` carry a staleness stamp at all. The data quality page probes a named sample of those routes live rather than averaging a figure across screens that do not report one.
Entity resolution, and why it refuses
The cross-cutting limitation that belongs to no single source.
Matching a company to a federal contract, a clinical trial, a job board or a patent is a judgement, and this terminal is deliberately biased toward refusing. backend/issuer_resolver.py once accepted 'Appleton Marine' and 'Applewood Towers' as Apple Inc. because the strings share a prefix. The rule is now single-token with a two-character extension — it accepts moderna → modernatx and rejects apple → appleton — and tests/test_issuer_resolver_matching.py locks it. A false reject shows an honest 'no match'. A false accept would put another company's contracts under your ticker, so it is the failure this is tuned against.
Filings & regulatory
6 sourcesFilings index, XBRL company facts, and full-text search across every US registrant.
- Redistribution
- Work of the US federal government. Under 17 U.S.C. §105 such works carry no US copyright — but an individual document can still contain third-party material that does. Filings themselves are authored by registrants.
- Publisher cadence
- Filings appear as they are accepted, through the day. XBRL company facts update behind the filing.
- Re-read by this app
- Full-text search 30 min (TTL_SECONDS, backend/edgar_fts.py). Company facts are cached by route key.
- Issued by
- app/services/sec.py, cf.py, fa.py, backend/edgar_fts.py
Known limitations
- SEC enforces a fair-access policy that requires a declared User-Agent and throttles automated readers.
- Measured here: www.sec.gov/Archives answers 403 to the rotating User-Agent in app/services/http.py — archive fetches go through the dedicated SecArchiveClient in cf.py. data.sec.gov is unaffected.
- Measured here: efts.sec.gov rejects a User-Agent containing a URL or a parenthetical with 403 'Undeclared Automated Tool', even a stable descriptive one.
Weekly ATS and over-the-counter share volume by security.
- Redistribution
- FINRA is a private self-regulatory organisation, not a government body. Its data is NOT public domain and carries FINRA's own terms of use. Redistribution terms are set by the operator. Read them at the linked page before republishing anything taken from it.
- Publisher cadence
- Weekly, with a publication delay that differs by tier — Tier 1 securities sooner than non-Tier 1.
- Re-read by this app
- 6 hours (FINRA_TTL_SECONDS, backend/dark_pool_live.py).
- Issued by
- app/services/finra.py, backend/dark_pool_live.py
Known limitations
- Measured here: the group endpoint ACCEPTS a compareFilters parameter, ignores it, and answers HTTP 200 with an unfiltered market-wide row carrying issueSymbolIdentifier: null. A filtered request that looks like it succeeded can be about the whole market rather than your security.
- Measured here: it content-negotiates on Accept. The shared default header list included text/csv, so it returned CSV under content-type: text/plain until Accept was narrowed to application/json.
- Weekly granularity only. It cannot answer an intraday question about off-exchange volume.
Rules, proposed rules and notices, used to date regulatory events.
- Redistribution
- Work of the US federal government. Under 17 U.S.C. §105 such works carry no US copyright — but an individual document can still contain third-party material that does.
- Publisher cadence
- Every federal business day.
- Re-read by this app
- 5 min for HTTP, 3.5 min for the assembled corpus (BI_HTTP_CACHE_SECONDS / BI_CORPUS_CACHE_SECONDS, app/services/phase8.py).
- Issued by
- app/services/phase8.py
Known limitations
- Indexes the notice, not the underlying agency action. A rule can be in force before or after the date shown.
- Matching a document to a company is entity resolution and can decline to answer — see the note on false accepts below.
Federal and state docket entries behind the litigation views.
- Redistribution
- Judicial opinions are public record; the API, the enhancements and the bulk data carry the Free Law Project's own terms. Redistribution terms are set by the operator. Read them at the linked page before republishing anything taken from it.
- Publisher cadence
- Continuously as dockets are ingested; coverage and lag differ sharply by court.
- Re-read by this app
- 5 min for HTTP, 3.5 min for the assembled corpus (app/services/phase8.py).
- Issued by
- app/services/phase8.py
Known limitations
- Coverage is uneven. An absent docket is not evidence that no case exists.
- PACER-derived entries can lag the court's own record.
Drug, device and enforcement records for healthcare coverage.
- Redistribution
- Work of the US federal government. Under 17 U.S.C. §105 such works carry no US copyright — but an individual document can still contain third-party material that does. openFDA states its data is not for clinical decision-making.
- Publisher cadence
- Dataset-dependent — enforcement reports weekly, other endpoints less regularly.
- Re-read by this app
- 5 min for HTTP, 3.5 min for the assembled corpus (app/services/phase8.py).
- Issued by
- app/services/phase8.py
Known limitations
- Adverse-event records are voluntary reports. Counts are not incidence rates and must not be read as one.
- Company names in the raw records are unnormalised, so issuer matching is lossy.
OFAC SDN and consolidated sanctions entries, screened against issuer and counterparty names.
- Redistribution
- —
- Publisher cadence
- —
- Re-read by this app
- —
- Issued by
- app/services/sanctions.py
No limitation notes have been written for this source yet.
Rates & macro
7 sourcesDaily par yield curve — the risk-free leg of every discounting screen.
- Redistribution
- Work of the US federal government. Under 17 U.S.C. §105 such works carry no US copyright — but an individual document can still contain third-party material that does.
- Publisher cadence
- Daily par yield curve rates each federal business day, after the close of the bond market.
- Re-read by this app
- Up to 7 days for the cached series (CACHE_MAX_AGE_SECONDS, app/services/macro_rates.py); live fetch first, cache only as fallback.
- Issued by
- app/services/macro_rates.py
Known limitations
- Par yields are a curve construction, not traded instruments. Discounting with them is a modelling choice this terminal makes explicit on every screen that does it.
- No intraday points. The curve moves once per business day.
FOMC meeting calendar and balance-sheet releases.
- Redistribution
- Published by a US federal body. Board material is generally free to reuse; Redistribution terms are set by the operator. Read them at the linked page before republishing anything taken from it.
- Publisher cadence
- FOMC calendar is published a year ahead; statements at the scheduled decision times; H.4.1 balance sheet weekly on Thursdays.
- Re-read by this app
- —
- Issued by
- app/services/fed.py, events_calendar.py
Known limitations
- The calendar carries scheduled dates. An unscheduled meeting appears only once the Fed publishes it.
Macro series. The CSV endpoints need no key; an optional FRED key unlocks the JSON API.
- Redistribution
- FRED aggregates series from many originators. Some series carry the ORIGINATING provider's copyright and restrictions, not the Fed's. Redistribution terms are set by the operator. Read them at the linked page before republishing anything taken from it.
- Publisher cadence
- Per series. Each series carries its own release schedule and revision history.
- Re-read by this app
- Up to 7 days for the cached series (CACHE_MAX_AGE_SECONDS, app/services/macro_rates.py).
- Issued by
- app/services/macro_rates.py
Known limitations
- Macro series are revised. A figure read today for a past period may not be the figure that was published at the time.
- The keyless CSV endpoints cover fewer series than the JSON API, which needs SINGULARITY_FRED_API_KEY.
Euro-area policy rates and FX reference rates.
- Redistribution
- The ECB permits reuse of its statistical data with attribution. Redistribution terms are set by the operator. Read them at the linked page before republishing anything taken from it.
- Publisher cadence
- Euro foreign-exchange reference rates once per TARGET business day, in the afternoon CET. Policy rates on decision days.
- Re-read by this app
- —
- Issued by
- app/services/phase6.py
Known limitations
- Reference rates are indicative, published for information. They are not dealable prices and must not be treated as a tradable quote.
- No rates on TARGET holidays, which are not the same as US market holidays.
Cross-border banking and effective exchange rate statistics.
- Redistribution
- BIS statistics carry the BIS's own terms. Redistribution terms are set by the operator. Read them at the linked page before republishing anything taken from it.
- Publisher cadence
- Quarterly for the banking statistics; monthly for effective exchange rates.
- Re-read by this app
- —
- Issued by
- app/services/phase6.py
Known limitations
- Long publication lag — cross-border banking statistics land months after the reference quarter.
- Revisions are routine and are not flagged in the series itself.
Country-level indicators behind the sovereign screens.
- Redistribution
- World Bank Open Data is published under Creative Commons Attribution 4.0 (CC BY 4.0). Attribution is required; a small number of datasets are excluded from that licence.
- Publisher cadence
- Indicator-dependent, most annually, with country coverage filling in over following years.
- Re-read by this app
- —
- Issued by
- app/services/phase7.py
Known limitations
- Country coverage is incomplete and uneven. A missing country-year is common and is rendered as absent, never as zero.
- Indicators are revised and rebased; a series can change shape between reads.
National accounts detail used in the economic calendar context.
- Redistribution
- Work of the US federal government. Under 17 U.S.C. §105 such works carry no US copyright — but an individual document can still contain third-party material that does.
- Publisher cadence
- On the BEA release calendar — GDP in three successive estimates per quarter, personal income monthly.
- Re-read by this app
- —
- Issued by
- app/services/phase6.py
Known limitations
- Successive estimates for the same quarter differ materially. Which vintage you are reading matters and is not always visible in the payload.
Market data
5 sourcesQuotes, OHLC history and fundamentals. Delayed, not a tick feed — see the limits below.
- Redistribution
- An unofficial public endpoint with no published licence for programmatic reuse. Treat any redistribution as prohibited. Redistribution terms are set by the operator. Read them at the linked page before republishing anything taken from it.
- Publisher cadence
- Continuously during market hours, delayed.
- Re-read by this app
- Cached by route key at the API layer; quote responses additionally cached under work/yahoo_quote_cache.
- Issued by
- app/services/yahoo.py, hp.py, phase5.py
Known limitations
- This is not a consolidated tape. There is no bid/ask depth, and the last bar can lag the market by minutes.
- No published contract. The endpoint can change shape or stop answering without notice, which is why app/services/market.py cross-checks against Stooq.
- Adequate for research. Not adequate for pricing a trade — and this product has no order routing.
Independent end-of-day history, used to cross-check the primary quote path.
- Redistribution
- Redistribution terms are set by the operator. Read them at the linked page before republishing anything taken from it.
- Publisher cadence
- End of day.
- Re-read by this app
- —
- Issued by
- app/services/market.py
Known limitations
- End-of-day only, so it can confirm a level but never an intraday move.
- Symbol coverage and suffix conventions differ from the primary quote path; a cross-check miss is not proof of a bad quote.
Index constituents and listing reference data.
- Redistribution
- Redistribution terms are set by the operator. Read them at the linked page before republishing anything taken from it.
- Publisher cadence
- Listing and constituent reference data on Nasdaq's own schedule.
- Re-read by this app
- —
- Issued by
- app/services/phase5.py, splc.py
Known limitations
- Reference data, not prices. Constituent lists lag index changes.
Token supply and unlock schedules for the digital-asset screens.
- Redistribution
- CoinGecko's public API carries its own terms and an attribution requirement; redistribution on the free tier is restricted. Redistribution terms are set by the operator. Read them at the linked page before republishing anything taken from it.
- Publisher cadence
- Continuously.
- Re-read by this app
- Markets 15 min, search 6 h, treasury 6 h (MARKETS_TTL / SEARCH_TTL / TREASURY_TTL, backend/token_supply_live.py).
- Issued by
- backend/token_supply_live.py
Known limitations
- Circulating-supply figures are compiled by CoinGecko from issuer disclosures. They are an aggregator's view, not an on-chain measurement.
- The keyless tier is rate limited; a refused request degrades the affected panel rather than the whole screen.
Bloomberg-operated open security identifiers, used to enrich a symbol with its FIGI, exchange and security type.
- Redistribution
- —
- Publisher cadence
- —
- Re-read by this app
- —
- Issued by
- app/services/symbology.py
No limitation notes have been written for this source yet.
Government & science
9 sourcesFederal contract and grant awards, resolved to the issuer under review.
- Redistribution
- Work of the US federal government. Under 17 U.S.C. §105 such works carry no US copyright — but an individual document can still contain third-party material that does.
- Publisher cadence
- Agencies submit monthly; award records appear and are corrected on a rolling basis.
- Re-read by this app
- 6 hours (TTL_SECONDS, backend/federal_awards_live.py); 5 min for the research corpus path (app/services/phase8.py).
- Issued by
- app/services/phase8.py, backend/federal_awards_live.py
Known limitations
- Measured here: spending_by_award ACCEPTS a recipient_id and returns HTTP 200 with awards for other companies — Humana and Lockheed came back for Microsoft's recipient hash. The echoed identifier must be asserted, not the status code.
- Award data is restated. A contract value read today can differ from the same award next month.
- Mapping an award recipient to a listed issuer is entity resolution — see the note below.
Registered trial phases, sponsors and status changes.
- Redistribution
- Work of the US federal government. Under 17 U.S.C. §105 such works carry no US copyright — but an individual document can still contain third-party material that does. Registered records are supplied by sponsors.
- Publisher cadence
- Continuously, as sponsors register and update records.
- Re-read by this app
- 6 hours (TTL_SECONDS, backend/clinical_trials_live.py).
- Issued by
- backend/clinical_trials_live.py
Known limitations
- Sponsors self-report. Status fields go stale when a sponsor does not update them, and 'active' can mean a record nobody has touched in a year.
- The sponsor name is free text, so issuer matching is lossy and deliberately strict.
Publication metadata behind the research-intensity views.
- Redistribution
- Crossref metadata is openly available; individual works are copyright of their publishers. Redistribution terms are set by the operator. Read them at the linked page before republishing anything taken from it.
- Publisher cadence
- Continuously as members deposit records.
- Re-read by this app
- 5 min for HTTP, 3.5 min for the assembled corpus (app/services/phase8.py).
- Issued by
- app/services/phase8.py
Known limitations
- Metadata only — no full text, and affiliation data is sparse and inconsistent.
- Deposit lag varies by publisher, so recent months are systematically under-counted.
Preprint activity for technology coverage.
- Redistribution
- Per-paper licences vary and are chosen by the author. The API and its metadata carry arXiv's own terms of use. Redistribution terms are set by the operator. Read them at the linked page before republishing anything taken from it.
- Publisher cadence
- Daily announcement cycle on weekdays.
- Re-read by this app
- 5 min for HTTP, 3.5 min for the assembled corpus (app/services/phase8.py).
- Issued by
- app/services/phase8.py
Known limitations
- Preprints are not peer reviewed.
- Coverage is concentrated in physics, maths and computer science. Absence of preprints says little about research activity in other fields.
Forecast and ERA5 reanalysis weather over agricultural and industrial polygons.
- Redistribution
- Open-Meteo publishes free access for non-commercial use with attribution, over data from national weather services which carry their own terms. Redistribution terms are set by the operator. Read them at the linked page before republishing anything taken from it.
- Publisher cadence
- Forecasts several times a day; ERA5 reanalysis with a lag of roughly five days.
- Re-read by this app
- Current 30 min, baseline 30 days (CURRENT_TTL_SECONDS / BASELINE_TTL_SECONDS, backend/crop_climate_live.py).
- Issued by
- backend/crop_climate_live.py, micro_climate.py
Known limitations
- Measured here: the ERA5 path returned HTTP 429. Because concurrent callers join ONE in-flight task per cache key, that single 429 propagated to a whole board until the failure was scoped — the affected part is now degraded and named on its own.
- Reanalysis is a model product, not a station reading. It is a best estimate over a grid cell.
Crowdsourced ADS-B aircraft positions for the corporate-jet screens.
- Redistribution
- Free for non-commercial and research use under OpenSky's own terms. Redistribution terms are set by the operator. Read them at the linked page before republishing anything taken from it.
- Publisher cadence
- Continuously, from crowdsourced receivers.
- Re-read by this app
- States 3 min; geocoding 30 days (STATES_TTL_SECONDS / GEOCODE_TTL_SECONDS, backend/jet_activity_live.py).
- Issued by
- backend/jet_activity_live.py, jet_tracker.py
Known limitations
- Coverage is crowdsourced and therefore geographically uneven — dense over Europe, sparse over oceans and much of the global south. An aircraft that disappears has probably left receiver coverage.
- Anonymous polling is rate limited.
- An aircraft's registered owner is not necessarily its operator on a given flight.
Seismic events on the global intelligence map.
- Redistribution
- Work of the US federal government. Under 17 U.S.C. §105 such works carry no US copyright — but an individual document can still contain third-party material that does.
- Publisher cadence
- Continuously; early magnitudes are automatic and are revised.
- Re-read by this app
- —
- Issued by
- app/services/geo.py
Known limitations
- The first magnitude published for an event is frequently revised within hours.
Wildfire, storm and volcanic event feeds.
- Redistribution
- Work of the US federal government. Under 17 U.S.C. §105 such works carry no US copyright — but an individual document can still contain third-party material that does.
- Publisher cadence
- Continuously, curated from source feeds.
- Re-read by this app
- —
- Issued by
- app/services/geo.py
Known limitations
- Curated event feed, not a sensor feed. An event appears when a curator's source reports it.
Published housing price and inventory panels by metro.
- Redistribution
- Zillow publishes these panels for research use under Zillow's own terms, which require attribution and restrict redistribution. Redistribution terms are set by the operator. Read them at the linked page before republishing anything taken from it.
- Publisher cadence
- Monthly, with the whole history restated on each release.
- Re-read by this app
- 24 hours (CSV_TTL_SECONDS, backend/housing_research_live.py).
- Issued by
- backend/housing_research_live.py
Known limitations
- The full back-history is restated every month, so a value read last month may not match the same cell today. This is a revision, not an error.
- Metro definitions are Zillow's own and do not align with census geographies.
- A model-derived index, not a transaction record.
News & chatter
5 sourcesWorldwide event and tone stream behind the geopolitical layer.
- Redistribution
- GDELT publishes for open use with attribution. Redistribution terms are set by the operator. Read them at the linked page before republishing anything taken from it.
- Publisher cadence
- Every 15 minutes.
- Re-read by this app
- —
- Issued by
- app/services/geo.py, news.py
Known limitations
- Tone and event codes are machine-extracted from news text. They measure coverage and framing, not what happened.
- Heavily skewed toward English-language and online-first publishers.
Public retail message stream, used for chatter volume — never as a signal on its own.
- Redistribution
- Redistribution terms are set by the operator. Read them at the linked page before republishing anything taken from it.
- Publisher cadence
- Continuously.
- Re-read by this app
- 5 min per page (PAGE_TTL_SECONDS, backend/retail_chatter_live.py).
- Issued by
- app/services/news.py, backend/retail_chatter_live.py
Known limitations
- Used for chatter VOLUME only. It is never treated as a signal on its own, and it is trivially manipulable by anyone willing to post.
- Self-declared bull/bear tags are unverified.
Headline feeds for the news hub, attributed per item.
- Redistribution
- Each publisher's feed carries that publisher's copyright and terms. Headlines are shown attributed and linked to the source; the article text is not copied. Redistribution terms are set by the operator. Read them at the linked page before republishing anything taken from it.
- Publisher cadence
- Continuously, per publisher.
- Re-read by this app
- —
- Issued by
- app/services/news.py
Known limitations
- A headline is an editorial judgement, not a fact about a company.
- Feed windows are short — RSS gives recent items only, so this cannot be used as a historical news archive.
Public post search. No account required.
- Redistribution
- Posts are their authors'. The public AppView carries Bluesky's own terms. Redistribution terms are set by the operator. Read them at the linked page before republishing anything taken from it.
- Publisher cadence
- Continuously.
- Re-read by this app
- —
- Issued by
- app/services/news.py
Known limitations
- Public search only — no view of private or blocked content, so the sample is not the whole network.
Public timeline search on mastodon.social.
- Redistribution
- Posts are their authors'; each instance sets its own terms. Redistribution terms are set by the operator. Read them at the linked page before republishing anything taken from it.
- Publisher cadence
- Continuously.
- Re-read by this app
- —
- Issued by
- app/services/news.py
Known limitations
- Reads one instance's public timeline (mastodon.social). It is a slice of a federated network, not the network.
Alternative data
4 sourcesPublic app reviews for the consumer-frustration index.
- Redistribution
- Apple's search API carries Apple's own terms. Reviews are their authors'. Redistribution terms are set by the operator. Read them at the linked page before republishing anything taken from it.
- Publisher cadence
- Continuously.
- Re-read by this app
- Search 6 h, reviews 1 h (SEARCH_TTL_SECONDS / REVIEWS_TTL_SECONDS, backend/app_store_live.py).
- Issued by
- backend/app_store_live.py
Known limitations
- iOS only, and the public review feed is paginated shallowly — this is a recent sample, not the full review corpus.
- Review volume tracks release events more than it tracks product quality.
Public hiring pages, read for headcount direction and stealth pivots.
- Redistribution
- Each board is the hiring company's own page, hosted by the vendor, and carries the vendor's and the company's terms. Redistribution terms are set by the operator. Read them at the linked page before republishing anything taken from it.
- Publisher cadence
- Continuously, as companies post and close roles.
- Re-read by this app
- 1 hour (BOARD_TTL_SECONDS, backend/job_board_live.py).
- Issued by
- backend/job_board_live.py
Known limitations
- Only companies that use these two vendors and leave the board public are visible. Absence of postings is not evidence of a hiring freeze.
- A posting can stay up after a role is filled.
Public repository activity. Unauthenticated requests are rate limited.
- Redistribution
- Repository contents carry each project's own licence. The API carries GitHub's terms. Redistribution terms are set by the operator. Read them at the linked page before republishing anything taken from it.
- Publisher cadence
- Continuously.
- Re-read by this app
- 5 min for HTTP, 3.5 min for the assembled corpus (app/services/phase8.py).
- Issued by
- app/services/phase8.py
Known limitations
- Unauthenticated requests are rate limited to a low ceiling, so this source degrades first under load.
- Public activity only. Most corporate engineering happens in private repositories and is invisible here.
Company reference text on the stock lab node graph.
- Redistribution
- Article text is licensed CC BY-SA 4.0 by the Wikimedia Foundation. Attribution and share-alike apply to any reuse.
- Publisher cadence
- Continuously.
- Re-read by this app
- —
- Issued by
- app/services/stock.py
Known limitations
- Anyone can edit it. It is used for reference text on a node graph, never as a source for a figure.