Backend-Kisum-MusicData — Analyst
Related: Promoters API · Data ownership
Machine calls accept the existing static key and Auth-issued X-Service-Token during the additive service-token rollout.
Backend-Kisum-MusicData — Analyst (artist_data)
Section titled “Backend-Kisum-MusicData — Analyst (artist_data)”Purpose
Section titled “Purpose”Centralize on-demand refresh of promoter-facing artist audience metrics into Mongo data_analyst.artist_data. No batch crontab — rows are created/updated only when a consumer requests stats for that artist_id.
Backend-Kisum-AI consumes this module through mcp-kisum-data on port 3011.
That MCP exposes exactly 40 read-only tools across analyst routes, chart
families, platform artist profiles, catalogs, social profiles and Bandsintown
events. It authenticates with MUSICDATA_INTERNAL_API_KEY; it does not query
MusicData MongoDB or third-party providers directly.
Platform tools receive each platform’s external artist ID. Only the four
/analyst/artists/{artistId}/* tools receive the canonical numeric Kisum
artist ID. YouTube channel IDs and YouTube Charts artist mids are different
identities and must never be interchanged.
Data ownership
Section titled “Data ownership”| Store | Key | Writer | Readers |
|---|---|---|---|
data_analyst.artist_data | artist_id (int32, kisum_data.artists_v2.id) | MusicData (GET /analyst/artists/:id/stats) when Promoters hybrid is enabled; else Promoters local refresh | Promoters GET /artists/:id/stats (fast path), agencies roster metrics, event-intelligence |
data_analyst.artist_popularity | artist_id + platform + date + source | MusicData (GET /analyst/artists/:id/popularity) on-demand backfill/incremental refresh | Same route (read path); Promoters GET /api/artists/:id/popularity/:platform (summarized BFF) |
Endpoints (machine auth)
Section titled “Endpoints (machine auth)”MUSICDATA_INTERNAL_API_KEY through its supported legacy headers, or an Auth-issued service token through X-Service-Token. Browser user tokens are not machine credentials.
| Method | Path | Behavior |
|---|---|---|
GET | /analyst/artists/{artistId}/stats | Load artist_data; if updatedAt within 7 days and no ?refresh=true, return mapped stats. Else 4-tier refresh (Viberate → Songstats → Soundcharts → native MusicData platforms), upsert Mongo, return flat stats in data. |
GET | /analyst/artists/{artistId}/stats/health | { exists, fresh, updatedAt, last_refresh_source, last_refresh_error } |
GET | /analyst/artists/{artistId}/chart-presence | Scan Mongo charts Kworb collections for placements of the Kisum artist. Resolves kisum_data.artists_v2 → artist_id, name, platforms.spotify.id; matches per collection (ID, slug, or normalized name in nested data[]). Returns data.charts[] grouped by chart family. |
GET | /analyst/artists/{artistId}/popularity | Mongo SoT: data_analyst.artist_popularity. Always returns stored items[]. On read: backfill from Soundcharts when no rows, or max(date) older than 60 days, or ?refresh=true. Full backfill 2020-01-01→today; incremental from max(date)→today. Query: platform (default spotify), optional read filters startDate/endDate. 404 when artist or external_data.soundcharts_uuid missing. A Soundcharts failure is reported in meta.last_refresh_error and only errors when nothing is stored. |
Query: ?refresh=true on stats forces external refresh (same as Promoters bypass_cache).
Popularity vs Promoters: MusicData returns stored items[] plus meta (Soundcharts refresh cost/debug). Promoters GET /artists/{id}/popularity/{platform} proxies this route (no direct Soundcharts), then summarizes into yearly / peak / current for the Rankings UI. See MusicData package doc.
Promoters integration
Section titled “Promoters integration”Env on Backend-Kisum-Promoters:
MUSICDATA_BASE_URL—https://api.kisum.io/data(prod) /http://localhost:3808(dev)MUSICDATA_INTERNAL_API_KEY— matches MusicData gate
Client: services/musicdata.client.js → ensureArtistStatsFresh(artistId, { refresh }).
Flow: Promoters serves fresh Mongo rows directly; delegates refresh to MusicData when stale/missing/bypass.
Outbound — Artists countries (YouTube charts)
Section titled “Outbound — Artists countries (YouTube charts)”MusicData does not own the country list. For GET /youtube/charts:
| Item | Detail |
|---|---|
| Purpose | Validate countryCode before scraping YouTube Charts |
| SoT | Backend-Kisum-Artists countries table |
| Call | Machine GET /api/v1/countries?iso2s={iso2}&limit=1 with X-Internal-API-Key |
| Env | ARTISTS_INTERNAL_BASE_URL + ARTISTS_INTERNAL_API_KEY on MusicData |
| Param | Canonical: countryCode (ISO2 or global). Legacy alias: regionCode |
| Special | countryCode=global is a YouTube Charts special value — allowed without Artists |
| Fail mode | Unset Artists env, upstream error, or unknown ISO2 → chart request rejected (fail-closed) |
Shazam charts country slug (local JSON)
Section titled “Shazam charts country slug (local JSON)”Shazam chart URLs need a country name slug (not only ISO2 existence). MusicData maps ISO2 → slug from local json/countries.json via src/services/shazam/countriesData.js (in-process, cached; no HTTP to /shazam/countries, no Artists call). global → world. Artists remains the platform country SoT for validation elsewhere (e.g. YouTube charts); Shazam URL naming stays on the local list because it matches Shazam’s path shape.
Shazam surface (what remains)
Section titled “Shazam surface (what remains)”Kept: chart routes (/shazam/charts*), plus /shazam/countries, /shazam/cities, /shazam/genres.
Removed (2026-07-30): /shazam/search, /shazam/artists/*, /shazam/albums/*, /shazam/songs/* — Shazam closed the upstream AMAPI endpoints those handlers called. Chart track href values now point at public https://www.shazam.com/song|artist/... pages, not MusicData detail routes.
Shazam charts track formatting (scrape only)
Section titled “Shazam charts track formatting (scrape only)”GET /shazam/charts/* formats each track from the chart HTML scrape only (song id/name, artist id/name, cover). Do not call Shazam AMAPI for enrichment (returns 405).
Missing scrape artistId → return artist name only; do not block the chart on enrichment.
Cover art: Chart HTML often serves absolute Apple CDN URLs (https://is1-ssl.mzstatic.com/...). Do not prefix https://www.shazam.com onto absolute URLs (that produced shazam.comhttps://…). Only relative paths get the Shazam host.
Same rule as country slug: no MusicData loopback inside MusicData.
TikTok user lookup (/tiktok/user/:id)
Section titled “TikTok user lookup (/tiktok/user/:id)”| Input | Mode | Provider |
|---|---|---|
Digits only (e.g. 6881…) | source=id (auto) | RapidAPI tiktok-scraper7 /user/info — requires active subscription |
Handle (e.g. taylorswift) | source=username (auto) | RapidAPI tiktok82 getProfile → alt RapidAPI → HTML scrape |
Override with ?source=id or ?source=username. Scrape needs a username, not a numeric id. If id API returns “not subscribed”, use the @handle instead.
Apple Music charts (/charts/applemusic)
Section titled “Apple Music charts (/charts/applemusic)”Live Apple charts use Apple Marketing Tools RSS most-played (rss.marketingtools.apple.com). There is no separate live /charts/itunes route — that path was removed (BREAKING) because it duplicated this feed. For historical Kworb iTunes scrapes, use crontab/charts.kworb_* Mongo data, not a browser /charts/itunes API.
TikTok charts (/charts/tiktok)
Section titled “TikTok charts (/charts/tiktok)”TikTok music charts are curated Spotify playlists (not TikTok Login / Display API).
| Call | Behavior |
|---|---|
GET /charts/tiktok | List playlists from Spotify curator user (in-process spotifyApi + catalog token) |
GET /charts/tiktok?chartId=… | Playlist detail |
GET /charts/tiktok?chartId=…&ext=tracks | Playlist tracks page (limit / offset) |
Do not HTTP-loopback to /spotify/charts/tiktok or SPOTIFY_HOST from inside MusicData (machine-only auth → 401). Alias: GET /spotify/charts/tiktok uses the same Spotify curator list in-process.
If Spotify returns 403 after a successful client-credentials grant, check the Spotify Developer app owner account: Development Mode now requires an active Spotify Premium subscription on the app owner. After upgrading, wait a few hours before retrying. This is upstream Spotify, not MusicData auth.
Genius query params (MusicData ↔ Genius API)
Section titled “Genius query params (MusicData ↔ Genius API)”Genius detail routes need numeric IDs (GET /artists/:id, /songs/:id, …). MusicData helpers:
| Route | id meaning |
|---|---|
/genius/artists | Numeric artist id or artist name (resolved via Genius /search) |
/genius/songs | Numeric song id or song title (resolved via /search) |
/genius/albums, /annotations, /users | Numeric id only — use /genius/search?q= first |
/genius/search | q (or id alias) |
Errors must never return axios config / Authorization (token leak). Shared: src/services/genius/helpers.js → safeGeniusError.
Last.fm query params (MusicData ↔ Last.fm API)
Section titled “Last.fm query params (MusicData ↔ Last.fm API)”Shared helper: src/services/lastfm/queryParams.js. Docs often use id as a name alias; Last.fm native params are artist / album / track / tag / mbid / country.
| Route family | Required | Notes |
|---|---|---|
| Artist info / similar / top* | id or artist, or mbid | id = artist name |
| Artist / album / track search | id or name param | Encoded before upstream call |
| Album info / top tags | artist + (album or id), or mbid | id = album name |
| Track info / similar / top tags | artist + (track or id), or mbid | id = track name |
| Tag top* / weekly chart list | tag or id | id = tag name; optional limit/page |
| Geo top artists / tracks | country or id | Full English country name (e.g. Spain), not ISO2 |
| Chart top artists / tags / tracks | optional limit, page | Worldwide only — no country on Last.fm chart methods |
chartWeeklyArtist / newReleases | scrape helpers | Not Last.fm JSON API; page (+ ext for releases) |
Missing required params → 400. Last.fm error 6 after a valid param map means upstream has no match.
Implementation (MusicData)
Section titled “Implementation (MusicData)”src/services/analyst/mongo.js—data_analyst,kisum_data,data_spotify,charts.kworb_spotify_listenerssrc/services/analyst/artistDataMapper.js— DB ↔ API mapping (sync with Promotersutils/artist-data-stats.utils.js)src/services/analyst/artistStatsRefresh.js— external provider pipeline (tiers 1–3 + native tier 4)src/services/analyst/platformNativeStats.js— per-field native gap-fill (in-process MusicData modules only)src/services/analyst/artistStatsService.js— freshness + upsert orchestrationsrc/services/analyst/chartPresenceService.js— Kworb chart presence acrosschartsDB (registry of per-collection lookups)src/services/analyst/chartPresenceMatch.js— normalized artist name / slug matching for Kworb rowssrc/services/analyst/artistPopularityService.js— Mongo SoT read + on-demand Soundcharts refreshsrc/services/analyst/soundchartsPopularity.js— Soundcharts popularity paging (fetchAllSoundchartsPopularityPages)src/services/analyst/soundchartsStats.js— Soundcharts current stats + UUID resolve helper
Artist popularity (GET /analyst/artists/{artistId}/popularity)
Section titled “Artist popularity (GET /analyst/artists/{artistId}/popularity)”Purpose: Machine-facing popularity time series. Mongo data_analyst.artist_popularity is the source of truth — the route always returns stored points. Soundcharts is used only to backfill or refresh on read.
Refresh rules (on-demand, no crontab):
| Condition | Soundcharts fetch window |
|---|---|
| No Mongo rows | 2020-01-01 → today (paginated, upsert per date) |
max(date) older than 60 days | max(date) → today |
| Fresh within 60 days | Skip Soundcharts |
?refresh=true | Same as stale (force refresh) |
Mongo document: { artist_id, platform, date, value, source: 'soundchart', soundcharts_uuid, updatedAt } — unique on { artist_id, platform, date, source }.
Response: data.items[] from Mongo; meta.fresh, meta.lastDate, meta.refreshed, meta.last_refresh_error.
Soundcharts failure behaviour (since 2026-08-06): a failed refresh degrades, it does not fail the request. When Mongo already holds a series, the stored points are returned with meta.refreshed: false and meta.last_refresh_error = { status, message }. Only an artist with no stored series propagates the error. Upstream 401 / 403 / 404 / 429 keep their status (everything else becomes 502) and the Soundcharts message is appended, e.g. Soundcharts popularity request failed (429): You have reached your monthly available quota of 0 requests. Same rule the stats route already followed with last_refresh_error.
UUID resolution: kisum_data → external_data.soundcharts_uuid (not data_platforms).
Env: SOUNDCHARTS_APP_ID, SOUNDCHARTS_API_KEY, DB_MONGO_PROD_KISUM.
Chart presence (GET /analyst/artists/{artistId}/chart-presence)
Section titled “Chart presence (GET /analyst/artists/{artistId}/chart-presence)”Purpose: Answer “which Kworb chart families does this artist appear in?” without the browser querying each charts.kworb_* collection.
Match strategy (per collection):
| Collection | Match |
|---|---|
kworb_spotify_top_artists | artist_id or spotify_id |
kworb_spotify_listeners | artist_id or spotifyId |
kworb_global_artists | artist_id, kworbSlug (from name), or name |
kworb_current_charts | artist_id or flat artist (#1 rows) |
kworb_spotify_country_charts | data[].artist_id, data[].spotifyArtistId, or normalized data[].artist |
billboard | data[].artist_id, data[].artist[].id (Kisum int32), or data[].artist[].name |
| radio / Spotify period / YouTube nested charts | data[].artist_id or data[].artist (primary artist before feat.) |
Crontab enrichment: All Kworb and Billboard chart scrapers (/crontab/kworb, /crontab/billboard) set artist_id on each row (kisum_data.artists_v2.id, or null). Lookup: Spotify platform id when present, else case-insensitive name / sortName. Shared helper: Backend-Kisum-MusicData/src/services/crontab/kworb/artistLookup.js.
Response (data): artistId, name, spotifyId, chartCount, families[], scrapedAtMax, charts[] with source, label, family, rank / entryCount / placements[].
Performance note: Name-based collections use regex pre-filter + row filter; for high traffic consider crontab-built charts.artist_chart_presence index (future).
Refresh pipeline (4 tiers)
Section titled “Refresh pipeline (4 tiers)”On external refresh, metrics are merged per field. A positive value from an earlier tier is never overwritten by a later tier.
| Tier | Source | When |
|---|---|---|
| 1 | Viberate fanbase-distribution | external_data.viberate_uuid present |
| 2 | Songstats /enterprise/v1/artists/stats | gaps after tier 1 |
| 3 | Soundcharts current/stats + YouTube audience | gaps after tiers 1–2 |
| 4 | Native MusicData platform modules | gaps after tiers 1–3 |
last_refresh_source examples: viberate+songstats+soundcharts+native(genius,lastfm,qq).
Tier 4 — native platform gap-fill
Section titled “Tier 4 — native platform gap-fill”ID source: kisum_data.artists_v2.platforms.{platform}.id (fallback streamingPlatforms.*). Last.fm: platforms.musicbrainz.id → platforms.lastfm.id → artist.name. Bandsintown: artist.name. QQ: platforms.qq.id = singermid.
Tier A (high confidence): Spotify followers (spotifyApi), Spotify monthly listeners (charts.kworb_spotify_listeners.listeners), YouTube subscribers/views, Deezer nb_fan (public API), Genius followers_count, Last.fm listeners/plays.
Tier B (fixture-verified extract paths): NetEase data.fansCnt, QQ response.num, Instagram followers_count / follower_count (Meta Business Discovery), TikTok (RapidAPI by id or username), Bandsintown tracker_count.
Skipped (no audience metric in MusicData): Shazam, Apple Music catalog, Facebook, X, SoundCloud, Amazon, Tidal, Beatport, Traxsource, Songkick, Kuwo.
Removed: Legacy HTTP self-call to music.kisum.dev/instagram/... during refresh — Instagram gap-fill uses in-process Meta Business Discovery (graph.facebook.com) in tier 4 (replaces Zylalabs).
Concurrency: max 4 parallel native fetches; 3s timeout per platform; failures are silent per platform.
New API ↔ DB fields (tier 4)
Section titled “New API ↔ DB fields (tier 4)”| API key | artist_data field |
|---|---|
genius | genius_followers |
lastfm | lastfm_listeners |
lastfm_plays | lastfm_plays |
netease | netease_fans |
qq | qq_followers |
Env (refresh)
Section titled “Env (refresh)”VIBERATE_TOKEN, SONGSTATS_API_KEY, SOUNDCHARTS_APP_ID, SOUNDCHARTS_API_KEY, DB_MONGO_PROD_KISUM, INSTAGRAM_BUSINESS_ACCOUNT_ID + FACEBOOK_USER_ACCESS_TOKEN (Instagram tier 4 Business Discovery), Spotify/YouTube/Genius/Last.fm/NetEase/QQ credentials as used by existing MusicData routes
Instagram OAuth (/instagram/me)
Section titled “Instagram OAuth (/instagram/me)”MusicData stores the Kisum Instagram Graph long-lived token in Mongo auth.instagram (not caller-supplied).
| Method | Path | Gate | Purpose |
|---|---|---|---|
GET | /instagram/login | Public | Browser OAuth start (Instagram Login — own account) |
GET | /instagram/callback | Public | Code exchange + long-lived token persist |
GET | /instagram/refresh | Public | Refresh long-lived token (~60 days) |
GET | /instagram/me | MusicData gate | Read authorized account profile; auto-refresh once on 401 |
GET | /instagram/user/:username | MusicData gate | Other public Business/Creator profile via Meta Business Discovery |
Env (own account): INSTAGRAM_CLIENT_ID, INSTAGRAM_CLIENT_SECRET, INSTAGRAM_REDIRECT_URI (must match Meta app redirect URI). Optional INSTAGRAM_SCOPES (default instagram_business_basic).
Env (Business Discovery — other accounts): INSTAGRAM_BUSINESS_ACCOUNT_ID (caller Professional IG User ID), FACEBOOK_USER_ACCESS_TOKEN (Facebook User token with instagram_basic, instagram_manage_insights, pages_read_engagement, pages_show_list). Optional: store the same token in Mongo auth.instagram with { type: 'docker', label: 'kisum-facebook' }. Host: graph.facebook.com (default API version v25.0).
Ops: expired Instagram Login token → open /instagram/login; expired Facebook Discovery token → regenerate User token in Graph API Explorer (or Facebook Login flow) and update FACEBOOK_USER_ACCESS_TOKEN. Live/external use of Business Discovery requires Meta App Review + Live Mode for permissions used by non-role users.
News RSS (GET /news)
Section titled “News RSS (GET /news)”Reads Mongo news.articles by default (populated by crontab). Aggregates many public music-industry RSS/Atom feeds (Billboard, HipHopDX, Music Business Worldwide, etc.).
| Item | Detail |
|---|---|
| Timeout | 10 seconds per feed (FEED_TIMEOUT_MS in news.js) — slow publishers often need more than 5s from Docker |
| MBW URL | https://www.musicbusinessworldwide.com/feed/ (trailing slash required) |
| Cache bypass | Header x-redis: bypass or query ?live=true |
| Live scrape | ?live=true scrapes feeds instead of Mongo |
| Legacy cron query | ?cron=true scrapes + upserts; response data is null (prefer /crontab/news) |
| Retention | /crontab/news keeps 10 days of articles |
Public base: https://api.kisum.io/data/…
Crontab jobs (GET /crontab/*)
Section titled “Crontab jobs (GET /crontab/*)”Write path for charts, news, and DB backups. Never Redis-cached. Default response is 202 Accepted (job continues in background — Cloudflare-safe on api.kisum.io). Use ?wait=true only on a direct/local host for a synchronous 200.
| Path | Purpose |
|---|---|
/crontab/billboard | Scrape allowlisted Billboard → charts.billboard |
/crontab/kworb | Scrape kworb.net → charts.kworb_* |
/crontab/news | Scrape RSS → news.articles; purge older than 10 days |
/crontab/backup/mongo | Dump all Mongo DBs (skip admin/local) → disk + AWS S3 + R2 |
/crontab/backup/sql | pg_dump allowlisted Postgres → disk + AWS S3 + R2 |
/crontab/instagram/refresh | Refresh Meta long-lived Instagram token (same handler as /instagram/refresh) |
/crontab/spotify/refresh | Refresh Spotify user access token (same handler as /spotify/refresh) |
/crontab/youtube/refresh | Refresh YouTube user access token (same handler as /youtube/refresh) |
/crontab/instagram/facebook/refresh | Re-derive Facebook Business Discovery Page token (same as /instagram/facebook/refresh) |
Docker image requirement: MusicData Alpine image must include postgresql-client (pg_dump) and mongodb-tools (mongodump). Without them, /crontab/backup/sql crashes the Node process (spawn pg_dump ENOENT).
Backup destinations: each dump uploads to AWS S3 (BUCKET_AWS_*) and Cloudflare R2 (R2_*) when both are configured. S3 keys: backup/MongoDB/ and backup/PostgreSQL/. R2 keys: MongoDB/ and SQL/.
Postgres allowlist (/crontab/backup/sql): ARTISTS_DATABASE_URL, AUTH_DATABASE_URL, CHAT_DATABASE_URL, CORE_DATABASE_URL, FINANCE_DATABASE_URL, KISUMAI_DATABASE_URL, PROMOTERS_DATABASE_URL, VENUES_DATABASE_URL — direct Postgres URLs (port 25060). Legacy MARKET_DIRECTORY_DATABASE_URL removed (artists DB replaced it).
Read companions: GET /charts/billboard, GET /charts/kworb, GET /charts/kworb/:collection, GET /news.
Package detail: Backend-Kisum-MusicData/docs/charts-mongo-crontab.md and regenerated openapi-docs/.