theme-detector

어떤 스킬인가요?

Detect and analyze trending market themes across sectors. Use when user asks about current market themes, trending sectors, sector rotation, thematic investing, what themes are hot or cold, or wants to identify bullish and bearish market narratives with lifecycle analysis.

설치 방법

프로젝트 폴더에서 아래 명령어를 실행하고, 설치할 에이전트를 선택하세요.

설치 명령을 직접 실행해야 적용됩니다. 지원 에이전트와 필요한 권한·라이선스는 제작자의 안내를 확인하세요.

스킬 원문 파일

# Theme Detection Report

**Date:** {{date}}
**Mode:** {{mode}}
**Themes Analyzed:** {{themes_analyzed}}
**Industries Scanned:** {{industries_scanned}}
**Total Stocks:** {{total_stocks}}
**Uptrend Data:** {{uptrend_data_status}}
**Execution Time:** {{execution_time}}

---

## 1. Theme Dashboard

| Rank | Theme | Direction | Heat | Lifecycle | Confidence |
|------|-------|-----------|------|-----------|------------|
{{#each top_themes}}
| {{rank}} | {{name}} | {{direction_emoji}} {{direction}} | {{heat}}/100 | {{lifecycle}} | {{confidence}} |
{{/each}}

**Legend:** Direction: LEAD/LAG (Dashboard), ^/v (Summary) | Heat: 0-100 scale | Lifecycle: Emerging/Accelerating/Trending/Mature/Exhausting

---

## 2. Bullish Themes Detail

{{#each bullish_themes}}
### {{name}} - Heat: {{heat}}/100

**Direction:** Bullish ({{direction_strength}}) | **Lifecycle:** {{lifecycle}} | **Confidence:** {{confidence}}

**Heat Components:**
- Momentum: {{momentum}}/100
- Volume: {{volume}}/100
- Uptrend: {{uptrend}}/100
- Breadth: {{breadth}}/100

**Lifecycle Factors:**
- Duration: {{duration_weeks}} weeks
- Extremity clustering: {{extremity_pct}}%
- Relative P/E: {{relative_pe}}x
- ETF proliferation: {{etf_count}} ETFs (score: {{etf_score}}/100)
- Narrative: {{narrative_assessment}}

**Top Industries:**
{{#each industries}}
- {{name}}: {{change_pct}}% ({{stock_count}} stocks, uptrend {{uptrend_ratio}}%)
{{/each}}

**Top Movers:**
{{#each top_stocks}}
- {{ticker}}: {{change_pct}}% | Vol: {{relative_volume}}x
{{/each}}

**Proxy ETFs:** {{proxy_etfs}}

**Assessment:** {{assessment}}

---
{{/each}}

## 3. Bearish Themes Detail

{{#each bearish_themes}}
### {{name}} - Heat: {{heat}}/100

**Direction:** Bearish ({{direction_strength}}) | **Lifecycle:** {{lifecycle}} | **Confidence:** {{confidence}}

**Heat Components:**
- Momentum: {{momentum}}/100
- Volume: {{volume}}/100
- Downtrend: {{downtrend}}/100
- Breadth: {{breadth}}/100

**Top Declining Industries:**
{{#each industries}}
- {{name}}: {{change_pct}}% ({{stock_count}} stocks, downtrend {{downtrend_ratio}}%)
{{/each}}

**Worst Performers:**
{{#each bottom_stocks}}
- {{ticker}}: {{change_pct}}% | Vol: {{relative_volume}}x
{{/each}}

**Assessment:** {{assessment}}

---
{{/each}}

## 4. All Themes Summary

| Theme | Direction | Heat | Momentum | Volume | Uptrend | Breadth | Lifecycle | Confidence |
|-------|-----------|------|----------|--------|---------|---------|-----------|------------|
{{#each all_themes}}
| {{name}} | {{direction}} | {{heat}} | {{momentum}} | {{volume}} | {{uptrend}} | {{breadth}} | {{lifecycle}} | {{confidence}} |
{{/each}}

---

## 5. Industry Rankings

### Top 10 Industries (by performance)

| Rank | Industry | Sector | Change % | Rel. Volume | Uptrend Ratio | Theme |
|------|----------|--------|----------|-------------|---------------|-------|
{{#each top_industries}}
| {{rank}} | {{name}} | {{sector}} | {{change_pct}}% | {{rel_volume}}x | {{uptrend_ratio}}% | {{theme}} |
{{/each}}

### Bottom 10 Industries (by performance)

| Rank | Industry | Sector | Change % | Rel. Volume | Downtrend Ratio | Theme |
|------|----------|--------|----------|-------------|-----------------|-------|
{{#each bottom_industries}}
| {{rank}} | {{name}} | {{sector}} | {{change_pct}}% | {{rel_volume}}x | {{downtrend_ratio}}% | {{theme}} |
{{/each}}

---

## 6. Sector Uptrend Ratios

| Sector | Uptrend Ratio | Avg Change % | Dominant Theme | Direction |
|--------|---------------|-------------|----------------|-----------|
{{#each sectors}}
| {{name}} | {{uptrend_ratio}}% | {{avg_change_pct}}% | {{dominant_theme}} | {{direction}} |
{{/each}}

---

## 7. Methodology Notes

**Scoring Model:** 3-Dimensional (Heat x Lifecycle x Confidence)
- **Theme Heat** (0-100): Momentum (30%) + Volume (25%) + Uptrend (25%) + Breadth (20%)
- **Lifecycle**: Duration + Extremity clustering + Valuation + ETF proliferation + Narrative saturation
- **Confidence**: Quantitative breadth + Cross-sector confirmation + Narrative verification

**Data Sources:**
- FINVIZ {{mode}} (industry performance, volume, 52-week data)
{{#if uptrend_data}}
- uptrend-dashboard (3-point technical evaluation)
{{/if}}
{{#if fmp_data}}
- FMP API (P/E ratios for valuation analysis)
{{/if}}
- WebSearch (narrative confirmation for top themes)

**Limitations:**
- Theme definitions are static (predefined in cross_sector_themes.md)
- {{mode_limitation}}
- Duration tracking requires historical baseline data
- Narrative assessment is subjective

---

*Generated by Theme Detector v1.0 | {{date}} {{time}}*

# Cross-Sector Theme Definitions

This reference defines market themes by their constituent FINVIZ industries, sectors, proxy ETFs, and representative stocks. The theme_classifier.py script reads these definitions to map industry-level data into theme-level aggregations.

**Usage Notes:**
- Industry names must exactly match FINVIZ industry names (see `finviz_industry_codes.md`)
- `min_matching_industries`: Design-intent minimum for each theme. In practice, theme detection uses the global `cross_sector_min_matches` setting (default: 2, configurable in themes.yaml) as the threshold. Per-theme values below are informational and not individually enforced in code
- `static_stocks`: Fallback representative stocks used when industry-level data is insufficient
- `proxy_etfs`: Used for quick volume/momentum checks and as user-facing exposure recommendations

---

## AI & Semiconductors

- **Direction bias**: Bullish (typically)
- **Industries**: Semiconductors, Software - Application, Software - Infrastructure, Information Technology Services, Electronic Components, Computer Hardware, Scientific & Technical Instruments
- **Sectors**: Technology (primary), Communication Services, Industrials
- **Proxy ETFs**: SMH, SOXX, AIQ, BOTZ, CHAT
- **Static stocks**: NVDA, AVGO, AMD, INTC, QCOM, MRVL, AMAT, LRCX, KLAC, TSM, MU, ARM, SNPS, CDNS, MCHP
- **Min matching industries**: 2

---

## Clean Energy & EV

- **Direction bias**: Bullish (typically)
- **Industries**: Solar, Utilities - Renewable, Auto Manufacturers, Auto Parts, Electrical Equipment & Parts, Specialty Chemicals
- **Sectors**: Utilities, Consumer Cyclical, Industrials, Basic Materials
- **Proxy ETFs**: ICLN, QCLN, TAN, DRIV, LIT
- **Static stocks**: ENPH, SEDG, FSLR, RUN, TSLA, RIVN, LCID, NIO, PLUG, BE, CHPT, ALB, SQM, LAC, LTHM
- **Min matching industries**: 2

---

## Cybersecurity

- **Direction bias**: Bullish (typically)
- **Industries**: Software - Infrastructure, Information Technology Services, Software - Application, Communication Equipment
- **Sectors**: Technology (primary)
- **Proxy ETFs**: CIBR, HACK, BUG
- **Static stocks**: CRWD, PANW, FTNT, ZS, NET, S, OKTA, CYBR, QLYS, RPD, TENB, VRNS, SAIL, MNDT, DDOG
- **Min matching industries**: 2

**Note:** Cybersecurity overlaps with broader software industries. Theme classification uses proxy ETF volume and static stock performance to differentiate from general software themes.

---

## Cloud Computing & SaaS

- **Direction bias**: Bullish (typically)
- **Industries**: Software - Application, Software - Infrastructure, Information Technology Services
- **Sectors**: Technology (primary), Communication Services
- **Proxy ETFs**: SKYY, WCLD, CLOU
- **Static stocks**: CRM, NOW, SNOW, DDOG, TEAM, MDB, ESTC, NET, ZS, HUBS, BILL, TTD, PLTR, DOCN, DT
- **Min matching industries**: 2

**Note:** Cloud/SaaS overlaps significantly with Cybersecurity and AI themes. When multiple themes share the same industries, proxy ETF performance differentiates them.

---

## Biotech & Genomics

- **Direction bias**: Bullish (typically, but highly volatile)
- **Industries**: Biotechnology, Drug Manufacturers - Specialty & Generic, Medical Devices, Diagnostics & Research, Drug Manufacturers - General
- **Sectors**: Healthcare (primary)
- **Proxy ETFs**: XBI, IBB, ARKG, GNOM
- **Static stocks**: AMGN, GILD, VRTX, REGN, MRNA, BIIB, ILMN, CRSP, NTLA, BEAM, EDIT, EXAS, TWST, SGEN, BMRN
- **Min matching industries**: 2

---

## Infrastructure & Construction

- **Direction bias**: Bullish (typically, policy-driven)
- **Industries**: Engineering & Construction, Building Materials, Industrial Distribution, Farm & Heavy Construction Machinery, Steel, Specialty Industrial Machinery, Railroads, Waste Management
- **Sectors**: Industrials (primary), Basic Materials
- **Proxy ETFs**: PAVE, IFRA, SIMS
- **Static stocks**: CAT, DE, VMC, MLM, URI, PWR, EME, MTZ, GVA, AECOM, STRL, GBX, NUE, CLF, RS
- **Min matching industries**: 3

---

## Gold & Precious Metals

- **Direction bias**: Bullish (typically in risk-off or inflation)
- **Industries**: Gold, Silver, Other Precious Metals & Mining
- **Sectors**: Basic Materials (primary)
- **Proxy ETFs**: GDX, GDXJ, RING, SIL
- **Static stocks**: NEM, GOLD, AEM, FNV, WPM, RGLD, KGC, AGI, AU, HMY, PAAS, CDE, HL, MAG, EQX
- **Min matching industries**: 2

---

## Oil & Gas (Energy Sector)

- **Direction bias**: Varies (cyclical)
- **Industries**: Oil & Gas E&P, Oil & Gas Equipment & Services, Oil & Gas Midstream, Oil & Gas Refining & Marketing, Oil & Gas Integrated, Oil & Gas Drilling
- **Sectors**: Energy (primary)
- **Proxy ETFs**: XLE, XOP, OIH
- **Static stocks**: XOM, CVX, COP, EOG, SLB, HAL, PXD, DVN, MPC, VLO, PSX, OXY, FANG, HES, WMB
- **Min matching industries**: 2

---

## Financial Services & Banks

- **Direction bias**: Varies (rate-sensitive)
- **Industries**: Banks - Diversified, Banks - Regional, Capital Markets, Insurance - Diversified, Insurance - Property & Casualty, Financial Data & Stock Exchanges, Credit Services, Asset Management, Insurance Brokers, Mortgage Finance
- **Sectors**: Financial Services (primary)
- **Proxy ETFs**: XLF, KBE, KRE, IAI
- **Static stocks**: JPM, BAC, WFC, GS, MS, C, SCHW, BLK, AXP, ICE, CME, MCO, SPGI, BX, KKR
- **Min matching industries**: 3

---

## Healthcare & Pharma

- **Direction bias**: Varies (defensive in downturns)
- **Industries**: Drug Manufacturers - General, Health Care Plans, Medical Care Facilities, Health Information Services, Medical Distribution, Medical Instruments & Supplies, Pharmaceutical Retailers
- **Sectors**: Healthcare (primary)
- **Proxy ETFs**: XLV, IHE, IHI
- **Static stocks**: UNH, JNJ, LLY, PFE, ABT, TMO, DHR, MDT, ISRG, SYK, BSX, EW, HCA, CVS, CI
- **Min matching industries**: 3

**Note:** Healthcare & Pharma is distinct from Biotech & Genomics. Healthcare focuses on established pharma, insurance, and medical devices; Biotech focuses on emerging drug development and genomics.

---

## Defense & Aerospace

- **Direction bias**: Bullish (typically in geopolitical tension)
- **Industries**: Aerospace & Defense, Airlines, Security & Protection Services
- **Sectors**: Industrials (primary)
- **Proxy ETFs**: ITA, PPA, ROKT, ARKX
- **Static stocks**: LMT, RTX, NOC, BA, GD, LHX, HII, TDG, HWM, AXON, LDOS, BWXT, KTOS, RKLB, SPR
- **Min matching industries**: 2

---

## Real Estate & REITs

- **Direction bias**: Varies (rate-sensitive)
- **Industries**: REIT - Residential, REIT - Industrial, REIT - Retail, REIT - Office, REIT - Healthcare Facilities, REIT - Diversified, REIT - Hotel & Motel, REIT - Specialty, Real Estate Services, Real Estate - Diversified, Real Estate - Development
- **Sectors**: Real Estate (primary)
- **Proxy ETFs**: VNQ, XLRE, IYR
- **Static stocks**: PLD, AMT, CCI, EQIX, SPG, O, WELL, DLR, PSA, VICI, EXR, AVB, ARE, MAA, IRM
- **Min matching industries**: 3

---

## Retail & Consumer

- **Direction bias**: Varies (consumer sentiment driven)
- **Industries**: Internet Retail, Specialty Retail, Apparel Retail, Home Improvement Retail, Department Stores, Discount Stores, Luxury Goods, Restaurants, Leisure, Resorts & Casinos, Gambling, Apparel Manufacturing, Footwear & Accessories
- **Sectors**: Consumer Cyclical (primary), Consumer Defensive
- **Proxy ETFs**: XLY, XRT, XLP, IBUY
- **Static stocks**: AMZN, HD, LOW, TJX, COST, WMT, TGT, NKE, SBUX, MCD, DPZ, LULU, ROST, BURL, DECK
- **Min matching industries**: 3

---

## Crypto & Blockchain

- **Direction bias**: Bullish (typically in risk-on)
- **Industries**: Capital Markets, Software - Application, Financial Data & Stock Exchanges, Information Technology Services
- **Sectors**: Financial Services, Technology
- **Proxy ETFs**: BITO, BLOK, BITQ, IBIT, DAPP
- **Static stocks**: COIN, MSTR, MARA, RIOT, CLSK, HUT, BITF, SQ, PYPL, HOOD, CIFR, IREN, HIVE, CORZ, BTBT
- **Min matching industries**: 2

**Note:** Crypto theme uses proxy ETFs and static stocks as primary signals rather than industry classification, since blockchain companies span multiple traditional industries.

---

## Nuclear Energy

- **Direction bias**: Bullish (policy-driven, AI data center demand)
- **Industries**: Uranium, Utilities - Independent Power Producers, Specialty Industrial Machinery, Electrical Equipment & Parts
- **Sectors**: Energy, Utilities, Industrials
- **Proxy ETFs**: URA, URNM, NLR, NUKZ
- **Static stocks**: CCJ, UEC, NXE, DNN, UUUU, LEU, SMR, OKLO, BWX, GEV, CEG, VST, TLN, NRG, BWXT
- **Min matching industries**: 2

---

## Uranium

- **Direction bias**: Bullish (supply deficit narrative)
- **Industries**: Uranium, Other Industrial Metals & Mining
- **Sectors**: Energy (primary), Basic Materials
- **Proxy ETFs**: URA, URNM, URNJ
- **Static stocks**: CCJ, UEC, NXE, DNN, UUUU, LEU, URG, GLATF, EU, PALAF, SRUUF, LTBR, FCUUF, AEC, WSTRF
- **Min matching industries**: 1

**Note:** Uranium is a sub-theme of Nuclear Energy but tracked separately due to its distinct commodity-driven dynamics. Requires only 1 matching industry due to narrow sector focus.

---

## Obesity & GLP-1

- **Direction bias**: Bullish (medical innovation)
- **Industries**: Drug Manufacturers - General, Drug Manufacturers - Specialty & Generic, Medical Devices, Biotechnology
- **Sectors**: Healthcare (primary)
- **Proxy ETFs**: SLIM, HRTS
- **Static stocks**: LLY, NVO, AMGN, VKTX, ALT, GPCR, SMLR, RVMD, PTGX, IVA
- **Min matching industries**: 2

**Note:** Obesity/GLP-1 is a narrow theme that overlaps with Healthcare & Pharma. It is differentiated primarily through proxy ETF volume and static stock performance rather than industry classification.

---

## Summary Table

| Theme | Industries | Sectors | Proxy ETFs | Static Stocks | Min Industries |
|-------|-----------|---------|-----------|--------------|----------------|
| AI & Semiconductors | 7 | 3 | 5 | 15 | 2 |
| Clean Energy & EV | 6 | 4 | 5 | 15 | 2 |
| Cybersecurity | 4 | 1 | 3 | 15 | 2 |
| Cloud Computing & SaaS | 3 | 2 | 3 | 15 | 2 |
| Biotech & Genomics | 5 | 1 | 4 | 15 | 2 |
| Infrastructure & Construction | 8 | 2 | 3 | 15 | 3 |
| Gold & Precious Metals | 3 | 1 | 4 | 15 | 2 |
| Oil & Gas (Energy) | 6 | 1 | 3 | 15 | 2 |
| Financial Services & Banks | 10 | 1 | 4 | 15 | 3 |
| Healthcare & Pharma | 7 | 1 | 3 | 15 | 3 |
| Defense & Aerospace | 3 | 1 | 4 | 15 | 2 |
| Real Estate & REITs | 11 | 1 | 3 | 15 | 3 |
| Retail & Consumer | 13 | 2 | 4 | 15 | 3 |
| Crypto & Blockchain | 4 | 2 | 5 | 15 | 2 |
| Nuclear Energy | 4 | 3 | 4 | 15 | 2 |
| Uranium | 2 | 2 | 3 | 15 | 1 |
| Obesity & GLP-1 | 4 | 1 | 2 | 10 | 2 |

**Total: 17 themes** covering all major market narratives.

---

## Theme Overlap Matrix

Some industries contribute to multiple themes. When scoring, each industry's data is used in all applicable themes:

| Industry | Themes |
|----------|--------|
| Software - Application | AI, Cybersecurity, Cloud, Crypto |
| Software - Infrastructure | AI, Cybersecurity, Cloud |
| Drug Manufacturers - General | Healthcare, Biotech, Obesity |
| Biotechnology | Biotech, Obesity |
| Capital Markets | Financial Services, Crypto |
| Electrical Equipment & Parts | Clean Energy, Nuclear |
| Uranium | Nuclear, Uranium |

This overlap is intentional - a strong software industry may boost multiple themes simultaneously, reflecting the interconnected nature of market narratives.

# FINVIZ Industry Filter Codes

This reference maps all FINVIZ industry names to their corresponding filter codes used in the FINVIZ Screener API and Elite CSV export URLs.

**URL Pattern (Elite CSV Export):**
```
https://elite.finviz.com/export.ashx?v=151&f=ind_{code},cap_smallover&auth={API_KEY}
```

**URL Pattern (Public Screener):**
```
https://finviz.com/screener.ashx?v=151&f=ind_{code},cap_smallover
```

**Filter Code Convention:** `ind_` + lowercase industry name with spaces, hyphens, ampersands, and special characters removed.

---

## Aerospace & Defense

| Industry Name | Filter Code |
|---|---|
| Aerospace & Defense | ind_aerospacedefense |

## Auto & Transport

| Industry Name | Filter Code |
|---|---|
| Airlines | ind_airlines |
| Auto Manufacturers | ind_automanufacturers |
| Auto Parts | ind_autoparts |
| Railroads | ind_railroads |
| Recreational Vehicles | ind_recreationalvehicles |
| Trucking | ind_trucking |
| Marine Shipping | ind_marineshipping |

## Basic Materials

| Industry Name | Filter Code |
|---|---|
| Aluminum | ind_aluminum |
| Building Materials | ind_buildingmaterials |
| Chemicals | ind_chemicals |
| Coking Coal | ind_cokingcoal |
| Copper | ind_copper |
| Gold | ind_gold |
| Lumber & Wood Production | ind_lumberwoodproduction |
| Other Industrial Metals & Mining | ind_otherindustrialmetalsmining |
| Other Precious Metals & Mining | ind_otherpreciousmetalsmining |
| Paper & Paper Products | ind_paperpaperproducts |
| Silver | ind_silver |
| Specialty Chemicals | ind_specialtychemicals |
| Steel | ind_steel |
| Uranium | ind_uranium |

## Communication Services

| Industry Name | Filter Code |
|---|---|
| Advertising Agencies | ind_advertisingagencies |
| Broadcasting | ind_broadcasting |
| Electronic Gaming & Multimedia | ind_electronicgamingmultimedia |
| Entertainment | ind_entertainment |
| Internet Content & Information | ind_internetcontentinformation |
| Publishing | ind_publishing |
| Telecom Services | ind_telecomservices |

## Consumer Cyclical

| Industry Name | Filter Code |
|---|---|
| Apparel Manufacturing | ind_apparelmanufacturing |
| Apparel Retail | ind_apparelretail |
| Auto & Truck Dealerships | ind_autotruckdealerships |
| Department Stores | ind_departmentstores |
| Discount Stores | ind_discountstores |
| Footwear & Accessories | ind_footwearaccessories |
| Furnishings, Fixtures & Appliances | ind_furnishingsfixturesappliances |
| Gambling | ind_gambling |
| Home Improvement Retail | ind_homeimprovementretail |
| Internet Retail | ind_internetretail |
| Leisure | ind_leisure |
| Lodging | ind_lodging |
| Luxury Goods | ind_luxurygoods |
| Packaging & Containers | ind_packagingcontainers |
| Personal Services | ind_personalservices |
| Residential Construction | ind_residentialconstruction |
| Resorts & Casinos | ind_resortscasinos |
| Restaurants | ind_restaurants |
| Specialty Retail | ind_specialtyretail |
| Textile Manufacturing | ind_textilemanufacturing |
| Travel Services | ind_travelservices |

## Consumer Defensive

| Industry Name | Filter Code |
|---|---|
| Beverages - Brewers | ind_beveragesbrewers |
| Beverages - Non-Alcoholic | ind_beveragesnonalcoholic |
| Beverages - Wineries & Distilleries | ind_beverageswineriesanddistilleries |
| Confectioners | ind_confectioners |
| Discount Stores | ind_discountstores |
| Education & Training Services | ind_educationtrainingservices |
| Farm Products | ind_farmproducts |
| Food Distribution | ind_fooddistribution |
| Grocery Stores | ind_grocerystores |
| Household & Personal Products | ind_householdpersonalproducts |
| Packaged Foods | ind_packagedfoods |
| Tobacco | ind_tobacco |

## Energy

| Industry Name | Filter Code |
|---|---|
| Oil & Gas Drilling | ind_oilgasdrilling |
| Oil & Gas E&P | ind_oilgasep |
| Oil & Gas Equipment & Services | ind_oilgasequipmentservices |
| Oil & Gas Integrated | ind_oilgasintegrated |
| Oil & Gas Midstream | ind_oilgasmidstream |
| Oil & Gas Refining & Marketing | ind_oilgasrefiningmarketing |
| Thermal Coal | ind_thermalcoal |
| Uranium | ind_uranium |

## Financial Services

| Industry Name | Filter Code |
|---|---|
| Asset Management | ind_assetmanagement |
| Banks - Diversified | ind_banksdiversified |
| Banks - Regional | ind_banksregional |
| Capital Markets | ind_capitalmarkets |
| Credit Services | ind_creditservices |
| Financial Conglomerates | ind_financialconglomerates |
| Financial Data & Stock Exchanges | ind_financialdatastockexchanges |
| Insurance - Diversified | ind_insurancediversified |
| Insurance - Life | ind_insurancelife |
| Insurance - Property & Casualty | ind_insurancepropertycasualty |
| Insurance - Reinsurance | ind_insurancereinsurance |
| Insurance - Specialty | ind_insurancespecialty |
| Insurance Brokers | ind_insurancebrokers |
| Mortgage Finance | ind_mortgagefinance |
| Shell Companies | ind_shellcompanies |

## Healthcare

| Industry Name | Filter Code |
|---|---|
| Biotechnology | ind_biotechnology |
| Diagnostics & Research | ind_diagnosticsresearch |
| Drug Manufacturers - General | ind_drugmanufacturersgeneral |
| Drug Manufacturers - Specialty & Generic | ind_drugmanufacturersspecialtygeneric |
| Health Information Services | ind_healthinformationservices |
| Health Care Plans | ind_healthcareplans |
| Medical Care Facilities | ind_medicalcarefacilities |
| Medical Devices | ind_medicaldevices |
| Medical Distribution | ind_medicaldistribution |
| Medical Instruments & Supplies | ind_medicalinstrumentssupplies |
| Pharmaceutical Retailers | ind_pharmaceuticalretailers |

## Industrials

| Industry Name | Filter Code |
|---|---|
| Aerospace & Defense | ind_aerospacedefense |
| Business Equipment & Supplies | ind_businessequipmentsupplies |
| Conglomerates | ind_conglomerates |
| Consulting Services | ind_consultingservices |
| Electrical Equipment & Parts | ind_electricalequipmentparts |
| Engineering & Construction | ind_engineeringconstruction |
| Farm & Heavy Construction Machinery | ind_farmheavyconstructionmachinery |
| Industrial Distribution | ind_industrialdistribution |
| Infrastructure Operations | ind_infrastructureoperations |
| Integrated Freight & Logistics | ind_integratedfreightlogistics |
| Metal Fabrication | ind_metalfabrication |
| Pollution & Treatment Controls | ind_pollutiontreatmentcontrols |
| Rental & Leasing Services | ind_rentalleasingservices |
| Security & Protection Services | ind_securityprotectionservices |
| Specialty Business Services | ind_specialtybusinessservices |
| Specialty Industrial Machinery | ind_specialtyindustrialmachinery |
| Staffing & Employment Services | ind_staffingemploymentservices |
| Tools & Accessories | ind_toolsaccessories |
| Waste Management | ind_wastemanagement |

## Real Estate

| Industry Name | Filter Code |
|---|---|
| Real Estate - Development | ind_realestatedevelopment |
| Real Estate - Diversified | ind_realestatediversified |
| Real Estate Services | ind_realestateservices |
| REIT - Diversified | ind_reitdiversified |
| REIT - Healthcare Facilities | ind_reithealthcarefacilities |
| REIT - Hotel & Motel | ind_reithotelandmotel |
| REIT - Industrial | ind_reitindustrial |
| REIT - Mortgage | ind_reitmortgage |
| REIT - Office | ind_reitoffice |
| REIT - Residential | ind_reitresidential |
| REIT - Retail | ind_reitretail |
| REIT - Specialty | ind_reitspecialty |

## Technology

| Industry Name | Filter Code |
|---|---|
| Communication Equipment | ind_communicationequipment |
| Computer Hardware | ind_computerhardware |
| Consumer Electronics | ind_consumerelectronics |
| Electronic Components | ind_electroniccomponents |
| Electronics & Computer Distribution | ind_electronicscomputerdistribution |
| Information Technology Services | ind_informationtechnologyservices |
| Scientific & Technical Instruments | ind_scientifictechnicalinstruments |
| Semiconductor Equipment & Materials | ind_semiconductorequipmentmaterials |
| Semiconductors | ind_semiconductors |
| Software - Application | ind_softwareapplication |
| Software - Infrastructure | ind_softwareinfrastructure |
| Solar | ind_solar |

## Utilities

| Industry Name | Filter Code |
|---|---|
| Utilities - Diversified | ind_utilitiesdiversified |
| Utilities - Independent Power Producers | ind_utilitiesindependentpowerproducers |
| Utilities - Regulated Electric | ind_utilitiesregulatedelectric |
| Utilities - Regulated Gas | ind_utilitiesregulatedgas |
| Utilities - Regulated Water | ind_utilitiesregulatedwater |
| Utilities - Renewable | ind_utilitiesrenewable |

---

## Complete Alphabetical Index

This flat list provides all industry filter codes in alphabetical order for quick lookup:

| # | Industry Name | Filter Code |
|---|---|---|
| 1 | Advertising Agencies | ind_advertisingagencies |
| 2 | Aerospace & Defense | ind_aerospacedefense |
| 3 | Airlines | ind_airlines |
| 4 | Aluminum | ind_aluminum |
| 5 | Apparel Manufacturing | ind_apparelmanufacturing |
| 6 | Apparel Retail | ind_apparelretail |
| 7 | Asset Management | ind_assetmanagement |
| 8 | Auto & Truck Dealerships | ind_autotruckdealerships |
| 9 | Auto Manufacturers | ind_automanufacturers |
| 10 | Auto Parts | ind_autoparts |
| 11 | Banks - Diversified | ind_banksdiversified |
| 12 | Banks - Regional | ind_banksregional |
| 13 | Beverages - Brewers | ind_beveragesbrewers |
| 14 | Beverages - Non-Alcoholic | ind_beveragesnonalcoholic |
| 15 | Beverages - Wineries & Distilleries | ind_beverageswineriesanddistilleries |
| 16 | Biotechnology | ind_biotechnology |
| 17 | Broadcasting | ind_broadcasting |
| 18 | Building Materials | ind_buildingmaterials |
| 19 | Business Equipment & Supplies | ind_businessequipmentsupplies |
| 20 | Capital Markets | ind_capitalmarkets |
| 21 | Chemicals | ind_chemicals |
| 22 | Coking Coal | ind_cokingcoal |
| 23 | Communication Equipment | ind_communicationequipment |
| 24 | Computer Hardware | ind_computerhardware |
| 25 | Confectioners | ind_confectioners |
| 26 | Conglomerates | ind_conglomerates |
| 27 | Consulting Services | ind_consultingservices |
| 28 | Consumer Electronics | ind_consumerelectronics |
| 29 | Copper | ind_copper |
| 30 | Credit Services | ind_creditservices |
| 31 | Department Stores | ind_departmentstores |
| 32 | Diagnostics & Research | ind_diagnosticsresearch |
| 33 | Discount Stores | ind_discountstores |
| 34 | Drug Manufacturers - General | ind_drugmanufacturersgeneral |
| 35 | Drug Manufacturers - Specialty & Generic | ind_drugmanufacturersspecialtygeneric |
| 36 | Education & Training Services | ind_educationtrainingservices |
| 37 | Electrical Equipment & Parts | ind_electricalequipmentparts |
| 38 | Electronic Components | ind_electroniccomponents |
| 39 | Electronic Gaming & Multimedia | ind_electronicgamingmultimedia |
| 40 | Electronics & Computer Distribution | ind_electronicscomputerdistribution |
| 41 | Engineering & Construction | ind_engineeringconstruction |
| 42 | Entertainment | ind_entertainment |
| 43 | Farm & Heavy Construction Machinery | ind_farmheavyconstructionmachinery |
| 44 | Farm Products | ind_farmproducts |
| 45 | Financial Conglomerates | ind_financialconglomerates |
| 46 | Financial Data & Stock Exchanges | ind_financialdatastockexchanges |
| 47 | Food Distribution | ind_fooddistribution |
| 48 | Footwear & Accessories | ind_footwearaccessories |
| 49 | Furnishings, Fixtures & Appliances | ind_furnishingsfixturesappliances |
| 50 | Gambling | ind_gambling |
| 51 | Gold | ind_gold |
| 52 | Grocery Stores | ind_grocerystores |
| 53 | Health Care Plans | ind_healthcareplans |
| 54 | Health Information Services | ind_healthinformationservices |
| 55 | Home Improvement Retail | ind_homeimprovementretail |
| 56 | Household & Personal Products | ind_householdpersonalproducts |
| 57 | Industrial Distribution | ind_industrialdistribution |
| 58 | Infrastructure Operations | ind_infrastructureoperations |
| 59 | Information Technology Services | ind_informationtechnologyservices |
| 60 | Insurance - Diversified | ind_insurancediversified |
| 61 | Insurance - Life | ind_insurancelife |
| 62 | Insurance - Property & Casualty | ind_insurancepropertycasualty |
| 63 | Insurance - Reinsurance | ind_insurancereinsurance |
| 64 | Insurance - Specialty | ind_insurancespecialty |
| 65 | Insurance Brokers | ind_insurancebrokers |
| 66 | Integrated Freight & Logistics | ind_integratedfreightlogistics |
| 67 | Internet Content & Information | ind_internetcontentinformation |
| 68 | Internet Retail | ind_internetretail |
| 69 | Leisure | ind_leisure |
| 70 | Lodging | ind_lodging |
| 71 | Lumber & Wood Production | ind_lumberwoodproduction |
| 72 | Luxury Goods | ind_luxurygoods |
| 73 | Marine Shipping | ind_marineshipping |
| 74 | Medical Care Facilities | ind_medicalcarefacilities |
| 75 | Medical Devices | ind_medicaldevices |
| 76 | Medical Distribution | ind_medicaldistribution |
| 77 | Medical Instruments & Supplies | ind_medicalinstrumentssupplies |
| 78 | Metal Fabrication | ind_metalfabrication |
| 79 | Mortgage Finance | ind_mortgagefinance |
| 80 | Oil & Gas Drilling | ind_oilgasdrilling |
| 81 | Oil & Gas E&P | ind_oilgasep |
| 82 | Oil & Gas Equipment & Services | ind_oilgasequipmentservices |
| 83 | Oil & Gas Integrated | ind_oilgasintegrated |
| 84 | Oil & Gas Midstream | ind_oilgasmidstream |
| 85 | Oil & Gas Refining & Marketing | ind_oilgasrefiningmarketing |
| 86 | Other Industrial Metals & Mining | ind_otherindustrialmetalsmining |
| 87 | Other Precious Metals & Mining | ind_otherpreciousmetalsmining |
| 88 | Packaged Foods | ind_packagedfoods |
| 89 | Packaging & Containers | ind_packagingcontainers |
| 90 | Paper & Paper Products | ind_paperpaperproducts |
| 91 | Personal Services | ind_personalservices |
| 92 | Pharmaceutical Retailers | ind_pharmaceuticalretailers |
| 93 | Pollution & Treatment Controls | ind_pollutiontreatmentcontrols |
| 94 | Publishing | ind_publishing |
| 95 | Railroads | ind_railroads |
| 96 | Real Estate - Development | ind_realestatedevelopment |
| 97 | Real Estate - Diversified | ind_realestatediversified |
| 98 | Real Estate Services | ind_realestateservices |
| 99 | Recreational Vehicles | ind_recreationalvehicles |
| 100 | REIT - Diversified | ind_reitdiversified |
| 101 | REIT - Healthcare Facilities | ind_reithealthcarefacilities |
| 102 | REIT - Hotel & Motel | ind_reithotelandmotel |
| 103 | REIT - Industrial | ind_reitindustrial |
| 104 | REIT - Mortgage | ind_reitmortgage |
| 105 | REIT - Office | ind_reitoffice |
| 106 | REIT - Residential | ind_reitresidential |
| 107 | REIT - Retail | ind_reitretail |
| 108 | REIT - Specialty | ind_reitspecialty |
| 109 | Rental & Leasing Services | ind_rentalleasingservices |
| 110 | Residential Construction | ind_residentialconstruction |
| 111 | Resorts & Casinos | ind_resortscasinos |
| 112 | Restaurants | ind_restaurants |
| 113 | Scientific & Technical Instruments | ind_scientifictechnicalinstruments |
| 114 | Security & Protection Services | ind_securityprotectionservices |
| 115 | Semiconductor Equipment & Materials | ind_semiconductorequipmentmaterials |
| 116 | Semiconductors | ind_semiconductors |
| 117 | Shell Companies | ind_shellcompanies |
| 118 | Silver | ind_silver |
| 119 | Software - Application | ind_softwareapplication |
| 120 | Software - Infrastructure | ind_softwareinfrastructure |
| 121 | Solar | ind_solar |
| 122 | Specialty Business Services | ind_specialtybusinessservices |
| 123 | Specialty Chemicals | ind_specialtychemicals |
| 124 | Specialty Industrial Machinery | ind_specialtyindustrialmachinery |
| 125 | Specialty Retail | ind_specialtyretail |
| 126 | Staffing & Employment Services | ind_staffingemploymentservices |
| 127 | Steel | ind_steel |
| 128 | Telecom Services | ind_telecomservices |
| 129 | Textile Manufacturing | ind_textilemanufacturing |
| 130 | Thermal Coal | ind_thermalcoal |
| 131 | Tobacco | ind_tobacco |
| 132 | Tools & Accessories | ind_toolsaccessories |
| 133 | Travel Services | ind_travelservices |
| 134 | Trucking | ind_trucking |
| 135 | Uranium | ind_uranium |
| 136 | Utilities - Diversified | ind_utilitiesdiversified |
| 137 | Utilities - Independent Power Producers | ind_utilitiesindependentpowerproducers |
| 138 | Utilities - Regulated Electric | ind_utilitiesregulatedelectric |
| 139 | Utilities - Regulated Gas | ind_utilitiesregulatedgas |
| 140 | Utilities - Regulated Water | ind_utilitiesregulatedwater |
| 141 | Utilities - Renewable | ind_utilitiesrenewable |
| 142 | Waste Management | ind_wastemanagement |

**Total: 142 industries**

---

## Notes

1. **Filter code derivation**: Remove spaces, hyphens, ampersands (`&`), and other special characters from the industry name, then lowercase the result. Prefix with `ind_`.

2. **Cross-sector industries**: Some industries (e.g., Uranium) appear under multiple FINVIZ sectors (Energy and Basic Materials). The filter code is the same regardless of which sector view you navigate from.

3. **Industry changes**: FINVIZ occasionally adds, removes, or renames industries. This list was compiled as of February 2026. If a filter code returns no results, verify the industry name on FINVIZ directly.

4. **Combining filters**: Multiple industry filters can be combined in FINVIZ URLs using comma separation:
   ```
   f=ind_semiconductors,ind_softwareapplication,cap_smallover
   ```

5. **Market cap filter**: For theme detection, always combine with `cap_smallover` (market cap > $300M) to filter out micro-caps that may distort aggregated metrics.

6. **Special cases**:
   - "E&P" becomes "ep" (ampersand removed)
   - "REIT - " prefix becomes "reit" (hyphen and spaces removed)
   - "Wineries & Distilleries" becomes "wineriesanddistilleries" (ampersand becomes "and" in this case)
   - Verify edge cases against FINVIZ directly if results seem incorrect

# Thematic ETF Catalog

This catalog maps market themes to their associated ETFs. The ETF count per theme is used for the `etf_proliferation_score` in lifecycle maturity assessment.

**ETF Proliferation Scoring:**
| ETF Count | Score |
|-----------|-------|
| 0         | 0     |
| 1         | 20    |
| 2-3       | 40    |
| 4-6       | 60    |
| 7-10      | 80    |
| >10       | 100   |

Higher ETF proliferation indicates more retail/institutional interest and a potentially more mature (crowded) theme.

---

## AI & Semiconductors

**ETF Count: 11** | Proliferation Score: 100

| Ticker | Name | Focus | AUM Tier |
|--------|------|-------|----------|
| SMH    | VanEck Semiconductor ETF | Semiconductor equities | Large |
| SOXX   | iShares Semiconductor ETF | Semiconductor equities | Large |
| SOXQ   | Invesco PHLX Semiconductor ETF | Semiconductor equities | Medium |
| AIQ    | Global X Artificial Intelligence & Technology ETF | AI & big data | Medium |
| BOTZ   | Global X Robotics & Artificial Intelligence ETF | AI & robotics | Medium |
| ROBT   | First Trust Nasdaq Artificial Intelligence & Robotics ETF | AI & robotics | Small |
| IRBO   | iShares Robotics and Artificial Intelligence Multisector ETF | AI multi-sector | Small |
| CHAT   | Roundhill Generative AI & Technology ETF | Generative AI | Small |
| WTAI   | WisdomTree Artificial Intelligence and Innovation Fund | AI innovation | Small |
| IGPT   | North Shore AI Powered Large Cap Growth ETF | AI large-cap | Small |
| AIVI   | WisdomTree International AI Enhanced Value Fund | AI value | Small |

---

## Clean Energy & EV

**ETF Count: 9** | Proliferation Score: 80

| Ticker | Name | Focus | AUM Tier |
|--------|------|-------|----------|
| ICLN   | iShares Global Clean Energy ETF | Global clean energy | Large |
| QCLN   | First Trust NASDAQ Clean Edge Green Energy ETF | Clean energy technology | Medium |
| TAN    | Invesco Solar ETF | Solar energy | Medium |
| FAN    | First Trust Global Wind Energy ETF | Wind energy | Small |
| LIT    | Global X Lithium & Battery Tech ETF | Lithium & batteries | Medium |
| DRIV   | Global X Autonomous & Electric Vehicles ETF | EVs & autonomous driving | Medium |
| IDRV   | iShares Self-Driving EV and Tech ETF | EVs & self-driving tech | Small |
| ACES   | ALPS Clean Energy ETF | US clean energy | Small |
| PBW    | Invesco WilderHill Clean Energy ETF | Clean energy innovation | Small |

---

## Cybersecurity

**ETF Count: 4** | Proliferation Score: 60

| Ticker | Name | Focus | AUM Tier |
|--------|------|-------|----------|
| CIBR   | First Trust NASDAQ Cybersecurity ETF | Cybersecurity equities | Large |
| HACK   | ETFMG Prime Cyber Security ETF | Cybersecurity equities | Medium |
| BUG    | Global X Cybersecurity ETF | Cybersecurity equities | Small |
| IHAK   | iShares Cybersecurity and Tech ETF | Cybersecurity & tech | Small |

---

## Cloud Computing & SaaS

**ETF Count: 4** | Proliferation Score: 60

| Ticker | Name | Focus | AUM Tier |
|--------|------|-------|----------|
| SKYY   | First Trust Cloud Computing ETF | Cloud computing | Medium |
| WCLD   | WisdomTree Cloud Computing Fund | Cloud software | Small |
| CLOU   | Global X Cloud Computing ETF | Cloud computing | Small |
| CLDL   | Direxion Daily Cloud Computing Bull 2X Shares | Cloud computing (leveraged) | Small |

---

## Biotech & Genomics

**ETF Count: 6** | Proliferation Score: 60

| Ticker | Name | Focus | AUM Tier |
|--------|------|-------|----------|
| XBI    | SPDR S&P Biotech ETF | Biotech equities (equal-weight) | Large |
| IBB    | iShares Biotechnology ETF | Biotech equities (cap-weight) | Large |
| ARKG   | ARK Genomic Revolution ETF | Genomics & biotech | Medium |
| GNOM   | Global X Genomics & Biotechnology ETF | Genomics | Small |
| IDNA   | iShares Genomics Immunology and Healthcare ETF | Genomics & immunology | Small |
| LABU   | Direxion Daily S&P Biotech Bull 3X Shares | Biotech (leveraged) | Medium |

---

## Infrastructure & Construction

**ETF Count: 3** | Proliferation Score: 40

| Ticker | Name | Focus | AUM Tier |
|--------|------|-------|----------|
| PAVE   | Global X U.S. Infrastructure Development ETF | US infrastructure | Large |
| IFRA   | iShares U.S. Infrastructure ETF | US infrastructure equities | Medium |
| SIMS   | SPDR S&P Kensho Intelligent Structures ETF | Smart infrastructure | Small |

---

## Cannabis

**ETF Count: 3** | Proliferation Score: 40

| Ticker | Name | Focus | AUM Tier |
|--------|------|-------|----------|
| MSOS   | AdvisorShares Pure US Cannabis ETF | US cannabis operators | Medium |
| MJ     | ETFMG Alternative Harvest ETF | Global cannabis | Small |
| THCX   | The Cannabis ETF | Cannabis equities | Small |

---

## Blockchain & Crypto

**ETF Count: 5** | Proliferation Score: 60

| Ticker | Name | Focus | AUM Tier |
|--------|------|-------|----------|
| BITO   | ProShares Bitcoin Strategy ETF | Bitcoin futures | Large |
| BLOK   | Amplify Transformational Data Sharing ETF | Blockchain companies | Medium |
| BITQ   | Bitwise Crypto Industry Innovators ETF | Crypto industry equities | Small |
| DAPP   | VanEck Digital Transformation ETF | Digital assets ecosystem | Small |
| IBIT   | iShares Bitcoin Trust ETF | Spot Bitcoin | Large |

---

## Defense & Aerospace

**ETF Count: 5** | Proliferation Score: 60

| Ticker | Name | Focus | AUM Tier |
|--------|------|-------|----------|
| ITA    | iShares U.S. Aerospace & Defense ETF | Aerospace & defense equities | Large |
| PPA    | Invesco Aerospace & Defense ETF | Aerospace & defense equities | Medium |
| ROKT   | SPDR S&P Kensho Final Frontiers ETF | Space & deep sea exploration | Small |
| UFO    | Procure Space ETF | Space industry | Small |
| ARKX   | ARK Space Exploration & Innovation ETF | Space exploration & innovation | Small |

---

## Obesity & GLP-1

**ETF Count: 2** | Proliferation Score: 40

| Ticker | Name | Focus | AUM Tier |
|--------|------|-------|----------|
| SLIM   | Themes Obesity & Cardiometabolic ETF | Obesity & cardiometabolic drugs | Small |
| HRTS   | Tema Cardiovascular and Metabolic ETF | Cardiovascular & metabolic health | Small |

---

## Nuclear Energy

**ETF Count: 3** | Proliferation Score: 40

| Ticker | Name | Focus | AUM Tier |
|--------|------|-------|----------|
| NLR    | VanEck Uranium+Nuclear Energy ETF | Uranium & nuclear energy | Medium |
| NUKZ   | Range Nuclear Renaissance Index ETF | Nuclear renaissance | Small |
| NUCL   | Horizons Global Uranium Index ETF | Global uranium | Small |

---

## Water Resources

**ETF Count: 3** | Proliferation Score: 40

| Ticker | Name | Focus | AUM Tier |
|--------|------|-------|----------|
| PHO    | Invesco Water Resources ETF | Water utilities & infrastructure | Medium |
| FIW    | First Trust Water ETF | Water industry | Medium |
| CGW    | Invesco S&P Global Water Index ETF | Global water equities | Small |

---

## Robotics & Automation

**ETF Count: 3** | Proliferation Score: 40

| Ticker | Name | Focus | AUM Tier |
|--------|------|-------|----------|
| ROBO   | Robo Global Robotics and Automation Index ETF | Robotics & automation | Medium |
| ARKQ   | ARK Autonomous Technology & Robotics ETF | Autonomous tech & robotics | Medium |
| IRBO   | iShares Robotics and Artificial Intelligence Multisector ETF | Robotics & AI (also in AI theme) | Small |

---

## Lithium & Battery Technology

**ETF Count: 2** | Proliferation Score: 40

| Ticker | Name | Focus | AUM Tier |
|--------|------|-------|----------|
| LIT    | Global X Lithium & Battery Tech ETF | Lithium mining & battery tech (also in Clean Energy) | Medium |
| BATT   | Amplify Lithium & Battery Technology ETF | Battery technology value chain | Small |

---

## Gold & Precious Metals

**ETF Count: 5** | Proliferation Score: 60

| Ticker | Name | Focus | AUM Tier |
|--------|------|-------|----------|
| GDX    | VanEck Gold Miners ETF | Gold mining equities | Large |
| GDXJ   | VanEck Junior Gold Miners ETF | Junior gold miners | Large |
| RING   | iShares MSCI Global Gold Miners ETF | Global gold miners | Medium |
| SGDM   | Sprott Gold Miners ETF | Gold miners (rules-based) | Small |
| SIL    | Global X Silver Miners ETF | Silver miners | Medium |

---

## Uranium

**ETF Count: 3** | Proliferation Score: 40

| Ticker | Name | Focus | AUM Tier |
|--------|------|-------|----------|
| URA    | Global X Uranium ETF | Uranium mining & nuclear components | Medium |
| URNM   | Sprott Uranium Miners ETF | Pure-play uranium miners | Medium |
| URNJ   | Sprott Junior Uranium Miners ETF | Junior uranium miners | Small |

---

## Oil & Gas (Energy)

**ETF Count: 6** | Proliferation Score: 60

| Ticker | Name | Focus | AUM Tier |
|--------|------|-------|----------|
| XLE    | Energy Select Sector SPDR Fund | US energy sector | Large |
| XOP    | SPDR S&P Oil & Gas Exploration & Production ETF | Oil & gas E&P | Large |
| OIH    | VanEck Oil Services ETF | Oil services equities | Medium |
| AMLP   | Alerian MLP ETF | Energy MLPs | Medium |
| FCG    | First Trust Natural Gas ETF | Natural gas equities | Small |
| CRAK   | VanEck Oil Refiners ETF | Oil refiners | Small |

---

## Financial Services & Banks

**ETF Count: 5** | Proliferation Score: 60

| Ticker | Name | Focus | AUM Tier |
|--------|------|-------|----------|
| XLF    | Financial Select Sector SPDR Fund | US financial sector | Large |
| KBE    | SPDR S&P Bank ETF | US bank equities | Medium |
| KRE    | SPDR S&P Regional Banking ETF | Regional bank equities | Large |
| IAI    | iShares U.S. Broker-Dealers & Securities Exchanges ETF | Broker-dealers | Small |
| KBWB   | Invesco KBW Bank ETF | Large-cap bank equities | Medium |

---

## Healthcare & Pharma

**ETF Count: 5** | Proliferation Score: 60

| Ticker | Name | Focus | AUM Tier |
|--------|------|-------|----------|
| XLV    | Health Care Select Sector SPDR Fund | US healthcare sector | Large |
| IHE    | iShares U.S. Pharmaceuticals ETF | US pharmaceutical equities | Medium |
| XHS    | SPDR S&P Health Care Services ETF | Healthcare services | Small |
| IHI    | iShares U.S. Medical Devices ETF | Medical devices | Medium |
| PPH    | VanEck Pharmaceutical ETF | Pharmaceutical equities | Small |

---

## Real Estate & REITs

**ETF Count: 4** | Proliferation Score: 60

| Ticker | Name | Focus | AUM Tier |
|--------|------|-------|----------|
| VNQ    | Vanguard Real Estate ETF | US REITs | Large |
| XLRE   | Real Estate Select Sector SPDR Fund | US real estate sector | Large |
| IYR    | iShares U.S. Real Estate ETF | US real estate equities | Large |
| MORT   | VanEck Mortgage REIT Income ETF | Mortgage REITs | Small |

---

## Retail & Consumer

**ETF Count: 4** | Proliferation Score: 60

| Ticker | Name | Focus | AUM Tier |
|--------|------|-------|----------|
| XLY    | Consumer Discretionary Select Sector SPDR Fund | Consumer discretionary | Large |
| XRT    | SPDR S&P Retail ETF | Retail equities | Medium |
| XLP    | Consumer Staples Select Sector SPDR Fund | Consumer staples | Large |
| IBUY   | Amplify Online Retail ETF | Online retail equities | Small |

---

## Summary Table

| Theme | ETF Count | Proliferation Score |
|-------|-----------|-------------------|
| AI & Semiconductors | 11 | 100 |
| Clean Energy & EV | 9 | 80 |
| Biotech & Genomics | 6 | 60 |
| Oil & Gas (Energy) | 6 | 60 |
| Gold & Precious Metals | 5 | 60 |
| Defense & Aerospace | 5 | 60 |
| Blockchain & Crypto | 5 | 60 |
| Financial Services & Banks | 5 | 60 |
| Healthcare & Pharma | 5 | 60 |
| Cloud Computing & SaaS | 4 | 60 |
| Cybersecurity | 4 | 60 |
| Real Estate & REITs | 4 | 60 |
| Retail & Consumer | 4 | 60 |
| Infrastructure & Construction | 3 | 40 |
| Cannabis | 3 | 40 |
| Nuclear Energy | 3 | 40 |
| Water Resources | 3 | 40 |
| Robotics & Automation | 3 | 40 |
| Uranium | 3 | 40 |
| Obesity & GLP-1 | 2 | 40 |
| Lithium & Battery Technology | 2 | 40 |

**Notes:**
- Some ETFs appear in multiple themes (e.g., LIT in both Clean Energy and Lithium, IRBO in both AI and Robotics)
- AUM Tier: Large (>$1B), Medium ($100M-$1B), Small (<$100M)
- Leveraged ETFs (LABU, CLDL) included for completeness but not recommended for theme exposure measurement
- Catalog should be updated quarterly to capture new ETF launches and closures

# Theme Detection Methodology

## Overview

The Theme Detector uses a **3-Dimensional Scoring Model** to identify, rank, and assess market themes. Unlike single-score ranking systems, this approach separates the intensity of a theme (heat), its maturity stage (lifecycle), and the reliability of the signal (confidence) into independent dimensions.

---

## Dimension 1: Theme Heat (0-100)

Theme Heat measures the **direction-neutral strength** of a theme. A high heat score means the theme is generating significant market activity, regardless of whether it is bullish or bearish.

### Components

#### 1.1 Momentum Strength (weight: 35%)

Measures the direction-neutral momentum strength using a log-sigmoid function on the absolute weighted return.

**Formula:**
```
weighted_return = perf_1w * 0.10 + perf_1m * 0.25 + perf_3m * 0.35 + perf_6m * 0.30
momentum_score = 100 / (1 + exp(-2.0 * (ln(1 + |weighted_return|) - ln(16))))
```

Midpoint at |15%| weighted return. Log transform compresses extreme values for better mid-range separation. Absolute value is used so both strong bullish and strong bearish moves generate high heat.

**Data Source:** FINVIZ industry performance (1W, 1M, 3M, 6M change %)

**Example Scores (continuous, not stepped):**
| |Weighted Return| | Score |
|---------------------|-------|
| 0% | ~3 |
| 5% | ~27 |
| 15% | 50 (midpoint) |
| 30% | ~73 |
| 50% | ~86 |

#### 1.2 Volume Intensity (weight: 20%)

Measures abnormal volume using sqrt scaling on the 20-day/60-day volume ratio.

**Formula:**
```
ratio = vol_20d / vol_60d
volume_score = min(100, sqrt(max(0, ratio - 0.8)) / sqrt(1.2) * 100)
```

Returns 50.0 if either volume input is None or zero.

**Data Source:** FINVIZ relative volume (volume / avg volume ratio)

**Example Scores:**
| Volume Ratio | Score |
|-------------|-------|
| 1.0 | ~37 |
| 1.2 | ~58 |
| 1.5 | ~76 |
| 2.0 | 100 (ceiling) |

#### 1.3 Uptrend Signal (weight: 25%)

Continuous score from sector uptrend data with base + bonus structure.

**Formula per sector:**
```
base = min(80, ratio * 100)        # continuous 0-80
ma_bonus = 10 if ratio > ma_10     # MA above bonus
slope_bonus = 10 if slope > 0      # positive slope bonus
sector_score = base + ma_bonus + slope_bonus  # 0-100
```

Final score = weighted average of sector scores. For bearish themes, result is inverted (100 - score).

**Data Source:** uptrend-dashboard output (ratio, ma_10, slope per sector)

Returns 50.0 if no sector data available.

#### 1.4 Breadth Signal (weight: 20%)

Measures directionally-aligned industry participation using a power curve with industry count bonus.

**Formula:**
```
breadth_score = min(100, ratio^2.5 * 80 + count_bonus)
count_bonus = min(20, industry_count * 2)
```

The ratio is the fraction of matched industries with directionally-aligned weighted returns (positive for LEAD themes, negative for LAG themes). Power curve (exponent 2.5) suppresses low ratios and amplifies high ones.

**Data Source:** Industry-level weighted returns (not stock-level)

**Example Scores (no count bonus):**
| Breadth Ratio | Score |
|-------------|-------|
| 0.5 | ~14 |
| 0.7 | ~33 |
| 0.9 | ~61 |
| 1.0 | 80 |

Returns 50.0 if ratio is None.

### Theme Heat Composite

```
theme_heat = (momentum * 0.35) + (volume * 0.20) + (uptrend * 0.25) + (breadth * 0.20)
```

Any None sub-score defaults to 50.0. Result clamped to 0-100.

---

## Dimension 2: Lifecycle Maturity

Lifecycle assessment classifies a theme into one of five stages: **Emerging**, **Accelerating**, **Trending**, **Mature**, or **Exhausting**. This is critical for distinguishing emerging opportunities from crowded trades.

### Components

#### 2.1 Duration Score (weight: 25%)

How long the theme has been active (consecutive weeks of elevated heat).

**Measurement:** Count weeks where theme_heat >= 40.

| Duration | Stage Signal |
|----------|-------------|
| 1-3 weeks | Emerging |
| 4-8 weeks | Accelerating |
| 9-12 weeks | Trending |
| 13-16 weeks | Mature |
| > 16 weeks | Exhausting |

**Limitation:** Duration tracking requires historical data. On first run, duration defaults to "Unknown" and lifecycle uses other factors only.

#### 2.2 Extremity Clustering Score (weight: 25%)

Percentage of stocks in the theme near 52-week highs or lows.

**Formula:**
```
extremity_pct = count(within_5pct_of_52wk_high_or_low) / count(total_stocks)
```

| Extremity % | Stage Signal |
|-------------|-------------|
| < 15%       | Emerging |
| 15-30%      | Accelerating |
| 30-45%      | Trending |
| 45-60%      | Mature |
| > 60%       | Exhausting |

**Data Source:** FINVIZ 52-week high/low data

#### 2.3 Price Extreme Saturation Score (weight: 25%)

Proportion of stocks near 52-week extremes (within 5%).

**Formula:**
```
bullish: pct = count(dist_from_52w_high <= 0.05) / total
bearish: pct = count(dist_from_52w_low <= 0.05) / total
score = min(100, pct * 200)
```

#### 2.4 Valuation Score (weight: 15%)

Average P/E ratio of theme constituents relative to S&P 500 P/E.

**Formula:**
```
relative_pe = avg_theme_pe / sp500_pe
```

| Relative P/E | Stage Signal |
|---------------|-------------|
| < 0.8         | Emerging (undervalued) |
| 0.8-1.0       | Accelerating (fair value) |
| 1.0-1.4       | Trending (slightly elevated) |
| 1.4-2.0       | Mature (overvalued) |
| > 2.0         | Exhausting (extreme) |

**Data Source:** FMP API for P/E ratios (optional; uses FINVIZ forward P/E as fallback)

#### 2.5 ETF Proliferation Score (weight: 10%)

Number of thematic ETFs tracking the theme. More ETFs indicate greater retail/institutional attention.

**Source:** `thematic_etf_catalog.md` (static reference)

| ETF Count | Score | Stage Signal |
|-----------|-------|-------------|
| 0         | 0     | Emerging |
| 1         | 20    | Emerging |
| 2-3       | 40    | Accelerating |
| 4-6       | 60    | Trending |
| 7-10      | 80    | Mature |
| > 10      | 100   | Exhausting |

### Lifecycle Maturity Composite

```
maturity = (duration * 0.25) + (extremity * 0.25) + (price_extreme * 0.25) + (valuation * 0.15) + (etf_proliferation * 0.10)
```

### Lifecycle Stage Classification

The lifecycle stage is classified from the maturity score:

| Maturity Score | Stage |
|----------------|-------|
| 0-20 | Emerging |
| 20-40 | Accelerating |
| 40-60 | Trending |
| 60-80 | Mature |
| 80-100 | Exhausting |

**Note:** Media/Narrative Saturation is not included in the automated maturity calculation. Claude's WebSearch narrative confirmation can be used to qualitatively adjust the lifecycle assessment.

---

## Dimension 3: Confidence (Low / Medium / High)

Confidence measures **how reliable** the theme detection is, based on data quality and confirmation signals.

### Layers

#### 3.1 Quantitative Layer (base)

Based on data breadth and consistency:

| Condition | Level |
|-----------|-------|
| >= 4 industries matching, >= 20 stocks analyzed | High |
| 2-3 industries matching, >= 10 stocks analyzed | Medium |
| 1 industry matching or < 10 stocks | Low |

#### 3.2 Breadth Layer (modifier)

Cross-sector participation adds confidence:

| Condition | Modifier |
|-----------|----------|
| Theme spans 3+ sectors | +1 level (cap at High) |
| Theme spans 2 sectors | No change |
| Theme in 1 sector only | -1 level (floor at Low) |

#### 3.3 Narrative Layer (modifier, applied in Step 4)

WebSearch confirmation adjusts confidence:

| Narrative Finding | Modifier |
|-------------------|----------|
| Strong confirmation (multiple sources, clear catalysts) | +1 level |
| Mixed signals | No change |
| Contradictory narrative (bearish articles for bullish theme) | -1 level |

### Final Confidence

```
confidence = apply_modifiers(quantitative_base, breadth_modifier, narrative_modifier)
confidence = clamp(confidence, Low, High)
```

---

## Direction Detection

Theme direction (**leading** vs. **lagging**) is determined by relative rank, not absolute price change:

### Algorithm

1. Each industry gets a `rank_direction` based on its position in the momentum-ranked list: top half = "bullish" (leading), bottom half = "bearish" (lagging).
2. Theme direction is the majority vote of its constituent industries' `rank_direction`.

```python
# Industry-level (industry_ranker.py)
rank_direction = "bullish" if rank <= len(industries) // 2 else "bearish"

# Theme-level (theme_classifier.py)
direction = majority_vote([ind.rank_direction for ind in theme.industries])
```

### Important: Relative, Not Absolute

**LEAD/LAG direction is relative.** A "lagging" theme may still have positive absolute returns — it simply underperforms relative to other themes. This means:
- **LEAD** themes: Suitable for overweight / new positions
- **LAG** themes: Candidates for underweight reduction, **not** short signals

---

## Data Sources

### Primary: FINVIZ

**Elite Mode (recommended):**
- CSV export endpoint: `https://elite.finviz.com/export.ashx?v=151&f=ind_{code},cap_smallover,...&auth=KEY`
- Provides: ticker, company, sector, industry, market cap, P/E, change%, volume, avg volume, 52wk high/low, RSI, SMA20/50/200
- Rate limit: 0.5s between requests
- Coverage: Full stock universe per industry

**Public Mode (fallback):**
- HTML scraping: `https://finviz.com/screener.ashx?v=151&f=ind_{code},cap_smallover`
- Provides: Same fields but limited to page 1 (~20 stocks per industry)
- Rate limit: 2.0s between requests (aggressive scraping may trigger blocks)
- Coverage: Top 20 stocks per industry by market cap

### Secondary: FMP API (optional)

- P/E ratios for valuation analysis
- Not required; FINVIZ forward P/E used as fallback
- Useful for more granular valuation metrics

### Tertiary: uptrend-dashboard (optional)

- CSV output from the uptrend-dashboard skill
- Provides 3-point technical evaluation per stock
- Significantly improves uptrend ratio accuracy
- If unavailable, FINVIZ price-vs-SMA200 is used as proxy

### Quaternary: WebSearch (narrative layer)

- Used for narrative confirmation in Step 4
- Not automated; Claude performs searches during workflow
- Subjective assessment of media coverage and analyst sentiment

---

## Integration with uptrend-dashboard

When uptrend-dashboard data is available, the theme detector uses it for enhanced 3-point evaluation:

**3-Point Evaluation Criteria:**
1. Price > 50-day SMA (short-term trend)
2. 50-day SMA > 200-day SMA (medium-term trend, golden/death cross)
3. 200-day SMA is rising (long-term trend confirmation)

**Stocks meeting all 3 points** = in confirmed uptrend
**Stocks meeting 0 points** = in confirmed downtrend

This provides more accurate uptrend ratios than simple price-above-SMA200 proxy.

---

## Output Schema

### JSON Output Structure

```json
{
  "metadata": {
    "date": "2026-02-16",
    "mode": "elite",
    "themes_analyzed": 14,
    "industries_scanned": 145,
    "total_stocks": 5200,
    "uptrend_data_available": true,
    "execution_time_seconds": 150
  },
  "themes": [
    {
      "name": "AI & Semiconductors",
      "direction": "Bullish",
      "direction_strength": "Strong",
      "theme_heat": 82,
      "heat_components": {
        "momentum": 90,
        "volume": 75,
        "uptrend": 85,
        "breadth": 70
      },
      "lifecycle": {
        "stage": "Mature",
        "duration_weeks": 12,
        "extremity_pct": 45,
        "relative_pe": 1.8,
        "etf_proliferation_score": 100,
        "etf_count": 11
      },
      "confidence": "High",
      "confidence_components": {
        "quantitative": "High",
        "breadth_modifier": "+1",
        "narrative_modifier": "pending"
      },
      "industries": [
        {
          "name": "Semiconductors",
          "change_pct": 4.2,
          "avg_relative_volume": 1.8,
          "uptrend_ratio": 0.75,
          "stock_count": 35
        }
      ],
      "top_stocks": [
        {"ticker": "NVDA", "change_pct": 6.5, "relative_volume": 2.1},
        {"ticker": "AVGO", "change_pct": 4.8, "relative_volume": 1.9}
      ],
      "proxy_etfs": ["SMH", "SOXX", "AIQ", "BOTZ"]
    }
  ],
  "industry_rankings": {
    "top_10": [...],
    "bottom_10": [...]
  },
  "sector_summary": {
    "Technology": {"uptrend_ratio": 0.65, "avg_change_pct": 2.1},
    "Energy": {"uptrend_ratio": 0.40, "avg_change_pct": -1.5}
  }
}
```

---

## Known Limitations

1. **Static theme definitions**: Cross-sector themes are predefined in `cross_sector_themes.md`. New themes (e.g., a sudden meme-stock theme) are not automatically detected.

2. **Industry granularity**: FINVIZ industry classification may not perfectly map to investment themes. Some industries span multiple themes.

3. **Single-stock dominance**: Large-cap stocks (e.g., NVDA in AI) can skew theme-level metrics. Market-cap weighting amplifies this effect.

4. **Temporal lag**: Weekly performance data does not capture intraday or same-day momentum shifts.

5. **ETF catalog staleness**: The thematic ETF catalog is manually maintained and may not reflect recent ETF launches or closures.

6. **Public mode data limits**: Only ~20 stocks per industry are captured, which may underrepresent small/mid-cap participation.

7. **Duration tracking**: First-run analysis cannot assess theme duration without historical baseline data.

8. **Narrative subjectivity**: Confidence adjustment from WebSearch is inherently subjective and depends on search result quality.

9. **Survivorship bias**: Analysis only covers currently listed stocks and active ETFs, missing delisted or closed instruments.

10. **FINVIZ data delays**: Public FINVIZ data is 15-minute delayed; Elite provides real-time during market hours.

# GENERATED by scripts/generate_market_calendar.py — do not edit.
# Source of truth: scripts/market_calendar/ and config/market-calendar-dependency.yaml.
pandas-market-calendars==5.2.2
curl-cffi>=0.7.0  # optional: yfinance transport exception support; FMP remains available without it
finvizfinance>=1.0.0  # optional: FINVIZ client degrades to static config (HAS_FINVIZFINANCE)
numpy>=1.26.0
pandas>=2.0.0
pyyaml>=6.0
requests>=2.31.0
yfinance>=0.2.0  # optional: ETF scanner falls back to FMP/static paths (HAS_YFINANCE)

# GENERATED by scripts/generate_market_calendar.py — do not edit.
# Source of truth: scripts/market_calendar/ and config/market-calendar-dependency.yaml.
"""Shared exchange-calendar and timezone contract for cash-equity skills.

The module is vendored into each standalone skill by
``scripts/generate_market_calendar.py``. Runtime code must not read repository
configuration: packaged skills contain this self-contained module plus an exact
runtime dependency declaration.
"""

from __future__ import annotations

from dataclasses import dataclass
from datetime import date, datetime, timedelta
from typing import Any, Optional
from zoneinfo import ZoneInfo


class CalendarUnavailableError(RuntimeError):
    """Raised when an exchange calendar cannot be loaded or queried safely."""


@dataclass(frozen=True)
class Session:
    """One regular cash-equity session in the venue's IANA timezone."""

    venue: str
    session_date: date
    timezone: str
    market_open: datetime
    market_close: datetime
    break_start: Optional[datetime] = None
    break_end: Optional[datetime] = None


_VENUES = {
    "XNYS": ("NYSE", "America/New_York"),
    # pandas-market-calendars exposes NASDAQ as an alias of its NYSE calendar.
    # Both US cash venues intentionally share this session set.
    "XNAS": ("NASDAQ", "America/New_York"),
    "XTKS": ("JPX", "Asia/Tokyo"),
    "XLON": ("LSE", "Europe/London"),
}


def supported_venues() -> tuple[str, ...]:
    """Return supported venue identifiers in deterministic order."""
    return tuple(sorted(_VENUES))


def _venue_contract(venue: str) -> tuple[str, str, str]:
    if not isinstance(venue, str):
        raise ValueError("venue must be a string")
    normalized = venue.strip().upper()
    try:
        provider_name, timezone_name = _VENUES[normalized]
    except KeyError as exc:
        raise ValueError(
            f"Unsupported venue {venue!r}; expected one of {', '.join(supported_venues())}"
        ) from exc
    return normalized, provider_name, timezone_name


def _require_date(value: date, *, name: str) -> date:
    if type(value) is not date:
        raise TypeError(f"{name} must be datetime.date (not datetime)")
    return value


def require_aware(value: datetime, *, name: str = "timestamp") -> datetime:
    """Reject naive datetimes at the shared boundary."""
    if not isinstance(value, datetime) or value.tzinfo is None:
        raise ValueError(f"{name} must be a timezone-aware datetime")
    if value.utcoffset() is None:
        raise ValueError(f"{name} must have a resolvable UTC offset")
    return value


def parse_aware_datetime(value: str) -> datetime:
    """Parse an offset-bearing ISO-8601 timestamp, accepting a trailing ``Z``."""
    if not isinstance(value, str) or not value.strip():
        raise ValueError("as-of must be a non-empty ISO-8601 timestamp")
    normalized = value.strip()
    if normalized.endswith(("Z", "z")):
        normalized = normalized[:-1] + "+00:00"
    try:
        parsed = datetime.fromisoformat(normalized)
    except ValueError as exc:
        raise ValueError(f"Invalid ISO-8601 as-of timestamp: {value!r}") from exc
    return require_aware(parsed, name="as-of")


def _provider_calendar(venue: str) -> tuple[str, str, Any]:
    normalized, provider_name, timezone_name = _venue_contract(venue)
    try:
        import pandas_market_calendars as market_calendars
    except (ImportError, TypeError) as exc:
        raise CalendarUnavailableError(
            "pandas-market-calendars==5.2.2 is required; install the skill's requirements.txt"
        ) from exc
    try:
        calendar = market_calendars.get_calendar(provider_name)
    except Exception as exc:
        raise CalendarUnavailableError(
            f"Could not load exchange calendar for {normalized} ({provider_name}): {exc}"
        ) from exc
    return normalized, timezone_name, calendar


def _schedule(venue: str, start_date: date, end_date: date):
    normalized, timezone_name, calendar = _provider_calendar(venue)
    try:
        schedule = calendar.schedule(
            start_date=start_date,
            end_date=end_date,
            tz=timezone_name,
            market_times="all",
        )
    except Exception as exc:
        raise CalendarUnavailableError(
            f"Could not query exchange calendar for {normalized} "
            f"from {start_date} through {end_date}: {exc}"
        ) from exc
    return normalized, timezone_name, schedule


def _as_datetime(value: Any, *, name: str) -> Optional[datetime]:
    if value is None or str(value) == "NaT":
        return None
    converter = getattr(value, "to_pydatetime", None)
    converted = converter() if converter is not None else value
    if not isinstance(converted, datetime):
        raise CalendarUnavailableError(f"Calendar returned invalid {name}: {value!r}")
    return require_aware(converted, name=name)


def _session_from_row(
    venue: str,
    timezone_name: str,
    session_date: date,
    row: Any,
) -> Session:
    market_open = _as_datetime(row.get("market_open"), name="market_open")
    market_close = _as_datetime(row.get("market_close"), name="market_close")
    if market_open is None or market_close is None:
        raise CalendarUnavailableError(
            f"Calendar returned an incomplete session for {venue} on {session_date}"
        )
    break_start = _as_datetime(row.get("break_start"), name="break_start")
    break_end = _as_datetime(row.get("break_end"), name="break_end")
    if (break_start is None) != (break_end is None):
        raise CalendarUnavailableError(
            f"Calendar returned an incomplete break for {venue} on {session_date}"
        )
    return Session(
        venue=venue,
        session_date=session_date,
        timezone=timezone_name,
        market_open=market_open,
        market_close=market_close,
        break_start=break_start,
        break_end=break_end,
    )


def session_for_date(venue: str, session_date: date) -> Optional[Session]:
    """Return a session, or ``None`` for a valid holiday/weekend."""
    session_date = _require_date(session_date, name="session_date")
    normalized, timezone_name, schedule = _schedule(venue, session_date, session_date)
    if schedule.empty:
        return None
    return _session_from_row(normalized, timezone_name, session_date, schedule.iloc[0])


def is_open_at(venue: str, timestamp: datetime) -> bool:
    """Return whether ``timestamp`` is inside regular trading hours."""
    timestamp = require_aware(timestamp)
    normalized, _, timezone_name = _venue_contract(venue)
    local = timestamp.astimezone(ZoneInfo(timezone_name))
    session = session_for_date(normalized, local.date())
    if session is None:
        return False
    if not (session.market_open <= local < session.market_close):
        return False
    if session.break_start is not None and session.break_end is not None:
        if session.break_start <= local < session.break_end:
            return False
    return True


def next_session_on_or_after(
    venue: str,
    start_date: date,
    *,
    max_search_days: int = 370,
) -> Session:
    """Return the first session on/after ``start_date`` within a finite horizon."""
    start_date = _require_date(start_date, name="start_date")
    if not isinstance(max_search_days, int) or isinstance(max_search_days, bool):
        raise TypeError("max_search_days must be an integer")
    if max_search_days < 0:
        raise ValueError("max_search_days must be non-negative")
    end_date = start_date + timedelta(days=max_search_days)
    normalized, timezone_name, schedule = _schedule(venue, start_date, end_date)
    if schedule.empty:
        raise CalendarUnavailableError(
            f"No {normalized} session found within {max_search_days} days of {start_date}"
        )
    label = schedule.index[0]
    label_date = label.date() if hasattr(label, "date") else date.fromisoformat(str(label)[:10])
    return _session_from_row(normalized, timezone_name, label_date, schedule.iloc[0])


def next_session_after(
    venue: str,
    session_date: date,
    *,
    max_search_days: int = 370,
) -> Session:
    """Return the first session strictly after ``session_date``."""
    session_date = _require_date(session_date, name="session_date")
    if max_search_days <= 0:
        raise ValueError("max_search_days must be positive")
    return next_session_on_or_after(
        venue,
        session_date + timedelta(days=1),
        max_search_days=max_search_days - 1,
    )


def previous_session_on_or_before(
    venue: str,
    end_date: date,
    *,
    max_search_days: int = 370,
) -> Session:
    """Return the last session on/before ``end_date`` within a finite horizon."""
    end_date = _require_date(end_date, name="end_date")
    if not isinstance(max_search_days, int) or isinstance(max_search_days, bool):
        raise TypeError("max_search_days must be an integer")
    if max_search_days < 0:
        raise ValueError("max_search_days must be non-negative")
    start_date = end_date - timedelta(days=max_search_days)
    normalized, timezone_name, schedule = _schedule(venue, start_date, end_date)
    if schedule.empty:
        raise CalendarUnavailableError(
            f"No {normalized} session found within {max_search_days} days of {end_date}"
        )
    label = schedule.index[-1]
    label_date = label.date() if hasattr(label, "date") else date.fromisoformat(str(label)[:10])
    return _session_from_row(normalized, timezone_name, label_date, schedule.iloc[-1])


def previous_session_before(
    venue: str,
    session_date: date,
    *,
    max_search_days: int = 370,
) -> Session:
    """Return the last session strictly before ``session_date``."""
    session_date = _require_date(session_date, name="session_date")
    if not isinstance(max_search_days, int) or isinstance(max_search_days, bool):
        raise TypeError("max_search_days must be an integer")
    if max_search_days <= 0:
        raise ValueError("max_search_days must be positive")
    return previous_session_on_or_before(
        venue,
        session_date - timedelta(days=1),
        max_search_days=max_search_days - 1,
    )


def current_or_next_session(venue: str, timestamp: datetime) -> Session:
    """Return today's not-yet-closed session, otherwise the next session."""
    timestamp = require_aware(timestamp)
    normalized, _, timezone_name = _venue_contract(venue)
    local = timestamp.astimezone(ZoneInfo(timezone_name))
    session = session_for_date(normalized, local.date())
    if session is not None and local < session.market_close:
        return session
    return next_session_after(normalized, local.date())


def count_sessions(
    venue: str,
    start_date: date,
    end_date: date,
    *,
    include_start: bool,
    include_end: bool,
) -> int:
    """Count sessions with explicit date-boundary inclusion semantics."""
    start_date = _require_date(start_date, name="start_date")
    end_date = _require_date(end_date, name="end_date")
    if start_date > end_date:
        raise ValueError("start_date must be on or before end_date")
    first = start_date if include_start else start_date + timedelta(days=1)
    last = end_date if include_end else end_date - timedelta(days=1)
    if first > last:
        return 0
    _, _, schedule = _schedule(venue, first, last)
    return int(len(schedule.index))

# Theme Detector calculators package

"""
Theme Heat Calculator (0-100)

ThemeHeat = momentum_strength * 0.35
           + volume_intensity * 0.20
           + uptrend_signal * 0.25
           + breadth_signal * 0.20

All sub-scores are direction-neutral (0-100 for "strength").
Formulas are designed to spread scores across the full 0-100 range,
avoiding ceiling effects that compress most themes into 80+.
"""

import math
from typing import Optional

HEAT_WEIGHTS = {
    "momentum_strength": 0.35,
    "volume_intensity": 0.20,
    "uptrend_signal": 0.25,
    "breadth_signal": 0.20,
}


def momentum_strength_score(weighted_return_pct: float) -> float:
    """Log-sigmoid score based on absolute weighted return.

    Formula: 100 / (1 + exp(-2.0 * (ln(1 + |wr|) - ln(16))))

    Midpoint at |wr| = 15% (typical strong industry weighted return).
    Log transform compresses extreme values for better mid-range separation.

    Examples:
        |0%|  -> ~3
        |5%|  -> ~27
        |15%| -> 50 (midpoint)
        |30%| -> ~73
        |50%| -> ~86
    """
    x = abs(weighted_return_pct)
    log_x = math.log(1.0 + x)
    log_mid = math.log(16.0)
    return 100.0 / (1.0 + math.exp(-2.0 * (log_x - log_mid)))


def volume_intensity_score(vol_20d: Optional[float], vol_60d: Optional[float]) -> Optional[float]:
    """Score based on short-term vs long-term volume ratio using sqrt scaling.

    Formula: min(100, sqrt(max(0, ratio - 0.8)) / sqrt(1.2) * 100)

    Ceiling at ratio=2.0 instead of 1.2, with better mid-range separation.

    Examples:
        ratio=1.0  -> ~37
        ratio=1.2  -> ~58
        ratio=1.5  -> ~76
        ratio=2.0  -> 100

    Returns None if either input is None or vol_60d == 0.
    """
    if vol_20d is None or vol_60d is None or vol_60d == 0:
        return None
    ratio = vol_20d / vol_60d
    raw = max(0.0, ratio - 0.8)
    return min(100.0, math.sqrt(raw) / math.sqrt(1.2) * 100.0)


def uptrend_signal_score(sector_data: list[dict], is_bearish: bool) -> Optional[float]:
    """Continuous score from sector uptrend data.

    Each sector entry: {"sector", "ratio", "ma_10", "slope", "weight"}

    Scoring per sector:
      base = min(80, ratio * 100)   # continuous 0-80
      +10 if ratio > ma_10          # MA above bonus
      +10 if slope > 0              # positive slope bonus
      Total: 0-100 continuous

    If is_bearish: result = 100 - weighted_average
    Returns None if empty.
    """
    if not sector_data:
        return None

    total_weight = 0.0
    weighted_sum = 0.0

    for entry in sector_data:
        ratio = entry.get("ratio") or 0
        ma_10 = entry.get("ma_10") or 0
        slope = entry.get("slope") or 0
        weight = entry.get("weight") or 1.0

        base = min(80.0, ratio * 100.0)
        ma_bonus = 10.0 if ratio > ma_10 else 0.0
        slope_bonus = 10.0 if slope > 0 else 0.0
        direction_score = base + ma_bonus + slope_bonus

        weighted_sum += direction_score * weight
        total_weight += weight

    if total_weight == 0:
        return None

    result = weighted_sum / total_weight

    if is_bearish:
        result = 100.0 - result

    return result


def breadth_signal_score(
    positive_ratio: Optional[float], industry_count: int = 0
) -> Optional[float]:
    """Score based on breadth ratio (0-1) with power curve and industry count bonus.

    Formula: min(100, ratio^2.5 * 80 + count_bonus)
    count_bonus = min(20, industry_count * 2)

    Power curve (exponent 2.5) suppresses low ratios and amplifies high ones,
    creating better separation in the 0.5-1.0 range.

    Examples (no bonus):
        ratio=0.5  -> ~14
        ratio=0.7  -> ~33
        ratio=0.9  -> ~61
        ratio=1.0  -> 80

    Returns None if positive_ratio is None.
    """
    if positive_ratio is None:
        return None
    count_bonus = min(20.0, industry_count * 2.0)
    raw = math.pow(max(0.0, positive_ratio), 2.5) * 80.0 + count_bonus
    return min(100.0, max(0.0, raw))


def calculate_theme_heat(
    momentum: Optional[float],
    volume: Optional[float],
    uptrend: Optional[float],
    breadth: Optional[float],
) -> float:
    """Weighted available sub-scores, clamped 0-100.

    Missing inputs are excluded from the denominator rather than defaulting
    to neutral 50.0. Returns 0.0 if no components are available.
    """
    detailed = calculate_theme_heat_detailed(momentum, volume, uptrend, breadth)
    return float(detailed["score"] or 0.0)


def calculate_theme_heat_detailed(
    momentum: Optional[float],
    volume: Optional[float],
    uptrend: Optional[float],
    breadth: Optional[float],
) -> dict:
    """Calculate heat score plus coverage metadata.

    Returns:
        {
            "score": float | None,
            "coverage": float,
            "missing_components": [str, ...],
            "components": {name: value | None},
        }
    """
    components = {
        "momentum_strength": momentum,
        "volume_intensity": volume,
        "uptrend_signal": uptrend,
        "breadth_signal": breadth,
    }
    score, coverage = weighted_available_score(components, HEAT_WEIGHTS)
    missing = [name for name, value in components.items() if value is None]
    return {
        "score": None if score is None else float(min(100.0, max(0.0, score))),
        "coverage": coverage,
        "missing_components": missing,
        "components": components,
    }


def weighted_available_score(
    components: dict[str, Optional[float]], weights: dict[str, float]
) -> tuple[Optional[float], float]:
    """Weighted score using only available components.

    Missing data reduces coverage instead of contributing a neutral score.
    """
    available = {k: v for k, v in components.items() if v is not None}
    total_possible = sum(weights.values())
    total_weight = sum(weights[k] for k in available if k in weights)
    if total_weight == 0 or total_possible == 0:
        return None, 0.0

    score = sum(float(available[k]) * weights[k] for k in available if k in weights) / total_weight
    return score, total_weight / total_possible

#!/usr/bin/env python3
"""
Industry Ranker - Momentum scoring and ranking for industries.

Calculates direction-neutral momentum strength using a sigmoid function,
then ranks industries by weighted multi-timeframe returns.

Timeframe Weights:
- 1W:  10%
- 1M:  25%
- 3M:  35%
- 6M:  30%
Total: 100%
"""

import math

TIMEFRAME_WEIGHTS: dict[str, float] = {
    "perf_1w": 0.10,
    "perf_1m": 0.25,
    "perf_3m": 0.35,
    "perf_6m": 0.30,
}


def momentum_strength_score(weighted_return_pct: float) -> float:
    """
    Direction-neutral sigmoid momentum score.

    Formula: 100 / (1 + exp(-0.15 * (abs(weighted_return) - 5.0)))

    Returns a score in [0, 100] where:
        |0%|  -> ~32
        |5%|  -> 50 (midpoint)
        |10%| -> ~68
        |15%| -> ~82
        |20%| -> ~90
    """
    x = abs(weighted_return_pct)
    return 100.0 / (1.0 + math.exp(-0.15 * (x - 5.0)))


def rank_industries(industries: list[dict]) -> list[dict]:
    """
    Rank industries by momentum strength score.

    Each input dict must have keys: name, perf_1w, perf_1m, perf_3m, perf_6m.
    Additional keys are preserved.

    Adds fields: weighted_return, momentum_score, direction, rank,
    strength_rank, signed_rank, rank_direction.
    Returns list sorted by momentum_score descending.
    """
    if not industries:
        return []

    scored = []
    for ind in industries:
        weighted_return = sum(
            ind.get(key, 0.0) * weight for key, weight in TIMEFRAME_WEIGHTS.items()
        )
        score = momentum_strength_score(weighted_return)
        direction = "bullish" if weighted_return > 0 else "bearish"

        entry = dict(ind)
        entry["weighted_return"] = round(weighted_return, 4)
        entry["momentum_score"] = round(score, 2)
        entry["direction"] = direction
        scored.append(entry)

    signed_ranked = sorted(scored, key=lambda x: x["weighted_return"], reverse=True)
    signed_mid = len(signed_ranked) // 2
    for i, entry in enumerate(signed_ranked, start=1):
        entry["signed_rank"] = i
        entry["rank_direction"] = "bullish" if i <= signed_mid else "bearish"

    scored.sort(key=lambda x: x["momentum_score"], reverse=True)

    for i, entry in enumerate(scored, start=1):
        entry["rank"] = i
        entry["strength_rank"] = i

    return scored


def get_top_bottom_industries(ranked: list[dict], n: int = 5) -> dict[str, list[dict]]:
    """
    Extract top N and bottom N industries from a ranked list.

    Args:
        ranked: List sorted by momentum_score descending (output of rank_industries).
        n: Number of industries for each group.

    Returns:
        {"top": [...], "bottom": [...]}
    """
    if not ranked:
        return {"top": [], "bottom": []}

    top = ranked[:n]
    bottom = ranked[-n:] if len(ranked) >= n else ranked[:]

    return {"top": top, "bottom": bottom}

"""
Lifecycle Maturity Calculator (0-100)

Maturity = duration * 0.25
         + extremity * 0.25
         + price_extreme * 0.25
         + valuation * 0.15
         + etf_proliferation * 0.10

All sub-scores are direction-aware.
"""

import statistics
from typing import Optional

LIFECYCLE_WEIGHTS = {
    "duration": 0.25,
    "extremity": 0.25,
    "price_extreme": 0.25,
    "valuation": 0.15,
    "etf_proliferation": 0.10,
}


def estimate_duration_score(
    perf_1m: Optional[float],
    perf_3m: Optional[float],
    perf_6m: Optional[float],
    perf_1y: Optional[float],
    is_bearish: bool,
) -> float:
    """Count horizons where trend is active. Each active = 25 points.

    Bullish: perf > 2%
    Bearish: perf < -2%
    None values treated as inactive.
    """
    horizons = [perf_1m, perf_3m, perf_6m, perf_1y]
    count = 0
    for p in horizons:
        if p is None:
            continue
        if is_bearish and p < -2.0:
            count += 1
        elif not is_bearish and p > 2.0:
            count += 1
    return float(count * 25)


def extremity_clustering_score(stock_metrics: list[dict], is_bearish: bool) -> Optional[float]:
    """Proportion of stocks at RSI extremes.

    Bullish: count RSI > 70
    Bearish: count RSI < 30
    Formula: min(100, pct * 200)
    Returns None if empty or no valid RSI values.
    """
    if not stock_metrics:
        return None

    valid = [s for s in stock_metrics if s.get("rsi") is not None]
    if not valid:
        return None

    if is_bearish:
        extreme_count = sum(1 for s in valid if s["rsi"] < 30)
    else:
        extreme_count = sum(1 for s in valid if s["rsi"] > 70)

    pct = extreme_count / len(valid)
    return min(100.0, pct * 200.0)


def price_extreme_saturation_score(stock_metrics: list[dict], is_bearish: bool) -> Optional[float]:
    """Proportion of stocks near 52-week extremes.

    Bullish: dist_from_52w_high <= 0.05
    Bearish: dist_from_52w_low <= 0.05
    Formula: min(100, pct * 200)
    Returns None if empty or no valid 52-week distance values.
    """
    if not stock_metrics:
        return None

    key = "dist_from_52w_low" if is_bearish else "dist_from_52w_high"
    valid = [s for s in stock_metrics if s.get(key) is not None]
    if not valid:
        return None

    near_count = sum(1 for s in valid if s[key] <= 0.05)
    pct = near_count / len(valid)
    return min(100.0, pct * 200.0)


def valuation_premium_score(stock_metrics: list[dict]) -> Optional[float]:
    """Score based on median P/E relative to market average (22).

    premium_ratio = median_PE / 22.0
    Score: min(100, max(0, (premium_ratio - 0.5) * 32))
    Needs 3+ valid P/E values, else returns None.
    """
    valid_pe = [
        s["pe_ratio"] for s in stock_metrics if s.get("pe_ratio") is not None and s["pe_ratio"] > 0
    ]

    if len(valid_pe) < 3:
        return None

    median_pe = statistics.median(valid_pe)
    premium_ratio = median_pe / 22.0
    return min(100.0, max(0.0, (premium_ratio - 0.5) * 32.0))


def etf_proliferation_score(etf_count: int) -> float:
    """Score based on number of theme-related ETFs.

    0 => 0, 1 => 20, <=3 => 40, <=6 => 60, <=10 => 80, >10 => 100
    """
    if etf_count == 0:
        return 0.0
    elif etf_count == 1:
        return 20.0
    elif etf_count <= 3:
        return 40.0
    elif etf_count <= 6:
        return 60.0
    elif etf_count <= 10:
        return 80.0
    else:
        return 100.0


def has_sufficient_lifecycle_data(
    extremity: Optional[float], price_extreme: Optional[float], valuation: Optional[float]
) -> bool:
    """Check whether stock-derived lifecycle sub-scores have real data.

    Returns False if all three stock-based sub-scores are None (indicating
    no stock metrics were available). Duration and etf_proliferation are
    industry-level scores and always available.
    """
    return not (extremity is None and price_extreme is None and valuation is None)


def classify_stage(maturity: float) -> str:
    """Classify lifecycle stage from maturity score.

    0-20: Emerging, 20-40: Accelerating, 40-60: Trending,
    60-80: Mature, 80-100: Exhausting
    """
    if maturity < 20:
        return "Emerging"
    elif maturity < 40:
        return "Accelerating"
    elif maturity < 60:
        return "Trending"
    elif maturity < 80:
        return "Mature"
    else:
        return "Exhausting"


def calculate_lifecycle_maturity(
    duration: Optional[float],
    extremity: Optional[float],
    price_extreme: Optional[float],
    valuation: Optional[float],
    etf_prolif: Optional[float],
) -> float:
    """Weighted available lifecycle sub-scores, clamped 0-100.

    Missing inputs are excluded from the denominator rather than defaulting
    to neutral 50.0. Returns 0.0 if no components are available.
    """
    detailed = calculate_lifecycle_maturity_detailed(
        duration, extremity, price_extreme, valuation, etf_prolif
    )
    return float(detailed["score"] or 0.0)


def calculate_lifecycle_maturity_detailed(
    duration: Optional[float],
    extremity: Optional[float],
    price_extreme: Optional[float],
    valuation: Optional[float],
    etf_prolif: Optional[float],
) -> dict:
    """Calculate lifecycle maturity plus coverage metadata."""
    components = {
        "duration": duration,
        "extremity": extremity,
        "price_extreme": price_extreme,
        "valuation": valuation,
        "etf_proliferation": etf_prolif,
    }
    score, coverage = _weighted_available_score(components, LIFECYCLE_WEIGHTS)
    missing = [name for name, value in components.items() if value is None]
    return {
        "score": None if score is None else float(min(100.0, max(0.0, score))),
        "coverage": coverage,
        "missing_components": missing,
        "components": components,
    }


def _weighted_available_score(
    components: dict[str, Optional[float]], weights: dict[str, float]
) -> tuple[Optional[float], float]:
    available = {k: v for k, v in components.items() if v is not None}
    total_possible = sum(weights.values())
    total_weight = sum(weights[k] for k in available if k in weights)
    if total_weight == 0 or total_possible == 0:
        return None, 0.0

    score = sum(float(available[k]) * weights[k] for k in available if k in weights) / total_weight
    return score, total_weight / total_possible

#!/usr/bin/env python3
"""
Theme Classifier - Detect cross-sector and vertical themes from ranked industries.

Cross-sector themes match industries against keyword templates (min 2 matches).
Vertical themes detect sector concentration (min 3 same-sector industries in top/bottom).

themes_config format:
{
    "cross_sector": [
        {
            "theme_name": str,
            "matching_keywords": [str, ...],
            "min_matching_industries": int,  # optional per-theme override
            "proxy_etfs": [str, ...],
            "static_stocks": [str, ...],
        },
        ...
    ],
    "vertical_min_industries": int,   # default 3
    "cross_sector_min_matches": int,  # default 2
}
"""

from collections import Counter


def classify_themes(
    ranked_industries: list[dict],
    themes_config: dict,
    top_n: int = 30,
) -> list[dict]:
    """
    Match ranked industries to cross-sector and vertical themes.

    Only the top N and bottom N industries (by momentum rank) are considered
    for theme matching. This prevents themes from always matching when using
    the full ~145 industry universe.

    Args:
        ranked_industries: Output of rank_industries (sorted by momentum_score desc).
        themes_config: Theme definitions with cross_sector templates and thresholds.
        top_n: Number of top/bottom industries to consider (default 30).

    Returns:
        List of theme result dicts, each with:
            theme_name, direction, matching_industries, sector_weights,
            proxy_etfs, static_stocks
    """
    if not ranked_industries:
        return []

    cross_sector_min = themes_config.get("cross_sector_min_matches", 2)
    vertical_min = themes_config.get("vertical_min_industries", 3)
    cross_sector_defs = themes_config.get("cross_sector", [])

    # Build active set from top N + bottom N (deduplicated)
    top = ranked_industries[:top_n]
    bottom = ranked_industries[-top_n:] if len(ranked_industries) > top_n else []
    active_set: dict[str, dict] = {ind["name"]: ind for ind in top}
    for ind in bottom:
        if ind["name"] not in active_set:
            active_set[ind["name"]] = ind

    themes = []

    # 1. Cross-sector theme matching (active set only)
    for theme_def in cross_sector_defs:
        keywords = theme_def.get("matching_keywords", [])
        matches = [kw for kw in keywords if kw in active_set]
        min_matches = theme_def.get("min_matching_industries", cross_sector_min)

        if len(matches) >= min_matches:
            matching_inds = [active_set[m] for m in matches]
            direction = _majority_direction(matching_inds)
            sector_weights = get_theme_sector_weights({"matching_industries": matching_inds})

            themes.append(
                {
                    "theme_name": theme_def["theme_name"],
                    "direction": direction,
                    "matching_industries": matching_inds,
                    "sector_weights": sector_weights,
                    "proxy_etfs": theme_def.get("proxy_etfs", []),
                    "static_stocks": theme_def.get("static_stocks", []),
                    "matching_keyword_count": len(keywords),
                    "min_matching_industries": min_matches,
                    "theme_origin": "seed",
                    "name_confidence": "high",
                }
            )

    # 2. Vertical (single-sector) theme detection
    # Count industries per sector in top N and bottom N separately
    top_set = set(ind["name"] for ind in top)

    # Top N sector groups
    top_sector_groups: dict[str, list[dict]] = {}
    for ind in top:
        sector = ind.get("sector")
        if sector is None:
            continue
        top_sector_groups.setdefault(sector, []).append(ind)

    for sector, inds in top_sector_groups.items():
        if len(inds) >= vertical_min:
            direction = _majority_direction(inds)
            sector_weights = get_theme_sector_weights({"matching_industries": inds})
            themes.append(
                {
                    "theme_name": f"{sector} Sector Concentration",
                    "direction": direction,
                    "matching_industries": inds,
                    "sector_weights": sector_weights,
                    "proxy_etfs": [],
                    "static_stocks": [],
                    "theme_origin": "vertical",
                    "name_confidence": "high",
                }
            )

    # Bottom N sector groups (excluding industries already in top N)
    bottom_sector_groups: dict[str, list[dict]] = {}
    for ind in bottom:
        if ind["name"] in top_set:
            continue
        sector = ind.get("sector")
        if sector is None:
            continue
        bottom_sector_groups.setdefault(sector, []).append(ind)

    for sector, inds in bottom_sector_groups.items():
        if len(inds) >= vertical_min:
            direction = _majority_direction(inds)
            sector_weights = get_theme_sector_weights({"matching_industries": inds})
            themes.append(
                {
                    "theme_name": f"{sector} Sector Concentration",
                    "direction": direction,
                    "matching_industries": inds,
                    "sector_weights": sector_weights,
                    "proxy_etfs": [],
                    "static_stocks": [],
                    "theme_origin": "vertical",
                    "name_confidence": "high",
                }
            )

    return themes


def get_matched_industry_names(themes: list[dict]) -> set[str]:
    """Return the set of all matched industry names across classified themes.

    Args:
        themes: Output of classify_themes().

    Returns:
        Set of industry name strings.
    """
    names: set[str] = set()
    for theme in themes:
        for ind in theme.get("matching_industries", []):
            name = ind.get("name", "")
            if name:
                names.add(name)
    return names


def get_theme_sector_weights(theme: dict) -> dict[str, float]:
    """
    Calculate sector weight distribution for a theme's matching industries.

    Args:
        theme: Dict with "matching_industries" key containing industry dicts
               (each may have a "sector" field).

    Returns:
        Dict mapping sector name to its proportion (0.0-1.0).
        Industries without a sector field are excluded.
    """
    matching = theme.get("matching_industries", [])
    sectors = [ind["sector"] for ind in matching if "sector" in ind]

    if not sectors:
        return {}

    counts = Counter(sectors)
    total = sum(counts.values())

    return {sector: count / total for sector, count in counts.items()}


# Sector-to-representative-stocks mapping for vertical theme enrichment
SECTOR_REPRESENTATIVE_STOCKS: dict[str, list[str]] = {
    "Technology": ["AAPL", "MSFT", "NVDA", "AVGO", "CRM"],
    "Consumer Cyclical": ["AMZN", "TSLA", "HD", "MCD", "NKE"],
    "Consumer Defensive": ["PG", "KO", "PEP", "WMT", "COST"],
    "Industrials": ["CAT", "HON", "UPS", "GE", "RTX"],
    "Healthcare": ["UNH", "JNJ", "LLY", "PFE", "ABBV"],
    "Financial": ["JPM", "BAC", "GS", "V", "MA"],
    "Energy": ["XOM", "CVX", "COP", "SLB", "EOG"],
    "Basic Materials": ["LIN", "APD", "NEM", "FCX", "NUE"],
    "Communication Services": ["META", "GOOGL", "NFLX", "DIS", "TMUS"],
    "Real Estate": ["PLD", "AMT", "EQIX", "SPG", "O"],
    "Utilities": ["NEE", "DUK", "SO", "D", "AEP"],
}

# Sector-to-ETF mapping for vertical theme enrichment
SECTOR_ETFS: dict[str, list[str]] = {
    "Energy": ["XLE"],
    "Technology": ["XLK"],
    "Basic Materials": ["XLB"],
    "Industrials": ["XLI"],
    "Consumer Cyclical": ["XLY"],
    "Consumer Defensive": ["XLP"],
    "Healthcare": ["XLV"],
    "Financial": ["XLF"],
    "Communication Services": ["XLC"],
    "Real Estate": ["XLRE"],
    "Utilities": ["XLU"],
}


def _industry_overlap_ratio(theme_a: dict, theme_b: dict) -> float:
    """Calculate industry name overlap ratio between two themes.

    Returns the Jaccard-like ratio: |intersection| / |smaller set|.
    """
    names_a = {ind.get("name") for ind in theme_a.get("matching_industries", [])}
    names_b = {ind.get("name") for ind in theme_b.get("matching_industries", [])}
    if not names_a or not names_b:
        return 0.0
    intersection = names_a & names_b
    smaller = min(len(names_a), len(names_b))
    return len(intersection) / smaller if smaller > 0 else 0.0


def enrich_vertical_themes(themes: list[dict]) -> None:
    """Add ETFs and stocks to vertical themes from overlapping seeds or sector mapping.

    Mutates vertical themes in place:
    1. If a seed theme shares >= 50% industry overlap, inherit its ETFs/stocks.
    2. Otherwise, assign sector ETF from SECTOR_ETFS mapping.
    """
    seed_themes = [t for t in themes if t.get("theme_origin") == "seed"]

    for theme in themes:
        if theme.get("theme_origin") != "vertical":
            continue
        if theme.get("proxy_etfs"):
            continue  # already has ETFs

        # Try inheriting from overlapping seed theme
        best_seed = None
        best_overlap = 0.0
        for seed in seed_themes:
            overlap = _industry_overlap_ratio(theme, seed)
            if overlap > best_overlap:
                best_overlap = overlap
                best_seed = seed

        if best_seed and best_overlap >= 0.5:
            theme["proxy_etfs"] = list(best_seed.get("proxy_etfs", []))
            theme["static_stocks"] = list(best_seed.get("static_stocks", []))
            continue

        # Sector ETF fallback
        sector_weights = theme.get("sector_weights", {})
        if sector_weights:
            primary_sector = max(sector_weights, key=sector_weights.get)
            etfs = SECTOR_ETFS.get(primary_sector, [])
            if etfs:
                theme["proxy_etfs"] = list(etfs)
        else:
            # Infer sector from matching industries
            sectors = [
                ind.get("sector")
                for ind in theme.get("matching_industries", [])
                if ind.get("sector")
            ]
            if sectors:
                primary = max(set(sectors), key=sectors.count)
                etfs = SECTOR_ETFS.get(primary, [])
                if etfs:
                    theme["proxy_etfs"] = list(etfs)

    # Fill empty static_stocks for vertical themes from sector mapping
    for theme in themes:
        if theme.get("theme_origin") != "vertical":
            continue
        if theme.get("static_stocks"):
            continue  # already has stocks

        sector_weights = theme.get("sector_weights", {})
        if sector_weights:
            primary_sector = max(sector_weights, key=sector_weights.get)
        else:
            sectors = [
                ind.get("sector")
                for ind in theme.get("matching_industries", [])
                if ind.get("sector")
            ]
            primary_sector = max(set(sectors), key=sectors.count) if sectors else None

        if primary_sector:
            stocks = SECTOR_REPRESENTATIVE_STOCKS.get(primary_sector, [])
            if stocks:
                theme["static_stocks"] = list(stocks)


def deduplicate_themes(themes: list[dict], overlap_threshold: float = 0.5) -> list[dict]:
    """Remove vertical themes that duplicate seed themes.

    A vertical theme is removed if:
    - Same direction as a seed theme
    - Industry overlap ratio >= overlap_threshold

    Seed themes are always kept. Returns a new list.
    """
    seed_themes = [t for t in themes if t.get("theme_origin") == "seed"]
    result = list(seed_themes)

    for theme in themes:
        if theme.get("theme_origin") == "seed":
            continue

        is_duplicate = False
        for seed in seed_themes:
            if theme.get("direction") != seed.get("direction"):
                continue
            overlap = _industry_overlap_ratio(theme, seed)
            if overlap >= overlap_threshold:
                is_duplicate = True
                break

        if not is_duplicate:
            result.append(theme)

    return result


def _majority_direction(industries: list[dict]) -> str:
    """Determine majority direction from rank_direction (falls back to direction)."""
    bullish = sum(
        1 for ind in industries if ind.get("rank_direction", ind.get("direction")) == "bullish"
    )
    bearish = len(industries) - bullish
    return "bullish" if bullish > bearish else "bearish"

"""Theme Discoverer - Automatic theme detection from unmatched industries.

Finds clusters of unmatched industries with similar performance patterns
and generates new theme entries compatible with classify_themes() output.

Algorithm:
1. Extract top_n and bottom_n from ranked_industries, exclude matched names.
2. Separate into bullish (top) and bearish (bottom) groups.
3. Sort each group by weighted_return.
4. Cluster adjacent industries that satisfy BOTH:
   a. weighted_return gap <= gap_threshold
   b. perf vector distance <= vector_threshold (normalized Euclidean)
5. Filter clusters by min_cluster_size.
6. Exclude clusters that overlap significantly with existing themes.
7. Auto-name clusters from industry name tokens.
"""

import math
from collections import Counter

from calculators.theme_classifier import get_theme_sector_weights

# ---------------------------------------------------------------------------
# Constants
# ---------------------------------------------------------------------------

_GAP_THRESHOLD_PCT = 3.0
_VECTOR_THRESHOLD = 0.5
_MIN_CLUSTER_SIZE = 2
_OVERLAP_THRESHOLD = 0.5

_STOP_WORDS = {
    "Services",
    "Products",
    "Equipment",
    "Materials",
    "General",
    "Other",
    "Specialty",
    "Diversified",
    "Regulated",
    "Independent",
    "&",
    "-",
    "and",
}


# ---------------------------------------------------------------------------
# Public API
# ---------------------------------------------------------------------------


def discover_themes(
    ranked_industries: list[dict],
    matched_names: set[str],
    existing_themes: list[dict],
    top_n: int = 30,
    gap_threshold: float = _GAP_THRESHOLD_PCT,
    vector_threshold: float = _VECTOR_THRESHOLD,
    min_cluster_size: int = _MIN_CLUSTER_SIZE,
) -> list[dict]:
    """Discover new themes from unmatched industries.

    Args:
        ranked_industries: Full ranked list (sorted by momentum_score desc).
        matched_names: Set of industry names already matched by seed/vertical.
        existing_themes: List of existing theme dicts (for overlap detection).
        top_n: Number of top/bottom industries to consider.
        gap_threshold: Max weighted_return gap between adjacent industries.
        vector_threshold: Max normalized Euclidean distance for perf vectors.
        min_cluster_size: Minimum industries to form a cluster.

    Returns:
        List of theme dicts compatible with classify_themes() output,
        each with theme_origin="discovered" and name_confidence="medium".
    """
    bullish, bearish = _get_unmatched_industries(ranked_industries, matched_names, top_n)

    discovered = []
    for group, direction in [(bullish, "bullish"), (bearish, "bearish")]:
        clusters = _cluster_by_proximity(group, gap_threshold, vector_threshold)
        for cluster in clusters:
            if len(cluster) < min_cluster_size:
                continue
            if _is_duplicate_of_existing(cluster, direction, existing_themes):
                continue
            name = _auto_name_cluster(cluster)
            theme = _build_theme_dict(name, cluster)
            discovered.append(theme)

    return discovered


# ---------------------------------------------------------------------------
# Internal helpers
# ---------------------------------------------------------------------------


def _get_unmatched_industries(
    ranked: list[dict],
    matched_names: set[str],
    top_n: int,
) -> tuple[list[dict], list[dict]]:
    """Extract unmatched industries from top N and bottom N.

    Returns:
        (bullish_unmatched, bearish_unmatched)
    """
    top = ranked[:top_n]
    bottom = ranked[-top_n:] if len(ranked) > top_n else []

    # Deduplicate: industries in both top and bottom only appear in top
    top_names = {ind["name"] for ind in top}

    bullish = [
        ind for ind in top if ind["name"] not in matched_names and ind.get("direction") == "bullish"
    ]
    bearish = [
        ind
        for ind in bottom
        if ind["name"] not in matched_names
        and ind["name"] not in top_names
        and ind.get("direction") == "bearish"
    ]

    return bullish, bearish


def _cluster_by_proximity(
    industries: list[dict],
    gap_threshold: float,
    vector_threshold: float,
) -> list[list[dict]]:
    """Cluster industries by weighted_return proximity and perf vector distance.

    Industries are sorted by weighted_return. Adjacent pairs are joined into
    the same cluster if BOTH conditions are met:
    1. abs(weighted_return difference) <= gap_threshold
    2. perf_vector_distance <= vector_threshold

    Returns:
        List of clusters (each cluster is a list of industry dicts).
        Only clusters with >= 2 members are returned.
    """
    if not industries:
        return []

    sorted_inds = sorted(industries, key=lambda x: x.get("weighted_return", 0), reverse=True)
    ranges = _compute_ranges(sorted_inds)

    clusters: list[list[dict]] = [[sorted_inds[0]]]

    for i in range(1, len(sorted_inds)):
        prev = sorted_inds[i - 1]
        curr = sorted_inds[i]

        gap = abs(prev.get("weighted_return", 0) - curr.get("weighted_return", 0))
        vec_dist = _perf_vector_distance(prev, curr, ranges)

        if gap <= gap_threshold and vec_dist <= vector_threshold:
            clusters[-1].append(curr)
        else:
            clusters.append([curr])

    # Filter to min size 2
    return [c for c in clusters if len(c) >= 2]


def _perf_vector_distance(a: dict, b: dict, ranges: dict) -> float:
    """Normalized Euclidean distance between perf vectors (1W, 1M, 3M).

    Each timeframe difference is normalized by the range (max - min) across
    all industries in the group. If range is 0, that dimension is ignored.

    Returns:
        0.0 (identical) to ~1.7 (maximally different across all 3 dimensions).
    """
    keys = ["perf_1w", "perf_1m", "perf_3m"]
    total = 0.0

    for key in keys:
        r = ranges.get(key, 0)
        if r == 0:
            continue
        diff = (a.get(key, 0) - b.get(key, 0)) / r
        total += diff * diff

    return math.sqrt(total)


def _compute_ranges(industries: list[dict]) -> dict[str, float]:
    """Compute value ranges for perf normalization."""
    keys = ["perf_1w", "perf_1m", "perf_3m"]
    ranges = {}
    for key in keys:
        vals = [ind.get(key, 0) for ind in industries]
        if vals:
            ranges[key] = max(vals) - min(vals)
        else:
            ranges[key] = 0
    return ranges


def _auto_name_cluster(industries: list[dict]) -> str:
    """Generate a descriptive name for a cluster from industry name tokens.

    Tokenizes all industry names, removes stop words, picks the top 2
    most frequent tokens and joins them.
    """
    tokens: list[str] = []
    for ind in industries:
        name = ind.get("name", "")
        for token in name.split():
            cleaned = token.strip("(),")
            if cleaned and cleaned not in _STOP_WORDS:
                tokens.append(cleaned)

    if not tokens:
        return "Unknown Cluster"

    counts = Counter(tokens)
    top = counts.most_common(3)

    if len(top) >= 2:
        return f"{top[0][0]} & {top[1][0]} Related"
    elif len(top) == 1:
        return f"{top[0][0]} Related"
    return "Unknown Cluster"


def _is_duplicate_of_existing(
    cluster_industries: list[dict],
    cluster_direction: str,
    existing_themes: list[dict],
    overlap_threshold: float = _OVERLAP_THRESHOLD,
) -> bool:
    """Check if a discovered cluster duplicates an existing theme.

    A cluster is considered duplicate if:
    1. Direction matches an existing theme, AND
    2. Jaccard similarity >= threshold OR overlap coefficient >= threshold.

    The overlap coefficient (intersection / min(|A|, |B|)) catches cases
    where a small cluster is a subset of a large existing theme, which
    Jaccard alone would miss because the union denominator dilutes the ratio.
    """
    cluster_names = {ind.get("name", "") for ind in cluster_industries}
    if not cluster_names:
        return False

    for theme in existing_themes:
        if theme.get("direction") != cluster_direction:
            continue
        theme_names = {ind.get("name", "") for ind in theme.get("matching_industries", [])}
        if not theme_names:
            continue
        intersection = cluster_names & theme_names
        union = cluster_names | theme_names
        jaccard = len(intersection) / len(union) if union else 0
        min_size = min(len(cluster_names), len(theme_names))
        overlap_coeff = len(intersection) / min_size if min_size else 0
        if jaccard >= overlap_threshold or overlap_coeff >= overlap_threshold:
            return True

    return False


def _build_theme_dict(name: str, industries: list[dict]) -> dict:
    """Build a theme dict compatible with classify_themes() output.

    Discovered themes have proxy_etfs=[], static_stocks=[],
    theme_origin="discovered", name_confidence="medium".
    """
    # Determine direction from majority
    bullish = sum(1 for ind in industries if ind.get("direction") == "bullish")
    bearish = len(industries) - bullish
    direction = "bullish" if bullish > bearish else "bearish"

    sector_weights = get_theme_sector_weights({"matching_industries": industries})

    return {
        "theme_name": name,
        "direction": direction,
        "matching_industries": industries,
        "sector_weights": sector_weights,
        "proxy_etfs": [],
        "static_stocks": [],
        "theme_origin": "discovered",
        "name_confidence": "medium",
    }

"""Theme configuration loader with YAML support and inline fallback.

Loads cross-sector theme definitions from:
1. User-specified YAML file (--themes-config)
2. Bundled themes.yaml (same directory as this module)
3. Inline DEFAULT_THEMES_CONFIG (fallback when YAML is unavailable)

Error handling:
- Explicit yaml_path: fail-fast on any error (FileNotFoundError, yaml.YAMLError)
- No yaml_path: bundled YAML -> inline fallback (safe degradation)
"""

import copy
import os
import sys
from typing import Optional

# Build tuple of catchable YAML errors (yaml may not be installed)
_YAML_ERRORS: tuple = ()
try:
    import yaml

    _YAML_ERRORS = (yaml.YAMLError,)
except ImportError:
    pass


def load_themes_config(
    yaml_path: Optional[str] = None,
) -> tuple[dict, dict[str, int]]:
    """Load themes config and ETF catalog.

    Args:
        yaml_path: Path to custom themes YAML. If None, tries bundled then inline.

    Returns:
        (themes_config, etf_catalog) where themes_config has etf_count stripped.

    Raises:
        FileNotFoundError: If explicit yaml_path doesn't exist.
        yaml.YAMLError: If explicit yaml_path contains invalid YAML.
        ValueError: If config fails validation.
    """
    if yaml_path is not None:
        # Explicit path -> fail-fast
        raw = _load_yaml(yaml_path)
        _validate_config(raw)
        catalog = _extract_etf_catalog(raw)
        config = _strip_etf_count(raw)
        return config, catalog

    # No path specified -> try bundled YAML, then inline fallback
    try:
        bundled = _get_bundled_yaml_path()
        raw = _load_yaml(bundled)
        _validate_config(raw)
        catalog = _extract_etf_catalog(raw)
        config = _strip_etf_count(raw)
        return config, catalog
    except (FileNotFoundError, ValueError, RuntimeError, ImportError, *_YAML_ERRORS) as exc:
        print(f"WARNING: YAML load failed ({exc}), using inline config", file=sys.stderr)
        from default_theme_config import DEFAULT_THEMES_CONFIG, ETF_CATALOG

        return copy.deepcopy(DEFAULT_THEMES_CONFIG), dict(ETF_CATALOG)


def _load_yaml(path: str) -> dict:
    """Load and parse a YAML file.

    Raises:
        FileNotFoundError: If file doesn't exist.
        RuntimeError: If PyYAML is not installed.
        yaml.YAMLError: If YAML is malformed.
    """
    if not os.path.exists(path):
        raise FileNotFoundError(f"YAML config not found: {path}")

    try:
        import yaml
    except ImportError:
        raise RuntimeError(
            "PyYAML is required but not installed. Install with: pip install pyyaml>=6.0"
        ) from None

    with open(path) as f:
        return yaml.safe_load(f)


def _validate_config(config: dict) -> None:
    """Validate required keys in themes config.

    Raises:
        ValueError: If required keys are missing.
    """
    if not isinstance(config, dict):
        raise ValueError("Config must be a dict")

    if "cross_sector" not in config:
        raise ValueError("Config missing required key: cross_sector")

    for i, theme in enumerate(config["cross_sector"]):
        if "theme_name" not in theme:
            raise ValueError(f"Theme at index {i} missing required key: theme_name")
        if "matching_keywords" not in theme:
            raise ValueError(
                f"Theme '{theme.get('theme_name', i)}' missing required key: matching_keywords"
            )


def _extract_etf_catalog(config: dict) -> dict[str, int]:
    """Extract ETF counts from config into a catalog dict.

    Each theme may have an `etf_count` field. Missing etf_count defaults to 0.

    Returns:
        {theme_name: etf_count}
    """
    catalog = {}
    for theme in config.get("cross_sector", []):
        name = theme.get("theme_name", "")
        catalog[name] = theme.get("etf_count", 0)
    return catalog


def _strip_etf_count(config: dict) -> dict:
    """Return a copy of config with etf_count removed from each theme.

    The etf_count is only used for ETF catalog extraction and should not
    be passed to classify_themes().
    """
    result = copy.deepcopy(config)
    for theme in result.get("cross_sector", []):
        theme.pop("etf_count", None)
    return result


def _get_bundled_yaml_path() -> str:
    """Return the absolute path to the bundled themes.yaml."""
    return os.path.join(os.path.dirname(os.path.abspath(__file__)), "themes.yaml")

"""Default inline theme configuration (fallback when YAML is unavailable).

This module holds DEFAULT_THEMES_CONFIG and ETF_CATALOG constants extracted
from theme_detector.py to avoid circular imports between theme_detector.py
and config_loader.py.
"""

DEFAULT_THEMES_CONFIG: dict = {
    "cross_sector_min_matches": 2,
    "vertical_min_industries": 3,
    "cross_sector": [
        {
            "theme_name": "AI & Semiconductors",
            "matching_keywords": [
                "Semiconductors",
                "Semiconductor Equipment & Materials",
                "Software - Application",
                "Software - Infrastructure",
                "Information Technology Services",
                "Computer Hardware",
                "Electronic Components",
            ],
            "proxy_etfs": ["SMH", "SOXX", "AIQ", "BOTZ"],
            "static_stocks": [
                "NVDA",
                "AVGO",
                "AMD",
                "INTC",
                "QCOM",
                "MRVL",
                "AMAT",
                "LRCX",
                "KLAC",
                "TSM",
            ],
        },
        {
            "theme_name": "Clean Energy & EV",
            "matching_keywords": [
                "Solar",
                "Utilities - Renewable",
                "Auto Manufacturers",
                "Electrical Equipment & Parts",
                "Specialty Chemicals",
            ],
            "proxy_etfs": ["ICLN", "QCLN", "TAN", "LIT"],
            "static_stocks": [
                "ENPH",
                "SEDG",
                "FSLR",
                "RUN",
                "TSLA",
                "RIVN",
                "ALB",
                "PLUG",
                "BE",
                "NEE",
            ],
        },
        {
            "theme_name": "Cybersecurity",
            "matching_keywords": [
                "Software - Infrastructure",
                "Software - Application",
                "Information Technology Services",
            ],
            "proxy_etfs": ["CIBR", "HACK", "BUG"],
            "static_stocks": [
                "CRWD",
                "PANW",
                "FTNT",
                "ZS",
                "S",
                "OKTA",
                "NET",
                "CYBR",
                "QLYS",
                "RPD",
            ],
        },
        {
            "theme_name": "Cloud Computing & SaaS",
            "matching_keywords": [
                "Software - Application",
                "Software - Infrastructure",
                "Information Technology Services",
            ],
            "proxy_etfs": ["SKYY", "WCLD", "CLOU"],
            "static_stocks": [
                "CRM",
                "NOW",
                "SNOW",
                "DDOG",
                "MDB",
                "NET",
                "ZS",
                "HUBS",
                "WDAY",
                "TEAM",
            ],
        },
        {
            "theme_name": "Biotech & Genomics",
            "matching_keywords": [
                "Biotechnology",
                "Drug Manufacturers - General",
                "Drug Manufacturers - Specialty & Generic",
                "Diagnostics & Research",
            ],
            "proxy_etfs": ["XBI", "IBB", "ARKG"],
            "static_stocks": [
                "AMGN",
                "GILD",
                "VRTX",
                "REGN",
                "MRNA",
                "BIIB",
                "ILMN",
                "SGEN",
                "ALNY",
                "BMRN",
            ],
        },
        {
            "theme_name": "Infrastructure & Construction",
            "matching_keywords": [
                "Engineering & Construction",
                "Building Products & Equipment",
                "Building Materials",
                "Specialty Industrial Machinery",
                "Farm & Heavy Construction Machinery",
                "Infrastructure Operations",
            ],
            "proxy_etfs": ["PAVE", "IFRA"],
            "static_stocks": [
                "CAT",
                "DE",
                "VMC",
                "MLM",
                "URI",
                "PWR",
                "EME",
                "J",
                "ACM",
                "FAST",
            ],
        },
        {
            "theme_name": "Gold & Precious Metals",
            "matching_keywords": [
                "Gold",
                "Silver",
                "Other Precious Metals & Mining",
            ],
            "proxy_etfs": ["GDX", "GDXJ", "SLV", "GLD"],
            "static_stocks": [
                "NEM",
                "GOLD",
                "AEM",
                "FNV",
                "WPM",
                "RGLD",
                "KGC",
                "AGI",
                "HL",
                "PAAS",
            ],
        },
        {
            "theme_name": "Oil & Gas (Energy)",
            "matching_keywords": [
                "Oil & Gas E&P",
                "Oil & Gas Equipment & Services",
                "Oil & Gas Integrated",
                "Oil & Gas Midstream",
                "Oil & Gas Refining & Marketing",
                "Oil & Gas Drilling",
            ],
            "proxy_etfs": ["XLE", "XOP", "OIH"],
            "static_stocks": [
                "XOM",
                "CVX",
                "COP",
                "EOG",
                "SLB",
                "PXD",
                "MPC",
                "PSX",
                "OXY",
                "DVN",
            ],
        },
        {
            "theme_name": "Financial Services & Banks",
            "matching_keywords": [
                "Banks - Diversified",
                "Banks - Regional",
                "Capital Markets",
                "Insurance - Diversified",
                "Financial Data & Stock Exchanges",
                "Asset Management",
            ],
            "proxy_etfs": ["XLF", "KBE", "KRE"],
            "static_stocks": [
                "JPM",
                "BAC",
                "GS",
                "MS",
                "WFC",
                "C",
                "BLK",
                "SCHW",
                "ICE",
                "CME",
            ],
        },
        {
            "theme_name": "Healthcare & Pharma",
            "matching_keywords": [
                "Drug Manufacturers - General",
                "Medical Devices",
                "Medical Instruments & Supplies",
                "Healthcare Plans",
                "Medical Care Facilities",
            ],
            "proxy_etfs": ["XLV", "IHI", "XBI"],
            "static_stocks": [
                "UNH",
                "JNJ",
                "LLY",
                "PFE",
                "ABT",
                "TMO",
                "DHR",
                "SYK",
                "ISRG",
                "MDT",
            ],
        },
        {
            "theme_name": "Defense & Aerospace",
            "matching_keywords": [
                "Aerospace & Defense",
                "Communication Equipment",
                "Scientific & Technical Instruments",
            ],
            "proxy_etfs": ["ITA", "PPA", "XAR"],
            "static_stocks": [
                "LMT",
                "RTX",
                "NOC",
                "GD",
                "BA",
                "LHX",
                "HII",
                "TDG",
                "HWM",
                "AXON",
            ],
        },
        {
            "theme_name": "Real Estate & REITs",
            "matching_keywords": [
                "REIT - Diversified",
                "REIT - Industrial",
                "REIT - Residential",
                "REIT - Retail",
                "REIT - Specialty",
                "REIT - Office",
                "Real Estate - Development",
                "Real Estate Services",
            ],
            "proxy_etfs": ["VNQ", "IYR", "XLRE"],
            "static_stocks": [
                "PLD",
                "AMT",
                "EQIX",
                "CCI",
                "SPG",
                "O",
                "WELL",
                "DLR",
                "PSA",
                "AVB",
            ],
        },
        {
            "theme_name": "Retail & Consumer",
            "matching_keywords": [
                "Internet Retail",
                "Specialty Retail",
                "Apparel Retail",
                "Discount Stores",
                "Home Improvement Retail",
                "Department Stores",
            ],
            "proxy_etfs": ["XRT", "XLY"],
            "static_stocks": [
                "AMZN",
                "HD",
                "LOW",
                "TJX",
                "COST",
                "WMT",
                "TGT",
                "ROST",
                "BURL",
                "LULU",
            ],
        },
        {
            "theme_name": "Crypto & Blockchain",
            "matching_keywords": [
                "Capital Markets",
                "Software - Infrastructure",
                "Financial Data & Stock Exchanges",
            ],
            "proxy_etfs": ["BITO", "BLOK", "BITQ"],
            "static_stocks": [
                "COIN",
                "MSTR",
                "MARA",
                "RIOT",
                "CLSK",
                "HUT",
                "BITF",
                "HIVE",
                "SQ",
                "PYPL",
            ],
        },
        {
            "theme_name": "Nuclear & Uranium",
            "min_matching_industries": 1,
            "matching_keywords": [
                "Uranium",
                "Utilities - Independent Power Producers",
                "Utilities - Regulated Electric",
            ],
            "proxy_etfs": ["URA", "URNM", "NLR"],
            "static_stocks": [
                "CCJ",
                "UEC",
                "NXE",
                "DNN",
                "LEU",
                "SMR",
                "UUUU",
                "URG",
                "OKLO",
                "VST",
            ],
        },
    ],
}

ETF_CATALOG: dict[str, int] = {
    "AI & Semiconductors": 8,
    "Clean Energy & EV": 7,
    "Cybersecurity": 4,
    "Cloud Computing & SaaS": 4,
    "Biotech & Genomics": 5,
    "Infrastructure & Construction": 3,
    "Gold & Precious Metals": 4,
    "Oil & Gas (Energy)": 5,
    "Financial Services & Banks": 5,
    "Healthcare & Pharma": 5,
    "Defense & Aerospace": 5,
    "Real Estate & REITs": 4,
    "Retail & Consumer": 3,
    "Crypto & Blockchain": 4,
    "Nuclear & Uranium": 3,
}

#!/usr/bin/env python3
"""
Theme Detector - ETF & Stock Metrics Scanner

Uses FMP API (preferred) with yfinance fallback for batch downloading
stock/ETF data and computing technical metrics: RSI-14, 52-week distance,
PE ratio, volume ratios.

FMP API key is optional; without it, yfinance is used exclusively.
"""

import sys
import time
from datetime import date, timedelta
from json import JSONDecodeError
from typing import Any, Optional

try:
    import numpy as np
    import pandas as pd
except ImportError:
    print("ERROR: pandas/numpy not found. Install with: pip install pandas numpy", file=sys.stderr)
    sys.exit(1)

try:
    import yfinance as yf

    HAS_YFINANCE = True
except ImportError:
    HAS_YFINANCE = False

# yfinance uses curl_cffi, whose transport errors are unrelated to requests.
_YF_ERRORS: tuple = ()
try:
    from yfinance.exceptions import YFException

    _YF_ERRORS += (YFException,)
except ImportError:
    pass
try:
    from curl_cffi.requests.exceptions import RequestException as CurlRequestException

    _YF_ERRORS += (CurlRequestException,)
except ImportError:
    pass

try:
    import requests as _requests_lib
    from requests.exceptions import RequestException

    HAS_REQUESTS = True
except ImportError:
    HAS_REQUESTS = False

_REQUEST_ERRORS: tuple = (RequestException,) if HAS_REQUESTS else ()


# ---------------------------------------------------------------------------
# FMP endpoint definitions (R2-1: callable URL builders for stable/v3)
# ---------------------------------------------------------------------------


def _stable_quote_url(base: str, symbols_str: str, params: dict) -> tuple[str, dict]:
    """stable/quote?symbol=A,B"""
    params["symbol"] = symbols_str
    return base, params


def _v3_quote_url(base: str, symbols_str: str, params: dict) -> tuple[str, dict]:
    """api/v3/quote/A,B"""
    return f"{base}/{symbols_str}", params


def _stable_hist_url(base: str, symbols_str: str, params: dict) -> tuple[str, dict]:
    """stable/historical-price-eod/full?symbol=SYM&from=&to="""
    params["symbol"] = symbols_str
    # The /stable EOD endpoint ignores `timeseries`; convert it to a from/to
    # range (~2x calendar days to cover N trading days, +10 slack).
    days = params.pop("timeseries", None)
    as_of = params.pop("_as_of", None)
    if days is not None:
        today = date.fromisoformat(as_of) if as_of else date.today()
        params["from"] = (today - timedelta(days=int(days) * 2 + 10)).isoformat()
        params["to"] = today.isoformat()
    return base, params


def _v3_hist_url(base: str, symbols_str: str, params: dict) -> tuple[str, dict]:
    """api/v3/historical-price-full/A,B?..."""
    params.pop("_as_of", None)
    return f"{base}/{symbols_str}", params


def _normalize_eod_flat_list(data: Any, symbols_str: str) -> Any:
    """Convert a /stable historical-price-eod/full flat list to the v3 shape.

    Input  : [{"symbol": "AAPL", "date": .., "close": .., ...}, ...]
    Output : {"symbol": "AAPL", "historical": [...]}
    Non-list input (v3 dict / historicalStockList) passes through unchanged;
    returns None when no row matches the requested symbol.
    """
    if not isinstance(data, list):
        return data
    norm = symbols_str.replace("-", ".")
    rows = [
        r
        for r in data
        if isinstance(r, dict) and (r.get("symbol") or symbols_str).replace("-", ".") == norm
    ]
    if not rows:
        return None
    return {"symbol": symbols_str, "historical": rows}


_FMP_ENDPOINTS = {
    "quote": [
        ("https://financialmodelingprep.com/stable/quote", _stable_quote_url),
        ("https://financialmodelingprep.com/api/v3/quote", _v3_quote_url),
    ],
    "historical": [
        ("https://financialmodelingprep.com/stable/historical-price-eod/full", _stable_hist_url),
        ("https://financialmodelingprep.com/api/v3/historical-price-full", _v3_hist_url),
    ],
}


def _history_row_on_or_before(row: Any, as_of_date: date) -> bool:
    """Accept only well-formed dated rows that do not exceed the PIT ceiling."""
    if not isinstance(row, dict):
        return False
    raw_date = row.get("date")
    if not isinstance(raw_date, str):
        return False
    try:
        row_date = date.fromisoformat(raw_date[:10])
    except ValueError:
        return False
    return row_date <= as_of_date


def _frame_on_or_before(data: pd.DataFrame, as_of_date: date) -> pd.DataFrame:
    """Remove invalid and post-as-of rows from a provider DataFrame.

    ``Timestamp.date()`` intentionally preserves the provider row's displayed
    date for both timezone-aware and naive indexes. Numeric indexes are not
    dates and must not be silently interpreted as Unix nanoseconds.
    """
    if data is None or data.empty:
        return data
    keep: list[bool] = []
    for raw_index in data.index:
        if isinstance(raw_index, (bool, int, float, complex)):
            keep.append(False)
            continue
        try:
            timestamp = pd.Timestamp(raw_index)
        except (TypeError, ValueError):
            keep.append(False)
            continue
        keep.append(not pd.isna(timestamp) and timestamp.date() <= as_of_date)
    return data.loc[keep].copy()


class ETFScanner:
    """Scans ETFs and stocks for volume ratios and technical metrics.

    Supports FMP API (preferred) with automatic yfinance fallback.
    """

    # One symbol per request: FMP's /stable endpoints do not support
    # comma-batched symbols (that is a paid legacy feature that silently
    # returns []), so quotes/history are fetched individually.
    FMP_HIST_BATCH_SIZE = 1
    FMP_QUOTE_BATCH_SIZE = 1

    _ENDPOINT_FAILURE_THRESHOLD = 3  # disable endpoint after N consecutive failures

    def __init__(
        self,
        fmp_api_key: Optional[str] = None,
        rate_limit_sec: float = 0.3,
        as_of_date: Optional[date] = None,
    ):
        self._cache: dict[str, pd.DataFrame] = {}
        self._fmp_api_key = fmp_api_key
        self._rate_limit_sec = rate_limit_sec
        self._as_of_date = as_of_date
        self._last_request_time = 0.0
        self._fmp_quote_cache: dict[str, dict] = {}  # normalized_symbol -> quote dict
        self._stats: dict[str, dict[str, int]] = {
            "stock": {"fmp_calls": 0, "fmp_failures": 0, "yf_calls": 0, "yf_fallbacks": 0},
            "etf": {"fmp_calls": 0, "fmp_failures": 0, "yf_calls": 0, "yf_fallbacks": 0},
        }
        self._current_stats_context: str = "stock"
        # Circuit breaker: track consecutive failures per endpoint URL prefix
        self._endpoint_failures: dict[str, int] = {}
        self._disabled_endpoints: set[str] = set()

    def backend_stats(self) -> dict[str, Any]:
        """Return backend usage statistics with both flat and nested formats.

        Flat keys (backward compatible): fmp_calls, fmp_failures, yf_calls, yf_fallbacks
        Nested keys: stock.{...}, etf.{...}
        """
        flat: dict[str, int] = {}
        for key in ["fmp_calls", "fmp_failures", "yf_calls", "yf_fallbacks"]:
            flat[key] = self._stats["stock"][key] + self._stats["etf"][key]
        return {
            **flat,
            "stock": dict(self._stats["stock"]),
            "etf": dict(self._stats["etf"]),
        }

    # -------------------------------------------------------------------
    # Symbol normalization (R2-4)
    # -------------------------------------------------------------------
    @staticmethod
    def _normalize_symbol_for_fmp(symbol: str) -> str:
        """Normalize symbol for FMP API (e.g., BRK-B -> BRK.B)."""
        return symbol.replace("-", ".")

    # -------------------------------------------------------------------
    # FMP infrastructure (R2-1: callable URL builder)
    # -------------------------------------------------------------------
    def _fmp_rate_limit(self):
        now = time.time()
        elapsed = now - self._last_request_time
        if elapsed < self._rate_limit_sec:
            time.sleep(self._rate_limit_sec - elapsed)
        self._last_request_time = time.time()

    def _fmp_request(
        self, endpoint_key: str, symbols_str: str, extra_params: Optional[dict] = None
    ) -> Optional[Any]:
        """Try each endpoint (stable -> v3) with circuit breaker.

        Endpoints that fail consecutively are disabled to avoid wasting
        API calls and triggering rate limits from the provider.

        Args:
            endpoint_key: "quote" or "historical"
            symbols_str: comma-separated symbols (already normalized)
            extra_params: e.g. {"timeseries": 20}

        Returns:
            Parsed JSON or None on all failures.
        """
        if not HAS_REQUESTS or not self._fmp_api_key:
            return None

        params = {}
        if extra_params:
            params.update(extra_params)

        ctx = self._current_stats_context
        for base_url, url_builder in _FMP_ENDPOINTS[endpoint_key]:
            # Circuit breaker: skip endpoints with too many consecutive failures
            if base_url in self._disabled_endpoints:
                continue

            url, final_params = url_builder(base_url, symbols_str, dict(params))
            self._fmp_rate_limit()
            self._stats[ctx]["fmp_calls"] += 1
            try:
                resp = _requests_lib.get(
                    url, params=final_params, headers={"apikey": self._fmp_api_key}, timeout=15
                )
                if resp.status_code == 200:
                    data = resp.json()
                    if endpoint_key == "historical":
                        # /stable returns a flat OHLCV list; reshape to the v3
                        # {"symbol", "historical"} dict the parser expects.
                        data = _normalize_eod_flat_list(data, symbols_str)
                    if data:
                        # Reset failure count on success
                        self._endpoint_failures[base_url] = 0
                        return data
            except (*_REQUEST_ERRORS, ValueError):
                print("WARN: Degraded FMP request; trying available fallback", file=sys.stderr)

            self._stats[ctx]["fmp_failures"] += 1
            failures = self._endpoint_failures.get(base_url, 0) + 1
            self._endpoint_failures[base_url] = failures
            if failures >= self._ENDPOINT_FAILURE_THRESHOLD:
                self._disabled_endpoints.add(base_url)

        return None

    # -------------------------------------------------------------------
    # FMP quote fetch (R2-4: normalized cache)
    # -------------------------------------------------------------------
    def _fetch_fmp_quotes(self, symbols: list[str]) -> dict[str, dict]:
        """Batch fetch quotes. Returns {original_symbol: quote_dict}."""
        result: dict[str, dict] = {}
        uncached = []
        for s in symbols:
            norm = self._normalize_symbol_for_fmp(s)
            if norm not in self._fmp_quote_cache:
                uncached.append(s)

        for i in range(0, len(uncached), self.FMP_QUOTE_BATCH_SIZE):
            batch = uncached[i : i + self.FMP_QUOTE_BATCH_SIZE]
            normalized = [self._normalize_symbol_for_fmp(s) for s in batch]
            data = self._fmp_request("quote", ",".join(normalized))
            if isinstance(data, list):
                for item in data:
                    sym = self._normalize_symbol_for_fmp(item.get("symbol", ""))
                    self._fmp_quote_cache[sym] = item

        # Per-symbol retry for missing: try original symbol if different
        for s in uncached:
            norm = self._normalize_symbol_for_fmp(s)
            if norm in self._fmp_quote_cache:
                continue
            if norm != s:
                data = self._fmp_request("quote", s)
                if isinstance(data, list):
                    for item in data:
                        sym = self._normalize_symbol_for_fmp(item.get("symbol", ""))
                        self._fmp_quote_cache[sym] = item

        for s in symbols:
            norm = self._normalize_symbol_for_fmp(s)
            cached = self._fmp_quote_cache.get(norm)
            if cached:
                result[s] = cached
        return result

    # -------------------------------------------------------------------
    # FMP historical fetch (R2-2: per-symbol retry)
    # -------------------------------------------------------------------
    def _fetch_fmp_historical(
        self, symbols: list[str], timeseries: int = 20
    ) -> dict[str, list[dict]]:
        """Batch fetch historical prices with per-symbol retry on partial failure."""
        result: dict[str, list[dict]] = {}
        extra = {"timeseries": timeseries}
        if self._as_of_date is not None:
            extra["_as_of"] = self._as_of_date.isoformat()

        # Phase 1: batch fetch
        for i in range(0, len(symbols), self.FMP_HIST_BATCH_SIZE):
            batch = symbols[i : i + self.FMP_HIST_BATCH_SIZE]
            normalized = [self._normalize_symbol_for_fmp(s) for s in batch]
            data = self._fmp_request("historical", ",".join(normalized), extra)
            if data is None:
                continue
            self._parse_historical_response(data, result)

        # Phase 2: per-symbol retry for missing symbols
        # Try normalized form first, then original if different
        missing = [s for s in symbols if self._normalize_symbol_for_fmp(s) not in result]
        for s in missing:
            norm = self._normalize_symbol_for_fmp(s)
            data = self._fmp_request("historical", norm, extra)
            if data is not None:
                self._parse_historical_response(data, result)
                continue
            # Retry with original symbol if normalization changed it
            if norm != s:
                data = self._fmp_request("historical", s, extra)
                if data is not None:
                    self._parse_historical_response(data, result)

        # Map normalized keys back to original symbols
        mapped: dict[str, list[dict]] = {}
        for s in symbols:
            norm = self._normalize_symbol_for_fmp(s)
            if norm in result:
                rows = result[norm]
                if self._as_of_date is not None:
                    rows = [row for row in rows if _history_row_on_or_before(row, self._as_of_date)]
                if rows:
                    mapped[s] = rows
        return mapped

    def _parse_historical_response(self, data: Any, result: dict) -> None:
        """Parse FMP historical response (batch or single format).

        Keys in result are always normalized (e.g., BRK.B not BRK-B).
        """
        if isinstance(data, dict):
            if "historicalStockList" in data:
                for entry in data["historicalStockList"]:
                    sym = entry.get("symbol", "")
                    if sym and "historical" in entry:
                        result[self._normalize_symbol_for_fmp(sym)] = entry["historical"]
            elif "historical" in data:
                sym = data.get("symbol", "")
                if sym:
                    result[self._normalize_symbol_for_fmp(sym)] = data["historical"]

    # -------------------------------------------------------------------
    # FMP-based stock metrics
    # -------------------------------------------------------------------
    def _batch_stock_metrics_fmp(self, symbols: list[str]) -> list[dict]:
        """Compute stock metrics using FMP quote + historical data."""
        quotes = self._fetch_fmp_quotes(symbols)
        historical = self._fetch_fmp_historical(symbols, timeseries=20)

        results = []
        for s in symbols:
            entry = {
                "symbol": s,
                "rsi_14": None,
                "dist_from_52w_high": None,
                "dist_from_52w_low": None,
                "pe_ratio": None,
            }

            # PE and 52w from quote
            q = quotes.get(s, {})
            pe = q.get("pe")
            if pe is not None:
                entry["pe_ratio"] = pe

            price = q.get("price")
            year_high = q.get("yearHigh")
            year_low = q.get("yearLow")
            if price and year_high and year_high > 0:
                entry["dist_from_52w_high"] = round((year_high - price) / year_high, 4)
            if price and year_low is not None and price > 0:
                entry["dist_from_52w_low"] = round((price - year_low) / price, 4)

            # RSI from historical close
            hist = historical.get(s, [])
            if hist:
                closes = [d.get("close") for d in hist if d.get("close") is not None]
                if len(closes) >= 15:
                    # Historical comes newest-first from FMP; reverse for RSI
                    prices_series = pd.Series(list(reversed(closes)))
                    entry["rsi_14"] = self._calculate_rsi(prices_series, period=14)

            results.append(entry)
        return results

    # -------------------------------------------------------------------
    # FMP-based ETF volume ratios
    # -------------------------------------------------------------------
    def _batch_etf_volume_ratios_fmp(self, symbols: list[str]) -> dict[str, dict]:
        """Compute ETF volume ratios using FMP historical data."""
        historical = self._fetch_fmp_historical(symbols, timeseries=60)

        result: dict[str, dict] = {}
        for s in symbols:
            entry = {"symbol": s, "vol_20d": None, "vol_60d": None, "vol_ratio": None}
            hist = historical.get(s, [])
            volumes = [d.get("volume") for d in hist if d.get("volume") is not None]

            if len(volumes) >= 20:
                # Historical comes newest-first
                vol_20d = float(np.mean(volumes[:20]))
                vol_60d = (
                    float(np.mean(volumes[:60])) if len(volumes) >= 60 else float(np.mean(volumes))
                )
                entry["vol_20d"] = vol_20d
                entry["vol_60d"] = vol_60d
                entry["vol_ratio"] = vol_20d / vol_60d if vol_60d > 0 else None

            result[s] = entry
        return result

    # -------------------------------------------------------------------
    # yfinance-based methods (original implementations)
    # -------------------------------------------------------------------
    def _get_etf_volume_ratio_yfinance(self, symbol: str) -> dict:
        """Get 20-day / 60-day average volume ratio via yfinance."""
        result = {"symbol": symbol, "vol_20d": None, "vol_60d": None, "vol_ratio": None}

        if not HAS_YFINANCE:
            print("WARNING: yfinance not installed.", file=sys.stderr)
            return result

        try:
            data = self._get_cached(symbol, period="6mo")
            if data is None or data.empty or "Volume" not in data.columns:
                return result

            volume = data["Volume"].dropna()
            if len(volume) < 20:
                return result

            vol_20d = float(volume.tail(20).mean())
            vol_60d = float(volume.tail(60).mean()) if len(volume) >= 60 else float(volume.mean())

            result["vol_20d"] = vol_20d
            result["vol_60d"] = vol_60d
            result["vol_ratio"] = vol_20d / vol_60d if vol_60d > 0 else None

        except Exception as e:
            print(f"WARNING: Volume ratio failed for {symbol}: {e}", file=sys.stderr)

        return result

    def _batch_stock_metrics_yfinance(self, symbols: list[str]) -> list[dict]:
        """Batch-download stock data and compute metrics via yfinance."""
        if not HAS_YFINANCE:
            print("WARNING: yfinance not installed.", file=sys.stderr)
            return [
                {
                    "symbol": s,
                    "rsi_14": None,
                    "dist_from_52w_high": None,
                    "dist_from_52w_low": None,
                    "pe_ratio": None,
                }
                for s in symbols
            ]

        try:
            kwargs = {
                "group_by": "ticker",
                "threads": True,
                "progress": False,
            }
            if self._as_of_date is None:
                kwargs["period"] = "1y"
            else:
                kwargs["start"] = (self._as_of_date - timedelta(days=370)).isoformat()
                kwargs["end"] = (self._as_of_date + timedelta(days=1)).isoformat()
            data = yf.download(symbols, **kwargs)
            if self._as_of_date is not None:
                data = _frame_on_or_before(data, self._as_of_date)
        except Exception as e:
            print(f"WARNING: Batch download failed: {e}", file=sys.stderr)
            return [
                {
                    "symbol": s,
                    "rsi_14": None,
                    "dist_from_52w_high": None,
                    "dist_from_52w_low": None,
                    "pe_ratio": None,
                }
                for s in symbols
            ]

        results = []
        for symbol in symbols:
            entry = {
                "symbol": symbol,
                "rsi_14": None,
                "dist_from_52w_high": None,
                "dist_from_52w_low": None,
                "pe_ratio": None,
            }
            try:
                if len(symbols) == 1:
                    sym_data = data
                else:
                    sym_data = data[symbol]

                if sym_data is None or sym_data.empty:
                    results.append(entry)
                    continue

                close = sym_data["Close"].dropna()
                high = sym_data["High"].dropna()
                low = sym_data["Low"].dropna()

                if len(close) < 2:
                    results.append(entry)
                    continue

                entry["rsi_14"] = self._calculate_rsi(close, period=14)

                distances = self._calculate_52w_distances(close, high, low)
                entry["dist_from_52w_high"] = distances["dist_from_52w_high"]
                entry["dist_from_52w_low"] = distances["dist_from_52w_low"]

                entry["pe_ratio"] = self._get_pe_ratio(symbol)

            except Exception as e:
                print(f"WARNING: Metrics failed for {symbol}: {e}", file=sys.stderr)

            results.append(entry)

        return results

    # -------------------------------------------------------------------
    # Public methods (FMP -> yfinance symbol-level fallback)
    # -------------------------------------------------------------------
    def get_etf_volume_ratio(self, symbol: str) -> dict:
        """Get 20-day / 60-day average volume ratio for an ETF.

        Args:
            symbol: Ticker symbol (e.g., "XLK")

        Returns:
            Dict with keys: symbol, vol_20d, vol_60d, vol_ratio.
            Values are None if data unavailable.
        """
        if self._fmp_api_key and HAS_REQUESTS:
            fmp_result = self._batch_etf_volume_ratios_fmp([symbol])
            data = fmp_result.get(symbol, {})
            if data.get("vol_20d") is not None:
                return data

        return self._get_etf_volume_ratio_yfinance(symbol)

    def batch_etf_volume_ratios(self, symbols: list[str]) -> dict[str, dict]:
        """Batch fetch ETF volume ratios with FMP -> yfinance fallback.

        Args:
            symbols: List of ETF ticker symbols

        Returns:
            Dict mapping symbol -> {symbol, vol_20d, vol_60d, vol_ratio}
        """
        if not symbols:
            return {}

        self._current_stats_context = "etf"
        # Phase 1: Try FMP batch
        fmp_results: dict[str, dict] = {}
        fmp_attempted = self._fmp_api_key and HAS_REQUESTS
        if fmp_attempted:
            fmp_results = self._batch_etf_volume_ratios_fmp(symbols)

        # Phase 2: yfinance for missing/empty
        result: dict[str, dict] = {}
        missing_for_yf = []
        for sym in symbols:
            fmp_data = fmp_results.get(sym, {})
            if fmp_data.get("vol_20d") is not None:
                result[sym] = fmp_data
            else:
                missing_for_yf.append(sym)

        if missing_for_yf:
            if fmp_attempted:
                self._stats["etf"]["yf_fallbacks"] += 1
            for sym in missing_for_yf:
                self._stats["etf"]["yf_calls"] += 1
                result[sym] = self._get_etf_volume_ratio_yfinance(sym)

        return result

    def batch_stock_metrics(self, symbols: list[str]) -> list[dict]:
        """Batch compute stock metrics with FMP -> yfinance symbol-level fallback.

        Args:
            symbols: List of ticker symbols

        Returns:
            List of dicts with keys: symbol, rsi_14, dist_from_52w_high,
            dist_from_52w_low, pe_ratio. Values are None if unavailable.
        """
        if not symbols:
            return []

        self._current_stats_context = "stock"
        # Phase 1: FMP results (accept any non-empty result)
        fmp_results: dict[str, dict] = {}
        fmp_attempted = self._fmp_api_key and HAS_REQUESTS
        metric_keys = ["pe_ratio", "rsi_14", "dist_from_52w_high", "dist_from_52w_low"]
        if fmp_attempted:
            fmp_list = self._batch_stock_metrics_fmp(symbols)
            for m in fmp_list:
                sym = m["symbol"]
                has_any = any(m.get(k) is not None for k in metric_keys)
                if has_any:
                    fmp_results[sym] = m

        # Phase 2: yfinance for symbols missing RSI (not just missing entirely)
        missing_rsi = [
            s for s in symbols if s not in fmp_results or fmp_results[s].get("rsi_14") is None
        ]
        yf_results: dict[str, dict] = {}
        if missing_rsi:
            if fmp_attempted:
                self._stats["stock"]["yf_fallbacks"] += 1
            yf_list = self._batch_stock_metrics_yfinance(missing_rsi)
            self._stats["stock"]["yf_calls"] += 1
            for m in yf_list:
                sym = m["symbol"]
                if sym in fmp_results:
                    # Field-level merge: fill only None fields from yfinance
                    for k, v in m.items():
                        if k != "symbol" and fmp_results[sym].get(k) is None and v is not None:
                            fmp_results[sym][k] = v
                else:
                    yf_results[sym] = m

        # Merge: FMP takes priority, yfinance fills gaps
        results = []
        for s in symbols:
            if s in fmp_results:
                results.append(fmp_results[s])
            elif s in yf_results:
                results.append(yf_results[s])
            else:
                results.append(
                    {
                        "symbol": s,
                        "rsi_14": None,
                        "dist_from_52w_high": None,
                        "dist_from_52w_low": None,
                        "pe_ratio": None,
                    }
                )
        return results

    # -------------------------------------------------------------------
    # Shared utilities (unchanged)
    # -------------------------------------------------------------------
    @staticmethod
    def _calculate_rsi(prices: pd.Series, period: int = 14) -> Optional[float]:
        """Calculate RSI using Wilder's smoothing method."""
        if prices is None or len(prices) < period + 1:
            return None

        deltas = prices.diff()

        gains = deltas.where(deltas > 0, 0.0)
        losses = (-deltas).where(deltas < 0, 0.0)

        first_avg_gain = gains.iloc[1 : period + 1].mean()
        first_avg_loss = losses.iloc[1 : period + 1].mean()

        avg_gain = first_avg_gain
        avg_loss = first_avg_loss

        for i in range(period + 1, len(prices)):
            avg_gain = (avg_gain * (period - 1) + gains.iloc[i]) / period
            avg_loss = (avg_loss * (period - 1) + losses.iloc[i]) / period

        if avg_loss == 0:
            return 100.0

        rs = avg_gain / avg_loss
        rsi = 100.0 - (100.0 / (1.0 + rs))
        return round(rsi, 2)

    @staticmethod
    def _calculate_52w_distances(close: pd.Series, high: pd.Series, low: pd.Series) -> dict:
        """Calculate distance from 52-week high and low."""
        result = {"dist_from_52w_high": None, "dist_from_52w_low": None}

        if close.empty:
            return result

        current = float(close.iloc[-1])
        if current <= 0:
            return result

        high_52w = float(high.max())
        low_52w = float(low.min())

        if high_52w > 0:
            result["dist_from_52w_high"] = round((high_52w - current) / high_52w, 4)

        if low_52w >= 0:
            result["dist_from_52w_low"] = (
                round((current - low_52w) / current, 4) if current > 0 else None
            )

        return result

    def _get_pe_ratio(self, symbol: str) -> Optional[float]:
        """Get trailing P/E ratio for a symbol via yfinance info."""
        try:
            ticker = yf.Ticker(symbol)
            info = ticker.info
        except (*_YF_ERRORS, *_REQUEST_ERRORS, JSONDecodeError):
            print("WARN: Degraded P/E lookup; provider unavailable", file=sys.stderr)
            return None
        pe = info.get("trailingPE")
        if pe is not None:
            try:
                return round(float(pe), 2)
            except (TypeError, ValueError, OverflowError):
                print("WARN: Degraded P/E lookup; invalid numeric value", file=sys.stderr)
        return None

    def _get_cached(self, symbol: str, period: str = "6mo") -> Optional[pd.DataFrame]:
        """Get cached download or fetch new data."""
        cache_key = f"{symbol}_{period}_{self._as_of_date or 'live'}"
        if cache_key in self._cache:
            return self._cache[cache_key]

        try:
            if self._as_of_date is None:
                data = yf.download(symbol, period=period, progress=False)
            else:
                lookback_days = 190 if period == "6mo" else 370
                data = yf.download(
                    symbol,
                    start=(self._as_of_date - timedelta(days=lookback_days)).isoformat(),
                    end=(self._as_of_date + timedelta(days=1)).isoformat(),
                    progress=False,
                )
            if self._as_of_date is not None:
                data = _frame_on_or_before(data, self._as_of_date)
            self._cache[cache_key] = data
            return data
        except Exception as e:
            print(f"WARNING: Download failed for {symbol}: {e}", file=sys.stderr)
            return None

#!/usr/bin/env python3
"""
Theme Detector - FINVIZ Performance Client

Fetches sector and industry performance data from FINVIZ using the
finvizfinance library. No API key required (public data).

Data Source: finvizfinance.group.performance
"""

import math
import sys
from typing import Optional

try:
    from finvizfinance.group import performance as fvperf

    HAS_FINVIZFINANCE = True
except ImportError:
    HAS_FINVIZFINANCE = False


# Hard caps for performance values (absolute %).
# Applied before z-score winsorization to handle corrupted FINVIZ data
# (e.g., Luxury Goods: -100%, Biotechnology: +87% in perf_1w).
HARD_CAPS = {
    "perf_1w": 30.0,
    "perf_1m": 60.0,
    "perf_3m": 100.0,
    "perf_6m": 100.0,
    "perf_1y": 200.0,
    "perf_ytd": 200.0,
}


# Mapping from finvizfinance DataFrame columns to standardized keys
COLUMN_MAP = {
    "Name": "name",
    "Perf Week": "perf_1w",
    "Perf Month": "perf_1m",
    "Perf Quart": "perf_3m",
    "Perf Half": "perf_6m",
    "Perf Year": "perf_1y",
    "Perf YTD": "perf_ytd",
}


def _parse_perf_value(val) -> Optional[float]:
    """Parse a performance value to float.

    The finvizfinance library may return:
    - float already (e.g., 0.12 for 12%)
    - string like "0.12%" or "12.34%"
    - None or NaN
    """
    if val is None:
        return None
    try:
        import math

        if isinstance(val, float) and math.isnan(val):
            return None
    except (TypeError, ValueError):
        pass
    if isinstance(val, (int, float)):
        return float(val)
    if isinstance(val, str):
        cleaned = val.strip().rstrip("%")
        if not cleaned:
            return None
        try:
            num = float(cleaned)
            # If original had % sign but value looks like it's already in
            # decimal form (e.g., "0.12%"), it's ambiguous.
            # finvizfinance typically returns decimal (0.12 = 12%).
            # If string had % and value > 1, it's likely a percentage.
            if "%" in val and abs(num) > 1:
                return num / 100.0
            return num
        except ValueError:
            return None
    return None


def _dataframe_to_dicts(df) -> list[dict]:
    """Convert a finvizfinance DataFrame to standardized list of dicts."""
    rows = []
    for _, row in df.iterrows():
        entry = {}
        for src_col, dst_key in COLUMN_MAP.items():
            if src_col in row.index:
                if dst_key == "name":
                    entry[dst_key] = str(row[src_col]).strip()
                else:
                    entry[dst_key] = _parse_perf_value(row[src_col])
            else:
                if dst_key != "name":
                    entry[dst_key] = None
        if entry.get("name"):
            rows.append(entry)
    return rows


def _apply_hard_caps(industries: list[dict]) -> list[dict]:
    """Apply absolute hard caps to performance values.

    Caps each perf_* field to ±HARD_CAPS[key]. Original values are saved in
    raw_perf_* only if raw_perf_* does not already exist (preserves the
    two-stage pipeline: hard cap -> z-score winsorization).

    Args:
        industries: List of industry dicts with perf_* fields.

    Returns:
        Same list, mutated in place.
    """
    for ind in industries:
        for key, cap in HARD_CAPS.items():
            val = ind.get(key)
            if val is None:
                continue
            if abs(val) > cap:
                raw_key = f"raw_{key}"
                if raw_key not in ind:
                    ind[raw_key] = val
                ind[key] = cap if val > 0 else -cap
    return industries


def cap_outlier_performances(industries: list[dict], z_threshold: float = 3.0) -> list[dict]:
    """Winsorize performance values exceeding z_threshold standard deviations.

    For each perf_* field, computes mean and std across all industries,
    then caps values exceeding |z| > z_threshold to the boundary value.
    Original values are preserved in raw_perf_* fields.

    Skips winsorization if fewer than 5 industries (insufficient for z-score).

    Args:
        industries: List of industry dicts with perf_* fields.
        z_threshold: Z-score threshold for outlier detection (default 3.0).

    Returns:
        Modified list (same objects, mutated in place).
    """
    # Stage 1: Apply hard caps before z-score calculation
    _apply_hard_caps(industries)

    if len(industries) < 5:
        return industries

    perf_keys = ["perf_1w", "perf_1m", "perf_3m", "perf_6m", "perf_1y", "perf_ytd"]

    for key in perf_keys:
        values = [ind[key] for ind in industries if ind.get(key) is not None]
        if len(values) < 5:
            continue

        mean = sum(values) / len(values)
        variance = sum((v - mean) ** 2 for v in values) / len(values)
        std = math.sqrt(variance) if variance > 0 else 0

        if std == 0:
            continue

        cap_high = mean + z_threshold * std
        cap_low = mean - z_threshold * std

        for ind in industries:
            val = ind.get(key)
            if val is None:
                continue
            z = (val - mean) / std
            if abs(z) > z_threshold:
                raw_key = f"raw_{key}"
                if raw_key not in ind:
                    ind[raw_key] = val
                ind[key] = cap_high if val > mean else cap_low

    return industries


def get_sector_performance() -> list[dict]:
    """Fetch sector-level performance data from FINVIZ.

    Returns:
        List of dicts with keys: name, perf_1w, perf_1m, perf_3m,
        perf_6m, perf_1y, perf_ytd. Values are floats in decimal
        form (e.g., 0.05 = 5%).
    """
    if not HAS_FINVIZFINANCE:
        print(
            "WARNING: finvizfinance not installed. Install with: pip install finvizfinance",
            file=sys.stderr,
        )
        return []

    try:
        perf = fvperf.Performance()
        df = perf.screener_view(group="Sector")
        return _dataframe_to_dicts(df)
    except Exception as e:
        print(f"WARNING: Failed to fetch sector performance: {e}", file=sys.stderr)
        return []


def get_industry_performance() -> list[dict]:
    """Fetch industry-level performance data from FINVIZ.

    Returns:
        List of dicts with same structure as get_sector_performance().
        Typically 140+ industries.
    """
    if not HAS_FINVIZFINANCE:
        print(
            "WARNING: finvizfinance not installed. Install with: pip install finvizfinance",
            file=sys.stderr,
        )
        return []

    try:
        perf = fvperf.Performance()
        df = perf.screener_view(group="Industry")
        return _dataframe_to_dicts(df)
    except Exception as e:
        print(f"WARNING: Failed to fetch industry performance: {e}", file=sys.stderr)
        return []

#!/usr/bin/env python3
"""Stock leadership evidence for Theme Detector v2.

The module accepts either pre-labeled scan hits or raw daily metric rows.
Raw rows are expanded into one ScanHit per matched condition so a single
symbol can count as both EP9M and range expansion on the same day.
"""

import csv
import json
import math
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Optional

SCAN_TYPES = ["five_day_20pct", "ep9m", "range_expansion", "new_high", "high_rs"]
LEADERSHIP_WEIGHTS = {
    "five_day_20pct": 0.30,
    "ep9m": 0.25,
    "range_expansion": 0.20,
    "new_high": 0.15,
    "high_rs": 0.10,
}


@dataclass
class ScanHit:
    date: str
    symbol: str
    scan_type: str
    return_5d: Optional[float] = None
    change_pct: Optional[float] = None
    volume: Optional[float] = None
    avg_volume_50d: Optional[float] = None
    relative_volume: Optional[float] = None
    dollar_volume: Optional[float] = None
    close_location: Optional[float] = None
    atr_expansion: Optional[float] = None
    rs_rating: Optional[float] = None
    market_cap: Optional[float] = None
    close: Optional[float] = None
    industry: Optional[str] = None
    sector: Optional[str] = None
    theme_guess: Optional[str] = None
    catalyst: Optional[str] = None

    def to_dict(self) -> dict:
        return asdict(self)


def load_scan_hits(path: str, run_date: str) -> tuple[list[ScanHit], dict]:
    """Load scan hits from JSON/JSONL/CSV and derive missing scan types."""
    rows = _load_rows(path)
    hits: list[ScanHit] = []
    skipped_rows = 0
    skipped_date_rows = 0
    for row in rows:
        if not _row_matches_run_date(row, run_date):
            skipped_date_rows += 1
            continue
        row_hits = detect_scan_hits_from_row(row, run_date)
        if row_hits:
            hits.extend(row_hits)
        else:
            skipped_rows += 1

    summary = {
        "path": path,
        "rows": len(rows),
        "hits": len(hits),
        "skipped_rows": skipped_rows,
        "skipped_date_rows": skipped_date_rows,
    }
    return hits, summary


def detect_scan_hits_from_row(row: dict, run_date: str) -> list[ScanHit]:
    """Derive one or more ScanHit records from a raw or pre-labeled row."""
    symbol = str(row.get("symbol") or "").strip().upper()
    if not symbol:
        return []
    if not _row_matches_run_date(row, run_date):
        return []

    enriched = _enrich_row(row)
    date = str(row.get("date") or run_date)
    scan_types = _extract_scan_types(row)
    if not scan_types:
        scan_types = _derive_scan_types(enriched)

    return [_scan_hit_from_row(enriched, date, symbol, scan_type) for scan_type in scan_types]


def aggregate_leadership(
    themes: list[dict], scan_hits: list[ScanHit], history: dict
) -> dict[str, dict]:
    """Aggregate leadership evidence for each theme."""
    result: dict[str, dict] = {}
    hits_by_theme: dict[str, list[ScanHit]] = {}

    for hit in scan_hits:
        matched_names = _matched_theme_names(hit, themes)
        for name in matched_names:
            hits_by_theme.setdefault(name, []).append(hit)

    for theme in themes:
        name = theme.get("theme_name") or theme.get("name")
        hits = hits_by_theme.get(name, [])
        counts = {scan_type: 0 for scan_type in SCAN_TYPES}
        for hit in hits:
            if hit.scan_type in counts:
                counts[hit.scan_type] += 1

        if hits:
            score = calculate_leadership_score(name, counts, history)
            coverage = 1.0
        else:
            score = None
            coverage = 0.0

        leader_candidates = calculate_leader_candidates(hits)

        result[name] = {
            "leadership_score": score,
            "leadership_coverage": coverage,
            "leadership_counts": counts,
            "leader_symbols": sorted({hit.symbol for hit in hits}),
            "leader_candidates": leader_candidates,
            "fresh_leadership_symbols": [
                candidate["symbol"]
                for candidate in leader_candidates
                if set(candidate.get("scan_types", [])) & {"ep9m", "range_expansion", "new_high"}
            ],
            "extended_symbols": [
                candidate["symbol"]
                for candidate in leader_candidates
                if candidate.get("is_extended")
            ],
            "scan_hits": [hit.to_dict() for hit in hits],
        }

    return result


def calculate_leadership_score(theme_name: str, counts: dict[str, int], history: dict) -> float:
    """Calculate 0-100 leadership score from hit counts and prior history."""
    weighted = 0.0
    for scan_type, weight in LEADERSHIP_WEIGHTS.items():
        value = counts.get(scan_type, 0)
        prior = _prior_counts(history, theme_name, scan_type)
        component = _count_component_score(value, prior)
        weighted += component * weight
    weight_sum = sum(LEADERSHIP_WEIGHTS.values())
    return round(min(100.0, max(0.0, weighted / weight_sum if weight_sum else 0.0)), 2)


def blend_theme_heat(base_heat: float, leadership_score: Optional[float]) -> float:
    """Blend leadership only when actual leadership evidence exists."""
    if leadership_score is None:
        return base_heat
    blended = 0.70 * base_heat + 0.30 * leadership_score
    return round(max(base_heat, blended), 2)


def calculate_leader_candidates(scan_hits: list[ScanHit]) -> list[dict]:
    """Build one ranked leadership candidate row per symbol."""
    by_symbol: dict[str, list[ScanHit]] = {}
    for hit in scan_hits:
        by_symbol.setdefault(hit.symbol, []).append(hit)

    candidates = []
    for symbol, hits in by_symbol.items():
        metrics = _merge_hit_metrics(hits)
        score, components, coverage = _leader_score(metrics)
        candidates.append(
            {
                "symbol": symbol,
                "leader_score": score,
                "leader_score_components": components,
                "leader_score_coverage": coverage,
                "scan_types": sorted({hit.scan_type for hit in hits}),
                "risk_bucket": _market_cap_bucket(metrics.get("market_cap")),
                "market_cap": metrics.get("market_cap"),
                "relative_volume": metrics.get("relative_volume"),
                "return_5d": metrics.get("return_5d"),
                "atr_expansion": metrics.get("atr_expansion"),
                "close_location": metrics.get("close_location"),
                "rs_rating": metrics.get("rs_rating"),
                "dollar_volume": metrics.get("dollar_volume"),
                "is_extended": _is_extended(metrics, hits),
            }
        )

    return sorted(candidates, key=lambda item: item.get("leader_score", 0), reverse=True)


def _load_rows(path: str) -> list[dict]:
    p = Path(path)
    if p.suffix.lower() == ".csv":
        with p.open(newline="") as f:
            return list(csv.DictReader(f))
    with p.open() as f:
        if p.suffix.lower() == ".jsonl":
            return [json.loads(line) for line in f if line.strip()]
        data = json.load(f)
    if isinstance(data, list):
        return data
    if isinstance(data, dict):
        for key in ("scan_hits", "rows", "data"):
            if isinstance(data.get(key), list):
                return data[key]
    return []


def _extract_scan_types(row: dict) -> list[str]:
    raw = row.get("scan_types", row.get("scan_type"))
    if raw is None:
        return []
    if isinstance(raw, str):
        values = [part.strip() for part in raw.replace(";", ",").split(",")]
    elif isinstance(raw, list):
        values = [str(part).strip() for part in raw]
    else:
        values = [str(raw).strip()]
    return [value for value in values if value in SCAN_TYPES]


def _derive_scan_types(row: dict) -> list[str]:
    scan_types = []
    if _is_five_day_20pct(row):
        scan_types.append("five_day_20pct")
    if _is_ep9m(row):
        scan_types.append("ep9m")
    if _is_range_expansion(row):
        scan_types.append("range_expansion")
    if _is_new_high(row):
        scan_types.append("new_high")
    if _is_high_rs(row):
        scan_types.append("high_rs")
    return scan_types


def _row_matches_run_date(row: dict, run_date: str) -> bool:
    row_date = _date_key(row.get("date"))
    if row_date is None:
        return True
    return row_date == _date_key(run_date)


def _date_key(value) -> Optional[str]:
    if value in (None, ""):
        return None
    text = str(value).strip()
    return text[:10] if len(text) >= 10 else text


def _scan_hit_from_row(row: dict, date: str, symbol: str, scan_type: str) -> ScanHit:
    return ScanHit(
        date=date,
        symbol=symbol,
        scan_type=scan_type,
        return_5d=_float(row.get("return_5d")),
        change_pct=_float(row.get("change_pct")),
        volume=_float(row.get("volume")),
        avg_volume_50d=_float(row.get("avg_volume_50d")),
        relative_volume=_float(row.get("relative_volume")),
        dollar_volume=_float(row.get("dollar_volume")),
        close_location=_float(row.get("close_location")),
        atr_expansion=_float(row.get("atr_expansion")),
        rs_rating=_relative_strength_score(row),
        market_cap=_float(row.get("market_cap")),
        close=_float(row.get("close")),
        industry=_clean(row.get("industry")),
        sector=_clean(row.get("sector")),
        theme_guess=_clean(row.get("theme_guess")),
        catalyst=_clean(row.get("catalyst")),
    )


def _enrich_row(row: dict) -> dict:
    enriched = dict(row)
    volume = _float(enriched.get("volume"))
    avg_volume = _float(enriched.get("avg_volume_50d"))
    close = _float(enriched.get("close"))
    true_range = _float(enriched.get("true_range"))
    atr_20 = _float(enriched.get("atr_20"))

    if _float(enriched.get("relative_volume")) is None and volume is not None and avg_volume:
        enriched["relative_volume"] = volume / avg_volume
    if _float(enriched.get("dollar_volume")) is None and volume is not None and close is not None:
        enriched["dollar_volume"] = volume * close
    if _float(enriched.get("atr_expansion")) is None and true_range is not None and atr_20:
        enriched["atr_expansion"] = true_range / atr_20
    return enriched


def _is_five_day_20pct(row: dict) -> bool:
    value = _float(row.get("return_5d"))
    return value is not None and value >= 20.0


def _is_ep9m(row: dict) -> bool:
    volume = _float(row.get("volume"))
    rel_volume = _float(row.get("relative_volume"))
    change = _float(row.get("change_pct"))
    return (
        volume is not None
        and rel_volume is not None
        and change is not None
        and volume >= 9_000_000
        and rel_volume >= 2.0
        and change >= 4.0
    )


def _is_range_expansion(row: dict) -> bool:
    change = _float(row.get("change_pct"))
    atr_expansion = _float(row.get("atr_expansion"))
    close_location = _float(row.get("close_location"))
    return (
        change is not None
        and atr_expansion is not None
        and close_location is not None
        and change >= 4.0
        and atr_expansion >= 1.5
        and close_location >= 0.75
    )


def _is_new_high(row: dict) -> bool:
    if _truthy(row.get("new_high")) or _truthy(row.get("is_new_high")):
        return True
    close = _float(row.get("close"))
    high_52w = _float(row.get("high_52w"))
    dist = _float(row.get("dist_from_52w_high"))
    if close is not None and high_52w is not None and high_52w > 0:
        return close >= high_52w
    return dist is not None and dist <= 0.01


def _is_high_rs(row: dict) -> bool:
    rs = _relative_strength_score(row)
    if rs is None:
        return False
    return rs >= 90.0


def _matched_theme_names(hit: ScanHit, themes: list[dict]) -> list[str]:
    if hit.theme_guess:
        return [hit.theme_guess]
    matches = []
    for theme in themes:
        name = theme.get("theme_name") or theme.get("name")
        industries = {ind.get("name") for ind in theme.get("matching_industries", [])}
        stocks = set(theme.get("static_stocks", []) or theme.get("representative_stocks", []))
        if (hit.industry and hit.industry in industries) or hit.symbol in stocks:
            matches.append(name)
    return matches


def _prior_counts(history: dict, theme_name: str, scan_type: str) -> list[int]:
    records = history.get(theme_name, [])
    counts = []
    for record in records:
        leadership_counts = record.get("leadership_counts") or {}
        counts.append(int(leadership_counts.get(scan_type, 0) or 0))
    return counts[-20:]


def _merge_hit_metrics(hits: list[ScanHit]) -> dict:
    metrics = {
        "relative_volume": None,
        "return_5d": None,
        "atr_expansion": None,
        "close_location": None,
        "rs_rating": None,
        "dollar_volume": None,
        "market_cap": None,
    }
    for hit in hits:
        for key in metrics:
            value = getattr(hit, key, None)
            if value is None:
                continue
            current = metrics[key]
            metrics[key] = value if current is None else max(current, value)
    return metrics


def _leader_score(metrics: dict) -> tuple[float, dict, float]:
    component_values = {
        "relative_volume_score": _score_relative_volume(metrics.get("relative_volume")),
        "return_5d_score": _score_return_5d(metrics.get("return_5d")),
        "range_expansion_score": _score_range_expansion(metrics.get("atr_expansion")),
        "close_location_score": _score_close_location(metrics.get("close_location")),
        "relative_strength_score": _score_relative_strength(metrics.get("rs_rating")),
        "liquidity_score": _score_liquidity(metrics.get("dollar_volume")),
    }
    weights = {
        "relative_volume_score": 0.25,
        "return_5d_score": 0.20,
        "range_expansion_score": 0.20,
        "close_location_score": 0.15,
        "relative_strength_score": 0.10,
        "liquidity_score": 0.10,
    }
    available = {key: value for key, value in component_values.items() if value is not None}
    total_weight = sum(weights[key] for key in available)
    total_possible = sum(weights.values())
    if total_weight <= 0:
        return 0.0, component_values, 0.0
    score = sum(available[key] * weights[key] for key in available) / total_possible
    coverage = total_weight / total_possible
    return round(min(100.0, max(0.0, score)), 2), component_values, round(coverage, 4)


def _score_relative_volume(value: Optional[float]) -> Optional[float]:
    if value is None:
        return None
    return round(min(100.0, max(0.0, (value / 3.0) * 100.0)), 2)


def _score_return_5d(value: Optional[float]) -> Optional[float]:
    if value is None:
        return None
    return round(min(100.0, max(0.0, (value / 25.0) * 100.0)), 2)


def _score_range_expansion(value: Optional[float]) -> Optional[float]:
    if value is None:
        return None
    return round(min(100.0, max(0.0, (value / 2.0) * 100.0)), 2)


def _score_close_location(value: Optional[float]) -> Optional[float]:
    if value is None:
        return None
    return round(min(100.0, max(0.0, value * 100.0)), 2)


def _score_relative_strength(value: Optional[float]) -> Optional[float]:
    if value is None:
        return None
    return round(min(100.0, max(0.0, value)), 2)


def _score_liquidity(value: Optional[float]) -> Optional[float]:
    if value is None:
        return None
    if value >= 100_000_000:
        return 100.0
    if value >= 20_000_000:
        return 75.0
    if value >= 5_000_000:
        return 50.0
    return 25.0


def _market_cap_bucket(value: Optional[float]) -> str:
    if value is None:
        return "unknown"
    if value >= 200_000_000_000:
        return "mega"
    if value >= 10_000_000_000:
        return "large"
    if value >= 2_000_000_000:
        return "mid"
    if value >= 300_000_000:
        return "small"
    return "micro"


def _is_extended(metrics: dict, hits: list[ScanHit]) -> bool:
    return (metrics.get("return_5d") or 0.0) >= 20.0 or any(
        hit.scan_type == "five_day_20pct" for hit in hits
    )


def _relative_strength_score(row: dict) -> Optional[float]:
    value = _float(row.get("rs_rating"))
    if value is None:
        value = _float(row.get("relative_strength"))
    if value is None:
        return None
    return value * 100.0 if value <= 1.0 else value


def _count_component_score(value: int, prior: list[int]) -> float:
    if not prior:
        return min(100.0, 50.0 + value * 25.0) if value > 0 else 0.0
    if len(prior) >= 2:
        mean = sum(prior) / len(prior)
        variance = sum((x - mean) ** 2 for x in prior) / len(prior)
        std = math.sqrt(variance)
        if std > 0:
            return min(100.0, max(0.0, 50.0 + ((value - mean) / std) * 15.0))
    baseline = max(sum(prior) / len(prior), 1.0)
    return min(100.0, max(0.0, (value / baseline) * 50.0))


def _float(value) -> Optional[float]:
    if value in (None, ""):
        return None
    try:
        return float(value)
    except (TypeError, ValueError):
        return None


def _clean(value) -> Optional[str]:
    if value in (None, ""):
        return None
    return str(value).strip() or None


def _truthy(value) -> bool:
    if isinstance(value, bool):
        return value
    if value is None:
        return False
    return str(value).strip().lower() in {"1", "true", "yes", "y"}

#!/usr/bin/env python3
"""
Theme Detector - Report Generator

Generates JSON and Markdown reports for detected market themes.
"""

import json
import os
from datetime import date, datetime
from typing import Optional
from zoneinfo import ZoneInfo

from _market_calendar import count_sessions


def generate_json_report(
    themes: list[dict],
    industry_rankings: dict,
    sector_uptrend: dict,
    metadata: dict,
    *,
    as_of_date: Optional[date] = None,
) -> dict:
    """Create the full JSON output structure.

    Args:
        themes: List of scored theme dicts from the classifier/scorer.
            Each theme has: name, direction, heat, maturity, stage,
            confidence, heat_breakdown, maturity_breakdown,
            representative_stocks, proxy_etfs, etc.
        industry_rankings: Dict with "top" and "bottom" lists of industries.
        sector_uptrend: Dict mapping sector name to uptrend data
            (ratio, ma_10, slope, trend, latest_date).
        metadata: Dict with run metadata (generated_at, data_sources, etc.)

    Returns:
        Complete JSON-serializable report dict.
    """
    bullish = [t for t in themes if t.get("direction") == "bullish"]
    bearish = [t for t in themes if t.get("direction") == "bearish"]

    # Sort by heat descending within each group
    bullish.sort(key=lambda t: t.get("heat", 0), reverse=True)
    bearish.sort(key=lambda t: t.get("heat", 0), reverse=True)
    match_ranked = sorted(
        themes,
        key=lambda t: (
            t.get("theme_match_score") is not None,
            t.get("theme_match_score") or -1.0,
            t.get("theme_rank_score", t.get("heat", 0)),
        ),
        reverse=True,
    )

    # Data quality flags
    data_quality = _assess_data_quality(
        themes,
        industry_rankings,
        sector_uptrend,
        metadata,
        as_of_date=as_of_date,
    )
    what_changed = _summarize_what_changed_today(themes)
    leadership_evidence = _build_leadership_evidence(themes)

    return {
        "report_type": "theme_detector",
        "generated_at": metadata.get("generated_at", datetime.now().strftime("%Y-%m-%d %H:%M:%S")),
        "metadata": metadata,
        "summary": {
            "total_themes": len(themes),
            "bullish_count": len(bullish),
            "bearish_count": len(bearish),
            "top_bullish": bullish[0]["name"] if bullish else None,
            "top_bearish": bearish[0]["name"] if bearish else None,
        },
        "themes": {
            "all": themes,
            "bullish": bullish,
            "bearish": bearish,
            "match_ranked": match_ranked,
        },
        "industry_rankings": industry_rankings,
        "sector_uptrend": sector_uptrend,
        "what_changed_today": what_changed,
        "leadership_evidence": leadership_evidence,
        "data_quality": data_quality,
    }


def generate_markdown_report(json_data: dict, top_n_detail: int = 3) -> str:
    """Generate a formatted Markdown report from JSON data.

    Args:
        json_data: Full JSON report dict from generate_json_report().
        top_n_detail: Number of top themes to show in detail sections
            (default 3, corresponds to --top CLI arg).

    Sections:
    1. Theme Dashboard (all themes table)
    2. Bullish Themes Detail (top N)
    3. Bearish Themes Detail (top N)
    4. All Themes Summary Table
    5. Industry Rankings (top/bottom 15)
    6. Sector Uptrend Ratios (3-point display)
    7. Methodology Notes + Data Quality Flags
    """
    lines = []
    themes_data = json_data.get("themes", {})
    all_themes = themes_data.get("all", [])
    bullish = themes_data.get("bullish", [])
    bearish = themes_data.get("bearish", [])
    summary = json_data.get("summary", {})

    # Header
    lines.append("# Theme Detector Report")
    lines.append("")
    lines.append(f"**Generated:** {json_data.get('generated_at', 'N/A')}")
    lines.append(
        f"**Themes Detected:** {summary.get('total_themes', 0)} "
        f"({summary.get('bullish_count', 0)} leading, "
        f"{summary.get('bearish_count', 0)} lagging)"
    )
    lines.append("")

    # Section 1: Theme Dashboard
    lines.append("---")
    lines.append("")
    lines.append("## 1. Theme Dashboard")
    lines.append("")

    if not all_themes:
        lines.append("**WARNING:** No themes detected. Check data sources.")
        lines.append("")
    else:
        lines.append(
            "| Theme | Origin | Direction | Heat | Match | Maturity | Stage | Confidence |"
        )
        lines.append(
            "|-------|--------|-----------|------|-------|----------|-------|------------|"
        )
        for t in all_themes:
            heat_bar = _heat_bar(t.get("heat", 0))
            origin = _origin_label(t.get("theme_origin", "seed"))
            match = t.get("theme_match_score")
            match_str = "N/A" if match is None else f"{match:.1f}"
            lines.append(
                f"| {t.get('name', 'N/A')} "
                f"| {origin} "
                f"| {_direction_label(t.get('direction'))} "
                f"| {heat_bar} {t.get('heat', 0):.1f} "
                f"| {match_str} "
                f"| {t.get('maturity', 0):.1f} "
                f"| {t.get('stage', 'N/A')} "
                f"| {t.get('confidence', 'N/A')} |"
            )
        lines.append("")

    # Section 2: What Changed Today
    lines.append("---")
    lines.append("")
    lines.append("## 2. What Changed Today")
    lines.append("")
    _add_what_changed_today(lines, json_data.get("what_changed_today", {}))

    # Section 3: Leadership Evidence
    lines.append("---")
    lines.append("")
    lines.append("## 3. Leadership Evidence")
    lines.append("")
    _add_leadership_evidence(lines, json_data.get("leadership_evidence", []))

    # Section 4: Bullish Themes Detail
    lines.append("---")
    lines.append("")
    lines.append(f"## 4. Leading Themes (Top {top_n_detail})")
    lines.append("")
    _add_theme_details(lines, bullish[:top_n_detail])

    # Section 5: Bearish Themes Detail
    lines.append("---")
    lines.append("")
    lines.append(f"## 5. Lagging Themes (Top {top_n_detail})")
    lines.append("")
    _add_theme_details(lines, bearish[:top_n_detail])

    # Section 6: All Themes Summary Table
    lines.append("---")
    lines.append("")
    lines.append("## 6. All Themes Summary")
    lines.append("")
    if all_themes:
        lines.append("| # | Theme | Dir | Heat | Maturity | Stage | Industries |")
        lines.append("|---|-------|-----|------|----------|-------|------------|")
        for i, t in enumerate(all_themes, 1):
            industries = t.get("industries", [])
            ind_str = ", ".join(industries[:3])
            if len(industries) > 3:
                ind_str += f" (+{len(industries) - 3})"
            lines.append(
                f"| {i} "
                f"| {t.get('name', 'N/A')} "
                f"| {_direction_arrow(t.get('direction'))} "
                f"| {t.get('heat', 0):.1f} "
                f"| {t.get('maturity', 0):.1f} "
                f"| {t.get('stage', 'N/A')} "
                f"| {ind_str} |"
            )
        lines.append("")
    else:
        lines.append("No themes detected.")
        lines.append("")

    # Section 7: Industry Rankings
    lines.append("---")
    lines.append("")
    lines.append("## 7. Industry Rankings")
    lines.append("")
    rankings = json_data.get("industry_rankings", {})
    top = rankings.get("top", [])
    bottom = rankings.get("bottom", [])

    if top:
        lines.append("### Top 15 Industries")
        lines.append("")
        lines.append("| # | Industry | Perf 1W | Perf 1M | Perf 3M | Score |")
        lines.append("|---|----------|---------|---------|---------|-------|")
        for i, ind in enumerate(top[:15], 1):
            lines.append(
                f"| {i} "
                f"| {ind.get('name', 'N/A')} "
                f"| {_fmt_pct(ind.get('perf_1w'))} "
                f"| {_fmt_pct(ind.get('perf_1m'))} "
                f"| {_fmt_pct(ind.get('perf_3m'))} "
                f"| {ind.get('momentum_score', 0):.2f} |"
            )
        lines.append("")

    if bottom:
        lines.append("### Bottom 15 Industries")
        lines.append("")
        lines.append("| # | Industry | Perf 1W | Perf 1M | Perf 3M | Score |")
        lines.append("|---|----------|---------|---------|---------|-------|")
        for i, ind in enumerate(bottom[:15], 1):
            lines.append(
                f"| {i} "
                f"| {ind.get('name', 'N/A')} "
                f"| {_fmt_pct(ind.get('perf_1w'))} "
                f"| {_fmt_pct(ind.get('perf_1m'))} "
                f"| {_fmt_pct(ind.get('perf_3m'))} "
                f"| {ind.get('momentum_score', 0):.2f} |"
            )
        lines.append("")

    if not top and not bottom:
        lines.append("Industry ranking data unavailable.")
        lines.append("")

    # Section 8: Sector Uptrend Ratios
    lines.append("---")
    lines.append("")
    lines.append("## 8. Sector Uptrend Ratios")
    lines.append("")
    uptrend = json_data.get("sector_uptrend", {})
    if uptrend:
        lines.append("| Sector | Ratio | 10MA | Slope | Trend | Date |")
        lines.append("|--------|-------|------|-------|-------|------|")
        for sector_name, data in sorted(uptrend.items()):
            if not isinstance(data, dict):
                continue
            ratio_pct = f"{data['ratio'] * 100:.1f}%" if data.get("ratio") is not None else "N/A"
            ma_pct = f"{data['ma_10'] * 100:.1f}%" if data.get("ma_10") is not None else "N/A"
            slope_str = f"{data['slope']:+.4f}" if data.get("slope") is not None else "N/A"
            lines.append(
                f"| {sector_name} "
                f"| {ratio_pct} "
                f"| {ma_pct} "
                f"| {slope_str} "
                f"| {data.get('trend', 'N/A')} "
                f"| {data.get('latest_date', 'N/A')} |"
            )
        lines.append("")
    else:
        lines.append("Sector uptrend data unavailable.")
        lines.append("")

    # Section 9: Methodology + Data Quality
    lines.append("---")
    lines.append("")
    lines.append("## 9. Methodology & Data Quality")
    lines.append("")
    lines.append("### Methodology")
    lines.append("")
    lines.append("The Theme Detector identifies market themes by:")
    lines.append("")
    lines.append(
        "1. **Industry Ranking:** FINVIZ performance data ranked by "
        "multi-timeframe composite score (1W 10%, 1M 25%, 3M 35%, 6M 30%)"
    )
    lines.append(
        "2. **Theme Classification:** Industries grouped into "
        "thematic clusters (AI/Semiconductors, Energy Transition, etc.)"
    )
    lines.append(
        "3. **Heat Scoring:** Theme strength measured by performance "
        "momentum, volume confirmation, and breadth"
    )
    lines.append(
        "4. **Lifecycle Stage:** Theme maturity assessed via "
        "Emerging/Accelerating/Trending/Mature/Exhausting classification"
    )
    lines.append("5. **Uptrend Overlay:** Monty's sector uptrend ratios provide breadth context")
    lines.append("")
    lines.append(
        "**Note on Confidence:** Script output confidence is capped at "
        "Medium. Claude's WebSearch narrative confirmation can elevate "
        "confidence to High."
    )
    lines.append("")
    lines.append(
        "**Note on Direction:** Theme direction (LEAD/LAG) is based on "
        "**relative performance of matched industries**, not absolute "
        "price change. A LAG theme may still have positive returns — it "
        "indicates relative underperformance, not a short signal."
    )
    lines.append("")

    data_quality = json_data.get("data_quality", {})
    flags = data_quality.get("flags", [])
    if flags:
        lines.append("### Data Quality Flags")
        lines.append("")
        for flag in flags:
            lines.append(f"- {flag}")
        lines.append("")
    else:
        lines.append("### Data Quality")
        lines.append("")
        lines.append("All data sources returned valid results.")
        lines.append("")

    # Disclaimer
    lines.append("---")
    lines.append("")
    lines.append(
        "**Disclaimer:** This analysis is for educational and "
        "informational purposes only. Not investment advice. "
        "Past patterns may not predict future outcomes."
    )
    lines.append("")

    return "\n".join(lines)


def save_reports(json_data: dict, markdown: str, output_dir: str) -> dict[str, str]:
    """Save JSON and Markdown report files.

    Args:
        json_data: Full JSON report dict
        markdown: Formatted markdown string
        output_dir: Directory to save reports

    Returns:
        Dict with "json" and "markdown" keys containing file paths.
    """
    os.makedirs(output_dir, exist_ok=True)

    timestamp = datetime.now().strftime("%Y-%m-%d_%H%M%S")
    json_path = os.path.join(output_dir, f"theme_detector_{timestamp}.json")
    md_path = os.path.join(output_dir, f"theme_detector_{timestamp}.md")

    with open(json_path, "w", encoding="utf-8") as f:
        json.dump(json_data, f, indent=2, default=str)

    with open(md_path, "w", encoding="utf-8") as f:
        f.write(markdown)

    return {"json": json_path, "markdown": md_path}


# --- Private helpers ---


def _assess_data_quality(
    themes: list[dict],
    industry_rankings: dict,
    sector_uptrend: dict,
    metadata: dict,
    *,
    as_of_date: Optional[date] = None,
) -> dict:
    """Assess data quality and return flags.

    Checks FINVIZ connectivity, uptrend data freshness, and scanner
    backend statistics (FMP failure rate, yfinance fallback usage).
    """
    flags = []

    if not themes:
        flags.append("No themes detected - check FINVIZ connectivity")

    top = industry_rankings.get("top", [])
    bottom = industry_rankings.get("bottom", [])
    if not top and not bottom:
        flags.append("Industry rankings unavailable - FINVIZ data may be missing")

    if not sector_uptrend:
        flags.append("Sector uptrend data unavailable - GitHub CSV fetch may have failed")
    else:
        # Check for stale data
        for sector, data in sector_uptrend.items():
            if isinstance(data, dict) and data.get("latest_date"):
                try:
                    latest = date.fromisoformat(data["latest_date"])
                    current_date = as_of_date or datetime.now(ZoneInfo("America/New_York")).date()
                    if latest > current_date:
                        flags.append(f"Uptrend data for {sector} is dated after the report as-of")
                        break
                    age = count_sessions(
                        "XNYS",
                        latest,
                        current_date,
                        include_start=False,
                        include_end=True,
                    )
                    if age > 3:
                        flags.append(f"Uptrend data for {sector} is {age} sessions old")
                        break  # One warning is enough
                except (ValueError, TypeError):
                    pass

    sources = metadata.get("data_sources", {})
    if sources.get("finviz_error"):
        flags.append(f"FINVIZ error: {sources['finviz_error']}")
    if sources.get("uptrend_error"):
        flags.append(f"Uptrend error: {sources['uptrend_error']}")
    scan_hits = sources.get("scan_hits", {})
    if isinstance(scan_hits, dict) and not scan_hits.get("path"):
        flags.append("Stock leadership scan hits unavailable - leadership evidence not blended")
    elif (
        isinstance(scan_hits, dict)
        and scan_hits.get("rows", 0) > 0
        and scan_hits.get("hits", 0) == 0
    ):
        flags.append("Stock leadership scan rows produced no qualifying hits")

    low_heat = [t.get("name") for t in themes if t.get("heat_coverage", 1.0) < 0.75]
    if low_heat:
        flags.append(f"Heat component coverage low for {len(low_heat)} theme(s)")
    low_lifecycle = [t.get("name") for t in themes if t.get("maturity_coverage", 1.0) < 0.60]
    if low_lifecycle:
        flags.append(f"Lifecycle component coverage low for {len(low_lifecycle)} theme(s)")

    # Scanner backend statistics (support both flat and nested formats)
    scanner = sources.get("scanner_backend", {})
    for ctx_label, ctx_key in [("Stock", "stock"), ("ETF", "etf")]:
        ctx_stats = scanner.get(ctx_key, {})
        ctx_fmp_calls = ctx_stats.get("fmp_calls", 0)
        ctx_fmp_failures = ctx_stats.get("fmp_failures", 0)
        if ctx_fmp_calls > 0 and ctx_fmp_failures / ctx_fmp_calls > 0.2:
            pct = ctx_fmp_failures / ctx_fmp_calls
            flags.append(
                f"FMP API ({ctx_label}): {ctx_fmp_failures}/{ctx_fmp_calls} "
                f"calls failed ({pct:.0%})"
            )
            # FMP uses dual-endpoint fallback (stable → v3); a ~50% failure
            # rate typically means one endpoint is consistently unavailable
            # while the other succeeds. Actual data loss is lower than the
            # failure count suggests.
            if pct >= 0.4:
                flags.append(
                    f"  Note: FMP uses dual-endpoint fallback; "
                    f"~{pct:.0%} failure rate likely reflects one endpoint "
                    f"being unavailable, not {pct:.0%} data loss"
                )

    # Flat fallback for backward compat (if nested not available)
    fmp_calls = scanner.get("fmp_calls", 0)
    fmp_failures = scanner.get("fmp_failures", 0)
    if fmp_calls > 0 and fmp_failures / fmp_calls > 0.2 and "stock" not in scanner:
        pct = fmp_failures / fmp_calls
        flags.append(f"FMP API: {fmp_failures}/{fmp_calls} calls failed ({pct:.0%})")

    yf_fallbacks = scanner.get("yf_fallbacks", 0)
    if yf_fallbacks > 0:
        flags.append(f"yfinance fallback used {yf_fallbacks} time(s) for missing FMP data")

    return {
        "status": "warning" if flags else "ok",
        "flags": flags,
    }


def _summarize_what_changed_today(themes: list[dict]) -> dict:
    newly_emerging = []
    accelerating = []
    ep9m_clusters = []
    fading = []

    for theme in themes:
        metrics = theme.get("history_metrics", {})
        delta_1d = metrics.get("heat_delta_1d")
        z_20d = metrics.get("heat_z_20d")
        counts = theme.get("leadership_counts", {})
        ep9m_count = counts.get("ep9m", 0)

        if metrics.get("prior_observations", 0) == 0 and theme.get("heat", 0) >= 40:
            newly_emerging.append(theme.get("name"))
        if z_20d is not None and z_20d >= 1.0:
            accelerating.append(
                {
                    "theme": theme.get("name"),
                    "heat_z_20d": z_20d,
                    "heat_delta_1d": delta_1d,
                }
            )
        if ep9m_count:
            ep9m_clusters.append(
                {
                    "theme": theme.get("name"),
                    "ep9m_count": ep9m_count,
                    "leader_symbols": theme.get("leader_symbols", []),
                }
            )
        if delta_1d is not None and delta_1d <= -10.0:
            fading.append({"theme": theme.get("name"), "heat_delta_1d": delta_1d})

    accelerating.sort(key=lambda item: item.get("heat_z_20d") or 0, reverse=True)
    ep9m_clusters.sort(key=lambda item: item.get("ep9m_count") or 0, reverse=True)
    fading.sort(key=lambda item: item.get("heat_delta_1d") or 0)
    return {
        "newly_emerging_themes": newly_emerging,
        "largest_heat_acceleration": accelerating[:5],
        "new_ep9m_clusters": ep9m_clusters[:5],
        "fading_after_leadership": fading[:5],
    }


def _build_leadership_evidence(themes: list[dict]) -> list[dict]:
    rows = []
    for theme in themes:
        counts = theme.get("leadership_counts", {})
        rows.append(
            {
                "theme": theme.get("name"),
                "five_day_20pct_count": counts.get("five_day_20pct", 0),
                "ep9m_count": counts.get("ep9m", 0),
                "range_expansion_count": counts.get("range_expansion", 0),
                "new_high_count": counts.get("new_high", 0),
                "high_rs_count": counts.get("high_rs", 0),
                "leadership_score": theme.get("leadership_score"),
                "leader_symbols": theme.get("leader_symbols", []),
                "leader_candidates": theme.get("leader_candidates", []),
            }
        )
    rows.sort(
        key=lambda row: (
            row.get("leadership_score") is not None,
            row.get("leadership_score") or 0,
            row.get("ep9m_count") or 0,
        ),
        reverse=True,
    )
    return rows


def _add_what_changed_today(lines: list[str], summary: dict) -> None:
    if not summary:
        lines.append("No history metrics available.")
        lines.append("")
        return
    emerging = summary.get("newly_emerging_themes", [])
    acceleration = summary.get("largest_heat_acceleration", [])
    ep9m = summary.get("new_ep9m_clusters", [])
    fading = summary.get("fading_after_leadership", [])

    lines.append(f"- **Newly emerging themes:** {', '.join(emerging) if emerging else 'None'}")
    if acceleration:
        accel = ", ".join(f"{item['theme']} (z={item['heat_z_20d']:+.2f})" for item in acceleration)
        lines.append(f"- **Largest heat acceleration:** {accel}")
    else:
        lines.append("- **Largest heat acceleration:** None")
    if ep9m:
        clusters = ", ".join(f"{item['theme']} ({item['ep9m_count']})" for item in ep9m)
        lines.append(f"- **New EP9M clusters:** {clusters}")
    else:
        lines.append("- **New EP9M clusters:** None")
    if fading:
        fade = ", ".join(f"{item['theme']} ({item['heat_delta_1d']:+.1f})" for item in fading)
        lines.append(f"- **Fading after prior leadership:** {fade}")
    else:
        lines.append("- **Fading after prior leadership:** None")
    lines.append("")


def _add_leadership_evidence(lines: list[str], evidence: list[dict]) -> None:
    if not evidence:
        lines.append("No leadership evidence available.")
        lines.append("")
        return
    lines.append(
        "| Theme | 5D+20% | EP9M | Range Expansion | New Highs | High RS | Score | Top Candidate Evidence |"
    )
    lines.append(
        "|-------|--------|------|-----------------|-----------|---------|-------|------------------------|"
    )
    for row in evidence:
        candidates = row.get("leader_candidates", [])
        if candidates:
            leaders = ", ".join(
                f"{candidate.get('symbol')} "
                f"({candidate.get('leader_score', 0):.1f}, "
                f"cov {candidate.get('leader_score_coverage', 0):.0%})"
                for candidate in candidates[:5]
            )
        else:
            leaders = ", ".join(row.get("leader_symbols", [])[:8]) or "N/A"
        score = row.get("leadership_score")
        score_str = "N/A" if score is None else f"{score:.1f}"
        lines.append(
            f"| {row.get('theme', 'N/A')} "
            f"| {row.get('five_day_20pct_count', 0)} "
            f"| {row.get('ep9m_count', 0)} "
            f"| {row.get('range_expansion_count', 0)} "
            f"| {row.get('new_high_count', 0)} "
            f"| {row.get('high_rs_count', 0)} "
            f"| {score_str} "
            f"| {leaders} |"
        )
    lines.append("")


def _heat_bar(heat: float) -> str:
    """Create a text bar for heat visualization (0-100 scale)."""
    if heat >= 80.0:
        return "████"
    elif heat >= 60.0:
        return "███░"
    elif heat >= 40.0:
        return "██░░"
    elif heat >= 20.0:
        return "█░░░"
    else:
        return "░░░░"


_ORIGIN_LABELS = {"seed": "Seed", "vertical": "Vertical", "discovered": "*NEW*"}


def _origin_label(origin: Optional[str]) -> str:
    """Format theme origin for display."""
    return _ORIGIN_LABELS.get(origin or "seed", "Seed")


def _direction_label(direction: Optional[str]) -> str:
    """Format direction for display.

    Uses LEAD/LAG instead of BULL/BEAR because direction is determined
    by relative performance of matched industries, not by absolute
    price change. A 'LAG' theme may still have positive returns.
    """
    if direction == "bullish":
        return "LEAD"
    elif direction == "bearish":
        return "LAG"
    return "N/A"


def _direction_arrow(direction: Optional[str]) -> str:
    """Format direction as arrow."""
    if direction == "bullish":
        return "^"
    elif direction == "bearish":
        return "v"
    return "-"


def _fmt_pct(value: Optional[float]) -> str:
    """Format a percent value as percentage string (no conversion)."""
    if value is None:
        return "N/A"
    return f"{value:+.1f}%"


def _heat_subscore_interpretation(key: str, value: float) -> str:
    """Provide a brief interpretation of a heat sub-score value."""
    if value >= 75:
        strength = "strong"
    elif value >= 50:
        strength = "moderate"
    elif value >= 25:
        strength = "weak"
    else:
        strength = "very weak"

    descriptions = {
        "momentum_strength": f"{strength} theme momentum",
        "volume_intensity": f"{strength} volume confirmation",
        "uptrend_signal": f"{strength} breadth alignment",
        "breadth_signal": f"{strength} directional consensus",
    }
    return descriptions.get(key, strength)


_SOURCE_LABELS = {
    "finviz_elite": "Fe",
    "finviz_public": "Fp",
    "etf_holdings": "E",
    "static": "S",
}


def _format_stock_list(theme_data: dict) -> str:
    """Format stock list with optional source labels.

    When stock_details is present and contains non-static sources,
    appends [Fe]/[Fp]/[E]/[S] labels. Otherwise plain comma-joined tickers.
    """
    details = theme_data.get("stock_details", [])
    tickers = theme_data.get("representative_stocks", [])

    if not details:
        return ", ".join(tickers) if tickers else "N/A"

    parts = []
    for d in details:
        label = _SOURCE_LABELS.get(d.get("source", ""), "?")
        parts.append(f"{d['symbol']}[{label}]")
    return ", ".join(parts) if parts else "N/A"


def _add_theme_details(lines: list[str], themes: list[dict]) -> None:
    """Add detailed theme sections to the report lines."""
    if not themes:
        lines.append("No themes in this category.")
        lines.append("")
        return

    for t in themes:
        lines.append(f"### {t.get('name', 'Unknown Theme')}")
        lines.append("")

        # Lifecycle data insufficient badge
        if t.get("lifecycle_data_quality") == "insufficient":
            lines.append(
                "> **[Lifecycle data insufficient]** "
                "Stock-level lifecycle components are missing or partial; "
                "maturity uses available components and coverage metadata."
            )
            lines.append("")

        # Discovered theme provisional badge
        if t.get("theme_origin") == "discovered":
            lines.append(
                "> **[Discovered - provisional]** Theme name and boundaries are auto-generated."
            )
            lines.append("")

        # Show origin line for discovered themes (seed/vertical are self-evident)
        t_origin = t.get("theme_origin", "seed")
        if t_origin == "discovered":
            name_conf = t.get("name_confidence", "medium")
            lines.append(f"- **Origin:** Discovered (name confidence: {name_conf})")
        lines.append(f"- **Direction:** {_direction_label(t.get('direction'))}")
        lines.append(f"- **Heat:** {t.get('heat', 0):.1f}/100 ({t.get('heat_label', 'N/A')})")
        if t.get("base_heat") is not None and t.get("base_heat") != t.get("heat"):
            lines.append(f"- **Base Heat:** {t.get('base_heat', 0):.1f}/100")
        if t.get("leadership_score") is not None:
            lines.append(f"- **Leadership Score:** {t.get('leadership_score', 0):.1f}/100")
        if t.get("theme_match_score") is not None:
            lines.append(f"- **Theme Match Score:** {t.get('theme_match_score', 0):.1f}/100")
        lines.append(f"- **Maturity:** {t.get('maturity', 0):.1f}/100")
        lines.append(f"- **Stage:** {t.get('stage', 'N/A')}")
        lines.append(f"- **Confidence:** {t.get('confidence', 'N/A')}")
        penalties = t.get("confidence_penalties", [])
        if penalties:
            lines.append(f"- **Confidence Penalties:** {', '.join(penalties)}")
        lines.append(
            f"- **Data Coverage:** Heat {t.get('heat_coverage', 0):.0%}, "
            f"Lifecycle {t.get('maturity_coverage', 0):.0%}, "
            f"Leadership {t.get('leadership_coverage', 0):.0%}"
        )
        lines.append("")

        candidates = t.get("leader_candidates", [])
        if candidates:
            lines.append("**Leader Candidate Evidence:**")
            lines.append("")
            for candidate in candidates[:5]:
                scan_types = ", ".join(candidate.get("scan_types", []))
                risk_bucket = candidate.get("risk_bucket", "unknown")
                lines.append(
                    f"- {candidate.get('symbol')}: "
                    f"score {candidate.get('leader_score', 0):.1f}, "
                    f"coverage {candidate.get('leader_score_coverage', 0):.0%}, "
                    f"signals {scan_types or 'N/A'}, risk {risk_bucket}"
                )
            lines.append("")

        # Divergence alert (shown before heat breakdown for visibility)
        div = t.get("divergence")
        if div:
            mom_val = t.get("heat_breakdown", {}).get("momentum_strength", "N/A")
            upt_val = t.get("heat_breakdown", {}).get("uptrend_signal", "N/A")
            if isinstance(mom_val, float):
                mom_val = f"{mom_val:.1f}"
            if isinstance(upt_val, float):
                upt_val = f"{upt_val:.1f}"
            lines.append(
                f"> **Divergence Alert:** {div['description']}. "
                f"Momentum: {mom_val}, Uptrend: {upt_val}."
            )
            lines.append("")

        # Heat breakdown with interpretive labels
        heat_bd = t.get("heat_breakdown", {})
        if heat_bd:
            lines.append("**Heat Breakdown:**")
            lines.append("")
            for key, val in heat_bd.items():
                label = key.replace("_", " ").title()
                if isinstance(val, float):
                    interp = _heat_subscore_interpretation(key, val)
                    lines.append(f"- {label}: {val:.2f} ({interp})")
                else:
                    lines.append(f"- {label}: {val}")
            lines.append("")

        # Maturity breakdown
        mat_bd = t.get("maturity_breakdown", {})
        if mat_bd:
            lines.append("**Maturity Breakdown:**")
            lines.append("")
            if t.get("lifecycle_data_quality") == "insufficient":
                lines.append(
                    "- *Note: Stock-level lifecycle components are missing "
                    "or partial; maturity is based on available components "
                    "and coverage metadata.*"
                )
            for key, val in mat_bd.items():
                label = key.replace("_", " ").title()
                if isinstance(val, float):
                    lines.append(f"- {label}: {val:.2f}")
                else:
                    lines.append(f"- {label}: {val}")
            lines.append("")

        # Representative stocks
        lines.append(f"**Representative Stocks:** {_format_stock_list(t)}")
        lines.append("")

        # Proxy ETFs
        etfs = t.get("proxy_etfs", [])
        if etfs:
            lines.append(f"**Proxy ETFs:** {', '.join(etfs)}")
            lines.append("")

#!/usr/bin/env python3
"""
Representative Stock Selector - Dynamic stock selection for theme detection.

Selects representative stocks for market themes using a fallback chain:
1. FINVIZ Elite industry screener (paid API)
2. FINVIZ Public screener (finvizfinance, free)
3. FMP ETF Holdings (paid API)
4. Static stocks (config fallback)
"""

import csv
import io
import logging
import time
from typing import Optional

try:
    import requests
except ImportError:
    requests = None  # type: ignore[assignment]

try:
    from finvizfinance.screener.overview import Overview
except ImportError:
    Overview = None  # type: ignore[assignment,misc]

logger = logging.getLogger(__name__)

_MAX_CONSECUTIVE_FAILURES = 3

# ---------------------------------------------------------------------------
# FINVIZ filter mappings
# ---------------------------------------------------------------------------

# finvizfinance filter_dict option names (public screener)
CAP_FILTER_MAP = {
    "micro": "+Micro (over $50mln)",
    "small": "+Small (over $300mln)",
    "mid": "+Mid (over $2bln)",
}

# FINVIZ Elite URL filter codes
CAP_ELITE_MAP = {
    "micro": "cap_microover",
    "small": "cap_smallover",
    "mid": "cap_midover",
}

# Industry name -> Elite URL code (lowercase, spaces/punctuation removed)
# Derived from finvizfinance.screener.base.filter_dict["Industry"]["option"]
_INDUSTRY_CODE_CACHE: dict[str, str] = {}


def _industry_to_code(industry: str) -> str:
    """Convert FINVIZ industry name to Elite URL code."""
    if industry in _INDUSTRY_CODE_CACHE:
        return _INDUSTRY_CODE_CACHE[industry]
    code = industry.lower()
    for ch in " &-/,'()":
        code = code.replace(ch, "")
    _INDUSTRY_CODE_CACHE[industry] = code
    return code


# ---------------------------------------------------------------------------
# Source state tracking
# ---------------------------------------------------------------------------


class _SourceState:
    """Track per-source failure state for circuit breaker."""

    def __init__(self):
        self.consecutive_failures: int = 0
        self.total_failures: int = 0
        self.total_queries: int = 0
        self.disabled: bool = False


# ---------------------------------------------------------------------------
# Parse helpers
# ---------------------------------------------------------------------------


def _parse_market_cap(value) -> int:
    """Parse FINVIZ Market Cap string to integer.

    "2.8T" -> 2_800_000_000_000
    "150B" -> 150_000_000_000
    "500M" -> 500_000_000
    "-"    -> 0
    None   -> 0
    """
    if value is None:
        return 0
    if isinstance(value, (int, float)):
        return int(value)

    s = str(value).strip()
    if not s or s == "-":
        return 0

    multiplier = 1
    if s.endswith("T"):
        multiplier = 1_000_000_000_000
        s = s[:-1]
    elif s.endswith("B"):
        multiplier = 1_000_000_000
        s = s[:-1]
    elif s.endswith("M"):
        multiplier = 1_000_000
        s = s[:-1]

    try:
        return int(float(s) * multiplier)
    except (ValueError, TypeError):
        return 0


def _parse_change(value) -> Optional[float]:
    """Parse FINVIZ Change value to float percentage.

    "12.50%"  -> 12.50
    "-3.20%"  -> -3.20
    0.125     -> 12.5   (finvizfinance 0-1 range)
    12.5      -> 12.5   (abs > 1, already percent)
    "-"       -> None
    None      -> None
    """
    if value is None:
        return None

    if isinstance(value, str):
        s = value.strip()
        if not s or s == "-":
            return None
        s = s.replace("%", "")
        try:
            return float(s)
        except (ValueError, TypeError):
            return None

    if isinstance(value, (int, float)):
        f = float(value)
        # finvizfinance returns 0.125 for 12.5%; detect by abs <= 1
        if abs(f) <= 1.0:
            return f * 100.0
        return f

    return None


def _parse_volume(value) -> Optional[int]:
    """Parse FINVIZ Volume value to integer.

    "1,234,567" -> 1234567
    1234567     -> 1234567
    "1.2M"      -> 1200000
    "-"         -> None
    None        -> None
    """
    if value is None:
        return None

    if isinstance(value, (int, float)):
        return int(value)

    s = str(value).strip()
    if not s or s == "-":
        return None

    # Handle M suffix
    if s.upper().endswith("M"):
        try:
            return int(float(s[:-1]) * 1_000_000)
        except (ValueError, TypeError):
            return None

    # Handle comma-separated
    s = s.replace(",", "")
    try:
        return int(float(s))
    except (ValueError, TypeError):
        return None


# ---------------------------------------------------------------------------
# Selector
# ---------------------------------------------------------------------------


class RepresentativeStockSelector:
    """Select representative stocks for themes dynamically.

    Fallback chain: FINVIZ Elite -> FINVIZ Public -> FMP ETF Holdings -> static_stocks
    """

    def __init__(
        self,
        finviz_elite_key: Optional[str] = None,
        fmp_api_key: Optional[str] = None,
        finviz_mode: str = "public",
        rate_limit_sec: float = 1.0,
        max_per_industry: int = 4,
        min_cap: str = "small",
    ):
        self._finviz_elite_key = finviz_elite_key
        self._fmp_api_key = fmp_api_key
        self._finviz_mode = finviz_mode
        self._rate_limit_sec = rate_limit_sec
        self._max_per_industry = max_per_industry
        self._min_cap = min_cap
        self._industry_cache: dict[tuple[str, bool], list[dict]] = {}
        self._etf_cache: dict[str, list[dict]] = {}
        self._last_request_time: float = 0.0
        self._source_states: dict[str, _SourceState] = {
            "elite": _SourceState(),
            "public": _SourceState(),
            "fmp": _SourceState(),
        }

    # -- Public properties ---------------------------------------------------

    @property
    def query_count(self) -> int:
        return sum(s.total_queries for s in self._source_states.values())

    @property
    def failure_count(self) -> int:
        return sum(s.total_failures for s in self._source_states.values())

    @property
    def source_states(self) -> dict[str, _SourceState]:
        return self._source_states

    @property
    def _active_sources(self) -> list[str]:
        """Sources actually available given current config."""
        sources: list[str] = []
        if self._finviz_mode == "elite" and self._finviz_elite_key:
            sources.append("elite")
        sources.append("public")  # always available
        if self._fmp_api_key:
            sources.append("fmp")
        return sources

    @property
    def status(self) -> str:
        """Overall health: active / degraded / circuit_broken."""
        active = self._active_sources
        disabled_count = sum(1 for name in active if self._source_states[name].disabled)
        if disabled_count == 0:
            return "active"
        elif disabled_count < len(active):
            return "degraded"
        else:
            return "circuit_broken"

    # -- Main entry point ----------------------------------------------------

    def select_stocks(self, theme: dict, max_stocks: int = 10) -> list[dict]:
        """Select representative stocks for a theme.

        Returns list of dicts with keys:
            symbol, source, market_cap, matched_industries, reasons, composite_score
        """
        is_bearish = theme.get("direction") == "bearish"
        candidates: list[dict] = []
        industries = [ind.get("name", "") for ind in theme.get("matching_industries", [])]

        # Priority 1: FINVIZ (Elite or Public) per industry
        for industry in industries:
            cache_key = (industry, is_bearish)
            if cache_key in self._industry_cache:
                candidates.extend(self._industry_cache[cache_key][: self._max_per_industry])
                continue

            stocks: list[dict] = []
            fetch_limit = max(max_stocks, self._max_per_industry * 2)

            # Elite attempt
            if (
                self._finviz_mode == "elite"
                and self._finviz_elite_key
                and not self._source_states["elite"].disabled
            ):
                stocks = self._fetch_finviz_elite(
                    industry, limit=fetch_limit, is_bearish=is_bearish
                )

            # Public fallback
            if not stocks and not self._source_states["public"].disabled:
                stocks = self._fetch_finviz_public(
                    industry, limit=fetch_limit, is_bearish=is_bearish
                )

            # Score and cache
            stocks = self._compute_composite_score(stocks, is_bearish)
            self._industry_cache[cache_key] = stocks
            candidates.extend(stocks[: self._max_per_industry])

        # Priority 1.5: 2nd pass - fill from cache for single-industry themes
        unique_syms = set(c["symbol"] for c in candidates)
        if len(unique_syms) < max_stocks:
            for industry in industries:
                cache_key = (industry, is_bearish)
                cached = self._industry_cache.get(cache_key, [])
                for stock in cached[self._max_per_industry :]:
                    if stock["symbol"] not in unique_syms:
                        candidates.append(stock)
                        unique_syms.add(stock["symbol"])
                        if len(unique_syms) >= max_stocks:
                            break
                if len(unique_syms) >= max_stocks:
                    break

        # Priority 2: FMP ETF Holdings
        unique_count = len(set(c["symbol"] for c in candidates))
        if (
            unique_count < max_stocks
            and self._fmp_api_key
            and not self._source_states["fmp"].disabled
        ):
            for etf in theme.get("proxy_etfs", []):
                if etf in self._etf_cache:
                    candidates.extend(self._etf_cache[etf])
                    continue
                holdings = self._fetch_etf_holdings(etf, limit=max_stocks)
                self._etf_cache[etf] = holdings
                candidates.extend(holdings)

        # Priority 3: static_stocks fallback
        if not candidates:
            for sym in theme.get("static_stocks", [])[:max_stocks]:
                candidates.append(
                    {
                        "symbol": sym,
                        "source": "static",
                        "market_cap": 0,
                        "matched_industries": [],
                        "reasons": ["Static fallback"],
                        "composite_score": 0,
                    }
                )

        return self._merge_and_rank(candidates, max_stocks)

    # -- Fetch methods -------------------------------------------------------

    def _fetch_finviz_elite(self, industry: str, limit: int, is_bearish: bool) -> list[dict]:
        """Fetch stocks via FINVIZ Elite CSV export."""
        if requests is None:
            return []

        self._rate_limit()
        self._source_states["elite"].total_queries += 1

        ind_code = _industry_to_code(industry)
        cap_code = CAP_ELITE_MAP.get(self._min_cap, "cap_smallover")
        perf_code = "ta_perf2_4wdown" if is_bearish else "ta_perf2_4wup"
        filter_str = f"ind_{ind_code},{cap_code},sh_avgvol_o100,sh_price_o10,{perf_code}"
        url = (
            f"https://elite.finviz.com/export.ashx"
            f"?v=151&f={filter_str}&ft=4&auth={self._finviz_elite_key}"
        )

        try:
            resp = requests.get(url, timeout=15)
            if resp.status_code in (401, 403):
                logger.warning("FINVIZ Elite auth failed (%s)", resp.status_code)
                self._record_failure("elite")
                return []
            if resp.status_code != 200:
                self._record_failure("elite")
                return []

            reader = csv.DictReader(io.StringIO(resp.text))
            stocks: list[dict] = []
            for row in reader:
                ticker = row.get("Ticker", "").strip()
                if not ticker:
                    continue
                stocks.append(
                    {
                        "symbol": ticker,
                        "source": "finviz_elite",
                        "market_cap": _parse_market_cap(row.get("Market Cap")),
                        "change": _parse_change(row.get("Change")),
                        "volume": _parse_volume(row.get("Volume")),
                        "matched_industries": [industry],
                        "reasons": [f"Elite screener: {industry}"],
                    }
                )
                if len(stocks) >= limit:
                    break

            self._record_success("elite")
            return stocks

        except Exception:
            logger.exception("FINVIZ Elite fetch failed for %s", industry)
            self._record_failure("elite")
            return []

    def _fetch_finviz_public(self, industry: str, limit: int, is_bearish: bool) -> list[dict]:
        """Fetch stocks via finvizfinance public screener."""
        if Overview is None:
            logger.warning("finvizfinance not installed")
            self._record_failure("public")
            return []

        self._rate_limit()
        self._source_states["public"].total_queries += 1

        perf_filter = "Month Down" if is_bearish else "Month Up"
        filters_dict = {
            "Industry": industry,
            "Market Cap.": CAP_FILTER_MAP.get(self._min_cap, "+Small (over $300mln)"),
            "Average Volume": "Over 100K",
            "Price": "Over $10",
            "Performance 2": perf_filter,
        }

        try:
            o = Overview()
            o.set_filter(filters_dict=filters_dict)
            df = o.screener_view(order="Market Cap.", limit=limit, verbose=0, ascend=False)

            if df is None or df.empty:
                self._record_success("public")  # Empty result is not a failure
                return []

            stocks: list[dict] = []
            for _, row in df.iterrows():
                ticker = str(row.get("Ticker", "")).strip()
                if not ticker:
                    continue
                stocks.append(
                    {
                        "symbol": ticker,
                        "source": "finviz_public",
                        "market_cap": _parse_market_cap(row.get("Market Cap")),
                        "change": _parse_change(row.get("Change")),
                        "volume": _parse_volume(row.get("Volume")),
                        "matched_industries": [industry],
                        "reasons": [f"Public screener: {industry}"],
                    }
                )

            self._record_success("public")
            return stocks

        except Exception:
            logger.exception("FINVIZ public fetch failed for %s", industry)
            self._record_failure("public")
            return []

    def _fetch_etf_holdings(self, etf_symbol: str, limit: int) -> list[dict]:
        """Fetch top ETF holdings via FMP API."""
        if requests is None or not self._fmp_api_key:
            return []

        self._rate_limit()
        self._source_states["fmp"].total_queries += 1

        # stable: /etf/holdings?symbol= ; v3 fallback (legacy keys): /etf-holder/{symbol}.
        # Both return a list of holdings with `asset` (ticker) and `marketValue`.
        endpoints = [
            ("https://financialmodelingprep.com/stable/etf/holdings", {"symbol": etf_symbol}),
            (f"https://financialmodelingprep.com/api/v3/etf-holder/{etf_symbol}", None),
        ]

        try:
            data = None
            for url, params in endpoints:
                resp = requests.get(
                    url, params=params, headers={"apikey": self._fmp_api_key}, timeout=15
                )
                if resp.status_code != 200:
                    continue
                payload = resp.json()
                if isinstance(payload, list) and payload:
                    data = payload
                    break

            if not data:
                self._record_failure("fmp")
                return []

            stocks: list[dict] = []
            for item in data[:limit]:
                ticker = item.get("asset", "").strip()
                if not ticker:
                    continue
                stocks.append(
                    {
                        "symbol": ticker,
                        "source": "etf_holdings",
                        "market_cap": _parse_market_cap(item.get("marketValue")),
                        "change": None,
                        "volume": None,
                        "matched_industries": [],
                        "reasons": [f"Held by {etf_symbol}"],
                    }
                )

            self._record_success("fmp")
            return stocks

        except Exception:
            logger.exception("FMP ETF holdings fetch failed for %s", etf_symbol)
            self._record_failure("fmp")
            return []

    # -- Scoring & ranking ---------------------------------------------------

    def _compute_composite_score(self, stocks: list[dict], is_bearish: bool) -> list[dict]:
        """Compute composite score: 0.4*cap_rank + 0.3*change_rank + 0.3*vol_rank.

        Missing fields are excluded and weights re-normalized.
        """
        if not stocks:
            return []

        n = len(stocks)
        if n == 1:
            stocks[0]["composite_score"] = 1.0
            return stocks

        # Build ranks for each metric (1 = best, n = worst)
        # market_cap: larger is better
        cap_vals = [(i, s.get("market_cap") or 0) for i, s in enumerate(stocks)]
        cap_sorted = sorted(cap_vals, key=lambda x: x[1], reverse=True)
        cap_rank = {idx: rank + 1 for rank, (idx, _) in enumerate(cap_sorted)}

        # change: direction-aware
        change_vals = []
        has_change = False
        for i, s in enumerate(stocks):
            c = s.get("change")
            if c is not None:
                has_change = True
                directional = abs(c) if is_bearish else c
                change_vals.append((i, directional))
            else:
                change_vals.append((i, None))

        change_rank: dict[int, Optional[int]] = {}
        if has_change:
            valid = [(i, v) for i, v in change_vals if v is not None]
            valid_sorted = sorted(valid, key=lambda x: x[1], reverse=True)
            for rank, (idx, _) in enumerate(valid_sorted):
                change_rank[idx] = rank + 1
            for i, v in change_vals:
                if v is None:
                    change_rank[i] = None
        else:
            for i in range(n):
                change_rank[i] = None

        # volume: larger is better
        vol_vals = []
        has_volume = False
        for i, s in enumerate(stocks):
            v = s.get("volume")
            if v is not None:
                has_volume = True
                vol_vals.append((i, v))
            else:
                vol_vals.append((i, None))

        vol_rank: dict[int, Optional[int]] = {}
        if has_volume:
            valid = [(i, v) for i, v in vol_vals if v is not None]
            valid_sorted = sorted(valid, key=lambda x: x[1], reverse=True)
            for rank, (idx, _) in enumerate(valid_sorted):
                vol_rank[idx] = rank + 1
            for i, v in vol_vals:
                if v is None:
                    vol_rank[i] = None
        else:
            for i in range(n):
                vol_rank[i] = None

        # Compute composite with re-normalization
        for i, s in enumerate(stocks):
            weights = {}
            ranks = {}

            # Cap is always available
            weights["cap"] = 0.4
            ranks["cap"] = cap_rank[i]

            cr = change_rank.get(i)
            if cr is not None:
                weights["change"] = 0.3
                ranks["change"] = cr

            vr = vol_rank.get(i)
            if vr is not None:
                weights["vol"] = 0.3
                ranks["vol"] = vr

            total_weight = sum(weights.values())

            # Normalized rank score: (n - rank + 1) / n => 1.0 for rank 1
            # Count of valid items for each metric
            valid_counts = {
                "cap": n,
                "change": sum(1 for _, v in change_vals if v is not None) if has_change else 0,
                "vol": sum(1 for _, v in vol_vals if v is not None) if has_volume else 0,
            }

            score = 0.0
            for key, w in weights.items():
                count = valid_counts.get(key, n)
                if count > 0:
                    normalized = (count - ranks[key] + 1) / count
                    score += (w / total_weight) * normalized

            s["composite_score"] = round(score, 4)

        # Sort by composite score descending
        stocks.sort(key=lambda x: x.get("composite_score", 0), reverse=True)
        return stocks

    def _merge_and_rank(self, candidates: list[dict], max_stocks: int) -> list[dict]:
        """Deduplicate by symbol, merge industries/reasons, rank by score."""
        merged: dict[str, dict] = {}
        for c in candidates:
            sym = c["symbol"]
            if sym in merged:
                for ind in c.get("matched_industries", []):
                    if ind not in merged[sym]["matched_industries"]:
                        merged[sym]["matched_industries"].append(ind)
                merged[sym]["reasons"].extend(c.get("reasons", []))
                merged[sym]["composite_score"] = max(
                    merged[sym]["composite_score"],
                    c.get("composite_score", 0),
                )
            else:
                merged[sym] = dict(c)

        ranked = sorted(
            merged.values(),
            key=lambda x: x.get("composite_score", 0),
            reverse=True,
        )
        return ranked[:max_stocks]

    # -- Rate limiting & circuit breaker -------------------------------------

    def _rate_limit(self):
        """Enforce minimum interval between requests."""
        now = time.time()
        elapsed = now - self._last_request_time
        if elapsed < self._rate_limit_sec:
            time.sleep(self._rate_limit_sec - elapsed)
        self._last_request_time = time.time()

    def _record_success(self, source: str):
        """Record successful query for source."""
        state = self._source_states[source]
        state.consecutive_failures = 0

    def _record_failure(self, source: str):
        """Record failed query for source. Disable after consecutive limit."""
        state = self._source_states[source]
        state.consecutive_failures += 1
        state.total_failures += 1
        if state.consecutive_failures >= _MAX_CONSECUTIVE_FAILURES:
            state.disabled = True
            logger.warning(
                "Source %s disabled after %d consecutive failures",
                source,
                _MAX_CONSECUTIVE_FAILURES,
            )

"""
Theme Detector - Scoring, Labeling & Confidence

Combines theme heat and lifecycle maturity into a final theme score dict.
Follows the pattern from macro-regime-detector/scripts/scorer.py.

Design Note on Confidence Levels:
    narrative_confirmed is intentionally set to False in the script output.
    This is by design: Claude performs WebSearch-based narrative confirmation
    as a post-processing step. Script-only output therefore has a maximum
    confidence of "Medium" (quant_confirmed + breadth_confirmed = 2 layers).
    After Claude confirms narrative alignment, confidence can reach "High".
"""

HEAT_LABELS = {
    80: "Hot",
    60: "Warm",
    40: "Neutral",
    20: "Cool",
    0: "Cold",
}


def get_heat_label(heat_score: float) -> str:
    """Map heat score to label: Hot/Warm/Neutral/Cool/Cold."""
    if heat_score >= 80:
        return "Hot"
    elif heat_score >= 60:
        return "Warm"
    elif heat_score >= 40:
        return "Neutral"
    elif heat_score >= 20:
        return "Cool"
    else:
        return "Cold"


def calculate_confidence(
    quant_confirmed: bool,
    breadth_confirmed: bool,
    narrative_confirmed: bool,
    stale_data_penalty: bool,
    coverage_penalty_count: int = 0,
) -> str:
    """Determine confidence level from confirmation layers.

    3 layers confirmed => High
    2 layers confirmed => Medium
    1 or 0 layers     => Low
    stale_data_penalty downgrades by 1 level (High->Medium, Medium->Low).
    coverage_penalty_count downgrades by up to 2 additional levels.
    """
    confirmed_count = sum([quant_confirmed, breadth_confirmed, narrative_confirmed])

    if confirmed_count >= 3:
        level = "High"
    elif confirmed_count >= 2:
        level = "Medium"
    else:
        level = "Low"

    if stale_data_penalty and level != "Low":
        level = "Medium" if level == "High" else "Low"

    for _ in range(min(max(coverage_penalty_count, 0), 2)):
        if level == "High":
            level = "Medium"
        elif level == "Medium":
            level = "Low"
        else:
            break

    return level


def determine_data_mode(fmp_available: bool, finviz_elite: bool) -> str:
    """Describe available data sources."""
    if fmp_available and finviz_elite:
        return "FINVIZ-Elite+FMP"
    elif fmp_available and not finviz_elite:
        return "FMP-only"
    elif not fmp_available and finviz_elite:
        return "FINVIZ-Elite"
    else:
        return "FINVIZ-Public"


def score_theme(
    theme_heat: float,
    lifecycle_maturity: float,
    lifecycle_stage: str,
    direction: str,
    confidence: str,
    data_mode: str,
) -> dict:
    """Combine all dimensions into final theme score dict."""
    return {
        "theme_heat": theme_heat,
        "heat_label": get_heat_label(theme_heat),
        "lifecycle_maturity": lifecycle_maturity,
        "lifecycle_stage": lifecycle_stage,
        "direction": direction,
        "confidence": confidence,
        "data_mode": data_mode,
    }

"""Shared fixtures for Theme Detector tests"""

import os
import sys

# Add scripts directory to path so modules can be imported
sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
# Add tests directory to path so helpers can be imported
sys.path.insert(0, os.path.dirname(__file__))

# Theme Detector Tests

## Unit Tests (Phase 1)
- test_industry_ranker.py
- test_theme_classifier.py
- test_heat_calculator.py
- test_lifecycle_calculator.py
- test_scorer.py
- test_report_generator.py
- test_uptrend_client.py

## Unit Tests (Phase 2)
- test_representative_stock_selector.py (dynamic stock selection, FINVIZ/FMP fallback, circuit breaker)

## Integration Tests
- test_theme_detector_e2e.py (full pipeline, mocked I/O, no network required)

## Network Integration Tests (Phase 2 TODO)
- test_finviz_performance_client.py (requires network)
- test_etf_scanner.py (requires network)
- test_uptrend_client_integration.py (requires network)

"""Tests for config_loader module.

Covers YAML loading, validation, ETF catalog extraction, and fallback behavior.
"""

import os
import tempfile

import pytest
import yaml
from config_loader import (
    _extract_etf_catalog,
    _get_bundled_yaml_path,
    _load_yaml,
    _strip_etf_count,
    _validate_config,
    load_themes_config,
)

# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------


def _write_yaml(tmpdir, content):
    path = os.path.join(tmpdir, "themes.yaml")
    with open(path, "w", encoding="utf-8") as f:
        f.write(content)
    return path


VALID_YAML = """\
cross_sector_min_matches: 2
vertical_min_industries: 3
cross_sector:
  - theme_name: "Test Theme"
    matching_keywords:
      - Semiconductors
      - Software - Application
    proxy_etfs: [SMH, SOXX]
    static_stocks: [NVDA, AMD]
    etf_count: 4
"""

VALID_YAML_NO_ETF_COUNT = """\
cross_sector_min_matches: 2
vertical_min_industries: 3
cross_sector:
  - theme_name: "No ETF Count"
    matching_keywords:
      - Gold
    proxy_etfs: [GDX]
    static_stocks: [NEM]
"""


# ---------------------------------------------------------------------------
# TestLoadYaml
# ---------------------------------------------------------------------------


class TestLoadYaml:
    def test_loads_valid_yaml(self):
        with tempfile.TemporaryDirectory() as tmpdir:
            path = _write_yaml(tmpdir, VALID_YAML)
            config = _load_yaml(path)
            assert config["cross_sector_min_matches"] == 2
            assert len(config["cross_sector"]) == 1

    def test_missing_file_raises(self):
        with pytest.raises(FileNotFoundError):
            _load_yaml("/nonexistent/path/themes.yaml")

    def test_invalid_yaml_raises(self):
        with tempfile.TemporaryDirectory() as tmpdir:
            path = _write_yaml(tmpdir, "cross_sector: [{{invalid")
            with pytest.raises(yaml.YAMLError):
                _load_yaml(path)


# ---------------------------------------------------------------------------
# TestValidateConfig
# ---------------------------------------------------------------------------


class TestValidateConfig:
    def test_valid_config_passes(self):
        config = {
            "cross_sector_min_matches": 2,
            "vertical_min_industries": 3,
            "cross_sector": [
                {
                    "theme_name": "Test",
                    "matching_keywords": ["A"],
                    "proxy_etfs": [],
                    "static_stocks": [],
                },
            ],
        }
        _validate_config(config)  # should not raise

    def test_missing_cross_sector_raises(self):
        config = {"cross_sector_min_matches": 2}
        with pytest.raises(ValueError, match="cross_sector"):
            _validate_config(config)

    def test_missing_theme_name_raises(self):
        config = {
            "cross_sector": [
                {"matching_keywords": ["A"]},
            ],
        }
        with pytest.raises(ValueError, match="theme_name"):
            _validate_config(config)

    def test_missing_matching_keywords_raises(self):
        config = {
            "cross_sector": [
                {"theme_name": "Test"},
            ],
        }
        with pytest.raises(ValueError, match="matching_keywords"):
            _validate_config(config)


# ---------------------------------------------------------------------------
# TestExtractEtfCatalog
# ---------------------------------------------------------------------------


class TestExtractEtfCatalog:
    def test_extracts_etf_counts(self):
        config = {
            "cross_sector": [
                {"theme_name": "A", "etf_count": 5},
                {"theme_name": "B", "etf_count": 3},
            ],
        }
        catalog = _extract_etf_catalog(config)
        assert catalog == {"A": 5, "B": 3}

    def test_missing_etf_count_defaults_to_zero(self):
        config = {
            "cross_sector": [
                {"theme_name": "A"},
            ],
        }
        catalog = _extract_etf_catalog(config)
        assert catalog == {"A": 0}

    def test_empty_config_returns_empty(self):
        config = {"cross_sector": []}
        catalog = _extract_etf_catalog(config)
        assert catalog == {}


# ---------------------------------------------------------------------------
# TestStripEtfCount
# ---------------------------------------------------------------------------


class TestStripEtfCount:
    def test_removes_etf_count_from_themes(self):
        config = {
            "cross_sector": [
                {"theme_name": "A", "etf_count": 5, "matching_keywords": ["X"]},
            ],
        }
        stripped = _strip_etf_count(config)
        assert "etf_count" not in stripped["cross_sector"][0]
        assert stripped["cross_sector"][0]["theme_name"] == "A"

    def test_preserves_other_keys(self):
        config = {
            "cross_sector_min_matches": 2,
            "vertical_min_industries": 3,
            "cross_sector": [
                {"theme_name": "A", "etf_count": 5, "matching_keywords": ["X"]},
            ],
        }
        stripped = _strip_etf_count(config)
        assert stripped["cross_sector_min_matches"] == 2
        assert stripped["vertical_min_industries"] == 3


# ---------------------------------------------------------------------------
# TestLoadThemesConfig (integration)
# ---------------------------------------------------------------------------


class TestLoadThemesConfig:
    def test_bundled_yaml_loads_successfully(self):
        """Default (no path) loads the bundled themes.yaml."""
        config, catalog = load_themes_config()
        assert "cross_sector" in config
        assert len(config["cross_sector"]) == 15
        assert "AI & Semiconductors" in catalog
        assert catalog["AI & Semiconductors"] == 8

    def test_custom_yaml_path(self):
        with tempfile.TemporaryDirectory() as tmpdir:
            path = _write_yaml(tmpdir, VALID_YAML)
            config, catalog = load_themes_config(yaml_path=path)
            assert len(config["cross_sector"]) == 1
            assert config["cross_sector"][0]["theme_name"] == "Test Theme"
            assert catalog == {"Test Theme": 4}

    def test_explicit_path_missing_raises_error(self):
        """Explicit path that doesn't exist -> fail-fast (not fallback)."""
        with pytest.raises(FileNotFoundError):
            load_themes_config(yaml_path="/nonexistent/themes.yaml")

    def test_explicit_path_invalid_yaml_raises_error(self):
        """Explicit path with invalid YAML -> fail-fast."""
        with tempfile.TemporaryDirectory() as tmpdir:
            path = _write_yaml(tmpdir, "cross_sector: [{{invalid")
            with pytest.raises(yaml.YAMLError):
                load_themes_config(yaml_path=path)

    def test_bundled_yaml_fallback_to_inline(self, monkeypatch):
        """When bundled YAML is broken, falls back to inline config."""

        def fake_get_path():
            return "/nonexistent/bundled/themes.yaml"

        monkeypatch.setattr("config_loader._get_bundled_yaml_path", fake_get_path)
        config, catalog = load_themes_config()
        # Should fall back to DEFAULT_THEMES_CONFIG from default_theme_config.py
        assert "cross_sector" in config
        assert len(config["cross_sector"]) == 15

    def test_config_has_no_etf_count_after_loading(self):
        """etf_count should be stripped from the returned config."""
        config, catalog = load_themes_config()
        for theme in config["cross_sector"]:
            assert "etf_count" not in theme

    def test_get_bundled_yaml_path_exists(self):
        path = _get_bundled_yaml_path()
        assert os.path.exists(path)
        assert path.endswith("themes.yaml")

"""Tests for detect_divergence function."""

from theme_detector import detect_divergence


class TestDetectDivergence:
    """Unit tests for ETF price vs breadth divergence detection."""

    def test_no_divergence_small_gap(self):
        """Gap < 25 returns None (no divergence detected)."""
        heat_breakdown = {"momentum_strength": 60, "uptrend_signal": 50}
        result = detect_divergence(heat_breakdown, "bullish")
        assert result is None

    def test_no_divergence_exact_threshold(self):
        """Gap == 24.9 (just under 25) returns None."""
        heat_breakdown = {"momentum_strength": 74.9, "uptrend_signal": 50}
        result = detect_divergence(heat_breakdown, "bullish")
        assert result is None

    def test_no_divergence_zero_gap(self):
        """Identical momentum and uptrend returns None."""
        heat_breakdown = {"momentum_strength": 50, "uptrend_signal": 50}
        result = detect_divergence(heat_breakdown, "bullish")
        assert result is None

    def test_narrow_rally_momentum_exceeds_uptrend(self):
        """Momentum >> uptrend returns narrow_rally type."""
        heat_breakdown = {"momentum_strength": 85, "uptrend_signal": 40}
        result = detect_divergence(heat_breakdown, "bullish")
        assert result is not None
        assert result["type"] == "narrow_rally"
        assert result["gap"] == 45.0
        assert "concentrated" in result["description"]
        assert "fragile" in result["description"]

    def test_narrow_rally_exact_threshold(self):
        """Gap == 25 returns narrow_rally (threshold is inclusive for abs >= 25)."""
        heat_breakdown = {"momentum_strength": 75, "uptrend_signal": 50}
        result = detect_divergence(heat_breakdown, "bullish")
        assert result is not None
        assert result["type"] == "narrow_rally"
        assert result["gap"] == 25.0

    def test_internal_recovery_bearish_direction(self):
        """Uptrend >> momentum with bearish direction returns reversal candidate."""
        heat_breakdown = {"momentum_strength": 30, "uptrend_signal": 70}
        result = detect_divergence(heat_breakdown, "bearish")
        assert result is not None
        assert result["type"] == "internal_recovery"
        assert result["gap"] == 40.0
        assert "reversal candidate" in result["description"]

    def test_internal_recovery_bullish_direction(self):
        """Uptrend >> momentum with bullish direction returns acceleration candidate."""
        heat_breakdown = {"momentum_strength": 30, "uptrend_signal": 70}
        result = detect_divergence(heat_breakdown, "bullish")
        assert result is not None
        assert result["type"] == "internal_recovery"
        assert result["gap"] == 40.0
        assert "acceleration candidate" in result["description"]

    def test_missing_keys_return_none(self):
        """Missing keys return None rather than inventing neutral coverage."""
        result = detect_divergence({}, "bullish")
        assert result is None

    def test_none_uptrend_returns_none(self):
        """Missing uptrend coverage does not crash divergence detection."""
        result = detect_divergence({"momentum_strength": 80, "uptrend_signal": None}, "bullish")
        assert result is None

    def test_gap_rounding(self):
        """Gap values are rounded to 1 decimal place."""
        heat_breakdown = {"momentum_strength": 80.333, "uptrend_signal": 45.777}
        result = detect_divergence(heat_breakdown, "bullish")
        assert result is not None
        assert result["gap"] == 34.6  # 80.333 - 45.777 = 34.556 -> 34.6

    def test_narrow_rally_bearish_direction(self):
        """Narrow rally detection works for bearish themes too."""
        heat_breakdown = {"momentum_strength": 90, "uptrend_signal": 30}
        result = detect_divergence(heat_breakdown, "bearish")
        assert result is not None
        assert result["type"] == "narrow_rally"
        assert result["gap"] == 60.0

"""Unit tests for ETFScanner FMP API migration.

Tests FMP backend, symbol normalization, caching, batching,
per-symbol retry, symbol-level fallback, and backend stats.
"""

from datetime import date
from json import JSONDecodeError
from unittest.mock import MagicMock, patch

import numpy as np
import pandas as pd
import pytest
import requests
from etf_scanner import ETFScanner, _frame_on_or_before


def test_dataframe_ceiling_handles_timezone_naive_aware_and_invalid_indexes():
    frame = pd.DataFrame(
        {"Close": [1.0, 2.0, 3.0, 4.0]},
        index=[
            "2026-04-06",
            pd.Timestamp("2026-04-06 23:30", tz="America/New_York"),
            "2026-04-07",
            "not-a-date",
        ],
    )

    filtered = _frame_on_or_before(frame, date(2026, 4, 6))

    assert filtered["Close"].tolist() == [1.0, 2.0]


def test_dataframe_ceiling_rejects_nat_numeric_and_all_future_rows():
    frame = pd.DataFrame(
        {"Close": [1.0, 2.0, 3.0]},
        index=[pd.NaT, 0, "2026-04-07"],
    )

    assert _frame_on_or_before(frame, date(2026, 4, 6)).empty


@patch("etf_scanner.yf")
def test_yfinance_batch_filters_provider_overfetch_before_metrics(mock_yf):
    scanner = ETFScanner(as_of_date=date(2026, 4, 6))
    mock_yf.download.return_value = pd.DataFrame(
        {
            "Close": list(range(1, 18)) + [1_000.0],
            "High": list(range(1, 18)) + [1_001.0],
            "Low": list(range(1, 18)) + [999.0],
        },
        index=pd.date_range("2026-03-21", periods=18, freq="D"),
    )
    mock_yf.Ticker.return_value.info = {}

    [result] = scanner._batch_stock_metrics_yfinance(["AAPL"])

    assert result["dist_from_52w_high"] == 0.0


@patch("etf_scanner.yf")
def test_yfinance_cache_filters_before_storing(mock_yf):
    scanner = ETFScanner(as_of_date=date(2026, 4, 6))
    mock_yf.download.return_value = pd.DataFrame(
        {"Close": [100.0, 999.0]},
        index=pd.to_datetime(["2026-04-06", "2026-04-07"]),
    )

    data = scanner._get_cached("AAPL")

    assert data is not None
    assert data["Close"].tolist() == [100.0]
    assert scanner._cache["AAPL_6mo_2026-04-06"].equals(data)


# ---------------------------------------------------------------------------
# TestCalculateRSI
# ---------------------------------------------------------------------------
class TestCalculateRSI:
    """Tests for the static _calculate_rsi method."""

    def test_known_sequence_rsi(self):
        """RSI of a known oscillating sequence returns a value in (0, 100)."""
        # 20 data points with alternating gains and losses
        prices = pd.Series(
            [
                44.0,
                44.34,
                44.09,
                44.15,
                43.61,
                44.33,
                44.83,
                45.10,
                45.42,
                45.84,
                46.08,
                45.89,
                46.03,
                45.61,
                46.28,
                46.28,
                46.00,
                46.03,
                46.41,
                46.22,
            ]
        )
        rsi = ETFScanner._calculate_rsi(prices, period=14)
        assert rsi is not None
        assert 0 < rsi < 100

    def test_insufficient_data_returns_none(self):
        """Fewer than period+1 data points returns None."""
        prices = pd.Series([10.0, 11.0, 12.0])
        assert ETFScanner._calculate_rsi(prices, period=14) is None

    def test_all_gains_returns_100(self):
        """Monotonically increasing prices produce RSI = 100."""
        prices = pd.Series([float(i) for i in range(1, 20)])
        rsi = ETFScanner._calculate_rsi(prices, period=14)
        assert rsi == 100.0


# ---------------------------------------------------------------------------
# TestCalculate52wDistances
# ---------------------------------------------------------------------------
class TestCalculate52wDistances:
    """Tests for the static _calculate_52w_distances method."""

    def test_at_52w_high_returns_zero(self):
        """When current price equals 52w high, dist_from_52w_high is 0."""
        close = pd.Series([90.0, 95.0, 100.0])
        high = pd.Series([92.0, 97.0, 100.0])
        low = pd.Series([88.0, 93.0, 98.0])
        result = ETFScanner._calculate_52w_distances(close, high, low)
        assert result["dist_from_52w_high"] == 0.0

    def test_at_52w_low_returns_zero(self):
        """When current price equals 52w low, dist_from_52w_low is 0."""
        close = pd.Series([100.0, 95.0, 88.0])
        high = pd.Series([102.0, 97.0, 90.0])
        low = pd.Series([98.0, 93.0, 88.0])
        result = ETFScanner._calculate_52w_distances(close, high, low)
        assert result["dist_from_52w_low"] == 0.0

    def test_midpoint_values(self):
        """Midpoint values produce non-zero distances for both."""
        close = pd.Series([50.0, 100.0, 75.0])
        high = pd.Series([52.0, 102.0, 77.0])
        low = pd.Series([48.0, 98.0, 73.0])
        result = ETFScanner._calculate_52w_distances(close, high, low)
        assert result["dist_from_52w_high"] is not None
        assert result["dist_from_52w_low"] is not None
        assert result["dist_from_52w_high"] > 0
        assert result["dist_from_52w_low"] > 0


# ---------------------------------------------------------------------------
# TestNormalizeSymbol
# ---------------------------------------------------------------------------
class TestNormalizeSymbol:
    """Tests for symbol normalization (dash to dot)."""

    def test_dash_to_dot(self):
        assert ETFScanner._normalize_symbol_for_fmp("BRK-B") == "BRK.B"

    def test_normal_symbol_unchanged(self):
        assert ETFScanner._normalize_symbol_for_fmp("AAPL") == "AAPL"

    def test_already_dot_unchanged(self):
        assert ETFScanner._normalize_symbol_for_fmp("BRK.B") == "BRK.B"


# ---------------------------------------------------------------------------
# TestFMPEndpointFallback
# ---------------------------------------------------------------------------
class TestFMPEndpointFallback:
    """Tests for _fmp_request stable -> v3 fallback."""

    def _make_scanner(self):
        return ETFScanner(fmp_api_key="test_key", rate_limit_sec=0)

    @patch("etf_scanner._requests_lib")
    def test_stable_success_uses_stable_url_format(self, mock_requests):
        """Stable endpoint uses ?symbol= query param format."""
        scanner = self._make_scanner()
        mock_resp = MagicMock()
        mock_resp.status_code = 200
        mock_resp.json.return_value = [{"symbol": "AAPL", "pe": 30}]
        mock_requests.get.return_value = mock_resp

        result = scanner._fmp_request("quote", "AAPL,MSFT")
        assert result is not None

        # Verify stable URL was called (base URL without symbols in path)
        called_url = mock_requests.get.call_args[0][0]
        assert "stable/quote" in called_url
        assert "AAPL,MSFT" not in called_url  # symbols in params, not path
        called_params = mock_requests.get.call_args[1]["params"]
        assert called_params["symbol"] == "AAPL,MSFT"

    @patch("etf_scanner._requests_lib")
    def test_stable_fails_falls_back_to_v3_path_format(self, mock_requests):
        """When stable fails, v3 endpoint uses /SYMBOLS path format."""
        scanner = self._make_scanner()

        # First call (stable) fails
        fail_resp = MagicMock()
        fail_resp.status_code = 500
        # Second call (v3) succeeds
        ok_resp = MagicMock()
        ok_resp.status_code = 200
        ok_resp.json.return_value = [{"symbol": "AAPL"}]
        mock_requests.get.side_effect = [fail_resp, ok_resp]

        result = scanner._fmp_request("quote", "AAPL,MSFT")
        assert result is not None

        # Verify v3 call uses path-based symbols
        v3_call = mock_requests.get.call_args_list[1]
        called_url = v3_call[0][0]
        assert "/api/v3/quote/AAPL,MSFT" in called_url
        # Symbols should NOT be in params for v3
        assert "symbol" not in v3_call[1]["params"]

    @patch("etf_scanner._requests_lib")
    def test_both_fail_returns_none(self, mock_requests):
        """When both stable and v3 fail, returns None."""
        scanner = self._make_scanner()
        fail_resp = MagicMock()
        fail_resp.status_code = 500
        mock_requests.get.return_value = fail_resp

        result = scanner._fmp_request("quote", "AAPL")
        assert result is None
        assert scanner.backend_stats()["fmp_failures"] == 2


# ---------------------------------------------------------------------------
# TestFMPQuoteFetch
# ---------------------------------------------------------------------------
class TestFMPQuoteFetch:
    """Tests for _fetch_fmp_quotes."""

    def _make_scanner(self):
        return ETFScanner(fmp_api_key="test_key", rate_limit_sec=0)

    @patch("etf_scanner._requests_lib")
    def test_batch_returns_mapped_dict(self, mock_requests):
        """Quote fetch returns {symbol: quote_dict} mapping."""
        scanner = self._make_scanner()
        mock_resp = MagicMock()
        mock_resp.status_code = 200
        mock_resp.json.return_value = [
            {"symbol": "AAPL", "pe": 30, "price": 150},
            {"symbol": "MSFT", "pe": 35, "price": 400},
        ]
        mock_requests.get.return_value = mock_resp

        result = scanner._fetch_fmp_quotes(["AAPL", "MSFT"])
        assert "AAPL" in result
        assert "MSFT" in result
        assert result["AAPL"]["pe"] == 30

    @patch("etf_scanner._requests_lib")
    def test_splits_large_batch_by_quote_batch_size(self, mock_requests):
        """Symbols exceeding FMP_QUOTE_BATCH_SIZE are split into batches."""
        scanner = self._make_scanner()
        scanner.FMP_QUOTE_BATCH_SIZE = 3  # small batch for testing

        symbols = ["A", "B", "C", "D", "E"]
        # Return non-empty data so stable succeeds (no v3 fallback)
        mock_resp = MagicMock()
        mock_resp.status_code = 200
        mock_resp.json.return_value = [{"symbol": "X"}]
        mock_requests.get.return_value = mock_resp

        scanner._fetch_fmp_quotes(symbols)
        # 5 symbols / batch_size 3 = 2 batches, stable succeeds = 2 API calls
        assert mock_requests.get.call_count == 2

    @patch("etf_scanner._requests_lib")
    def test_cache_uses_normalized_key(self, mock_requests):
        """BRK-B is cached as BRK.B (normalized)."""
        scanner = self._make_scanner()
        mock_resp = MagicMock()
        mock_resp.status_code = 200
        mock_resp.json.return_value = [
            {"symbol": "BRK.B", "pe": 10, "price": 400},
        ]
        mock_requests.get.return_value = mock_resp

        result = scanner._fetch_fmp_quotes(["BRK-B"])
        # Cached under normalized key
        assert "BRK.B" in scanner._fmp_quote_cache
        # But result maps back to original symbol
        assert "BRK-B" in result

    @patch("etf_scanner._requests_lib")
    def test_cache_hit_prevents_duplicate_call(self, mock_requests):
        """Second call for same symbol uses cache, no API call."""
        scanner = self._make_scanner()
        mock_resp = MagicMock()
        mock_resp.status_code = 200
        mock_resp.json.return_value = [
            {"symbol": "AAPL", "pe": 30},
        ]
        mock_requests.get.return_value = mock_resp

        scanner._fetch_fmp_quotes(["AAPL"])
        call_count_1 = mock_requests.get.call_count

        scanner._fetch_fmp_quotes(["AAPL"])
        call_count_2 = mock_requests.get.call_count

        # No additional API call for cached symbol
        assert call_count_2 == call_count_1

    @patch("etf_scanner._requests_lib")
    def test_api_error_returns_empty(self, mock_requests):
        """API error returns empty dict."""
        scanner = self._make_scanner()
        mock_requests.get.side_effect = requests.ConnectionError("Connection error")

        result = scanner._fetch_fmp_quotes(["AAPL"])
        assert result == {}

    @patch("etf_scanner._requests_lib")
    def test_original_symbol_retry_normalizes_cache_key(self, mock_requests):
        """When retry returns BRK-B, it is cached under normalized BRK.B."""
        scanner = self._make_scanner()

        # Batch call with normalized BRK.B: stable fails, v3 fails
        fail_resp = MagicMock()
        fail_resp.status_code = 500
        # Retry with original BRK-B: stable returns data with BRK-B symbol
        retry_resp = MagicMock()
        retry_resp.status_code = 200
        retry_resp.json.return_value = [{"symbol": "BRK-B", "pe": 10}]
        mock_requests.get.side_effect = [fail_resp, fail_resp, retry_resp]

        result = scanner._fetch_fmp_quotes(["BRK-B"])
        # Cache key is normalized to BRK.B despite API returning BRK-B
        assert "BRK.B" in scanner._fmp_quote_cache
        # Result maps back to original symbol
        assert "BRK-B" in result
        assert result["BRK-B"]["pe"] == 10


# ---------------------------------------------------------------------------
# TestFMPHistoricalFetch
# ---------------------------------------------------------------------------
class TestFMPHistoricalFetch:
    """Tests for _fetch_fmp_historical."""

    def _make_scanner(self):
        return ETFScanner(fmp_api_key="test_key", rate_limit_sec=0)

    def test_as_of_filters_future_and_malformed_rows_fail_closed(self, monkeypatch):
        scanner = ETFScanner(
            fmp_api_key="test_key",
            rate_limit_sec=0,
            as_of_date=date(2026, 2, 13),
        )
        rows = [
            {"date": "2026-02-14", "close": 151},
            {"date": "not-a-date", "close": 999},
            {"close": 998},
            {"date": "2026-02-13", "close": 150},
        ]
        monkeypatch.setattr(
            scanner,
            "_fmp_request",
            lambda *_args, **_kwargs: {"symbol": "AAPL", "historical": rows},
        )

        result = scanner._fetch_fmp_historical(["AAPL"], timeseries=20)

        assert result == {"AAPL": [{"date": "2026-02-13", "close": 150}]}

    @patch("etf_scanner._requests_lib")
    def test_multi_symbol_parses_historicalStockList(self, mock_requests):
        """Multi-symbol response uses historicalStockList format."""
        scanner = self._make_scanner()
        mock_resp = MagicMock()
        mock_resp.status_code = 200
        mock_resp.json.return_value = {
            "historicalStockList": [
                {
                    "symbol": "AAPL",
                    "historical": [
                        {"date": "2026-02-14", "close": 150},
                        {"date": "2026-02-13", "close": 148},
                    ],
                },
                {
                    "symbol": "MSFT",
                    "historical": [
                        {"date": "2026-02-14", "close": 400},
                    ],
                },
            ]
        }
        mock_requests.get.return_value = mock_resp

        result = scanner._fetch_fmp_historical(["AAPL", "MSFT"], timeseries=20)
        assert "AAPL" in result
        assert "MSFT" in result
        assert len(result["AAPL"]) == 2

    @patch("etf_scanner._requests_lib")
    def test_single_symbol_parses_historical(self, mock_requests):
        """Single-symbol response uses {symbol, historical} format."""
        scanner = self._make_scanner()
        mock_resp = MagicMock()
        mock_resp.status_code = 200
        mock_resp.json.return_value = {
            "symbol": "AAPL",
            "historical": [
                {"date": "2026-02-14", "close": 150},
            ],
        }
        mock_requests.get.return_value = mock_resp

        result = scanner._fetch_fmp_historical(["AAPL"], timeseries=20)
        assert "AAPL" in result

    @patch("etf_scanner._requests_lib")
    def test_batches_in_groups_of_5(self, mock_requests):
        """Historical fetch batches symbols in groups of FMP_HIST_BATCH_SIZE."""
        scanner = self._make_scanner()
        scanner.FMP_HIST_BATCH_SIZE = 2  # small batch for testing

        # Each batch returns empty but valid response
        mock_resp = MagicMock()
        mock_resp.status_code = 200
        mock_resp.json.return_value = {"historicalStockList": []}
        mock_requests.get.return_value = mock_resp

        scanner._fetch_fmp_historical(["A", "B", "C", "D", "E"], timeseries=20)
        # 5 symbols / batch_size 2 = 3 batch calls
        # No per-symbol retry since all symbols missing -> 5 more calls
        # But with empty data, all symbols are missing -> retry each
        assert mock_requests.get.call_count == 3 + 5

    @patch("etf_scanner._requests_lib")
    def test_batch_incomplete_retries_missing_per_symbol(self, mock_requests):
        """Missing symbols from batch get retried individually."""
        scanner = self._make_scanner()

        # First batch call returns only AAPL (MSFT missing)
        batch_resp = MagicMock()
        batch_resp.status_code = 200
        batch_resp.json.return_value = {
            "historicalStockList": [
                {"symbol": "AAPL", "historical": [{"close": 150}]},
            ]
        }
        # Per-symbol retry for MSFT succeeds
        retry_resp = MagicMock()
        retry_resp.status_code = 200
        retry_resp.json.return_value = {
            "symbol": "MSFT",
            "historical": [{"close": 400}],
        }
        mock_requests.get.side_effect = [batch_resp, retry_resp]

        result = scanner._fetch_fmp_historical(["AAPL", "MSFT"], timeseries=20)
        assert "AAPL" in result
        assert "MSFT" in result

    @patch("etf_scanner._requests_lib")
    def test_api_error_returns_empty(self, mock_requests):
        """Total failure returns empty dict."""
        scanner = self._make_scanner()
        mock_requests.get.side_effect = requests.Timeout("Timeout")

        result = scanner._fetch_fmp_historical(["AAPL"], timeseries=20)
        assert result == {}

    @patch("etf_scanner._requests_lib")
    def test_original_symbol_retry_normalizes_result_key(self, mock_requests):
        """When retry returns BRK-B, result key is normalized to BRK.B."""
        scanner = self._make_scanner()

        # Batch with normalized BRK.B: stable+v3 both fail
        fail_resp = MagicMock()
        fail_resp.status_code = 500
        # Per-symbol retry with normalized BRK.B: fails
        # Per-symbol retry with original BRK-B: returns data
        retry_resp = MagicMock()
        retry_resp.status_code = 200
        retry_resp.json.return_value = {
            "symbol": "BRK-B",
            "historical": [{"close": 400}],
        }
        mock_requests.get.side_effect = [
            fail_resp,
            fail_resp,  # batch stable+v3
            fail_resp,
            fail_resp,  # per-symbol BRK.B stable+v3
            retry_resp,  # per-symbol BRK-B stable succeeds
        ]

        result = scanner._fetch_fmp_historical(["BRK-B"], timeseries=20)
        # Result maps to original symbol despite API returning BRK-B
        assert "BRK-B" in result
        assert result["BRK-B"][0]["close"] == 400


# ---------------------------------------------------------------------------
# TestBatchStockMetricsFMP
# ---------------------------------------------------------------------------
class TestBatchStockMetricsFMP:
    """Tests for FMP-based batch_stock_metrics internal path."""

    def _make_scanner(self):
        return ETFScanner(fmp_api_key="test_key", rate_limit_sec=0)

    @patch("etf_scanner._requests_lib")
    def test_pe_from_quote(self, mock_requests):
        """PE ratio is extracted from FMP quote data."""
        scanner = self._make_scanner()

        # Quote response
        quote_resp = MagicMock()
        quote_resp.status_code = 200
        quote_resp.json.return_value = [
            {"symbol": "AAPL", "pe": 30.5, "price": 150, "yearHigh": 180, "yearLow": 120},
        ]
        # Historical response for RSI
        hist_resp = MagicMock()
        hist_resp.status_code = 200
        hist_resp.json.return_value = {
            "historicalStockList": [
                {"symbol": "AAPL", "historical": [{"close": 150 - i * 0.5} for i in range(20)]},
            ]
        }
        mock_requests.get.side_effect = [quote_resp, hist_resp]

        results = scanner._batch_stock_metrics_fmp(["AAPL"])
        assert len(results) == 1
        assert results[0]["pe_ratio"] == 30.5

    @patch("etf_scanner._requests_lib")
    def test_52w_distances_from_quote(self, mock_requests):
        """52-week distances come from yearHigh/yearLow in quote."""
        scanner = self._make_scanner()

        quote_resp = MagicMock()
        quote_resp.status_code = 200
        quote_resp.json.return_value = [
            {"symbol": "AAPL", "pe": 30, "price": 150, "yearHigh": 200, "yearLow": 100},
        ]
        hist_resp = MagicMock()
        hist_resp.status_code = 200
        hist_resp.json.return_value = {"historicalStockList": []}
        mock_requests.get.side_effect = [quote_resp] + [hist_resp] * 20

        results = scanner._batch_stock_metrics_fmp(["AAPL"])
        assert results[0]["dist_from_52w_high"] == round((200 - 150) / 200, 4)
        assert results[0]["dist_from_52w_low"] == round((150 - 100) / 150, 4)

    @patch("etf_scanner._requests_lib")
    def test_rsi_from_historical_close(self, mock_requests):
        """RSI is calculated from historical close prices."""
        scanner = self._make_scanner()

        quote_resp = MagicMock()
        quote_resp.status_code = 200
        quote_resp.json.return_value = [
            {"symbol": "AAPL", "pe": 30, "price": 150, "yearHigh": 180, "yearLow": 120},
        ]
        # FMP returns newest-first; code reverses to oldest-first for RSI.
        # Decreasing close here -> reversed = increasing -> RSI=100
        hist_resp = MagicMock()
        hist_resp.status_code = 200
        hist_resp.json.return_value = {
            "historicalStockList": [
                {"symbol": "AAPL", "historical": [{"close": float(150 - i)} for i in range(20)]},
            ]
        }
        mock_requests.get.side_effect = [quote_resp, hist_resp]

        results = scanner._batch_stock_metrics_fmp(["AAPL"])
        # After reversal: monotonic increase -> RSI = 100
        assert results[0]["rsi_14"] is not None
        assert results[0]["rsi_14"] == 100.0

    @patch("etf_scanner._requests_lib")
    def test_missing_fields_return_none(self, mock_requests):
        """Missing quote fields produce None for those metrics."""
        scanner = self._make_scanner()

        quote_resp = MagicMock()
        quote_resp.status_code = 200
        quote_resp.json.return_value = [
            {"symbol": "AAPL"},  # No pe, price, yearHigh, yearLow
        ]
        hist_resp = MagicMock()
        hist_resp.status_code = 200
        hist_resp.json.return_value = {"historicalStockList": []}
        mock_requests.get.side_effect = [quote_resp] + [hist_resp] * 20

        results = scanner._batch_stock_metrics_fmp(["AAPL"])
        assert results[0]["pe_ratio"] is None
        assert results[0]["dist_from_52w_high"] is None
        assert results[0]["rsi_14"] is None

    def test_empty_symbols_returns_empty(self):
        """Empty input returns empty list."""
        scanner = self._make_scanner()
        results = scanner.batch_stock_metrics([])
        assert results == []


# ---------------------------------------------------------------------------
# TestETFVolumeRatioFMP
# ---------------------------------------------------------------------------
class TestETFVolumeRatioFMP:
    """Tests for FMP-based ETF volume ratio calculation."""

    def _make_scanner(self):
        return ETFScanner(fmp_api_key="test_key", rate_limit_sec=0)

    @patch("etf_scanner._requests_lib")
    def test_20d_60d_from_historical(self, mock_requests):
        """Volume ratio is calculated from 20d/60d averages."""
        scanner = self._make_scanner()

        # 60 days of volume data
        hist_data = [
            {
                "date": f"2026-02-{14 - i:02d}" if i < 14 else f"2026-01-{31 - (i - 14):02d}",
                "volume": 1_000_000 + i * 10_000,
            }
            for i in range(60)
        ]
        mock_resp = MagicMock()
        mock_resp.status_code = 200
        mock_resp.json.return_value = {
            "historicalStockList": [
                {"symbol": "XLK", "historical": hist_data},
            ]
        }
        mock_requests.get.return_value = mock_resp

        result = scanner._batch_etf_volume_ratios_fmp(["XLK"])
        assert "XLK" in result
        assert result["XLK"]["vol_20d"] is not None
        assert result["XLK"]["vol_60d"] is not None
        assert result["XLK"]["vol_ratio"] is not None

    @patch("etf_scanner._requests_lib")
    def test_insufficient_data_returns_none(self, mock_requests):
        """Fewer than 20 data points returns None values."""
        scanner = self._make_scanner()

        mock_resp = MagicMock()
        mock_resp.status_code = 200
        mock_resp.json.return_value = {
            "historicalStockList": [
                {
                    "symbol": "XLK",
                    "historical": [
                        {"date": "2026-02-14", "volume": 1000000},
                    ],
                },
            ]
        }
        mock_requests.get.return_value = mock_resp

        result = scanner._batch_etf_volume_ratios_fmp(["XLK"])
        assert result["XLK"]["vol_20d"] is None


# ---------------------------------------------------------------------------
# TestBatchETFVolumeRatios
# ---------------------------------------------------------------------------
class TestBatchETFVolumeRatios:
    """Tests for the public batch_etf_volume_ratios method."""

    @patch("etf_scanner._requests_lib")
    def test_batch_returns_all_etfs(self, mock_requests):
        """Batch fetches volume ratios for multiple ETFs."""
        scanner = ETFScanner(fmp_api_key="test_key", rate_limit_sec=0)

        hist_data = [{"volume": 1_000_000} for _ in range(60)]
        mock_resp = MagicMock()
        mock_resp.status_code = 200
        mock_resp.json.return_value = {
            "historicalStockList": [
                {"symbol": "XLK", "historical": hist_data},
                {"symbol": "SMH", "historical": hist_data},
            ]
        }
        mock_requests.get.return_value = mock_resp

        result = scanner.batch_etf_volume_ratios(["XLK", "SMH"])
        assert "XLK" in result
        assert "SMH" in result


# ---------------------------------------------------------------------------
# TestSymbolLevelFallback
# ---------------------------------------------------------------------------
class TestSymbolLevelFallback:
    """Tests for symbol-level FMP -> yfinance fallback."""

    @patch("etf_scanner._requests_lib")
    @patch("etf_scanner.yf")
    def test_partial_fmp_success_fills_missing_from_yfinance(self, mock_yf, mock_requests):
        """Partial FMP success -> missing symbols fall back to yfinance."""
        scanner = ETFScanner(fmp_api_key="test_key", rate_limit_sec=0)

        # FMP: only AAPL succeeds
        quote_resp = MagicMock()
        quote_resp.status_code = 200
        quote_resp.json.return_value = [
            {"symbol": "AAPL", "pe": 30, "price": 150, "yearHigh": 180, "yearLow": 120},
        ]

        # Symbols are fetched one per request (no comma-batching on /stable),
        # so route by symbol/endpoint rather than a fixed call sequence: AAPL
        # succeeds via FMP, MSFT fails (→ yfinance fallback).
        def fake_get(url, params=None, headers=None, timeout=None):
            p = params or {}
            # symbol is in params for /stable, in the path for the v3 fallback
            symbol = p.get("symbol") or url.rstrip("/").rsplit("/", 1)[-1]
            resp = MagicMock()
            if "AAPL" in symbol:
                resp.status_code = 200
                if "quote" in url:
                    resp.json.return_value = [
                        {"symbol": "AAPL", "pe": 30, "price": 150, "yearHigh": 180, "yearLow": 120}
                    ]
                else:  # /stable historical returns a flat OHLCV list
                    resp.json.return_value = [
                        {"symbol": "AAPL", "close": float(150 - i)} for i in range(20)
                    ]
            else:  # MSFT → FMP fails on both stable and v3
                resp.status_code = 500
            return resp

        mock_requests.get.side_effect = fake_get

        # yfinance fallback: download for MSFT
        mock_df = pd.DataFrame(
            {
                "Close": np.linspace(300, 400, 20),
                "High": np.linspace(305, 405, 20),
                "Low": np.linspace(295, 395, 20),
                "Volume": [1_000_000] * 20,
            }
        )
        mock_yf.download.return_value = mock_df
        mock_ticker = MagicMock()
        mock_ticker.info = {"trailingPE": 35.0}
        mock_yf.Ticker.return_value = mock_ticker

        results = scanner.batch_stock_metrics(["AAPL", "MSFT"])
        assert len(results) == 2
        # AAPL from FMP
        aapl = [r for r in results if r["symbol"] == "AAPL"][0]
        assert aapl["pe_ratio"] == 30
        # MSFT from yfinance fallback
        msft = [r for r in results if r["symbol"] == "MSFT"][0]
        assert msft["pe_ratio"] == 35.0
        # Stats show fallback occurred
        assert scanner.backend_stats()["yf_fallbacks"] >= 1

    @patch("etf_scanner._requests_lib")
    def test_all_fmp_success_no_yfinance_calls(self, mock_requests):
        """When FMP succeeds for all symbols, yfinance is not called."""
        scanner = ETFScanner(fmp_api_key="test_key", rate_limit_sec=0)

        quote_resp = MagicMock()
        quote_resp.status_code = 200
        quote_resp.json.return_value = [
            {"symbol": "AAPL", "pe": 30, "price": 150, "yearHigh": 180, "yearLow": 120},
        ]
        hist_resp = MagicMock()
        hist_resp.status_code = 200
        hist_resp.json.return_value = {
            "historicalStockList": [
                {"symbol": "AAPL", "historical": [{"close": float(150 - i)} for i in range(20)]},
            ]
        }
        mock_requests.get.side_effect = [quote_resp, hist_resp]

        with patch("etf_scanner.yf") as mock_yf:
            scanner.batch_stock_metrics(["AAPL"])
            mock_yf.download.assert_not_called()

        assert scanner.backend_stats()["yf_calls"] == 0

    @patch("etf_scanner.HAS_REQUESTS", False)
    @patch("etf_scanner.yf")
    def test_all_fmp_fail_falls_back_entirely(self, mock_yf):
        """Without requests library, falls back entirely to yfinance."""
        scanner = ETFScanner(fmp_api_key="test_key", rate_limit_sec=0)

        mock_df = pd.DataFrame(
            {
                "Close": np.linspace(100, 150, 20),
                "High": np.linspace(105, 155, 20),
                "Low": np.linspace(95, 145, 20),
                "Volume": [1_000_000] * 20,
            }
        )
        mock_yf.download.return_value = mock_df
        mock_ticker = MagicMock()
        mock_ticker.info = {"trailingPE": 25.0}
        mock_yf.Ticker.return_value = mock_ticker

        results = scanner.batch_stock_metrics(["AAPL"])
        assert len(results) == 1
        assert scanner.backend_stats()["yf_calls"] == 1


# ---------------------------------------------------------------------------
# TestBackendStats
# ---------------------------------------------------------------------------
class TestBackendStats:
    """Tests for backend_stats() tracking."""

    def test_initial_stats_all_zero(self):
        scanner = ETFScanner(fmp_api_key="test_key")
        stats = scanner.backend_stats()
        assert stats["fmp_calls"] == 0
        assert stats["fmp_failures"] == 0
        assert stats["yf_calls"] == 0
        assert stats["yf_fallbacks"] == 0

    @patch("etf_scanner._requests_lib")
    def test_stats_after_fmp_success(self, mock_requests):
        """FMP calls are counted after successful requests."""
        scanner = ETFScanner(fmp_api_key="test_key", rate_limit_sec=0)

        mock_resp = MagicMock()
        mock_resp.status_code = 200
        mock_resp.json.return_value = [
            {"symbol": "AAPL", "pe": 30, "price": 150, "yearHigh": 180, "yearLow": 120},
        ]
        mock_requests.get.return_value = mock_resp

        scanner._fmp_request("quote", "AAPL")
        stats = scanner.backend_stats()
        assert stats["fmp_calls"] == 1
        assert stats["fmp_failures"] == 0

    @patch("etf_scanner._requests_lib")
    @patch("etf_scanner.yf")
    def test_stats_after_yfinance_fallback(self, mock_yf, mock_requests):
        """yf_calls and yf_fallbacks are counted after fallback."""
        scanner = ETFScanner(fmp_api_key="test_key", rate_limit_sec=0)

        # FMP fails for everything
        fail_resp = MagicMock()
        fail_resp.status_code = 500
        mock_requests.get.return_value = fail_resp

        # yfinance works
        mock_df = pd.DataFrame(
            {
                "Close": np.linspace(100, 150, 20),
                "High": np.linspace(105, 155, 20),
                "Low": np.linspace(95, 145, 20),
                "Volume": [1_000_000] * 20,
            }
        )
        mock_yf.download.return_value = mock_df
        mock_ticker = MagicMock()
        mock_ticker.info = {"trailingPE": 25.0}
        mock_yf.Ticker.return_value = mock_ticker

        scanner.batch_stock_metrics(["AAPL"])
        stats = scanner.backend_stats()
        assert stats["yf_calls"] == 1
        assert stats["yf_fallbacks"] == 1  # FMP attempted but failed entirely
        assert stats["fmp_failures"] > 0

    @patch("etf_scanner._requests_lib")
    @patch("etf_scanner.yf")
    def test_yf_calls_and_yf_fallbacks_increment(self, mock_yf, mock_requests):
        """yf_fallbacks increments when partial FMP data triggers fallback."""
        scanner = ETFScanner(fmp_api_key="test_key", rate_limit_sec=0)

        # FMP: AAPL succeeds, MSFT missing
        quote_resp = MagicMock()
        quote_resp.status_code = 200
        quote_resp.json.return_value = [
            {"symbol": "AAPL", "pe": 30, "price": 150, "yearHigh": 180, "yearLow": 120},
        ]
        hist_resp = MagicMock()
        hist_resp.status_code = 200
        hist_resp.json.return_value = {
            "historicalStockList": [
                {"symbol": "AAPL", "historical": [{"close": float(150 - i)} for i in range(20)]},
            ]
        }
        mock_requests.get.side_effect = [
            quote_resp,
            hist_resp,
            # Per-symbol retry for MSFT fails
            MagicMock(status_code=500),
            MagicMock(status_code=500),
        ] + [MagicMock(status_code=503)] * 20

        mock_df = pd.DataFrame(
            {
                "Close": np.linspace(300, 400, 20),
                "High": np.linspace(305, 405, 20),
                "Low": np.linspace(295, 395, 20),
                "Volume": [1_000_000] * 20,
            }
        )
        mock_yf.download.return_value = mock_df
        mock_ticker = MagicMock()
        mock_ticker.info = {"trailingPE": 35.0}
        mock_yf.Ticker.return_value = mock_ticker

        scanner.batch_stock_metrics(["AAPL", "MSFT"])
        stats = scanner.backend_stats()
        assert stats["yf_fallbacks"] == 1
        assert stats["yf_calls"] == 1


@pytest.mark.parametrize("failure", [requests.Timeout("secret-url"), ValueError("secret-json")])
def test_fmp_expected_failure_warns_and_retains_fallback(monkeypatch, capsys, failure):
    import etf_scanner

    scanner = ETFScanner(fmp_api_key="test_key", rate_limit_sec=0)
    response = MagicMock(status_code=200)
    response.json.return_value = [{"symbol": "AAPL"}]
    get = MagicMock(side_effect=[failure, response])
    monkeypatch.setattr(etf_scanner._requests_lib, "get", get)
    assert scanner._fmp_request("quote", "AAPL") == [{"symbol": "AAPL"}]
    assert scanner.backend_stats()["fmp_failures"] == 1
    err = capsys.readouterr().err
    assert "degraded" in err.lower()
    assert "secret" not in err and "test_key" not in err


def test_fmp_programming_error_propagates(monkeypatch):
    import etf_scanner

    monkeypatch.setattr(etf_scanner._requests_lib, "get", MagicMock(side_effect=AssertionError))
    with pytest.raises(AssertionError):
        ETFScanner(fmp_api_key="test_key", rate_limit_sec=0)._fmp_request("quote", "AAPL")


@pytest.mark.parametrize(
    "failure", [requests.ConnectionError("secret-pe"), requests.Timeout("secret-url")]
)
def test_pe_expected_failure_warns(monkeypatch, capsys, failure):
    import etf_scanner

    monkeypatch.setattr(etf_scanner.yf, "Ticker", MagicMock(side_effect=failure))
    assert ETFScanner(rate_limit_sec=0)._get_pe_ratio("FAKE") is None
    err = capsys.readouterr().err
    assert "degraded" in err.lower() and "secret" not in err


def test_pe_programming_error_propagates(monkeypatch):
    import etf_scanner

    monkeypatch.setattr(etf_scanner.yf, "Ticker", MagicMock(side_effect=AssertionError))
    with pytest.raises(AssertionError):
        ETFScanner(rate_limit_sec=0)._get_pe_ratio("FAKE")


@pytest.mark.parametrize("value", ["invalid", {}, []])
def test_invalid_pe_conversion_warns(monkeypatch, capsys, value):
    import etf_scanner

    monkeypatch.setattr(
        etf_scanner.yf, "Ticker", MagicMock(return_value=MagicMock(info={"trailingPE": value}))
    )
    assert ETFScanner(rate_limit_sec=0)._get_pe_ratio("FAKE") is None
    assert "Degraded" in capsys.readouterr().err


@pytest.mark.parametrize("family", ["curl", "yfinance", "json"])
def test_yfinance_provider_exception_families(monkeypatch, capsys, family):
    import etf_scanner
    from curl_cffi.requests.exceptions import RequestException
    from yfinance.exceptions import YFException

    failure = {
        "curl": RequestException("secret-transport"),
        "yfinance": YFException("secret-provider"),
        "json": JSONDecodeError("secret-response", "secret-body", 0),
    }[family]
    monkeypatch.setattr(etf_scanner.yf, "Ticker", MagicMock(side_effect=failure))
    assert ETFScanner(rate_limit_sec=0)._get_pe_ratio("FAKE") is None
    err = capsys.readouterr().err
    assert "Degraded" in err and "secret" not in err


def test_curl_cffi_absence_preserves_requests_fallback(monkeypatch, capsys):
    """Optional curl-cffi absence must not prevent offline/FMP use."""
    import builtins
    import importlib.util
    from pathlib import Path

    real_import = builtins.__import__

    def without_curl(name, *args, **kwargs):
        if name.startswith("curl_cffi"):
            raise ImportError("fictional absent optional transport")
        return real_import(name, *args, **kwargs)

    monkeypatch.setattr(builtins, "__import__", without_curl)
    spec = importlib.util.spec_from_file_location(
        "scanner_without_curl", Path(__file__).parents[1] / "etf_scanner.py"
    )
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)
    assert module.HAS_REQUESTS
    assert not any(error.__module__.startswith("curl_cffi") for error in module._YF_ERRORS)
    monkeypatch.setattr(module.yf, "Ticker", MagicMock(side_effect=requests.Timeout("private")))
    assert module.ETFScanner(rate_limit_sec=0)._get_pe_ratio("FAKE") is None
    assert "Degraded" in capsys.readouterr().err


@pytest.mark.parametrize("missing", ["yfinance", "yfinance.exceptions"])
def test_optional_yfinance_imports_preserve_fmp(monkeypatch, missing):
    import builtins
    import importlib.util
    from pathlib import Path

    original = builtins.__import__

    def without_optional(name, *args, **kwargs):
        if name == missing:
            raise ImportError("fictional optional absence")
        return original(name, *args, **kwargs)

    monkeypatch.setattr(builtins, "__import__", without_optional)
    spec = importlib.util.spec_from_file_location(
        "scanner_without_yf", Path(__file__).parents[1] / "etf_scanner.py"
    )
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)
    assert module.HAS_REQUESTS
    assert module.HAS_YFINANCE is (missing != "yfinance")
    response = MagicMock(status_code=200)
    response.json.return_value = [{"symbol": "FAKE"}]
    monkeypatch.setattr(module._requests_lib, "get", MagicMock(return_value=response))
    assert module.ETFScanner(fmp_api_key="test_key", rate_limit_sec=0)._fmp_request(
        "quote", "FAKE"
    ) == [{"symbol": "FAKE"}]

"""Tests for finviz_performance_client outlier winsorization."""

from finviz_performance_client import HARD_CAPS, _apply_hard_caps, cap_outlier_performances


class TestCapOutlierPerformances:
    def test_no_outliers(self):
        """Normal data should pass through unchanged."""
        data = [
            {"name": "A", "perf_1w": 2.0, "perf_1m": 5.0, "perf_3m": 10.0, "perf_6m": 15.0},
            {"name": "B", "perf_1w": -1.0, "perf_1m": 3.0, "perf_3m": 8.0, "perf_6m": 12.0},
        ]
        result = cap_outlier_performances(data)
        assert result[0]["perf_1w"] == 2.0
        assert result[1]["perf_1w"] == -1.0

    def test_outlier_capped(self):
        """Extreme outlier should be capped to z_threshold boundary."""
        data = [
            {"name": f"I{i}", "perf_1w": 2.0, "perf_1m": 5.0, "perf_3m": 10.0, "perf_6m": 15.0}
            for i in range(20)
        ]
        # Add extreme outlier
        data.append(
            {"name": "Outlier", "perf_1w": 99.0, "perf_1m": 5.0, "perf_3m": 10.0, "perf_6m": 15.0}
        )
        result = cap_outlier_performances(data)
        outlier = next(r for r in result if r["name"] == "Outlier")
        # Should be capped and original preserved
        assert outlier["perf_1w"] < 99.0
        assert outlier.get("raw_perf_1w") == 99.0

    def test_raw_fields_preserved(self):
        """When an outlier is capped, raw_perf_* should store original value."""
        data = [
            {"name": f"I{i}", "perf_1w": 2.0, "perf_1m": 5.0, "perf_3m": 10.0, "perf_6m": 15.0}
            for i in range(20)
        ]
        data.append(
            {"name": "Extreme", "perf_1w": -99.0, "perf_1m": 5.0, "perf_3m": 10.0, "perf_6m": 15.0}
        )
        result = cap_outlier_performances(data)
        extreme = next(r for r in result if r["name"] == "Extreme")
        assert "raw_perf_1w" in extreme
        assert extreme["raw_perf_1w"] == -99.0

    def test_none_values_skipped(self):
        """None performance values should not cause errors."""
        data = [
            {"name": "A", "perf_1w": 2.0, "perf_1m": None, "perf_3m": 10.0, "perf_6m": 15.0},
            {"name": "B", "perf_1w": None, "perf_1m": 3.0, "perf_3m": 8.0, "perf_6m": 12.0},
        ]
        result = cap_outlier_performances(data)
        assert result[0]["perf_1m"] is None
        assert result[1]["perf_1w"] is None

    def test_empty_input(self):
        assert cap_outlier_performances([]) == []

    def test_small_dataset_skipped(self):
        """With fewer than 5 entries, z-score winsorization should not apply
        but hard caps still apply."""
        data = [
            {"name": "A", "perf_1w": 99.0, "perf_1m": 5.0, "perf_3m": 10.0, "perf_6m": 15.0},
        ]
        result = cap_outlier_performances(data)
        # Hard cap clips to 30.0 even for small datasets
        assert result[0]["perf_1w"] == HARD_CAPS["perf_1w"]


class TestApplyHardCaps:
    def test_perf_1w_capped_positive(self):
        """perf_1w exceeding +30% should be capped to +30%."""
        data = [{"name": "A", "perf_1w": 87.0}]
        _apply_hard_caps(data)
        assert data[0]["perf_1w"] == 30.0
        assert data[0]["raw_perf_1w"] == 87.0

    def test_perf_1w_capped_negative(self):
        """perf_1w below -30% should be capped to -30%."""
        data = [{"name": "A", "perf_1w": -100.0}]
        _apply_hard_caps(data)
        assert data[0]["perf_1w"] == -30.0
        assert data[0]["raw_perf_1w"] == -100.0

    def test_within_cap_unchanged(self):
        """Values within hard cap range should not be modified."""
        data = [{"name": "A", "perf_1w": 15.0, "perf_1m": -40.0, "perf_3m": 50.0}]
        _apply_hard_caps(data)
        assert data[0]["perf_1w"] == 15.0
        assert data[0]["perf_1m"] == -40.0
        assert data[0]["perf_3m"] == 50.0
        assert "raw_perf_1w" not in data[0]
        assert "raw_perf_1m" not in data[0]
        assert "raw_perf_3m" not in data[0]

    def test_raw_not_overwritten_by_second_stage(self):
        """raw_perf_* set by hard cap should not be overwritten by z-score stage."""
        data = [{"name": f"I{i}", "perf_1w": 5.0} for i in range(20)]
        # Add one that triggers hard cap
        data.append({"name": "Extreme", "perf_1w": 50.0})
        cap_outlier_performances(data)
        extreme = next(r for r in data if r["name"] == "Extreme")
        # raw_perf_1w should store original 50.0 (from hard cap), not 30.0
        assert extreme["raw_perf_1w"] == 50.0

    def test_none_values_skipped(self):
        """None perf values should be skipped without error."""
        data = [{"name": "A", "perf_1w": None, "perf_1m": 70.0}]
        _apply_hard_caps(data)
        assert data[0]["perf_1w"] is None
        assert data[0]["perf_1m"] == 60.0

    def test_all_perf_keys_capped(self):
        """All perf_* keys should be capped at their respective limits."""
        data = [
            {
                "name": "A",
                "perf_1w": 999.0,
                "perf_1m": 999.0,
                "perf_3m": 999.0,
                "perf_6m": 999.0,
                "perf_1y": 999.0,
                "perf_ytd": 999.0,
            }
        ]
        _apply_hard_caps(data)
        for key, cap in HARD_CAPS.items():
            assert data[0][key] == cap

"""FMP /stable migration for theme-detector.

Keys issued after 2025-08-31 lose v3 access (403). Fixes:
- Quotes/history are fetched one symbol per request (FMP /stable does not
  support comma-batched symbols — that silently returns []).
- Historical uses /stable/historical-price-eod/full (the prior stable URL,
  historical-price-full, 404s) with timeseries->from/to and flat-list
  normalization back to the v3 {"symbol","historical"} shape.
- ETF holdings: /api/v3/etf-holder/{sym} -> /stable/etf/holdings?symbol=
  (+ v3 fallback); both expose `asset` and `marketValue`.
"""

from unittest.mock import MagicMock, patch

import representative_stock_selector as rss
from etf_scanner import ETFScanner, _normalize_eod_flat_list, _stable_hist_url


class TestHistoricalNormalize:
    def test_flat_list_to_v3_dict(self):
        flat = [
            {"symbol": "AAPL", "date": "2026-05-19", "close": 298.97},
            {"symbol": "AAPL", "date": "2026-05-18", "close": 296.0},
        ]
        out = _normalize_eod_flat_list(flat, "AAPL")
        assert out["symbol"] == "AAPL"
        assert len(out["historical"]) == 2

    def test_no_match_returns_none(self):
        assert _normalize_eod_flat_list([{"symbol": "MSFT", "close": 1}], "AAPL") is None

    def test_dict_passthrough(self):
        d = {"symbol": "AAPL", "historical": [{"close": 1}]}
        assert _normalize_eod_flat_list(d, "AAPL") is d

    def test_stable_hist_url_converts_timeseries(self):
        base = "https://financialmodelingprep.com/stable/historical-price-eod/full"
        url, params = _stable_hist_url(base, "AAPL", {"timeseries": 20})
        assert url == base
        assert params["symbol"] == "AAPL"
        assert "from" in params and "to" in params
        assert "timeseries" not in params  # converted to a from/to range


class TestDeBatch:
    def test_batch_sizes_are_one(self):
        assert ETFScanner.FMP_QUOTE_BATCH_SIZE == 1
        assert ETFScanner.FMP_HIST_BATCH_SIZE == 1

    def test_quotes_fetched_one_symbol_per_request(self):
        scanner = ETFScanner(fmp_api_key="k")
        seen = []

        def fake_req(endpoint_key, symbols_str, extra_params=None):
            seen.append((endpoint_key, symbols_str))
            return [{"symbol": symbols_str, "price": 100.0}]

        scanner._fmp_request = fake_req
        scanner._fetch_fmp_quotes(["AAPL", "MSFT", "NVDA"])
        quote_syms = [s for e, s in seen if e == "quote"]
        assert all("," not in s for s in quote_syms)  # never comma-batched
        assert set(quote_syms) >= {"AAPL", "MSFT", "NVDA"}


class TestEtfHoldings:
    def _resp(self, status, payload):
        r = MagicMock()
        r.status_code = status
        r.json.return_value = payload
        return r

    def test_stable_etf_holdings_first(self):
        sel = rss.RepresentativeStockSelector(fmp_api_key="k")
        sel._rate_limit = lambda: None
        with patch.object(rss, "requests") as mock_requests:
            mock_requests.get.return_value = self._resp(
                200, [{"asset": "NVDA", "marketValue": 1_000_000_000}]
            )
            holds = sel._fetch_etf_holdings("SPY", limit=5)
        assert holds[0]["symbol"] == "NVDA"
        assert mock_requests.get.call_args[0][0].endswith("/stable/etf/holdings")
        assert mock_requests.get.call_args[1]["params"] == {"symbol": "SPY"}

    def test_falls_back_to_v3_etf_holder(self):
        sel = rss.RepresentativeStockSelector(fmp_api_key="k")
        sel._rate_limit = lambda: None

        def fake_get(url, params=None, headers=None, timeout=None):
            if "/stable/etf/holdings" in url:
                return self._resp(403, {})
            return self._resp(200, [{"asset": "AAPL", "marketValue": 5}])

        with patch.object(rss, "requests") as mock_requests:
            mock_requests.get.side_effect = fake_get
            holds = sel._fetch_etf_holdings("XLK", limit=5)
        assert holds[0]["symbol"] == "AAPL"
        urls = [c[0][0] for c in mock_requests.get.call_args_list]
        assert any(u.endswith("/api/v3/etf-holder/XLK") for u in urls)

"""Tests for heat_calculator.py - Theme Heat Score (0-100)"""

import pytest
from calculators.heat_calculator import (
    breadth_signal_score,
    calculate_theme_heat,
    calculate_theme_heat_detailed,
    momentum_strength_score,
    uptrend_signal_score,
    volume_intensity_score,
)

# ── momentum_strength_score ──────────────────────────────────────────


class TestMomentumStrengthScore:
    """sigmoid: 100 / (1 + exp(-0.15 * (abs(wr%) - 5.0)))"""

    def test_zero_return(self):
        score = momentum_strength_score(0.0)
        assert 0 <= score <= 1

    def test_five_percent(self):
        score = momentum_strength_score(5.0)
        assert score == pytest.approx(12.33, abs=0.1)

    def test_negative_five_percent(self):
        score = momentum_strength_score(-5.0)
        assert score == pytest.approx(12.33, abs=0.1)

    def test_fifteen_percent(self):
        score = momentum_strength_score(15.0)
        assert score == pytest.approx(50.0, abs=0.1)

    def test_twenty_percent(self):
        score = momentum_strength_score(20.0)
        assert 62 <= score <= 65

    def test_negative_twenty_percent(self):
        score = momentum_strength_score(-20.0)
        assert 62 <= score <= 65

    def test_returns_float(self):
        assert isinstance(momentum_strength_score(3.0), float)


# ── volume_intensity_score ───────────────────────────────────────────


class TestVolumeIntensityScore:
    """min(100, max(0, (vol_20d/vol_60d - 0.8) * 250))"""

    def test_ratio_0_8_returns_zero(self):
        # (0.8 - 0.8) * 250 = 0
        assert volume_intensity_score(80.0, 100.0) == pytest.approx(0.0)

    def test_ratio_1_0_returns_50(self):
        assert volume_intensity_score(100.0, 100.0) == pytest.approx(40.8248)

    def test_ratio_1_2_returns_100(self):
        assert volume_intensity_score(120.0, 100.0) == pytest.approx(57.7350)

    def test_ratio_above_cap_clamped_to_100(self):
        # (2.0 - 0.8) * 250 = 300 => clamped 100
        assert volume_intensity_score(200.0, 100.0) == pytest.approx(100.0)

    def test_ratio_below_floor_clamped_to_0(self):
        # (0.5 - 0.8) * 250 = -75 => clamped 0
        assert volume_intensity_score(50.0, 100.0) == pytest.approx(0.0)

    def test_none_vol_20d(self):
        assert volume_intensity_score(None, 100.0) is None

    def test_none_vol_60d(self):
        assert volume_intensity_score(100.0, None) is None

    def test_zero_vol_60d(self):
        assert volume_intensity_score(100.0, 0.0) is None


# ── uptrend_signal_score ─────────────────────────────────────────────


class TestUptrendSignalScore:
    def _make_sector(self, ratio, ma_10, slope, weight=1.0):
        return {
            "sector": "test",
            "ratio": ratio,
            "ma_10": ma_10,
            "slope": slope,
            "weight": weight,
        }

    def test_both_positive_gives_80(self):
        # ratio>ma_10 AND slope>0 => 80
        data = [self._make_sector(ratio=50, ma_10=40, slope=0.5)]
        score = uptrend_signal_score(data, is_bearish=False)
        assert score == pytest.approx(100.0)

    def test_ratio_only_gives_60(self):
        # ratio>ma_10 but slope<=0 => 60
        data = [self._make_sector(ratio=50, ma_10=40, slope=-0.1)]
        score = uptrend_signal_score(data, is_bearish=False)
        assert score == pytest.approx(90.0)

    def test_slope_only_gives_60(self):
        # ratio<=ma_10 but slope>0 => 60
        data = [self._make_sector(ratio=30, ma_10=40, slope=0.5)]
        score = uptrend_signal_score(data, is_bearish=False)
        assert score == pytest.approx(90.0)

    def test_neither_gives_20(self):
        # ratio<=ma_10 AND slope<=0 => 20
        data = [self._make_sector(ratio=30, ma_10=40, slope=-0.1)]
        score = uptrend_signal_score(data, is_bearish=False)
        assert score == pytest.approx(80.0)

    def test_weighted_average(self):
        # sector A: both positive => 80, weight 2
        # sector B: neither => 20, weight 1
        # weighted = (80*2 + 20*1) / 3 = 60
        data = [
            self._make_sector(ratio=50, ma_10=40, slope=0.5, weight=2.0),
            self._make_sector(ratio=30, ma_10=40, slope=-0.1, weight=1.0),
        ]
        score = uptrend_signal_score(data, is_bearish=False)
        assert score == pytest.approx(93.33333333333333)

    def test_bearish_inversion(self):
        # both positive => 80, bearish => 100-80 = 20
        data = [self._make_sector(ratio=50, ma_10=40, slope=0.5)]
        score = uptrend_signal_score(data, is_bearish=True)
        assert score == pytest.approx(0.0)

    def test_empty_list(self):
        assert uptrend_signal_score([], is_bearish=False) is None

    def test_equal_ratio_and_ma10(self):
        # ratio == ma_10 => not > => slope<=0 => 20
        data = [self._make_sector(ratio=40, ma_10=40, slope=0)]
        score = uptrend_signal_score(data, is_bearish=False)
        assert score == pytest.approx(80.0)


# ── breadth_signal_score ─────────────────────────────────────────────


class TestBreadthSignalScore:
    def test_zero(self):
        assert breadth_signal_score(0.0) == pytest.approx(0.0)

    def test_half(self):
        assert breadth_signal_score(0.5) == pytest.approx(14.142135623730951)

    def test_full(self):
        assert breadth_signal_score(1.0) == pytest.approx(80.0)

    def test_above_one_clamped(self):
        assert breadth_signal_score(1.5) == pytest.approx(100.0)

    def test_negative_clamped(self):
        assert breadth_signal_score(-0.3) == pytest.approx(0.0)

    def test_none(self):
        assert breadth_signal_score(None) is None


# ── calculate_theme_heat ─────────────────────────────────────────────


class TestCalculateThemeHeat:
    def test_weighted_sum(self):
        result = calculate_theme_heat(80.0, 60.0, 70.0, 50.0)
        assert result == pytest.approx(67.5)

    def test_all_100(self):
        result = calculate_theme_heat(100.0, 100.0, 100.0, 100.0)
        assert result == pytest.approx(100.0)

    def test_all_zero(self):
        result = calculate_theme_heat(0.0, 0.0, 0.0, 0.0)
        assert result == pytest.approx(0.0)

    def test_all_none_has_zero_score_and_zero_coverage(self):
        result = calculate_theme_heat(None, None, None, None)
        assert result == pytest.approx(0.0)
        detailed = calculate_theme_heat_detailed(None, None, None, None)
        assert detailed["score"] is None
        assert detailed["coverage"] == pytest.approx(0.0)
        assert set(detailed["missing_components"]) == {
            "momentum_strength",
            "volume_intensity",
            "uptrend_signal",
            "breadth_signal",
        }

    def test_partial_none(self):
        result = calculate_theme_heat(80.0, None, None, None)
        assert result == pytest.approx(80.0)
        detailed = calculate_theme_heat_detailed(80.0, None, None, None)
        assert detailed["coverage"] == pytest.approx(0.35)
        assert detailed["missing_components"] == [
            "volume_intensity",
            "uptrend_signal",
            "breadth_signal",
        ]

    def test_clamped_above_100(self):
        result = calculate_theme_heat(200.0, 200.0, 200.0, 200.0)
        assert result == 100.0

    def test_clamped_below_0(self):
        result = calculate_theme_heat(-50.0, -50.0, -50.0, -50.0)
        assert result == 0.0

    def test_returns_float(self):
        assert isinstance(calculate_theme_heat(50, 50, 50, 50), float)


# ── uptrend_signal_score with None values ────────────────────────────


class TestUptrendSignalNoneValues:
    """Ensure None values in sector_data don't cause TypeError."""

    def test_none_ma_10(self):
        """ma_10=None should not crash (treated as 0)."""
        data = [{"sector": "Tech", "ratio": 0.5, "ma_10": None, "slope": 0.01, "weight": 1.0}]
        score = uptrend_signal_score(data, is_bearish=False)
        assert score == pytest.approx(70.0)

    def test_none_slope(self):
        """slope=None should not crash (treated as 0)."""
        data = [{"sector": "Tech", "ratio": 0.5, "ma_10": 0.3, "slope": None, "weight": 1.0}]
        score = uptrend_signal_score(data, is_bearish=False)
        # ratio(0.5) > ma_10(0.3) but slope(0) not > 0 => 60
        assert score == pytest.approx(60.0)

    def test_none_ratio(self):
        """ratio=None should not crash (treated as 0)."""
        data = [{"sector": "Tech", "ratio": None, "ma_10": 0.3, "slope": 0.01, "weight": 1.0}]
        score = uptrend_signal_score(data, is_bearish=False)
        assert score == pytest.approx(10.0)

    def test_all_none(self):
        """All values None should not crash."""
        data = [{"sector": "Tech", "ratio": None, "ma_10": None, "slope": None, "weight": 1.0}]
        score = uptrend_signal_score(data, is_bearish=False)
        assert score == pytest.approx(0.0)

#!/usr/bin/env python3
"""
Tests for industry_ranker module.

Covers momentum_strength_score sigmoid, weighted return calculation,
ranking, direction classification, and top/bottom extraction.
"""

from calculators.industry_ranker import (
    TIMEFRAME_WEIGHTS,
    get_top_bottom_industries,
    momentum_strength_score,
    rank_industries,
)

# ---------------------------------------------------------------------------
# momentum_strength_score tests
# ---------------------------------------------------------------------------


class TestMomentumStrengthScore:
    """Test the direction-neutral sigmoid scoring function."""

    def test_zero_return_gives_about_32(self):
        """abs(0%) -> ~32 (sigmoid at -5.0 midpoint)."""
        score = momentum_strength_score(0.0)
        assert 30 <= score <= 34, f"Expected ~32, got {score}"

    def test_five_pct_gives_50(self):
        """abs(5%) is the midpoint -> exactly 50."""
        score = momentum_strength_score(5.0)
        assert abs(score - 50.0) < 0.1, f"Expected 50, got {score}"

    def test_ten_pct_gives_about_68(self):
        """abs(10%) -> ~68."""
        score = momentum_strength_score(10.0)
        assert 66 <= score <= 70, f"Expected ~68, got {score}"

    def test_fifteen_pct_gives_about_82(self):
        """abs(15%) -> ~82."""
        score = momentum_strength_score(15.0)
        assert 80 <= score <= 84, f"Expected ~82, got {score}"

    def test_twenty_pct_gives_about_90(self):
        """abs(20%) -> ~90."""
        score = momentum_strength_score(20.0)
        assert 88 <= score <= 92, f"Expected ~90, got {score}"

    def test_negative_return_same_as_positive(self):
        """Negative and positive returns with same magnitude give same score."""
        pos = momentum_strength_score(12.0)
        neg = momentum_strength_score(-12.0)
        assert abs(pos - neg) < 0.01

    def test_large_return_caps_near_100(self):
        """Very large returns approach but don't exceed 100."""
        score = momentum_strength_score(50.0)
        assert 99 <= score <= 100

    def test_score_always_between_0_and_100(self):
        """Score is bounded [0, 100] for any input."""
        for ret in [-100, -50, -10, 0, 10, 50, 100]:
            score = momentum_strength_score(ret)
            assert 0 <= score <= 100, f"Out of bounds for {ret}: {score}"

    def test_monotonically_increasing_with_abs(self):
        """Higher absolute return -> higher score."""
        s1 = momentum_strength_score(5.0)
        s2 = momentum_strength_score(10.0)
        s3 = momentum_strength_score(20.0)
        assert s1 < s2 < s3


# ---------------------------------------------------------------------------
# TIMEFRAME_WEIGHTS validation
# ---------------------------------------------------------------------------


class TestTimeframeWeights:
    """Verify timeframe weights are correct and sum to 1.0."""

    def test_weights_sum_to_one(self):
        total = sum(TIMEFRAME_WEIGHTS.values())
        assert abs(total - 1.0) < 0.001

    def test_expected_keys(self):
        expected = {"perf_1w", "perf_1m", "perf_3m", "perf_6m"}
        assert set(TIMEFRAME_WEIGHTS.keys()) == expected

    def test_expected_values(self):
        assert TIMEFRAME_WEIGHTS["perf_1w"] == 0.10
        assert TIMEFRAME_WEIGHTS["perf_1m"] == 0.25
        assert TIMEFRAME_WEIGHTS["perf_3m"] == 0.35
        assert TIMEFRAME_WEIGHTS["perf_6m"] == 0.30


# ---------------------------------------------------------------------------
# rank_industries tests
# ---------------------------------------------------------------------------


def _make_industry(name, perf_1w=0.0, perf_1m=0.0, perf_3m=0.0, perf_6m=0.0):
    """Helper to build a synthetic industry dict."""
    return {
        "name": name,
        "perf_1w": perf_1w,
        "perf_1m": perf_1m,
        "perf_3m": perf_3m,
        "perf_6m": perf_6m,
    }


class TestRankIndustries:
    """Test rank_industries function."""

    def test_adds_required_fields(self):
        """Each result dict gets ranking and direction fields."""
        industries = [_make_industry("Tech", 5, 10, 15, 20)]
        ranked = rank_industries(industries)
        assert len(ranked) == 1
        item = ranked[0]
        assert "momentum_score" in item
        assert "weighted_return" in item
        assert "direction" in item
        assert "rank" in item
        assert "strength_rank" in item
        assert "signed_rank" in item
        assert "rank_direction" in item

    def test_weighted_return_calculation(self):
        """Verify weighted return = sum(perf_x * weight_x)."""
        ind = _make_industry("Test", perf_1w=10.0, perf_1m=20.0, perf_3m=30.0, perf_6m=40.0)
        ranked = rank_industries([ind])
        expected = 10 * 0.10 + 20 * 0.25 + 30 * 0.35 + 40 * 0.30
        assert abs(ranked[0]["weighted_return"] - expected) < 0.01

    def test_sorted_by_momentum_score_desc(self):
        """Industries are sorted by momentum_score descending."""
        industries = [
            _make_industry("Low", perf_1w=1, perf_1m=1, perf_3m=1, perf_6m=1),
            _make_industry("High", perf_1w=20, perf_1m=30, perf_3m=40, perf_6m=50),
            _make_industry("Mid", perf_1w=5, perf_1m=10, perf_3m=10, perf_6m=10),
        ]
        ranked = rank_industries(industries)
        assert ranked[0]["name"] == "High"
        assert ranked[-1]["name"] == "Low"
        # Verify descending order
        scores = [r["momentum_score"] for r in ranked]
        assert scores == sorted(scores, reverse=True)

    def test_rank_starts_at_one(self):
        """Rank should be 1-based."""
        industries = [
            _make_industry("A", perf_3m=10),
            _make_industry("B", perf_3m=20),
        ]
        ranked = rank_industries(industries)
        assert ranked[0]["rank"] == 1
        assert ranked[1]["rank"] == 2

    def test_bullish_direction_for_positive_return(self):
        """Positive weighted return -> direction='bullish'."""
        ranked = rank_industries([_make_industry("Up", perf_3m=15, perf_6m=10)])
        assert ranked[0]["direction"] == "bullish"

    def test_bearish_direction_for_negative_return(self):
        """Negative weighted return -> direction='bearish'."""
        ranked = rank_industries([_make_industry("Down", perf_3m=-15, perf_6m=-10)])
        assert ranked[0]["direction"] == "bearish"

    def test_zero_return_direction(self):
        """Zero weighted return -> direction='bearish' (non-positive)."""
        ranked = rank_industries([_make_industry("Flat")])
        assert ranked[0]["direction"] == "bearish"

    def test_empty_list_returns_empty(self):
        """Empty input returns empty list."""
        assert rank_industries([]) == []

    def test_preserves_original_fields(self):
        """Original industry dict fields are preserved in output."""
        ind = _make_industry("Test", perf_1w=5, perf_1m=10, perf_3m=15, perf_6m=20)
        ind["sector"] = "Technology"  # extra field
        ranked = rank_industries([ind])
        assert ranked[0]["name"] == "Test"
        assert ranked[0]["perf_1w"] == 5
        assert ranked[0]["sector"] == "Technology"

    def test_mixed_bullish_bearish(self):
        """Mix of positive and negative returns handled correctly."""
        industries = [
            _make_industry("Bull", perf_1w=10, perf_1m=15, perf_3m=20, perf_6m=25),
            _make_industry("Bear", perf_1w=-10, perf_1m=-15, perf_3m=-20, perf_6m=-25),
        ]
        ranked = rank_industries(industries)
        # Both should have same momentum_score (direction-neutral)
        assert abs(ranked[0]["momentum_score"] - ranked[1]["momentum_score"]) < 0.01
        # But different directions
        directions = {r["name"]: r["direction"] for r in ranked}
        assert directions["Bull"] == "bullish"
        assert directions["Bear"] == "bearish"

    def test_severe_decliner_keeps_bearish_rank_direction(self):
        """Strong negative momentum can rank high by strength but not as bullish."""
        industries = [
            _make_industry("Crash", perf_1w=-30, perf_1m=-30, perf_3m=-30, perf_6m=-30),
            _make_industry("Leader", perf_1w=20, perf_1m=20, perf_3m=20, perf_6m=20),
            _make_industry("Mild", perf_1w=2, perf_1m=2, perf_3m=2, perf_6m=2),
            _make_industry("Flat"),
        ]
        ranked = rank_industries(industries)
        crash = next(item for item in ranked if item["name"] == "Crash")
        leader = next(item for item in ranked if item["name"] == "Leader")
        assert crash["strength_rank"] < leader["strength_rank"]
        assert crash["rank_direction"] == "bearish"
        assert leader["rank_direction"] == "bullish"


# ---------------------------------------------------------------------------
# get_top_bottom_industries tests
# ---------------------------------------------------------------------------


class TestGetTopBottomIndustries:
    """Test top/bottom extraction."""

    def test_top_and_bottom_n(self):
        """Returns correct top N and bottom N."""
        industries = [
            _make_industry(f"Ind{i}", perf_3m=20 - i * 2, perf_6m=15 - i) for i in range(10)
        ]
        ranked = rank_industries(industries)
        result = get_top_bottom_industries(ranked, n=3)
        assert len(result["top"]) == 3
        assert len(result["bottom"]) == 3
        # Top should have rank 1,2,3
        assert result["top"][0]["rank"] == 1
        assert result["top"][2]["rank"] == 3
        # Bottom should have rank 8,9,10
        assert result["bottom"][0]["rank"] == 8
        assert result["bottom"][2]["rank"] == 10

    def test_n_greater_than_half(self):
        """When n > len/2, top and bottom may overlap; should handle gracefully."""
        industries = [_make_industry(f"I{i}", perf_3m=i) for i in range(4)]
        ranked = rank_industries(industries)
        result = get_top_bottom_industries(ranked, n=3)
        assert len(result["top"]) == 3
        assert len(result["bottom"]) == 3

    def test_empty_ranked_list(self):
        """Empty input returns empty top and bottom."""
        result = get_top_bottom_industries([], n=5)
        assert result["top"] == []
        assert result["bottom"] == []

    def test_n_exceeds_list_length(self):
        """n > len(ranked) returns all available."""
        industries = [_make_industry(f"I{i}", perf_3m=i * 5) for i in range(3)]
        ranked = rank_industries(industries)
        result = get_top_bottom_industries(ranked, n=10)
        assert len(result["top"]) == 3
        assert len(result["bottom"]) == 3

    def test_default_n_is_5(self):
        """Default n should be 5."""
        industries = [_make_industry(f"I{i}", perf_3m=i * 3) for i in range(20)]
        ranked = rank_industries(industries)
        result = get_top_bottom_industries(ranked)
        assert len(result["top"]) == 5
        assert len(result["bottom"]) == 5

"""Tests for stock leadership scan-hit detection and aggregation."""

import json

from leadership import (
    ScanHit,
    aggregate_leadership,
    blend_theme_heat,
    calculate_leader_candidates,
    calculate_leadership_score,
    detect_scan_hits_from_row,
    load_scan_hits,
)


def test_raw_row_expands_to_multiple_scan_hits():
    row = {
        "date": "2026-07-04",
        "symbol": "NVDA",
        "return_5d": 24,
        "change_pct": 7,
        "volume": 12_000_000,
        "avg_volume_50d": 4_000_000,
        "true_range": 6,
        "atr_20": 3,
        "close_location": 0.9,
        "new_high": True,
        "rs_rating": 95,
        "industry": "Semiconductors",
    }
    hits = detect_scan_hits_from_row(row, "2026-07-04")
    scan_types = {hit.scan_type for hit in hits}
    assert scan_types == {
        "five_day_20pct",
        "ep9m",
        "range_expansion",
        "new_high",
        "high_rs",
    }
    assert all(hit.rs_rating == 95 for hit in hits)


def test_relative_strength_alias_normalized_for_high_rs():
    hits = detect_scan_hits_from_row(
        {"date": "2026-07-04", "symbol": "CRWD", "relative_strength": 0.96},
        "2026-07-04",
    )

    assert [hit.scan_type for hit in hits] == ["high_rs"]
    assert hits[0].rs_rating == 96.0


def test_load_scan_hits_accepts_prelabeled_json(tmp_path):
    path = tmp_path / "scan_hits.json"
    path.write_text(
        json.dumps(
            [
                {
                    "date": "2026-07-04",
                    "symbol": "CCJ",
                    "scan_type": "ep9m,range_expansion",
                    "theme_guess": "Nuclear & Uranium",
                }
            ]
        ),
        encoding="utf-8",
    )
    hits, summary = load_scan_hits(str(path), "2026-07-04")
    assert summary["rows"] == 1
    assert summary["hits"] == 2
    assert {hit.scan_type for hit in hits} == {"ep9m", "range_expansion"}


def test_load_scan_hits_filters_non_run_date_rows(tmp_path):
    path = tmp_path / "scan_hits.json"
    path.write_text(
        json.dumps(
            [
                {
                    "date": "2026-07-03",
                    "symbol": "NVDA",
                    "scan_type": "five_day_20pct",
                    "theme_guess": "AI & Semiconductors",
                },
                {
                    "date": "2026-07-04",
                    "symbol": "AVGO",
                    "scan_type": "ep9m",
                    "theme_guess": "AI & Semiconductors",
                },
                {
                    "symbol": "AMD",
                    "scan_type": "range_expansion",
                    "theme_guess": "AI & Semiconductors",
                },
            ]
        ),
        encoding="utf-8",
    )

    hits, summary = load_scan_hits(str(path), "2026-07-04")

    assert summary["rows"] == 3
    assert summary["skipped_date_rows"] == 1
    assert [hit.symbol for hit in hits] == ["AVGO", "AMD"]
    assert [hit.date for hit in hits] == ["2026-07-04", "2026-07-04"]


def test_detect_scan_hits_ignores_stale_row_date():
    hits = detect_scan_hits_from_row(
        {"date": "2026-07-03", "symbol": "NVDA", "return_5d": 25},
        "2026-07-04",
    )

    assert hits == []


def test_aggregate_leadership_scores_theme_hits():
    themes = [
        {
            "theme_name": "AI & Semiconductors",
            "matching_industries": [{"name": "Semiconductors"}],
            "static_stocks": ["NVDA"],
        }
    ]
    hits = detect_scan_hits_from_row(
        {
            "symbol": "NVDA",
            "return_5d": 25,
            "change_pct": 6,
            "volume": 10_000_000,
            "relative_volume": 2.5,
            "atr_expansion": 1.8,
            "close_location": 0.8,
            "industry": "Semiconductors",
        },
        "2026-07-04",
    )
    result = aggregate_leadership(themes, hits, history={})
    evidence = result["AI & Semiconductors"]
    assert evidence["leadership_score"] is not None
    assert evidence["leadership_coverage"] == 1.0
    assert evidence["leadership_counts"]["five_day_20pct"] == 1
    assert evidence["leadership_counts"]["ep9m"] == 1
    assert evidence["leadership_counts"]["range_expansion"] == 1
    assert evidence["leader_symbols"] == ["NVDA"]


def test_missing_leadership_does_not_depress_heat():
    assert blend_theme_heat(72.0, None) == 72.0
    assert blend_theme_heat(70.0, 100.0) == 79.0


def test_high_rs_only_hit_is_scored_not_punitive():
    themes = [
        {
            "theme_name": "AI & Semiconductors",
            "matching_industries": [{"name": "Semiconductors"}],
            "static_stocks": ["NVDA"],
        }
    ]
    hits = detect_scan_hits_from_row(
        {"symbol": "NVDA", "rs_rating": 95, "industry": "Semiconductors"},
        "2026-07-04",
    )
    result = aggregate_leadership(themes, hits, history={})
    score = result["AI & Semiconductors"]["leadership_score"]
    assert score is not None
    assert score == 7.5
    assert blend_theme_heat(70.0, score) >= 70.0


def test_leadership_weights_distinguish_sparse_evidence():
    high_rs_only = calculate_leadership_score(
        "AI & Semiconductors",
        {
            "five_day_20pct": 0,
            "ep9m": 0,
            "range_expansion": 0,
            "new_high": 0,
            "high_rs": 1,
        },
        history={},
    )
    five_day_only = calculate_leadership_score(
        "AI & Semiconductors",
        {
            "five_day_20pct": 1,
            "ep9m": 0,
            "range_expansion": 0,
            "new_high": 0,
            "high_rs": 0,
        },
        history={},
    )
    all_once = calculate_leadership_score(
        "AI & Semiconductors",
        {
            "five_day_20pct": 1,
            "ep9m": 1,
            "range_expansion": 1,
            "new_high": 1,
            "high_rs": 1,
        },
        history={},
    )

    assert high_rs_only == 7.5
    assert five_day_only == 22.5
    assert all_once == 75.0
    assert high_rs_only < five_day_only < all_once
    assert blend_theme_heat(70.0, high_rs_only) == 70.0
    assert blend_theme_heat(70.0, all_once) == 71.5


def test_leader_candidates_rank_abnormal_evidence_not_market_cap():
    hits = [
        ScanHit(
            date="2026-07-04",
            symbol="MEGA",
            scan_type="high_rs",
            relative_volume=1.1,
            return_5d=3.0,
            atr_expansion=1.0,
            close_location=0.55,
            rs_rating=82.0,
            dollar_volume=500_000_000,
            market_cap=500_000_000_000,
        ),
        ScanHit(
            date="2026-07-04",
            symbol="SMID",
            scan_type="ep9m",
            relative_volume=3.0,
            return_5d=18.0,
            atr_expansion=1.9,
            close_location=0.92,
            rs_rating=96.0,
            dollar_volume=30_000_000,
            market_cap=3_000_000_000,
        ),
    ]

    candidates = calculate_leader_candidates(hits)

    assert candidates[0]["symbol"] == "SMID"
    assert candidates[0]["risk_bucket"] == "mid"
    assert candidates[1]["symbol"] == "MEGA"
    assert candidates[1]["risk_bucket"] == "mega"
    assert candidates[0]["leader_score"] > candidates[1]["leader_score"]


def test_leader_candidates_do_not_overstate_sparse_high_rs_only():
    candidates = calculate_leader_candidates(
        [
            ScanHit(
                date="2026-07-04",
                symbol="RS",
                scan_type="high_rs",
                rs_rating=95.0,
            ),
            ScanHit(
                date="2026-07-04",
                symbol="FULL",
                scan_type="ep9m",
                relative_volume=3.0,
                return_5d=18.0,
                atr_expansion=1.8,
                close_location=0.9,
                rs_rating=95.0,
                dollar_volume=30_000_000,
            ),
        ]
    )

    by_symbol = {candidate["symbol"]: candidate for candidate in candidates}

    assert by_symbol["RS"]["leader_score"] == 9.5
    assert by_symbol["RS"]["leader_score_coverage"] == 0.1
    assert by_symbol["FULL"]["leader_score"] > by_symbol["RS"]["leader_score"]


def test_leadership_aggregation_exposes_fresh_and_extended_symbols():
    themes = [
        {
            "theme_name": "AI & Semiconductors",
            "matching_industries": [{"name": "Semiconductors"}],
            "static_stocks": ["NVDA"],
        }
    ]
    hits = [
        ScanHit(date="2026-07-04", symbol="NVDA", scan_type="ep9m", return_5d=10),
        ScanHit(date="2026-07-04", symbol="NVDA", scan_type="five_day_20pct", return_5d=22),
    ]

    evidence = aggregate_leadership(themes, hits, history={})["AI & Semiconductors"]

    assert evidence["fresh_leadership_symbols"] == ["NVDA"]
    assert evidence["extended_symbols"] == ["NVDA"]
    assert evidence["leader_candidates"][0]["is_extended"] is True

"""Tests for lifecycle_calculator.py - Lifecycle Maturity Score (0-100)"""

import pytest
from calculators.lifecycle_calculator import (
    calculate_lifecycle_maturity,
    calculate_lifecycle_maturity_detailed,
    classify_stage,
    estimate_duration_score,
    etf_proliferation_score,
    extremity_clustering_score,
    price_extreme_saturation_score,
    valuation_premium_score,
)

# ── estimate_duration_score ──────────────────────────────────────────


class TestEstimateDurationScore:
    def test_bullish_all_active(self):
        # All 4 horizons > 2% => 100
        score = estimate_duration_score(3.0, 5.0, 10.0, 20.0, is_bearish=False)
        assert score == pytest.approx(100.0)

    def test_bullish_none_active(self):
        # All < 2% => 0
        score = estimate_duration_score(0.5, 1.0, 1.5, 1.9, is_bearish=False)
        assert score == pytest.approx(0.0)

    def test_bullish_two_active(self):
        # 2 out of 4 => 50
        score = estimate_duration_score(3.0, 5.0, 1.0, 0.5, is_bearish=False)
        assert score == pytest.approx(50.0)

    def test_bearish_all_active(self):
        # All < -2% => 100
        score = estimate_duration_score(-3.0, -5.0, -10.0, -20.0, is_bearish=True)
        assert score == pytest.approx(100.0)

    def test_bearish_none_active(self):
        # None < -2%
        score = estimate_duration_score(-0.5, -1.0, 0.0, 1.0, is_bearish=True)
        assert score == pytest.approx(0.0)

    def test_bearish_mixed(self):
        # 3 out of 4 < -2% => 75
        score = estimate_duration_score(-3.0, -5.0, -10.0, 0.0, is_bearish=True)
        assert score == pytest.approx(75.0)

    def test_boundary_exactly_2_percent(self):
        # 2.0 is NOT > 2 => not active
        score = estimate_duration_score(2.0, 2.0, 2.0, 2.0, is_bearish=False)
        assert score == pytest.approx(0.0)

    def test_boundary_just_above(self):
        score = estimate_duration_score(2.1, 2.1, 2.1, 2.1, is_bearish=False)
        assert score == pytest.approx(100.0)

    def test_none_values_treated_as_inactive(self):
        score = estimate_duration_score(None, None, 5.0, 10.0, is_bearish=False)
        assert score == pytest.approx(50.0)


# ── extremity_clustering_score ───────────────────────────────────────


class TestExtremityClusteringScore:
    def _make_stocks(self, rsi_values):
        return [{"rsi": r} for r in rsi_values]

    def test_bullish_50pct_above_70(self):
        # 5 out of 10 RSI > 70 => 50% * 200 = 100
        stocks = self._make_stocks([75, 80, 72, 85, 71, 40, 50, 60, 30, 20])
        score = extremity_clustering_score(stocks, is_bearish=False)
        assert score == pytest.approx(100.0)

    def test_bullish_25pct_above_70(self):
        # 2 out of 8 => 25% * 200 = 50
        stocks = self._make_stocks([75, 80, 40, 50, 60, 30, 20, 65])
        score = extremity_clustering_score(stocks, is_bearish=False)
        assert score == pytest.approx(50.0)

    def test_bearish_50pct_below_30(self):
        # 5 out of 10 RSI < 30
        stocks = self._make_stocks([10, 15, 20, 25, 29, 40, 50, 60, 70, 80])
        score = extremity_clustering_score(stocks, is_bearish=True)
        assert score == pytest.approx(100.0)

    def test_empty_returns_none(self):
        assert extremity_clustering_score([], is_bearish=False) is None

    def test_none_rsi_ignored(self):
        stocks = [{"rsi": None}, {"rsi": 75}, {"rsi": 80}]
        # 2 out of 3 valid, both > 70 => 2/3 ≈ 66.7% * 200 ≈ 133 => clamped 100
        score = extremity_clustering_score(stocks, is_bearish=False)
        assert score == pytest.approx(100.0)

    def test_zero_extreme(self):
        stocks = self._make_stocks([50, 55, 60, 65])
        score = extremity_clustering_score(stocks, is_bearish=False)
        assert score == pytest.approx(0.0)

    def test_boundary_rsi_70_not_counted_bullish(self):
        # RSI == 70 is NOT > 70
        stocks = self._make_stocks([70])
        score = extremity_clustering_score(stocks, is_bearish=False)
        assert score == pytest.approx(0.0)

    def test_boundary_rsi_30_not_counted_bearish(self):
        # RSI == 30 is NOT < 30
        stocks = self._make_stocks([30])
        score = extremity_clustering_score(stocks, is_bearish=True)
        assert score == pytest.approx(0.0)


# ── price_extreme_saturation_score ───────────────────────────────────


class TestPriceExtremeSaturationScore:
    def test_bullish_50pct_near_high(self):
        # 5 out of 10 within 5% of 52w high
        stocks = [
            {"dist_from_52w_high": d}
            for d in [0.01, 0.02, 0.03, 0.04, 0.05, 0.10, 0.15, 0.20, 0.25, 0.30]
        ]
        score = price_extreme_saturation_score(stocks, is_bearish=False)
        assert score == pytest.approx(100.0)

    def test_bullish_25pct_near_high(self):
        stocks = [
            {"dist_from_52w_high": d} for d in [0.01, 0.02, 0.10, 0.15, 0.20, 0.25, 0.30, 0.35]
        ]
        score = price_extreme_saturation_score(stocks, is_bearish=False)
        assert score == pytest.approx(50.0)

    def test_bullish_zero_near_high(self):
        stocks = [{"dist_from_52w_high": d} for d in [0.10, 0.20, 0.30]]
        score = price_extreme_saturation_score(stocks, is_bearish=False)
        assert score == pytest.approx(0.0)

    def test_bearish_50pct_near_low(self):
        stocks = [
            {"dist_from_52w_low": d}
            for d in [0.01, 0.02, 0.03, 0.04, 0.05, 0.10, 0.15, 0.20, 0.25, 0.30]
        ]
        score = price_extreme_saturation_score(stocks, is_bearish=True)
        assert score == pytest.approx(100.0)

    def test_empty_returns_none(self):
        assert price_extreme_saturation_score([], is_bearish=False) is None

    def test_boundary_exactly_5pct(self):
        # dist <= 0.05 => counted
        stocks = [{"dist_from_52w_high": 0.05}]
        score = price_extreme_saturation_score(stocks, is_bearish=False)
        # 1/1 = 100% * 200 = 200 => clamped 100
        assert score == pytest.approx(100.0)

    def test_boundary_just_above_5pct(self):
        stocks = [{"dist_from_52w_high": 0.051}]
        score = price_extreme_saturation_score(stocks, is_bearish=False)
        assert score == pytest.approx(0.0)


# ── valuation_premium_score ──────────────────────────────────────────


class TestValuationPremiumScore:
    def _make_stocks(self, pe_values):
        return [{"pe_ratio": p} for p in pe_values]

    def test_half_market_premium(self):
        # median PE = 11 => ratio = 0.5 => (0.5-0.5)*32 = 0
        stocks = self._make_stocks([11, 11, 11])
        assert valuation_premium_score(stocks) == pytest.approx(0.0)

    def test_market_premium(self):
        # median PE = 22 => ratio = 1.0 => (1.0-0.5)*32 = 16
        stocks = self._make_stocks([22, 22, 22])
        assert valuation_premium_score(stocks) == pytest.approx(16.0)

    def test_double_premium(self):
        # median PE = 44 => ratio = 2.0 => (2.0-0.5)*32 = 48
        stocks = self._make_stocks([44, 44, 44])
        assert valuation_premium_score(stocks) == pytest.approx(48.0)

    def test_triple_premium(self):
        # median PE = 66 => ratio = 3.0 => (3.0-0.5)*32 = 80
        stocks = self._make_stocks([66, 66, 66])
        assert valuation_premium_score(stocks) == pytest.approx(80.0)

    def test_capped_at_100(self):
        # median PE = 110 => ratio = 5.0 => (5.0-0.5)*32 = 144 => clamped 100
        stocks = self._make_stocks([110, 110, 110])
        assert valuation_premium_score(stocks) == pytest.approx(100.0)

    def test_fewer_than_3_valid_returns_none(self):
        stocks = self._make_stocks([22, 22])
        assert valuation_premium_score(stocks) is None

    def test_none_pe_excluded(self):
        stocks = self._make_stocks([None, 22, 22])
        assert valuation_premium_score(stocks) is None

    def test_negative_pe_excluded(self):
        stocks = self._make_stocks([-10, 22, 22, 22])
        # valid: [22, 22, 22] => median 22 => ratio 1.0 => 16
        assert valuation_premium_score(stocks) == pytest.approx(16.0)

    def test_empty_returns_none(self):
        assert valuation_premium_score([]) is None


# ── etf_proliferation_score ──────────────────────────────────────────


class TestEtfProliferationScore:
    def test_zero(self):
        assert etf_proliferation_score(0) == pytest.approx(0.0)

    def test_one(self):
        assert etf_proliferation_score(1) == pytest.approx(20.0)

    def test_three(self):
        assert etf_proliferation_score(3) == pytest.approx(40.0)

    def test_six(self):
        assert etf_proliferation_score(6) == pytest.approx(60.0)

    def test_ten(self):
        assert etf_proliferation_score(10) == pytest.approx(80.0)

    def test_fifteen(self):
        assert etf_proliferation_score(15) == pytest.approx(100.0)

    def test_two(self):
        # 1 < 2 <= 3 => 40
        assert etf_proliferation_score(2) == pytest.approx(40.0)

    def test_five(self):
        # 3 < 5 <= 6 => 60
        assert etf_proliferation_score(5) == pytest.approx(60.0)


# ── classify_stage ───────────────────────────────────────────────────


class TestClassifyStage:
    def test_emerging(self):
        assert classify_stage(0) == "Emerging"
        assert classify_stage(10) == "Emerging"
        assert classify_stage(19.9) == "Emerging"

    def test_accelerating(self):
        assert classify_stage(20) == "Accelerating"
        assert classify_stage(30) == "Accelerating"
        assert classify_stage(39.9) == "Accelerating"

    def test_trending(self):
        assert classify_stage(40) == "Trending"
        assert classify_stage(50) == "Trending"
        assert classify_stage(59.9) == "Trending"

    def test_mature(self):
        assert classify_stage(60) == "Mature"
        assert classify_stage(70) == "Mature"
        assert classify_stage(79.9) == "Mature"

    def test_exhausting(self):
        assert classify_stage(80) == "Exhausting"
        assert classify_stage(90) == "Exhausting"
        assert classify_stage(100) == "Exhausting"


# ── calculate_lifecycle_maturity ─────────────────────────────────────


class TestCalculateLifecycleMaturity:
    def test_weighted_sum(self):
        # 60*0.25 + 80*0.25 + 40*0.25 + 50*0.15 + 30*0.10
        # = 15 + 20 + 10 + 7.5 + 3 = 55.5
        result = calculate_lifecycle_maturity(60.0, 80.0, 40.0, 50.0, 30.0)
        assert result == pytest.approx(55.5)

    def test_all_100(self):
        result = calculate_lifecycle_maturity(100.0, 100.0, 100.0, 100.0, 100.0)
        assert result == pytest.approx(100.0)

    def test_all_zero(self):
        result = calculate_lifecycle_maturity(0.0, 0.0, 0.0, 0.0, 0.0)
        assert result == pytest.approx(0.0)

    def test_none_has_zero_score_and_zero_coverage(self):
        result = calculate_lifecycle_maturity(None, None, None, None, None)
        assert result == pytest.approx(0.0)
        detailed = calculate_lifecycle_maturity_detailed(None, None, None, None, None)
        assert detailed["score"] is None
        assert detailed["coverage"] == pytest.approx(0.0)
        assert set(detailed["missing_components"]) == {
            "duration",
            "extremity",
            "price_extreme",
            "valuation",
            "etf_proliferation",
        }

    def test_clamped_above_100(self):
        result = calculate_lifecycle_maturity(200.0, 200.0, 200.0, 200.0, 200.0)
        assert result == 100.0

    def test_clamped_below_0(self):
        result = calculate_lifecycle_maturity(-50.0, -50.0, -50.0, -50.0, -50.0)
        assert result == 0.0

    def test_returns_float(self):
        assert isinstance(calculate_lifecycle_maturity(50, 50, 50, 50, 50), float)

# GENERATED by scripts/generate_market_calendar.py — do not edit.
# Source of truth: scripts/market_calendar/ and config/market-calendar-dependency.yaml.
"""Generated contract tests for the vendored exchange calendar."""

from datetime import date, datetime
from zoneinfo import ZoneInfo

import _market_calendar as calendar
import pytest


def test_supported_cash_equity_venues():
    assert calendar.supported_venues() == ("XLON", "XNAS", "XNYS", "XTKS")


def test_jpx_close_cutover_and_lunch_break():
    before = calendar.session_for_date("XTKS", date(2024, 11, 1))
    after = calendar.session_for_date("XTKS", date(2024, 11, 5))
    assert before.market_close.strftime("%H:%M") == "15:00"
    assert after.market_close.strftime("%H:%M") == "15:30"
    assert after.break_start.strftime("%H:%M") == "11:30"
    assert after.break_end.strftime("%H:%M") == "12:30"
    assert (
        calendar.is_open_at("XTKS", datetime(2024, 11, 5, 12, 0, tzinfo=ZoneInfo("Asia/Tokyo")))
        is False
    )


def test_us_holiday_early_close_and_dst():
    assert calendar.session_for_date("XNYS", date(2026, 4, 3)) is None
    early = calendar.session_for_date("XNYS", date(2026, 11, 27))
    assert early.market_close.strftime("%H:%M %z") == "13:00 -0500"
    winter = calendar.session_for_date("XNYS", date(2026, 1, 5))
    summer = calendar.session_for_date("XNAS", date(2026, 7, 6))
    assert winter.market_open.utcoffset().total_seconds() == -5 * 3600
    assert summer.market_open.utcoffset().total_seconds() == -4 * 3600


def test_lse_uses_dst_aware_timezone():
    winter = calendar.session_for_date("XLON", date(2026, 1, 5))
    summer = calendar.session_for_date("XLON", date(2026, 7, 6))
    assert winter.market_open.tzname() == "GMT"
    assert summer.market_open.tzname() == "BST"


def test_next_session_skips_weekend_and_holiday():
    session = calendar.next_session_on_or_after("XNYS", date(2026, 7, 3))
    assert session.session_date == date(2026, 7, 6)
    later = calendar.next_session_after("XNYS", date(2026, 7, 6))
    assert later.session_date == date(2026, 7, 7)


def test_previous_session_skips_weekend_and_holiday():
    session = calendar.previous_session_before("XNYS", date(2026, 7, 6))
    assert session.session_date == date(2026, 7, 2)
    same = calendar.previous_session_on_or_before("XNYS", date(2026, 7, 6))
    assert same.session_date == date(2026, 7, 6)


@pytest.mark.parametrize(
    ("include_start", "include_end", "expected"),
    [(False, False, 0), (False, True, 1), (True, False, 1), (True, True, 2)],
)
def test_count_session_boundaries(include_start, include_end, expected):
    assert (
        calendar.count_sessions(
            "XNYS",
            date(2026, 7, 2),
            date(2026, 7, 6),
            include_start=include_start,
            include_end=include_end,
        )
        == expected
    )


def test_same_day_and_reverse_contracts():
    assert (
        calendar.count_sessions(
            "XNYS",
            date(2026, 7, 6),
            date(2026, 7, 6),
            include_start=True,
            include_end=True,
        )
        == 1
    )
    with pytest.raises(ValueError, match="on or before"):
        calendar.count_sessions(
            "XNYS",
            date(2026, 7, 7),
            date(2026, 7, 6),
            include_start=False,
            include_end=True,
        )


def test_naive_datetime_and_unknown_venue_fail_closed():
    with pytest.raises(ValueError, match="timezone-aware"):
        calendar.is_open_at("XNYS", datetime(2026, 7, 6, 10, 0))
    with pytest.raises(ValueError, match="Unsupported venue"):
        calendar.session_for_date("NOPE", date(2026, 7, 6))


def test_parse_aware_datetime_accepts_z_and_rejects_date_only():
    parsed = calendar.parse_aware_datetime("2026-07-06T14:00:00Z")
    assert parsed.utcoffset().total_seconds() == 0
    with pytest.raises(ValueError, match="timezone-aware"):
        calendar.parse_aware_datetime("2026-07-06")


def test_close_is_exclusive_and_current_session_rolls_forward():
    at_close = datetime(2026, 11, 27, 13, 0, tzinfo=ZoneInfo("America/New_York"))
    assert calendar.is_open_at("XNYS", at_close) is False
    assert calendar.current_or_next_session("XNYS", at_close).session_date == date(2026, 11, 30)


def test_provider_failure_is_not_treated_as_a_holiday(monkeypatch):
    def fail_provider(_venue):
        raise calendar.CalendarUnavailableError("calendar unavailable")

    monkeypatch.setattr(calendar, "_provider_calendar", fail_provider)
    with pytest.raises(calendar.CalendarUnavailableError, match="unavailable"):
        calendar.session_for_date("XNYS", date(2026, 7, 6))


def test_search_horizon_failure_is_explicit(monkeypatch):
    class EmptySchedule:
        empty = True

    monkeypatch.setattr(
        calendar,
        "_schedule",
        lambda venue, start, end: ("XNYS", "America/New_York", EmptySchedule()),
    )
    with pytest.raises(calendar.CalendarUnavailableError, match="No XNYS session"):
        calendar.next_session_on_or_after("XNYS", date(2026, 7, 6), max_search_days=1)

"""Tests for report_generator module."""

import json
import os
import tempfile
from datetime import date

import pytest
from report_generator import (
    _fmt_pct,
    _format_stock_list,
    _origin_label,
    generate_json_report,
    generate_markdown_report,
    save_reports,
)

# --- Fixtures ---


@pytest.fixture
def sample_themes():
    """Sample theme data for testing (0-100 scale for heat/maturity)."""
    return [
        {
            "name": "AI / Semiconductors",
            "direction": "bullish",
            "heat": 85.0,
            "maturity": 62.0,
            "stage": "growth",
            "confidence": "High",
            "industries": [
                "Semiconductors",
                "Software - Infrastructure",
                "Information Technology Services",
            ],
            "heat_breakdown": {
                "performance_momentum": 75.0,
                "volume_confirmation": 80.0,
                "breadth_score": 90.0,
            },
            "maturity_breakdown": {
                "duration_score": 60.0,
                "crowding_score": 55.0,
                "acceleration": 70.0,
            },
            "representative_stocks": ["NVDA", "AVGO", "AMD", "MSFT"],
            "proxy_etfs": ["SMH", "SOXX", "XLK"],
        },
        {
            "name": "Energy Transition",
            "direction": "bullish",
            "heat": 63.0,
            "maturity": 41.0,
            "stage": "early",
            "confidence": "Medium",
            "industries": ["Solar", "Uranium"],
            "heat_breakdown": {
                "performance_momentum": 60.0,
                "volume_confirmation": 55.0,
                "breadth_score": 70.0,
            },
            "maturity_breakdown": {},
            "representative_stocks": ["FSLR", "ENPH"],
            "proxy_etfs": ["TAN", "URA"],
        },
        {
            "name": "Traditional Retail",
            "direction": "bearish",
            "heat": 58.0,
            "maturity": 75.0,
            "stage": "decline",
            "confidence": "Low",
            "industries": ["Specialty Retail", "Department Stores"],
            "heat_breakdown": {
                "performance_momentum": -40.0,
                "volume_confirmation": 30.0,
                "breadth_score": 25.0,
            },
            "maturity_breakdown": {},
            "representative_stocks": ["M", "KSS"],
            "proxy_etfs": ["XRT"],
        },
    ]


@pytest.fixture
def sample_industry_rankings():
    """Sample industry rankings (perf values in percent, score is momentum_score)."""
    return {
        "top": [
            {
                "name": "Semiconductors",
                "perf_1w": 5.0,
                "perf_1m": 12.0,
                "perf_3m": 25.0,
                "momentum_score": 0.89,
            },
            {
                "name": "Software - Infrastructure",
                "perf_1w": 3.0,
                "perf_1m": 8.0,
                "perf_3m": 18.0,
                "momentum_score": 0.75,
            },
        ],
        "bottom": [
            {
                "name": "Department Stores",
                "perf_1w": -4.0,
                "perf_1m": -10.0,
                "perf_3m": -15.0,
                "momentum_score": -0.65,
            },
            {
                "name": "Specialty Retail",
                "perf_1w": -3.0,
                "perf_1m": -7.0,
                "perf_3m": -12.0,
                "momentum_score": -0.50,
            },
        ],
    }


@pytest.fixture
def sample_sector_uptrend():
    """Sample sector uptrend data."""
    from datetime import datetime, timedelta

    recent_date = (datetime.now() - timedelta(days=1)).strftime("%Y-%m-%d")
    return {
        "Technology": {
            "ratio": 0.35,
            "ma_10": 0.32,
            "slope": 0.0025,
            "trend": "up",
            "latest_date": recent_date,
        },
        "Healthcare": {
            "ratio": 0.22,
            "ma_10": 0.24,
            "slope": -0.0015,
            "trend": "down",
            "latest_date": recent_date,
        },
    }


@pytest.fixture
def sample_metadata():
    """Sample metadata."""
    return {
        "generated_at": "2026-02-16 10:00:00",
        "data_sources": {
            "finviz": "ok",
            "uptrend": "ok",
            "scan_hits": {"path": "scan_hits.json", "rows": 3, "hits": 3, "skipped_rows": 0},
        },
    }


@pytest.fixture
def sample_json_report(
    sample_themes, sample_industry_rankings, sample_sector_uptrend, sample_metadata
):
    """Full JSON report generated from sample data."""
    return generate_json_report(
        sample_themes,
        sample_industry_rankings,
        sample_sector_uptrend,
        sample_metadata,
    )


# --- Tests for generate_json_report ---


class TestGenerateJsonReport:
    def test_report_structure(self, sample_json_report):
        """JSON report has all required top-level keys."""
        required_keys = [
            "report_type",
            "generated_at",
            "metadata",
            "summary",
            "themes",
            "industry_rankings",
            "sector_uptrend",
            "what_changed_today",
            "leadership_evidence",
            "data_quality",
        ]
        for key in required_keys:
            assert key in sample_json_report, f"Missing key: {key}"

    def test_report_type(self, sample_json_report):
        assert sample_json_report["report_type"] == "theme_detector"

    def test_summary_counts(self, sample_json_report):
        summary = sample_json_report["summary"]
        assert summary["total_themes"] == 3
        assert summary["bullish_count"] == 2
        assert summary["bearish_count"] == 1
        assert summary["top_bullish"] == "AI / Semiconductors"
        assert summary["top_bearish"] == "Traditional Retail"

    def test_themes_grouped(self, sample_json_report):
        themes = sample_json_report["themes"]
        assert "all" in themes
        assert "bullish" in themes
        assert "bearish" in themes
        assert "match_ranked" in themes
        assert len(themes["all"]) == 3
        assert len(themes["bullish"]) == 2
        assert len(themes["bearish"]) == 1

    def test_bullish_sorted_by_heat(self, sample_json_report):
        bullish = sample_json_report["themes"]["bullish"]
        heats = [t["heat"] for t in bullish]
        assert heats == sorted(heats, reverse=True)

    def test_match_ranked_preserves_separate_consumer_rank(self, sample_industry_rankings):
        themes = [
            {
                "name": "Higher Heat",
                "direction": "bullish",
                "heat": 90.0,
                "theme_match_score": 40.0,
            },
            {
                "name": "Higher Match",
                "direction": "bullish",
                "heat": 70.0,
                "theme_match_score": 95.0,
            },
        ]
        report = generate_json_report(themes, sample_industry_rankings, {}, {"data_sources": {}})

        assert [theme["name"] for theme in report["themes"]["all"]] == [
            "Higher Heat",
            "Higher Match",
        ]
        assert [theme["name"] for theme in report["themes"]["match_ranked"]] == [
            "Higher Match",
            "Higher Heat",
        ]

    def test_data_quality_ok(self, sample_json_report):
        dq = sample_json_report["data_quality"]
        assert dq["status"] == "ok"
        assert dq["flags"] == []

    def test_report_freshness_uses_xnys_sessions(
        self, sample_themes, sample_industry_rankings, sample_metadata
    ):
        sector = {"Technology": {"latest_date": "2026-04-02"}}
        report = generate_json_report(
            sample_themes,
            sample_industry_rankings,
            sector,
            sample_metadata,
            as_of_date=date(2026, 4, 6),
        )
        assert not any("sessions old" in flag for flag in report["data_quality"]["flags"])

    def test_report_freshness_rejects_future_data(
        self, sample_themes, sample_industry_rankings, sample_metadata
    ):
        sector = {"Technology": {"latest_date": "2026-04-07"}}
        report = generate_json_report(
            sample_themes,
            sample_industry_rankings,
            sector,
            sample_metadata,
            as_of_date=date(2026, 4, 6),
        )
        assert any("after the report as-of" in flag for flag in report["data_quality"]["flags"])

    def test_empty_themes(self, sample_industry_rankings, sample_sector_uptrend, sample_metadata):
        """Empty themes list produces warning, not error."""
        report = generate_json_report(
            [],
            sample_industry_rankings,
            sample_sector_uptrend,
            sample_metadata,
        )
        assert report["summary"]["total_themes"] == 0
        assert report["summary"]["top_bullish"] is None
        assert report["data_quality"]["status"] == "warning"
        assert any("No themes" in f for f in report["data_quality"]["flags"])

    def test_json_serializable(self, sample_json_report):
        """Report can be serialized to JSON."""
        serialized = json.dumps(sample_json_report, default=str)
        deserialized = json.loads(serialized)
        assert deserialized["report_type"] == "theme_detector"


# --- Tests for generate_markdown_report ---


class TestGenerateMarkdownReport:
    def test_contains_all_sections(self, sample_json_report):
        """Markdown contains all required sections."""
        md = generate_markdown_report(sample_json_report)
        assert "## 1. Theme Dashboard" in md
        assert "## 2. What Changed Today" in md
        assert "## 3. Leadership Evidence" in md
        assert "## 4. Leading Themes (Top 3)" in md
        assert "## 5. Lagging Themes (Top 3)" in md
        assert "## 6. All Themes Summary" in md
        assert "## 7. Industry Rankings" in md
        assert "## 8. Sector Uptrend Ratios" in md
        assert "## 9. Methodology & Data Quality" in md

    def test_header_info(self, sample_json_report):
        md = generate_markdown_report(sample_json_report)
        assert "# Theme Detector Report" in md
        assert "2026-02-16 10:00:00" in md

    def test_theme_dashboard_table(self, sample_json_report):
        md = generate_markdown_report(sample_json_report)
        assert "AI / Semiconductors" in md
        assert "LEAD" in md
        assert "LAG" in md

    def test_bullish_detail_stocks(self, sample_json_report):
        md = generate_markdown_report(sample_json_report)
        assert "NVDA" in md
        assert "SMH" in md

    def test_industry_rankings_present(self, sample_json_report):
        md = generate_markdown_report(sample_json_report)
        assert "Semiconductors" in md
        assert "Department Stores" in md
        assert "Top 15" in md
        assert "Bottom 15" in md

    def test_sector_uptrend_table(self, sample_json_report):
        md = generate_markdown_report(sample_json_report)
        assert "Technology" in md
        assert "Healthcare" in md
        assert "35.0%" in md  # Technology ratio

    def test_methodology_present(self, sample_json_report):
        md = generate_markdown_report(sample_json_report)
        assert "Methodology" in md
        assert "Disclaimer" in md

    def test_empty_themes_warning(
        self, sample_industry_rankings, sample_sector_uptrend, sample_metadata
    ):
        """Empty themes show warning, not crash."""
        report = generate_json_report(
            [],
            sample_industry_rankings,
            sample_sector_uptrend,
            sample_metadata,
        )
        md = generate_markdown_report(report)
        assert "WARNING" in md or "No themes" in md

    def test_no_industry_data(self, sample_themes, sample_sector_uptrend, sample_metadata):
        """Missing industry rankings handled gracefully."""
        report = generate_json_report(
            sample_themes,
            {"top": [], "bottom": []},
            sample_sector_uptrend,
            sample_metadata,
        )
        md = generate_markdown_report(report)
        assert "## 7. Industry Rankings" in md
        assert "unavailable" in md.lower() or "Industry" in md

    def test_top_n_detail_default_3(self, sample_json_report):
        """Default top_n_detail=3 shows 'Top 3' in section headers."""
        md = generate_markdown_report(sample_json_report)
        assert "## 4. Leading Themes (Top 3)" in md
        assert "## 5. Lagging Themes (Top 3)" in md

    def test_top_n_detail_custom(self, sample_json_report):
        """Custom top_n_detail changes section headers and limits themes."""
        md = generate_markdown_report(sample_json_report, top_n_detail=5)
        assert "## 4. Leading Themes (Top 5)" in md
        assert "## 5. Lagging Themes (Top 5)" in md

    def test_top_n_detail_1(self, sample_json_report):
        """top_n_detail=1 only shows 1 theme per direction."""
        md = generate_markdown_report(sample_json_report, top_n_detail=1)
        assert "## 4. Leading Themes (Top 1)" in md
        # Only the top bullish theme detail section should appear
        # AI / Semiconductors (heat 85) should appear, Energy Transition (heat 63) should not
        assert "### AI / Semiconductors" in md
        assert "### Energy Transition" not in md


# --- Tests for save_reports ---


class TestSaveReports:
    def test_save_creates_files(self, sample_json_report):
        """save_reports creates both JSON and MD files."""
        md = generate_markdown_report(sample_json_report)
        with tempfile.TemporaryDirectory() as tmpdir:
            paths = save_reports(sample_json_report, md, tmpdir)
            assert os.path.exists(paths["json"])
            assert os.path.exists(paths["markdown"])

    def test_filename_convention(self, sample_json_report):
        """Filenames follow theme_detector_YYYY-MM-DD_HHMMSS pattern."""
        md = generate_markdown_report(sample_json_report)
        with tempfile.TemporaryDirectory() as tmpdir:
            paths = save_reports(sample_json_report, md, tmpdir)
            json_name = os.path.basename(paths["json"])
            md_name = os.path.basename(paths["markdown"])
            assert json_name.startswith("theme_detector_")
            assert json_name.endswith(".json")
            assert md_name.startswith("theme_detector_")
            assert md_name.endswith(".md")

    def test_json_file_valid(self, sample_json_report):
        """Saved JSON file is valid JSON."""
        md = generate_markdown_report(sample_json_report)
        with tempfile.TemporaryDirectory() as tmpdir:
            paths = save_reports(sample_json_report, md, tmpdir)
            with open(paths["json"]) as f:
                loaded = json.load(f)
            assert loaded["report_type"] == "theme_detector"

    def test_creates_output_dir(self, sample_json_report):
        """save_reports creates output dir if it doesn't exist."""
        md = generate_markdown_report(sample_json_report)
        with tempfile.TemporaryDirectory() as tmpdir:
            nested = os.path.join(tmpdir, "reports", "sub")
            paths = save_reports(sample_json_report, md, nested)
            assert os.path.exists(paths["json"])


# --- Tests for _fmt_pct ---


class TestFmtPct:
    """_fmt_pct formats percent values directly (no *100 conversion)."""

    def test_positive_value(self):
        assert _fmt_pct(5.0) == "+5.0%"

    def test_negative_value(self):
        assert _fmt_pct(-3.2) == "-3.2%"

    def test_zero(self):
        assert _fmt_pct(0.0) == "+0.0%"

    def test_none_returns_na(self):
        assert _fmt_pct(None) == "N/A"

    def test_large_value(self):
        assert _fmt_pct(25.7) == "+25.7%"

    def test_small_negative(self):
        assert _fmt_pct(-0.5) == "-0.5%"


# --- Tests for Industry Rankings key usage ---


class TestIndustryRankingsKey:
    """Verify industry rankings use momentum_score, not composite_score."""

    def test_top_industries_show_momentum_score(self, sample_json_report):
        md = generate_markdown_report(sample_json_report)
        # momentum_score=0.89 should appear, not composite_score
        assert "0.89" in md

    def test_bottom_industries_show_momentum_score(self, sample_json_report):
        md = generate_markdown_report(sample_json_report)
        assert "-0.65" in md

    def test_industry_perf_not_multiplied(self, sample_json_report):
        """Perf values should appear as-is (already in percent)."""
        md = generate_markdown_report(sample_json_report)
        # 5.0% should appear, not 500.0%
        assert "+5.0%" in md
        assert "500.0%" not in md


# --- Tests for _format_stock_list ---


class TestFormatStockList:
    def test_no_details_uses_tickers(self):
        """Without stock_details, plain tickers are shown."""
        theme = {"representative_stocks": ["NVDA", "AMD", "AVGO"]}
        assert _format_stock_list(theme) == "NVDA, AMD, AVGO"

    def test_empty_stocks_returns_na(self):
        theme = {"representative_stocks": []}
        assert _format_stock_list(theme) == "N/A"

    def test_finviz_public_labels(self):
        """stock_details with finviz_public source shows [Fp] labels."""
        theme = {
            "representative_stocks": ["NEM", "GOLD"],
            "stock_details": [
                {"symbol": "NEM", "source": "finviz_public"},
                {"symbol": "GOLD", "source": "finviz_public"},
            ],
        }
        result = _format_stock_list(theme)
        assert "NEM[Fp]" in result
        assert "GOLD[Fp]" in result

    def test_finviz_elite_labels(self):
        theme = {
            "representative_stocks": ["NVDA"],
            "stock_details": [
                {"symbol": "NVDA", "source": "finviz_elite"},
            ],
        }
        assert "NVDA[Fe]" in _format_stock_list(theme)

    def test_etf_holdings_labels(self):
        theme = {
            "representative_stocks": ["AAPL"],
            "stock_details": [
                {"symbol": "AAPL", "source": "etf_holdings"},
            ],
        }
        assert "AAPL[E]" in _format_stock_list(theme)

    def test_static_labels(self):
        theme = {
            "representative_stocks": ["XOM"],
            "stock_details": [
                {"symbol": "XOM", "source": "static"},
            ],
        }
        assert "XOM[S]" in _format_stock_list(theme)

    def test_mixed_sources(self):
        theme = {
            "representative_stocks": ["NVDA", "AAPL", "XOM"],
            "stock_details": [
                {"symbol": "NVDA", "source": "finviz_public"},
                {"symbol": "AAPL", "source": "etf_holdings"},
                {"symbol": "XOM", "source": "static"},
            ],
        }
        result = _format_stock_list(theme)
        assert "NVDA[Fp]" in result
        assert "AAPL[E]" in result
        assert "XOM[S]" in result

    def test_unknown_source_shows_question_mark(self):
        theme = {
            "representative_stocks": ["X"],
            "stock_details": [
                {"symbol": "X", "source": "unknown_source"},
            ],
        }
        assert "X[?]" in _format_stock_list(theme)


class TestMarkdownWithStockDetails:
    def test_markdown_shows_source_labels(self):
        """Markdown report includes source labels when stock_details present."""
        themes = [
            {
                "name": "Gold Theme",
                "direction": "bullish",
                "heat": 70.0,
                "maturity": 40.0,
                "stage": "growth",
                "confidence": "Medium",
                "industries": ["Gold"],
                "heat_breakdown": {},
                "maturity_breakdown": {},
                "representative_stocks": ["NEM", "GOLD"],
                "stock_details": [
                    {"symbol": "NEM", "source": "finviz_public"},
                    {"symbol": "GOLD", "source": "finviz_public"},
                ],
                "proxy_etfs": ["GDX"],
            },
        ]
        metadata = {"generated_at": "2026-02-16 10:00:00", "data_sources": {}}
        report = generate_json_report(themes, {"top": [], "bottom": []}, {}, metadata)
        md = generate_markdown_report(report)
        assert "NEM[Fp]" in md
        assert "GOLD[Fp]" in md

    def test_markdown_no_details_plain_tickers(self):
        """Without stock_details, plain tickers are shown in markdown."""
        themes = [
            {
                "name": "Test Theme",
                "direction": "bullish",
                "heat": 60.0,
                "maturity": 30.0,
                "stage": "early",
                "confidence": "Low",
                "industries": ["Tech"],
                "heat_breakdown": {},
                "maturity_breakdown": {},
                "representative_stocks": ["AAPL", "MSFT"],
                "proxy_etfs": [],
            },
        ]
        metadata = {"generated_at": "2026-02-16 10:00:00", "data_sources": {}}
        report = generate_json_report(themes, {"top": [], "bottom": []}, {}, metadata)
        md = generate_markdown_report(report)
        assert "AAPL, MSFT" in md
        assert "[Fp]" not in md
        assert "[S]" not in md


# --- Tests for theme origin display ---


class TestOriginLabel:
    def test_seed_label(self):
        assert _origin_label("seed") == "Seed"

    def test_vertical_label(self):
        assert _origin_label("vertical") == "Vertical"

    def test_discovered_label(self):
        assert _origin_label("discovered") == "*NEW*"

    def test_none_defaults_to_seed(self):
        assert _origin_label(None) == "Seed"

    def test_unknown_defaults_to_seed(self):
        assert _origin_label("unknown") == "Seed"


class TestOriginInDashboard:
    def test_dashboard_shows_origin_column(self):
        themes = [
            {
                "name": "Seed Theme",
                "direction": "bullish",
                "heat": 70.0,
                "maturity": 40.0,
                "stage": "growth",
                "confidence": "Medium",
                "industries": ["Tech"],
                "heat_breakdown": {},
                "maturity_breakdown": {},
                "representative_stocks": ["AAPL"],
                "proxy_etfs": [],
                "theme_origin": "seed",
            },
            {
                "name": "New Theme",
                "direction": "bullish",
                "heat": 60.0,
                "maturity": 30.0,
                "stage": "early",
                "confidence": "Low",
                "industries": ["Mining"],
                "heat_breakdown": {},
                "maturity_breakdown": {},
                "representative_stocks": ["NEM"],
                "proxy_etfs": [],
                "theme_origin": "discovered",
                "name_confidence": "medium",
            },
        ]
        metadata = {"generated_at": "2026-02-16", "data_sources": {}}
        report = generate_json_report(themes, {"top": [], "bottom": []}, {}, metadata)
        md = generate_markdown_report(report)
        assert "| Origin |" in md
        assert "| Seed |" in md
        assert "| *NEW* |" in md

    def test_discovered_detail_shows_origin_line(self):
        themes = [
            {
                "name": "Discovered Theme",
                "direction": "bullish",
                "heat": 65.0,
                "maturity": 35.0,
                "stage": "early",
                "confidence": "Low",
                "industries": ["X"],
                "heat_breakdown": {},
                "maturity_breakdown": {},
                "representative_stocks": ["XYZ"],
                "proxy_etfs": [],
                "theme_origin": "discovered",
                "name_confidence": "medium",
            },
        ]
        metadata = {"generated_at": "2026-02-16", "data_sources": {}}
        report = generate_json_report(themes, {"top": [], "bottom": []}, {}, metadata)
        md = generate_markdown_report(report)
        assert "**Origin:** Discovered (name confidence: medium)" in md

    def test_seed_detail_no_origin_line(self):
        themes = [
            {
                "name": "Seed Theme",
                "direction": "bullish",
                "heat": 70.0,
                "maturity": 40.0,
                "stage": "growth",
                "confidence": "Medium",
                "industries": ["Tech"],
                "heat_breakdown": {},
                "maturity_breakdown": {},
                "representative_stocks": ["AAPL"],
                "proxy_etfs": [],
                "theme_origin": "seed",
            },
        ]
        metadata = {"generated_at": "2026-02-16", "data_sources": {}}
        report = generate_json_report(themes, {"top": [], "bottom": []}, {}, metadata)
        md = generate_markdown_report(report)
        assert "**Origin:**" not in md


# --- Tests for data quality badges ---


class TestDataQualityBadges:
    """Verify lifecycle insufficient and discovered-provisional badges."""

    def _make_theme(self, **overrides):
        base = {
            "name": "Test Theme",
            "direction": "bullish",
            "heat": 65.0,
            "maturity": 35.0,
            "stage": "early",
            "confidence": "Low",
            "industries": ["Tech"],
            "heat_breakdown": {},
            "maturity_breakdown": {},
            "representative_stocks": ["AAPL"],
            "proxy_etfs": [],
        }
        base.update(overrides)
        return base

    def _render(self, theme):
        metadata = {"generated_at": "2026-03-17", "data_sources": {}}
        report = generate_json_report([theme], {"top": [], "bottom": []}, {}, metadata)
        return generate_markdown_report(report)

    def test_lifecycle_insufficient_badge(self):
        md = self._render(self._make_theme(lifecycle_data_quality="insufficient"))
        assert "[Lifecycle data insufficient]" in md

    def test_lifecycle_sufficient_no_badge(self):
        md = self._render(self._make_theme(lifecycle_data_quality="sufficient"))
        assert "[Lifecycle data insufficient]" not in md

    def test_discovered_provisional_badge(self):
        md = self._render(self._make_theme(theme_origin="discovered"))
        assert "[Discovered - provisional]" in md

    def test_seed_no_provisional_badge(self):
        md = self._render(self._make_theme(theme_origin="seed"))
        assert "[Discovered - provisional]" not in md

"""Tests for representative_stock_selector module."""

import time
from unittest.mock import MagicMock, patch

import pytest
from representative_stock_selector import (
    _MAX_CONSECUTIVE_FAILURES,
    RepresentativeStockSelector,
    _parse_change,
    _parse_market_cap,
    _parse_volume,
)

# ---------------------------------------------------------------------------
# Parse helpers
# ---------------------------------------------------------------------------


class TestParseMarketCap:
    def test_trillions(self):
        assert _parse_market_cap("2.8T") == 2_800_000_000_000

    def test_billions(self):
        assert _parse_market_cap("150B") == 150_000_000_000

    def test_millions(self):
        assert _parse_market_cap("500M") == 500_000_000

    def test_none_returns_zero(self):
        assert _parse_market_cap(None) == 0

    def test_dash_returns_zero(self):
        assert _parse_market_cap("-") == 0

    def test_numeric_passthrough(self):
        assert _parse_market_cap(1_000_000) == 1_000_000

    def test_float_passthrough(self):
        assert _parse_market_cap(1.5e9) == 1_500_000_000


class TestParseChange:
    def test_percent_string(self):
        assert _parse_change("12.50%") == pytest.approx(12.50)

    def test_negative_percent(self):
        assert _parse_change("-3.20%") == pytest.approx(-3.20)

    def test_float_fraction(self):
        # finvizfinance returns 0.125 meaning 12.5%
        assert _parse_change(0.125) == pytest.approx(12.5)

    def test_negative_float_fraction(self):
        assert _parse_change(-0.032) == pytest.approx(-3.2)

    def test_already_float_large(self):
        # abs > 1 means already in percent form
        assert _parse_change(12.5) == pytest.approx(12.5)

    def test_negative_already_float_large(self):
        assert _parse_change(-5.3) == pytest.approx(-5.3)

    def test_dash_returns_none(self):
        assert _parse_change("-") is None

    def test_none_returns_none(self):
        assert _parse_change(None) is None


class TestParseVolume:
    def test_comma_string(self):
        assert _parse_volume("1,234,567") == 1234567

    def test_int_passthrough(self):
        assert _parse_volume(1234567) == 1234567

    def test_float_passthrough(self):
        assert _parse_volume(1234567.0) == 1234567

    def test_m_suffix(self):
        assert _parse_volume("1.2M") == 1200000

    def test_dash_returns_none(self):
        assert _parse_volume("-") is None

    def test_none_returns_none(self):
        assert _parse_volume(None) is None


# ---------------------------------------------------------------------------
# Composite score
# ---------------------------------------------------------------------------


class TestCompositeScore:
    def _make_selector(self):
        return RepresentativeStockSelector()

    def test_ranks_by_composite_not_market_cap_alone(self):
        """A stock with large market_cap but low change/volume is not #1."""
        sel = self._make_selector()
        stocks = [
            {
                "symbol": "BIG",
                "source": "finviz_public",
                "market_cap": 2_000_000_000_000,
                "change": 1.0,
                "volume": 100_000,
                "matched_industries": ["X"],
                "reasons": [],
            },
            {
                "symbol": "SMALL",
                "source": "finviz_public",
                "market_cap": 10_000_000_000,
                "change": 20.0,
                "volume": 5_000_000,
                "matched_industries": ["X"],
                "reasons": [],
            },
        ]
        scored = sel._compute_composite_score(stocks, is_bearish=False)
        # SMALL ranks #1: cap_rank=2 (0.4*0) but change_rank=1 (0.3*1) + vol_rank=1 (0.3*1)
        # BIG ranks #2: cap_rank=1 (0.4*1) but change_rank=2 (0.3*0) + vol_rank=2 (0.3*0)
        assert scored[0]["symbol"] == "SMALL"
        assert scored[1]["symbol"] == "BIG"
        assert scored[0]["composite_score"] > scored[1]["composite_score"]

    def test_weights_are_applied(self):
        """composite = 0.4 * cap_rank + 0.3 * change_rank + 0.3 * vol_rank"""
        sel = self._make_selector()
        stocks = [
            {
                "symbol": "A",
                "source": "finviz_public",
                "market_cap": 100_000_000_000,
                "change": 10.0,
                "volume": 1_000_000,
                "matched_industries": [],
                "reasons": [],
            },
        ]
        scored = sel._compute_composite_score(stocks, is_bearish=False)
        # Single stock => rank 1/1 => score = 0.4*1 + 0.3*1 + 0.3*1 = 1.0
        assert scored[0]["composite_score"] == pytest.approx(1.0)

    def test_bearish_uses_abs_change(self):
        """is_bearish=True => abs(change) ranks descending."""
        sel = self._make_selector()
        stocks = [
            {
                "symbol": "DROP",
                "source": "finviz_public",
                "market_cap": 50_000_000_000,
                "change": -15.0,
                "volume": 1_000_000,
                "matched_industries": [],
                "reasons": [],
            },
            {
                "symbol": "FLAT",
                "source": "finviz_public",
                "market_cap": 50_000_000_000,
                "change": -1.0,
                "volume": 1_000_000,
                "matched_industries": [],
                "reasons": [],
            },
        ]
        scored = sel._compute_composite_score(stocks, is_bearish=True)
        # DROP has larger abs(change), should score higher
        drop = next(s for s in scored if s["symbol"] == "DROP")
        flat = next(s for s in scored if s["symbol"] == "FLAT")
        assert drop["composite_score"] > flat["composite_score"]

    def test_missing_fields_renormalize(self):
        """change/volume=None => re-normalize with available metrics only."""
        sel = self._make_selector()
        stocks = [
            {
                "symbol": "A",
                "source": "etf_holdings",
                "market_cap": 100_000_000_000,
                "change": None,
                "volume": None,
                "matched_industries": [],
                "reasons": [],
            },
            {
                "symbol": "B",
                "source": "etf_holdings",
                "market_cap": 50_000_000_000,
                "change": None,
                "volume": None,
                "matched_industries": [],
                "reasons": [],
            },
        ]
        scored = sel._compute_composite_score(stocks, is_bearish=False)
        # Should not crash; A has bigger cap so should rank higher
        assert scored[0]["symbol"] == "A"
        assert scored[0]["composite_score"] > scored[1]["composite_score"]

    def test_empty_input(self):
        sel = self._make_selector()
        assert sel._compute_composite_score([], is_bearish=False) == []


# ---------------------------------------------------------------------------
# Merge and rank
# ---------------------------------------------------------------------------


class TestMergeAndRank:
    def _make_selector(self):
        return RepresentativeStockSelector()

    def test_deduplicates_by_symbol(self):
        sel = self._make_selector()
        candidates = [
            {
                "symbol": "NVDA",
                "source": "finviz_public",
                "market_cap": 100,
                "matched_industries": ["Semi"],
                "reasons": ["reason1"],
                "composite_score": 0.9,
            },
            {
                "symbol": "NVDA",
                "source": "finviz_public",
                "market_cap": 100,
                "matched_industries": ["Hardware"],
                "reasons": ["reason2"],
                "composite_score": 0.8,
            },
        ]
        result = sel._merge_and_rank(candidates, max_stocks=10)
        assert len(result) == 1
        assert result[0]["symbol"] == "NVDA"

    def test_merges_matched_industries_on_duplicate(self):
        sel = self._make_selector()
        candidates = [
            {
                "symbol": "NVDA",
                "source": "finviz_public",
                "market_cap": 100,
                "matched_industries": ["Semi"],
                "reasons": [],
                "composite_score": 0.9,
            },
            {
                "symbol": "NVDA",
                "source": "finviz_public",
                "market_cap": 100,
                "matched_industries": ["Hardware"],
                "reasons": [],
                "composite_score": 0.8,
            },
        ]
        result = sel._merge_and_rank(candidates, max_stocks=10)
        assert "Semi" in result[0]["matched_industries"]
        assert "Hardware" in result[0]["matched_industries"]

    def test_accumulates_reasons_on_duplicate(self):
        sel = self._make_selector()
        candidates = [
            {
                "symbol": "X",
                "source": "finviz_public",
                "market_cap": 0,
                "matched_industries": [],
                "reasons": ["r1"],
                "composite_score": 0.5,
            },
            {
                "symbol": "X",
                "source": "finviz_public",
                "market_cap": 0,
                "matched_industries": [],
                "reasons": ["r2"],
                "composite_score": 0.4,
            },
        ]
        result = sel._merge_and_rank(candidates, max_stocks=10)
        assert "r1" in result[0]["reasons"]
        assert "r2" in result[0]["reasons"]

    def test_sorts_by_composite_score_descending(self):
        sel = self._make_selector()
        candidates = [
            {
                "symbol": "A",
                "source": "s",
                "market_cap": 0,
                "matched_industries": [],
                "reasons": [],
                "composite_score": 0.3,
            },
            {
                "symbol": "B",
                "source": "s",
                "market_cap": 0,
                "matched_industries": [],
                "reasons": [],
                "composite_score": 0.9,
            },
            {
                "symbol": "C",
                "source": "s",
                "market_cap": 0,
                "matched_industries": [],
                "reasons": [],
                "composite_score": 0.6,
            },
        ]
        result = sel._merge_and_rank(candidates, max_stocks=10)
        assert [r["symbol"] for r in result] == ["B", "C", "A"]

    def test_respects_max_stocks(self):
        sel = self._make_selector()
        candidates = [
            {
                "symbol": f"S{i}",
                "source": "s",
                "market_cap": 0,
                "matched_industries": [],
                "reasons": [],
                "composite_score": i * 0.1,
            }
            for i in range(20)
        ]
        result = sel._merge_and_rank(candidates, max_stocks=5)
        assert len(result) == 5

    def test_empty_input(self):
        sel = self._make_selector()
        assert sel._merge_and_rank([], max_stocks=10) == []

    def test_duplicate_uses_max_composite_score(self):
        sel = self._make_selector()
        candidates = [
            {
                "symbol": "X",
                "source": "s",
                "market_cap": 0,
                "matched_industries": [],
                "reasons": [],
                "composite_score": 0.3,
            },
            {
                "symbol": "X",
                "source": "s",
                "market_cap": 0,
                "matched_industries": [],
                "reasons": [],
                "composite_score": 0.9,
            },
        ]
        result = sel._merge_and_rank(candidates, max_stocks=10)
        assert result[0]["composite_score"] == 0.9


# ---------------------------------------------------------------------------
# select_stocks fallback chain
# ---------------------------------------------------------------------------


def _mock_finviz_public_stocks(industry, limit, is_bearish):
    """Return fake stocks for FINVIZ public."""
    return [
        {
            "symbol": f"{industry[:3].upper()}{i}",
            "source": "finviz_public",
            "market_cap": (10 - i) * 1_000_000_000,
            "change": 5.0 + i,
            "volume": 1_000_000,
            "matched_industries": [industry],
            "reasons": [f"Top in {industry}"],
        }
        for i in range(min(limit, 8))
    ]


def _mock_finviz_elite_stocks(industry, limit, is_bearish):
    """Return fake stocks for FINVIZ elite."""
    return [
        {
            "symbol": f"E{industry[:2].upper()}{i}",
            "source": "finviz_elite",
            "market_cap": (10 - i) * 2_000_000_000,
            "change": 8.0 + i,
            "volume": 2_000_000,
            "matched_industries": [industry],
            "reasons": [f"Elite top in {industry}"],
        }
        for i in range(min(limit, 8))
    ]


def _mock_etf_holdings(etf, limit):
    """Return fake ETF holdings."""
    return [
        {
            "symbol": f"ETF{etf}{i}",
            "source": "etf_holdings",
            "market_cap": (5 - i) * 500_000_000,
            "change": None,
            "volume": None,
            "matched_industries": [],
            "reasons": [f"Held by {etf}"],
        }
        for i in range(min(limit, 5))
    ]


class TestSelectStocks:
    def test_finviz_elite_priority(self):
        """finviz_mode=elite + key => Elite is used."""
        sel = RepresentativeStockSelector(
            finviz_elite_key="test_key",
            finviz_mode="elite",
        )
        with (
            patch.object(sel, "_fetch_finviz_elite", side_effect=_mock_finviz_elite_stocks),
            patch.object(sel, "_fetch_finviz_public", side_effect=_mock_finviz_public_stocks),
            patch.object(sel, "_rate_limit"),
        ):
            theme = {
                "direction": "bullish",
                "matching_industries": [{"name": "Gold"}],
                "proxy_etfs": [],
                "static_stocks": [],
            }
            result = sel.select_stocks(theme, max_stocks=5)
            assert len(result) > 0
            assert all(d["source"] == "finviz_elite" for d in result)

    def test_finviz_mode_public_ignores_elite_key(self):
        """finviz_mode=public + key => Public is used, not Elite."""
        sel = RepresentativeStockSelector(
            finviz_elite_key="test_key",
            finviz_mode="public",
        )
        with (
            patch.object(
                sel, "_fetch_finviz_elite", side_effect=_mock_finviz_elite_stocks
            ) as elite_mock,
            patch.object(sel, "_fetch_finviz_public", side_effect=_mock_finviz_public_stocks),
            patch.object(sel, "_rate_limit"),
        ):
            theme = {
                "direction": "bullish",
                "matching_industries": [{"name": "Gold"}],
                "proxy_etfs": [],
                "static_stocks": [],
            }
            result = sel.select_stocks(theme, max_stocks=5)
            elite_mock.assert_not_called()
            assert all(d["source"] == "finviz_public" for d in result)

    def test_finviz_public_fallback(self):
        """No elite key => public screener."""
        sel = RepresentativeStockSelector()
        with (
            patch.object(sel, "_fetch_finviz_public", side_effect=_mock_finviz_public_stocks),
            patch.object(sel, "_rate_limit"),
        ):
            theme = {
                "direction": "bullish",
                "matching_industries": [{"name": "Gold"}],
                "proxy_etfs": [],
                "static_stocks": [],
            }
            result = sel.select_stocks(theme, max_stocks=5)
            assert len(result) > 0
            assert all(d["source"] == "finviz_public" for d in result)

    def test_etf_holdings_supplement(self):
        """FINVIZ returns few stocks => ETF holdings supplement."""
        sel = RepresentativeStockSelector(fmp_api_key="test_key")

        def empty_finviz(industry, limit, is_bearish):
            return []

        with (
            patch.object(sel, "_fetch_finviz_public", side_effect=empty_finviz),
            patch.object(sel, "_fetch_etf_holdings", side_effect=_mock_etf_holdings),
            patch.object(sel, "_rate_limit"),
        ):
            theme = {
                "direction": "bullish",
                "matching_industries": [{"name": "Gold"}],
                "proxy_etfs": ["GDX", "GLD"],
                "static_stocks": [],
            }
            result = sel.select_stocks(theme, max_stocks=5)
            assert len(result) > 0
            assert any(d["source"] == "etf_holdings" for d in result)

    def test_static_final_fallback(self):
        """All sources fail => static_stocks."""
        sel = RepresentativeStockSelector()

        def fail_finviz(industry, limit, is_bearish):
            return []

        with (
            patch.object(sel, "_fetch_finviz_public", side_effect=fail_finviz),
            patch.object(sel, "_rate_limit"),
        ):
            theme = {
                "direction": "bullish",
                "matching_industries": [{"name": "Gold"}],
                "proxy_etfs": [],
                "static_stocks": ["NEM", "GOLD", "AEM"],
            }
            result = sel.select_stocks(theme, max_stocks=5)
            assert len(result) == 3
            assert all(d["source"] == "static" for d in result)
            assert [d["symbol"] for d in result] == ["NEM", "GOLD", "AEM"]

    def test_vertical_theme_gets_stocks(self):
        """Vertical theme (static_stocks=[]) gets stocks from FINVIZ."""
        sel = RepresentativeStockSelector()
        with (
            patch.object(sel, "_fetch_finviz_public", side_effect=_mock_finviz_public_stocks),
            patch.object(sel, "_rate_limit"),
        ):
            theme = {
                "direction": "bullish",
                "matching_industries": [
                    {"name": "Gold"},
                    {"name": "Silver"},
                    {"name": "Copper"},
                ],
                "proxy_etfs": [],
                "static_stocks": [],
            }
            result = sel.select_stocks(theme, max_stocks=10)
            assert len(result) > 0

    def test_bearish_theme_uses_month_down_filter(self):
        """Bearish theme passes is_bearish=True to fetch methods."""
        sel = RepresentativeStockSelector()
        calls = []

        def track_finviz(industry, limit, is_bearish):
            calls.append(is_bearish)
            return _mock_finviz_public_stocks(industry, limit, is_bearish)

        with (
            patch.object(sel, "_fetch_finviz_public", side_effect=track_finviz),
            patch.object(sel, "_rate_limit"),
        ):
            theme = {
                "direction": "bearish",
                "matching_industries": [{"name": "Retail"}],
                "proxy_etfs": [],
                "static_stocks": [],
            }
            sel.select_stocks(theme, max_stocks=5)
            assert all(c is True for c in calls)

    def test_bearish_composite_ranks_by_abs_change(self):
        """Bearish theme: abs(change) ranks stocks by drop magnitude."""
        sel = RepresentativeStockSelector()

        def bearish_stocks(industry, limit, is_bearish):
            return [
                {
                    "symbol": "BIG_DROP",
                    "source": "finviz_public",
                    "market_cap": 10_000_000_000,
                    "change": -20.0,
                    "volume": 1_000_000,
                    "matched_industries": [industry],
                    "reasons": [],
                },
                {
                    "symbol": "SMALL_DROP",
                    "source": "finviz_public",
                    "market_cap": 10_000_000_000,
                    "change": -2.0,
                    "volume": 1_000_000,
                    "matched_industries": [industry],
                    "reasons": [],
                },
            ]

        with (
            patch.object(sel, "_fetch_finviz_public", side_effect=bearish_stocks),
            patch.object(sel, "_rate_limit"),
        ):
            theme = {
                "direction": "bearish",
                "matching_industries": [{"name": "Retail"}],
                "proxy_etfs": [],
                "static_stocks": [],
            }
            result = sel.select_stocks(theme, max_stocks=5)
            assert result[0]["symbol"] == "BIG_DROP"

    def test_max_per_industry_quota(self):
        """Each industry contributes at most max_per_industry in 1st pass.

        With max_per_industry=2 and max_stocks=4 (== 2 industries * 2),
        no 2nd pass occurs, so exactly 4 stocks are returned with each
        industry providing exactly 2.
        """
        sel = RepresentativeStockSelector(max_per_industry=2)

        def distinct_stocks(industry, limit, is_bearish):
            """Return stocks with industry-unique prefixes."""
            prefix = industry[:3].upper()
            return [
                {
                    "symbol": f"{prefix}{i}",
                    "source": "finviz_public",
                    "market_cap": (10 - i) * 1_000_000_000,
                    "change": 5.0,
                    "volume": 1_000_000,
                    "matched_industries": [industry],
                    "reasons": [],
                }
                for i in range(8)
            ]

        with (
            patch.object(sel, "_fetch_finviz_public", side_effect=distinct_stocks),
            patch.object(sel, "_rate_limit"),
        ):
            theme = {
                "direction": "bullish",
                "matching_industries": [
                    {"name": "Gold"},
                    {"name": "Silver"},
                ],
                "proxy_etfs": [],
                "static_stocks": [],
            }
            result = sel.select_stocks(theme, max_stocks=4)
            assert len(result) == 4
            # Verify each industry contributes exactly 2 (no more)
            gold_count = sum(1 for r in result if "Gold" in r.get("matched_industries", []))
            silver_count = sum(1 for r in result if "Silver" in r.get("matched_industries", []))
            assert gold_count == 2
            assert silver_count == 2

    def test_single_industry_theme_2nd_pass_fills(self):
        """Single industry theme: 2nd pass fills up to max_stocks."""
        sel = RepresentativeStockSelector(max_per_industry=4)

        def many_stocks(industry, limit, is_bearish):
            return [
                {
                    "symbol": f"S{i}",
                    "source": "finviz_public",
                    "market_cap": (20 - i) * 1_000_000_000,
                    "change": 5.0,
                    "volume": 1_000_000,
                    "matched_industries": [industry],
                    "reasons": [],
                }
                for i in range(min(limit, 15))
            ]

        with (
            patch.object(sel, "_fetch_finviz_public", side_effect=many_stocks),
            patch.object(sel, "_rate_limit"),
        ):
            theme = {
                "direction": "bullish",
                "matching_industries": [{"name": "Gold"}],
                "proxy_etfs": [],
                "static_stocks": [],
            }
            result = sel.select_stocks(theme, max_stocks=10)
            assert len(result) == 10

    def test_fetch_limit_at_least_max_stocks(self):
        """fetch_limit = max(max_stocks, max_per_industry*2)."""
        sel = RepresentativeStockSelector(max_per_industry=4)
        fetch_limits = []

        def track_limit(industry, limit, is_bearish):
            fetch_limits.append(limit)
            return []

        with (
            patch.object(sel, "_fetch_finviz_public", side_effect=track_limit),
            patch.object(sel, "_rate_limit"),
        ):
            theme = {
                "direction": "bullish",
                "matching_industries": [{"name": "Gold"}],
                "proxy_etfs": [],
                "static_stocks": ["A"],
            }
            sel.select_stocks(theme, max_stocks=10)
            assert all(fl >= 10 for fl in fetch_limits)

    def test_cache_prevents_duplicate_queries(self):
        """Same (industry, direction) only queried once."""
        sel = RepresentativeStockSelector()
        call_count = 0

        def count_calls(industry, limit, is_bearish):
            nonlocal call_count
            call_count += 1
            return _mock_finviz_public_stocks(industry, limit, is_bearish)

        with (
            patch.object(sel, "_fetch_finviz_public", side_effect=count_calls),
            patch.object(sel, "_rate_limit"),
        ):
            theme1 = {
                "direction": "bullish",
                "matching_industries": [{"name": "Gold"}],
                "proxy_etfs": [],
                "static_stocks": [],
            }
            theme2 = {
                "direction": "bullish",
                "matching_industries": [{"name": "Gold"}],
                "proxy_etfs": [],
                "static_stocks": [],
            }
            sel.select_stocks(theme1, max_stocks=5)
            sel.select_stocks(theme2, max_stocks=5)
            assert call_count == 1  # 2nd call uses cache

    def test_selector_none_uses_static(self):
        """When selector is None, static_stocks are used directly."""
        # This tests the _get_representative_stocks wrapper in theme_detector
        # but we verify the static fallback path in select_stocks
        sel = RepresentativeStockSelector()

        def fail_finviz(industry, limit, is_bearish):
            return []

        with (
            patch.object(sel, "_fetch_finviz_public", side_effect=fail_finviz),
            patch.object(sel, "_rate_limit"),
        ):
            theme = {
                "direction": "bullish",
                "matching_industries": [],
                "proxy_etfs": [],
                "static_stocks": ["A", "B", "C"],
            }
            result = sel.select_stocks(theme, max_stocks=5)
            assert [d["symbol"] for d in result] == ["A", "B", "C"]


# ---------------------------------------------------------------------------
# Circuit breaker
# ---------------------------------------------------------------------------


class TestCircuitBreaker:
    def test_consecutive_failures_disables_elite_only(self):
        """Elite 3 consecutive failures => Elite disabled, Public still active."""
        sel = RepresentativeStockSelector(
            finviz_elite_key="key",
            finviz_mode="elite",
        )
        for _ in range(_MAX_CONSECUTIVE_FAILURES):
            sel._record_failure("elite")
        assert sel._source_states["elite"].disabled is True
        assert sel._source_states["public"].disabled is False

    def test_mixed_source_failures_independent(self):
        """Elite fail -> Public success -> FMP fail: independent counters."""
        sel = RepresentativeStockSelector(
            finviz_elite_key="key",
            fmp_api_key="key",
            finviz_mode="elite",
        )
        sel._record_failure("elite")
        sel._record_success("public")
        sel._record_failure("fmp")

        assert sel._source_states["elite"].consecutive_failures == 1
        assert sel._source_states["public"].consecutive_failures == 0
        assert sel._source_states["fmp"].consecutive_failures == 1

    def test_success_resets_own_source_count(self):
        """Success resets only that source's consecutive counter."""
        sel = RepresentativeStockSelector(
            finviz_elite_key="key",
            finviz_mode="elite",
        )
        sel._record_failure("elite")
        sel._record_failure("elite")
        assert sel._source_states["elite"].consecutive_failures == 2
        sel._record_success("elite")
        assert sel._source_states["elite"].consecutive_failures == 0
        # total_failures stays
        assert sel._source_states["elite"].total_failures == 2

    def test_status_degraded_when_one_active_source_disabled(self):
        """One active source disabled => status='degraded'."""
        sel = RepresentativeStockSelector(
            finviz_elite_key="key",
            fmp_api_key="key",
            finviz_mode="elite",
        )
        for _ in range(_MAX_CONSECUTIVE_FAILURES):
            sel._record_failure("elite")
        assert sel.status == "degraded"

    def test_status_circuit_broken_when_all_active_disabled(self):
        """All active sources disabled => status='circuit_broken'."""
        sel = RepresentativeStockSelector(
            finviz_elite_key="key",
            fmp_api_key="key",
            finviz_mode="elite",
        )
        for source in ["elite", "public", "fmp"]:
            for _ in range(_MAX_CONSECUTIVE_FAILURES):
                sel._record_failure(source)
        assert sel.status == "circuit_broken"

    def test_status_active_ignores_elite_when_mode_public(self):
        """finviz_mode=public => elite not in active sources."""
        sel = RepresentativeStockSelector(
            finviz_elite_key="key",
            finviz_mode="public",
        )
        for _ in range(_MAX_CONSECUTIVE_FAILURES):
            sel._record_failure("elite")
        # elite is disabled but not in active sources
        assert sel.status == "active"

    def test_status_active_ignores_fmp_when_no_key(self):
        """fmp_api_key=None => fmp not in active sources."""
        sel = RepresentativeStockSelector()
        for _ in range(_MAX_CONSECUTIVE_FAILURES):
            sel._record_failure("fmp")
        assert sel.status == "active"


# ---------------------------------------------------------------------------
# FINVIZ Public fetch
# ---------------------------------------------------------------------------


class TestFetchFinvizPublic:
    def test_returns_stock_dicts_with_schema(self):
        """Each element has required keys."""
        sel = RepresentativeStockSelector()
        pd = pytest.importorskip("pandas")
        mock_df = pd.DataFrame(
            {
                "Ticker": ["NVDA", "AMD"],
                "Company": ["NVIDIA", "AMD Inc"],
                "Sector": ["Technology", "Technology"],
                "Industry": ["Semiconductors", "Semiconductors"],
                "Country": ["USA", "USA"],
                "Market Cap": [2_800_000_000_000, 200_000_000_000],
                "P/E": [60.0, 40.0],
                "Price": [800.0, 150.0],
                "Change": [0.05, 0.03],
                "Volume": [50_000_000, 30_000_000],
            }
        )
        with patch("representative_stock_selector.Overview") as MockOverview:
            mock_instance = MockOverview.return_value
            mock_instance.screener_view.return_value = mock_df
            with patch.object(sel, "_rate_limit"):
                result = sel._fetch_finviz_public("Semiconductors", limit=10, is_bearish=False)
        assert len(result) == 2
        for stock in result:
            assert "symbol" in stock
            assert "source" in stock
            assert stock["source"] == "finviz_public"
            assert "market_cap" in stock
            assert "matched_industries" in stock

    def test_uses_correct_filter_option_names(self):
        """filter_dict uses exact finvizfinance option names."""
        sel = RepresentativeStockSelector(min_cap="small")
        pd = pytest.importorskip("pandas")
        mock_df = pd.DataFrame(
            {
                "Ticker": ["X"],
                "Market Cap": [1_000_000_000],
                "Change": [0.01],
                "Volume": [100_000],
            }
        )
        with patch("representative_stock_selector.Overview") as MockOverview:
            mock_instance = MockOverview.return_value
            mock_instance.screener_view.return_value = mock_df
            with patch.object(sel, "_rate_limit"):
                sel._fetch_finviz_public("Gold", limit=10, is_bearish=False)
            # Verify set_filter was called with correct filter names
            call_args = mock_instance.set_filter.call_args
            filters = (
                call_args[1].get("filters_dict", {})
                if call_args[1]
                else call_args[0][0]
                if call_args[0]
                else {}
            )
            assert filters.get("Market Cap.") == "+Small (over $300mln)"
            assert filters.get("Average Volume") == "Over 100K"
            assert filters.get("Price") == "Over $10"
            assert filters.get("Performance 2") == "Month Up"

    def test_bearish_uses_month_down_filter(self):
        """Bearish => Performance 2: Month Down."""
        sel = RepresentativeStockSelector()
        pd = pytest.importorskip("pandas")
        mock_df = pd.DataFrame(
            {
                "Ticker": ["X"],
                "Market Cap": [1_000_000_000],
                "Change": [-0.05],
                "Volume": [100_000],
            }
        )
        with patch("representative_stock_selector.Overview") as MockOverview:
            mock_instance = MockOverview.return_value
            mock_instance.screener_view.return_value = mock_df
            with patch.object(sel, "_rate_limit"):
                sel._fetch_finviz_public("Retail", limit=10, is_bearish=True)
            call_args = mock_instance.set_filter.call_args
            filters = (
                call_args[1].get("filters_dict", {})
                if call_args[1]
                else call_args[0][0]
                if call_args[0]
                else {}
            )
            assert filters.get("Performance 2") == "Month Down"

    def test_rate_limiting(self):
        """Consecutive calls have rate_limit_sec delay."""
        sel = RepresentativeStockSelector(rate_limit_sec=0.1)
        pd = pytest.importorskip("pandas")
        mock_df = pd.DataFrame(
            {
                "Ticker": ["X"],
                "Market Cap": [1_000_000_000],
                "Change": [0.01],
                "Volume": [100_000],
            }
        )
        with patch("representative_stock_selector.Overview") as MockOverview:
            mock_instance = MockOverview.return_value
            mock_instance.screener_view.return_value = mock_df
            start = time.time()
            sel._fetch_finviz_public("Gold", limit=5, is_bearish=False)
            sel._fetch_finviz_public("Silver", limit=5, is_bearish=False)
            elapsed = time.time() - start
            assert elapsed >= 0.1  # At least one rate limit pause

    def test_failure_returns_empty_and_records(self):
        """Exception => empty list + failure recorded."""
        sel = RepresentativeStockSelector()
        with patch("representative_stock_selector.Overview") as MockOverview:
            mock_instance = MockOverview.return_value
            mock_instance.screener_view.side_effect = Exception("network error")
            with patch.object(sel, "_rate_limit"):
                result = sel._fetch_finviz_public("Gold", limit=10, is_bearish=False)
        assert result == []
        assert sel._source_states["public"].consecutive_failures == 1


# ---------------------------------------------------------------------------
# FINVIZ Elite fetch
# ---------------------------------------------------------------------------


class TestFetchFinvizElite:
    def test_csv_parsing(self):
        """CSV response is correctly parsed."""
        sel = RepresentativeStockSelector(
            finviz_elite_key="test_key",
            finviz_mode="elite",
        )
        csv_content = (
            "No.,Ticker,Company,Sector,Industry,Country,Market Cap,P/E,Price,Change,Volume\n"
            '1,NEM,Newmont,Basic Materials,Gold,USA,50.5B,20.5,45.30,5.20%,"1,234,567"\n'
            '2,GOLD,Barrick Gold,Basic Materials,Gold,Canada,30.2B,15.3,18.50,3.10%,"2,345,678"\n'
        )
        mock_response = MagicMock()
        mock_response.status_code = 200
        mock_response.text = csv_content
        with (
            patch("representative_stock_selector.requests.get", return_value=mock_response),
            patch.object(sel, "_rate_limit"),
        ):
            result = sel._fetch_finviz_elite("Gold", limit=10, is_bearish=False)
        assert len(result) == 2
        assert result[0]["symbol"] == "NEM"
        assert result[0]["source"] == "finviz_elite"

    @pytest.mark.parametrize(
        "min_cap,expected_code",
        [
            ("micro", "cap_microover"),
            ("small", "cap_smallover"),
            ("mid", "cap_midover"),
        ],
    )
    def test_filter_string_format(self, min_cap, expected_code):
        """Filter string contains correct cap code."""
        sel = RepresentativeStockSelector(
            finviz_elite_key="test_key",
            finviz_mode="elite",
            min_cap=min_cap,
        )
        mock_response = MagicMock()
        mock_response.status_code = 200
        mock_response.text = "No.,Ticker,Company\n"
        with (
            patch(
                "representative_stock_selector.requests.get", return_value=mock_response
            ) as mock_get,
            patch.object(sel, "_rate_limit"),
        ):
            sel._fetch_finviz_elite("Gold", limit=10, is_bearish=False)
            url = mock_get.call_args[0][0]
            assert expected_code in url

    def test_bearish_filter_uses_4wdown(self):
        """is_bearish=True => ta_perf2_4wdown in URL."""
        sel = RepresentativeStockSelector(
            finviz_elite_key="test_key",
            finviz_mode="elite",
        )
        mock_response = MagicMock()
        mock_response.status_code = 200
        mock_response.text = "No.,Ticker,Company\n"
        with (
            patch(
                "representative_stock_selector.requests.get", return_value=mock_response
            ) as mock_get,
            patch.object(sel, "_rate_limit"),
        ):
            sel._fetch_finviz_elite("Gold", limit=10, is_bearish=True)
            url = mock_get.call_args[0][0]
            assert "ta_perf2_4wdown" in url

    def test_auth_failure_returns_empty(self):
        """401/403 => empty list."""
        sel = RepresentativeStockSelector(
            finviz_elite_key="bad_key",
            finviz_mode="elite",
        )
        mock_response = MagicMock()
        mock_response.status_code = 403
        mock_response.text = "Forbidden"
        with (
            patch("representative_stock_selector.requests.get", return_value=mock_response),
            patch.object(sel, "_rate_limit"),
        ):
            result = sel._fetch_finviz_elite("Gold", limit=10, is_bearish=False)
        assert result == []
        assert sel._source_states["elite"].consecutive_failures == 1


# ---------------------------------------------------------------------------
# Properties
# ---------------------------------------------------------------------------


class TestProperties:
    def test_query_count(self):
        sel = RepresentativeStockSelector()
        sel._source_states["public"].total_queries = 5
        sel._source_states["fmp"].total_queries = 2
        assert sel.query_count == 7

    def test_failure_count(self):
        sel = RepresentativeStockSelector()
        sel._source_states["public"].total_failures = 3
        sel._source_states["elite"].total_failures = 1
        assert sel.failure_count == 4

    def test_active_sources_public_mode(self):
        sel = RepresentativeStockSelector(finviz_mode="public")
        assert sel._active_sources == ["public"]

    def test_active_sources_public_mode_with_fmp(self):
        sel = RepresentativeStockSelector(finviz_mode="public", fmp_api_key="key")
        assert sel._active_sources == ["public", "fmp"]

    def test_active_sources_elite_mode(self):
        sel = RepresentativeStockSelector(
            finviz_elite_key="key",
            finviz_mode="elite",
            fmp_api_key="key",
        )
        assert sel._active_sources == ["elite", "public", "fmp"]

    def test_active_sources_elite_mode_no_key(self):
        """elite mode but no key => elite not in active sources."""
        sel = RepresentativeStockSelector(finviz_mode="elite")
        assert "elite" not in sel._active_sources

    def test_source_states_property(self):
        sel = RepresentativeStockSelector()
        states = sel.source_states
        assert "elite" in states
        assert "public" in states
        assert "fmp" in states

"""Tests for scorer.py - Theme Detector scoring and labeling"""

from scorer import (
    calculate_confidence,
    determine_data_mode,
    get_heat_label,
    score_theme,
)

# ── get_heat_label ───────────────────────────────────────────────────


class TestGetHeatLabel:
    def test_hot(self):
        assert get_heat_label(80) == "Hot"
        assert get_heat_label(90) == "Hot"
        assert get_heat_label(100) == "Hot"

    def test_warm(self):
        assert get_heat_label(60) == "Warm"
        assert get_heat_label(70) == "Warm"
        assert get_heat_label(79.9) == "Warm"

    def test_neutral(self):
        assert get_heat_label(40) == "Neutral"
        assert get_heat_label(50) == "Neutral"
        assert get_heat_label(59.9) == "Neutral"

    def test_cool(self):
        assert get_heat_label(20) == "Cool"
        assert get_heat_label(30) == "Cool"
        assert get_heat_label(39.9) == "Cool"

    def test_cold(self):
        assert get_heat_label(0) == "Cold"
        assert get_heat_label(10) == "Cold"
        assert get_heat_label(19.9) == "Cold"


# ── calculate_confidence ─────────────────────────────────────────────


class TestCalculateConfidence:
    def test_three_layers_high(self):
        assert (
            calculate_confidence(
                quant_confirmed=True,
                breadth_confirmed=True,
                narrative_confirmed=True,
                stale_data_penalty=False,
            )
            == "High"
        )

    def test_two_layers_medium(self):
        assert (
            calculate_confidence(
                quant_confirmed=True,
                breadth_confirmed=True,
                narrative_confirmed=False,
                stale_data_penalty=False,
            )
            == "Medium"
        )

    def test_one_layer_low(self):
        assert (
            calculate_confidence(
                quant_confirmed=True,
                breadth_confirmed=False,
                narrative_confirmed=False,
                stale_data_penalty=False,
            )
            == "Low"
        )

    def test_zero_layers_low(self):
        assert (
            calculate_confidence(
                quant_confirmed=False,
                breadth_confirmed=False,
                narrative_confirmed=False,
                stale_data_penalty=False,
            )
            == "Low"
        )

    def test_stale_penalty_downgrades_high_to_medium(self):
        assert (
            calculate_confidence(
                quant_confirmed=True,
                breadth_confirmed=True,
                narrative_confirmed=True,
                stale_data_penalty=True,
            )
            == "Medium"
        )

    def test_stale_penalty_downgrades_medium_to_low(self):
        assert (
            calculate_confidence(
                quant_confirmed=True,
                breadth_confirmed=True,
                narrative_confirmed=False,
                stale_data_penalty=True,
            )
            == "Low"
        )

    def test_stale_penalty_low_stays_low(self):
        assert (
            calculate_confidence(
                quant_confirmed=True,
                breadth_confirmed=False,
                narrative_confirmed=False,
                stale_data_penalty=True,
            )
            == "Low"
        )

    def test_all_combinations_without_penalty(self):
        # 3 confirmed => High
        assert calculate_confidence(True, True, True, False) == "High"
        # 2 confirmed => Medium (multiple combos)
        assert calculate_confidence(True, True, False, False) == "Medium"
        assert calculate_confidence(True, False, True, False) == "Medium"
        assert calculate_confidence(False, True, True, False) == "Medium"
        # 1 confirmed => Low
        assert calculate_confidence(True, False, False, False) == "Low"
        assert calculate_confidence(False, True, False, False) == "Low"
        assert calculate_confidence(False, False, True, False) == "Low"


# ── determine_data_mode ──────────────────────────────────────────────


class TestDetermineDataMode:
    def test_both_available(self):
        result = determine_data_mode(fmp_available=True, finviz_elite=True)
        assert "FINVIZ" in result and "FMP" in result

    def test_fmp_only(self):
        result = determine_data_mode(fmp_available=True, finviz_elite=False)
        assert "FMP" in result

    def test_finviz_public_only(self):
        result = determine_data_mode(fmp_available=False, finviz_elite=False)
        assert "Public" in result or "public" in result.lower()

    def test_finviz_elite_no_fmp(self):
        result = determine_data_mode(fmp_available=False, finviz_elite=True)
        assert "FINVIZ" in result

    def test_returns_string(self):
        assert isinstance(determine_data_mode(True, True), str)


# ── score_theme ──────────────────────────────────────────────────────


class TestScoreTheme:
    def test_returns_complete_dict(self):
        result = score_theme(
            theme_heat=75.0,
            lifecycle_maturity=45.0,
            lifecycle_stage="Trending",
            direction="bullish",
            confidence="High",
            data_mode="FINVIZ-Elite+FMP",
        )
        assert "theme_heat" in result
        assert "heat_label" in result
        assert "lifecycle_maturity" in result
        assert "lifecycle_stage" in result
        assert "direction" in result
        assert "confidence" in result
        assert "data_mode" in result

    def test_heat_label_populated(self):
        result = score_theme(
            theme_heat=85.0,
            lifecycle_maturity=50.0,
            lifecycle_stage="Trending",
            direction="bullish",
            confidence="Medium",
            data_mode="FMP-only",
        )
        assert result["heat_label"] == "Hot"

    def test_all_fields_passed_through(self):
        result = score_theme(
            theme_heat=30.0,
            lifecycle_maturity=15.0,
            lifecycle_stage="Emerging",
            direction="bearish",
            confidence="Low",
            data_mode="FINVIZ-Public",
        )
        assert result["theme_heat"] == 30.0
        assert result["lifecycle_maturity"] == 15.0
        assert result["lifecycle_stage"] == "Emerging"
        assert result["direction"] == "bearish"
        assert result["confidence"] == "Low"
        assert result["data_mode"] == "FINVIZ-Public"

    def test_heat_label_matches_score(self):
        for heat, expected_label in [
            (5, "Cold"),
            (25, "Cool"),
            (45, "Neutral"),
            (65, "Warm"),
            (85, "Hot"),
        ]:
            result = score_theme(
                theme_heat=heat,
                lifecycle_maturity=50.0,
                lifecycle_stage="Trending",
                direction="bullish",
                confidence="Medium",
                data_mode="FMP-only",
            )
            assert result["heat_label"] == expected_label

#!/usr/bin/env python3
"""
Tests for theme_classifier module.

Covers cross-sector theme matching, vertical (single-sector) theme detection,
sector weight calculation, and edge cases.
"""

from calculators.industry_ranker import rank_industries
from calculators.theme_classifier import (
    classify_themes,
    get_matched_industry_names,
    get_theme_sector_weights,
)

# ---------------------------------------------------------------------------
# Sample themes_config for testing
# ---------------------------------------------------------------------------

SAMPLE_THEMES_CONFIG = {
    "cross_sector": [
        {
            "theme_name": "AI & Automation",
            "matching_keywords": ["Semiconductor", "Software - Application", "IT Services"],
            "proxy_etfs": ["BOTZ", "ROBO"],
            "static_stocks": ["NVDA", "MSFT"],
        },
        {
            "theme_name": "Green Energy Transition",
            "matching_keywords": ["Solar", "Utilities - Renewable", "Auto Manufacturers"],
            "proxy_etfs": ["ICLN", "TAN"],
            "static_stocks": ["ENPH", "TSLA"],
        },
        {
            "theme_name": "Infrastructure Boom",
            "matching_keywords": ["Building Materials", "Engineering & Construction", "Steel"],
            "proxy_etfs": ["PAVE", "IFRA"],
            "static_stocks": ["VMC", "NUE"],
        },
    ],
    "vertical_min_industries": 3,
    "cross_sector_min_matches": 2,
}


# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------


def _make_ranked_industry(name, weighted_return, momentum_score, direction, rank, sector=None):
    """Build a ranked industry dict (simulates output of rank_industries)."""
    entry = {
        "name": name,
        "perf_1w": 0.0,
        "perf_1m": 0.0,
        "perf_3m": 0.0,
        "perf_6m": 0.0,
        "weighted_return": weighted_return,
        "momentum_score": momentum_score,
        "direction": direction,
        "rank": rank,
    }
    if sector is not None:
        entry["sector"] = sector
    return entry


# ---------------------------------------------------------------------------
# Cross-sector theme matching
# ---------------------------------------------------------------------------


class TestCrossSectorThemes:
    """Test cross-sector theme detection (min 2 matching industries)."""

    def test_ai_theme_detected_with_two_matches(self):
        """Two matching industries from AI theme -> theme detected."""
        ranked = [
            _make_ranked_industry("Semiconductor", 15.0, 82.0, "bullish", 1, "Technology"),
            _make_ranked_industry("Software - Application", 12.0, 75.0, "bullish", 2, "Technology"),
            _make_ranked_industry(
                "Banks - Regional", 8.0, 65.0, "bullish", 3, "Financial Services"
            ),
        ]
        themes = classify_themes(ranked, SAMPLE_THEMES_CONFIG)
        theme_names = [t["theme_name"] for t in themes]
        assert "AI & Automation" in theme_names

    def test_ai_theme_not_detected_with_one_match(self):
        """Only one matching industry -> theme NOT detected."""
        ranked = [
            _make_ranked_industry("Semiconductor", 15.0, 82.0, "bullish", 1, "Technology"),
            _make_ranked_industry(
                "Banks - Regional", 8.0, 65.0, "bullish", 2, "Financial Services"
            ),
            _make_ranked_industry("Oil & Gas E&P", 5.0, 55.0, "bullish", 3, "Energy"),
        ]
        themes = classify_themes(ranked, SAMPLE_THEMES_CONFIG)
        theme_names = [t["theme_name"] for t in themes]
        assert "AI & Automation" not in theme_names

    def test_per_theme_min_matching_industries_override(self):
        """Theme-level minimum can detect a narrow one-industry theme."""
        config = {
            "cross_sector_min_matches": 2,
            "vertical_min_industries": 3,
            "cross_sector": [
                {
                    "theme_name": "Nuclear & Uranium",
                    "min_matching_industries": 1,
                    "matching_keywords": ["Uranium", "Other Industrial Metals & Mining"],
                    "proxy_etfs": ["URA"],
                    "static_stocks": ["CCJ"],
                },
                {
                    "theme_name": "Default Needs Two",
                    "matching_keywords": ["Uranium", "Gold"],
                    "proxy_etfs": [],
                    "static_stocks": [],
                },
            ],
        }
        ranked = [
            _make_ranked_industry("Uranium", 20.0, 90.0, "bullish", 1, "Energy"),
            _make_ranked_industry("Banks - Regional", 8.0, 65.0, "bullish", 2, "Financial"),
        ]
        themes = classify_themes(ranked, config)
        names = [theme["theme_name"] for theme in themes]
        assert "Nuclear & Uranium" in names
        assert "Default Needs Two" not in names

    def test_all_three_matches(self):
        """All three keywords match -> theme detected with all three matching."""
        ranked = [
            _make_ranked_industry("Semiconductor", 15.0, 82.0, "bullish", 1, "Technology"),
            _make_ranked_industry("Software - Application", 12.0, 75.0, "bullish", 2, "Technology"),
            _make_ranked_industry("IT Services", 10.0, 70.0, "bullish", 3, "Technology"),
        ]
        themes = classify_themes(ranked, SAMPLE_THEMES_CONFIG)
        ai_theme = [t for t in themes if t["theme_name"] == "AI & Automation"][0]
        assert len(ai_theme["matching_industries"]) == 3

    def test_theme_direction_bullish(self):
        """Theme direction is bullish when matching industries are bullish."""
        ranked = [
            _make_ranked_industry("Semiconductor", 15.0, 82.0, "bullish", 1, "Technology"),
            _make_ranked_industry("Software - Application", 12.0, 75.0, "bullish", 2, "Technology"),
        ]
        themes = classify_themes(ranked, SAMPLE_THEMES_CONFIG)
        ai_theme = [t for t in themes if t["theme_name"] == "AI & Automation"][0]
        assert ai_theme["direction"] == "bullish"

    def test_theme_direction_bearish(self):
        """Theme direction is bearish when matching industries are bearish."""
        ranked = [
            _make_ranked_industry("Semiconductor", -15.0, 82.0, "bearish", 1, "Technology"),
            _make_ranked_industry(
                "Software - Application", -12.0, 75.0, "bearish", 2, "Technology"
            ),
        ]
        themes = classify_themes(ranked, SAMPLE_THEMES_CONFIG)
        ai_theme = [t for t in themes if t["theme_name"] == "AI & Automation"][0]
        assert ai_theme["direction"] == "bearish"

    def test_theme_direction_mixed(self):
        """Mixed directions -> majority determines theme direction."""
        ranked = [
            _make_ranked_industry("Semiconductor", 15.0, 82.0, "bullish", 1, "Technology"),
            _make_ranked_industry("Software - Application", -5.0, 50.0, "bearish", 5, "Technology"),
            _make_ranked_industry("IT Services", 8.0, 65.0, "bullish", 3, "Technology"),
        ]
        themes = classify_themes(ranked, SAMPLE_THEMES_CONFIG)
        ai_theme = [t for t in themes if t["theme_name"] == "AI & Automation"][0]
        # 2 bullish vs 1 bearish -> bullish
        assert ai_theme["direction"] == "bullish"

    def test_proxy_etfs_and_static_stocks_included(self):
        """Theme result includes proxy_etfs and static_stocks from config."""
        ranked = [
            _make_ranked_industry("Semiconductor", 15.0, 82.0, "bullish", 1, "Technology"),
            _make_ranked_industry("Software - Application", 12.0, 75.0, "bullish", 2, "Technology"),
        ]
        themes = classify_themes(ranked, SAMPLE_THEMES_CONFIG)
        ai_theme = [t for t in themes if t["theme_name"] == "AI & Automation"][0]
        assert ai_theme["proxy_etfs"] == ["BOTZ", "ROBO"]
        assert ai_theme["static_stocks"] == ["NVDA", "MSFT"]

    def test_multiple_themes_detected(self):
        """Multiple themes can be detected simultaneously."""
        ranked = [
            _make_ranked_industry("Semiconductor", 15.0, 82.0, "bullish", 1, "Technology"),
            _make_ranked_industry("Software - Application", 12.0, 75.0, "bullish", 2, "Technology"),
            _make_ranked_industry("Solar", 18.0, 85.0, "bullish", 3, "Energy"),
            _make_ranked_industry(
                "Auto Manufacturers", 10.0, 70.0, "bullish", 4, "Consumer Cyclical"
            ),
        ]
        themes = classify_themes(ranked, SAMPLE_THEMES_CONFIG)
        theme_names = [t["theme_name"] for t in themes]
        assert "AI & Automation" in theme_names
        assert "Green Energy Transition" in theme_names


# ---------------------------------------------------------------------------
# Vertical (single-sector) theme detection
# ---------------------------------------------------------------------------


class TestVerticalThemes:
    """Test vertical theme detection (min 3 same-sector industries in top/bottom)."""

    def test_vertical_theme_detected_with_three_same_sector(self):
        """3+ industries from same sector in top ranks -> vertical theme detected."""
        ranked = [
            _make_ranked_industry("Semiconductor", 20.0, 90.0, "bullish", 1, "Technology"),
            _make_ranked_industry("Software - Application", 18.0, 85.0, "bullish", 2, "Technology"),
            _make_ranked_industry("IT Services", 15.0, 82.0, "bullish", 3, "Technology"),
            _make_ranked_industry(
                "Banks - Regional", 12.0, 75.0, "bullish", 4, "Financial Services"
            ),
            _make_ranked_industry("Oil & Gas E&P", 10.0, 70.0, "bullish", 5, "Energy"),
        ]
        themes = classify_themes(ranked, SAMPLE_THEMES_CONFIG)
        vertical_names = [t["theme_name"] for t in themes if "Sector" in t["theme_name"]]
        assert any("Technology" in n for n in vertical_names)

    def test_vertical_theme_not_detected_with_two_same_sector(self):
        """Only 2 same-sector industries -> no vertical theme."""
        ranked = [
            _make_ranked_industry("Semiconductor", 20.0, 90.0, "bullish", 1, "Technology"),
            _make_ranked_industry("Software - Application", 18.0, 85.0, "bullish", 2, "Technology"),
            _make_ranked_industry(
                "Banks - Regional", 12.0, 75.0, "bullish", 3, "Financial Services"
            ),
            _make_ranked_industry("Oil & Gas E&P", 10.0, 70.0, "bullish", 4, "Energy"),
            _make_ranked_industry("Gold", 8.0, 65.0, "bullish", 5, "Basic Materials"),
        ]
        themes = classify_themes(ranked, SAMPLE_THEMES_CONFIG)
        vertical_names = [t["theme_name"] for t in themes if "Sector" in t["theme_name"]]
        assert not any("Technology" in n for n in vertical_names)

    def test_vertical_bearish_bottom_sector(self):
        """3+ same-sector industries at bottom -> bearish vertical theme."""
        ranked = [
            _make_ranked_industry(
                "Banks - Regional", 12.0, 75.0, "bullish", 1, "Financial Services"
            ),
            _make_ranked_industry("Oil & Gas E&P", 10.0, 70.0, "bullish", 2, "Energy"),
            # Bottom 3 are all Technology bearish
            _make_ranked_industry("IT Services", -10.0, 70.0, "bearish", 3, "Technology"),
            _make_ranked_industry(
                "Software - Application", -15.0, 82.0, "bearish", 4, "Technology"
            ),
            _make_ranked_industry("Semiconductor", -20.0, 90.0, "bearish", 5, "Technology"),
        ]
        themes = classify_themes(ranked, SAMPLE_THEMES_CONFIG)
        vertical_themes = [t for t in themes if "Sector" in t["theme_name"]]
        tech_vertical = [t for t in vertical_themes if "Technology" in t["theme_name"]]
        assert len(tech_vertical) >= 1
        assert tech_vertical[0]["direction"] == "bearish"

    def test_no_sector_field_skips_vertical(self):
        """Industries without sector field don't contribute to vertical themes."""
        ranked = [
            _make_ranked_industry("Semiconductor", 20.0, 90.0, "bullish", 1),
            _make_ranked_industry("Software - Application", 18.0, 85.0, "bullish", 2),
            _make_ranked_industry("IT Services", 15.0, 82.0, "bullish", 3),
        ]
        themes = classify_themes(ranked, SAMPLE_THEMES_CONFIG)
        vertical_themes = [t for t in themes if "Sector" in t["theme_name"]]
        assert len(vertical_themes) == 0


# ---------------------------------------------------------------------------
# get_theme_sector_weights
# ---------------------------------------------------------------------------


class TestGetThemeSectorWeights:
    """Test sector weight calculation for a theme."""

    def test_single_sector(self):
        """All matching industries from one sector -> 100% weight."""
        theme = {
            "matching_industries": [
                {"name": "Semiconductor", "sector": "Technology"},
                {"name": "Software - Application", "sector": "Technology"},
            ]
        }
        weights = get_theme_sector_weights(theme)
        assert weights == {"Technology": 1.0}

    def test_two_sectors_equal(self):
        """Two sectors with equal industries -> 50/50."""
        theme = {
            "matching_industries": [
                {"name": "Semiconductor", "sector": "Technology"},
                {"name": "Solar", "sector": "Energy"},
            ]
        }
        weights = get_theme_sector_weights(theme)
        assert abs(weights["Technology"] - 0.5) < 0.01
        assert abs(weights["Energy"] - 0.5) < 0.01

    def test_three_sectors_uneven(self):
        """3 industries: 2 Tech, 1 Energy -> Tech=0.67, Energy=0.33."""
        theme = {
            "matching_industries": [
                {"name": "Semiconductor", "sector": "Technology"},
                {"name": "Software - Application", "sector": "Technology"},
                {"name": "Solar", "sector": "Energy"},
            ]
        }
        weights = get_theme_sector_weights(theme)
        assert abs(weights["Technology"] - 2 / 3) < 0.01
        assert abs(weights["Energy"] - 1 / 3) < 0.01

    def test_empty_matching_industries(self):
        """No matching industries -> empty weights."""
        theme = {"matching_industries": []}
        weights = get_theme_sector_weights(theme)
        assert weights == {}

    def test_weights_sum_to_one(self):
        """Sector weights always sum to 1.0."""
        theme = {
            "matching_industries": [
                {"name": "A", "sector": "Tech"},
                {"name": "B", "sector": "Energy"},
                {"name": "C", "sector": "Tech"},
                {"name": "D", "sector": "Healthcare"},
            ]
        }
        weights = get_theme_sector_weights(theme)
        total = sum(weights.values())
        assert abs(total - 1.0) < 0.001

    def test_missing_sector_field_skipped(self):
        """Industries without sector field are excluded from weight calc."""
        theme = {
            "matching_industries": [
                {"name": "A", "sector": "Tech"},
                {"name": "B"},  # no sector
            ]
        }
        weights = get_theme_sector_weights(theme)
        assert weights == {"Tech": 1.0}


# ---------------------------------------------------------------------------
# Edge cases
# ---------------------------------------------------------------------------


class TestClassifyThemesEdgeCases:
    """Edge cases for classify_themes."""

    def test_empty_ranked_list(self):
        """Empty ranked list -> no themes detected."""
        themes = classify_themes([], SAMPLE_THEMES_CONFIG)
        assert themes == []

    def test_empty_themes_config(self):
        """Empty config -> no cross-sector themes but vertical may still detect."""
        ranked = [
            _make_ranked_industry("Semiconductor", 20.0, 90.0, "bullish", 1, "Technology"),
            _make_ranked_industry("Software - Application", 18.0, 85.0, "bullish", 2, "Technology"),
            _make_ranked_industry("IT Services", 15.0, 82.0, "bullish", 3, "Technology"),
        ]
        config = {"cross_sector": [], "vertical_min_industries": 3, "cross_sector_min_matches": 2}
        themes = classify_themes(ranked, config)
        # Only vertical themes possible
        theme_names = [t["theme_name"] for t in themes]
        assert any("Technology" in n for n in theme_names)

    def test_theme_result_has_sector_weights(self):
        """Each theme result includes sector_weights dict."""
        ranked = [
            _make_ranked_industry("Semiconductor", 15.0, 82.0, "bullish", 1, "Technology"),
            _make_ranked_industry("Software - Application", 12.0, 75.0, "bullish", 2, "Technology"),
        ]
        themes = classify_themes(ranked, SAMPLE_THEMES_CONFIG)
        ai_theme = [t for t in themes if t["theme_name"] == "AI & Automation"][0]
        assert "sector_weights" in ai_theme
        assert isinstance(ai_theme["sector_weights"], dict)

    def test_integration_with_rank_industries(self):
        """Test full pipeline: raw industries -> ranked -> classified."""
        raw_industries = [
            {
                "name": "Semiconductor",
                "perf_1w": 5,
                "perf_1m": 10,
                "perf_3m": 20,
                "perf_6m": 25,
                "sector": "Technology",
            },
            {
                "name": "Software - Application",
                "perf_1w": 4,
                "perf_1m": 8,
                "perf_3m": 15,
                "perf_6m": 20,
                "sector": "Technology",
            },
            {
                "name": "Banks - Regional",
                "perf_1w": 2,
                "perf_1m": 5,
                "perf_3m": 8,
                "perf_6m": 10,
                "sector": "Financial Services",
            },
        ]
        ranked = rank_industries(raw_industries)
        themes = classify_themes(ranked, SAMPLE_THEMES_CONFIG)
        theme_names = [t["theme_name"] for t in themes]
        assert "AI & Automation" in theme_names


# ---------------------------------------------------------------------------
# top_n filtering
# ---------------------------------------------------------------------------


class TestTopNFiltering:
    """Test that classify_themes only considers top/bottom N industries."""

    def test_middle_industries_excluded_with_small_top_n(self):
        """With 100 industries and top_n=5, middle industries are excluded."""
        # Build 100 industries: top 5 include AI keywords, bottom 5 include Green keywords
        # Middle 90 include Infrastructure keywords that should NOT match
        ranked = []

        # Top 5 (highest momentum)
        ranked.append(
            _make_ranked_industry("Semiconductor", 20.0, 95.0, "bullish", 1, "Technology")
        )
        ranked.append(
            _make_ranked_industry("Software - Application", 18.0, 90.0, "bullish", 2, "Technology")
        )
        ranked.append(_make_ranked_industry("IT Services", 16.0, 88.0, "bullish", 3, "Technology"))
        ranked.append(
            _make_ranked_industry("Banks - Diversified", 15.0, 85.0, "bullish", 4, "Financial")
        )
        ranked.append(_make_ranked_industry("Gold", 14.0, 82.0, "bullish", 5, "Basic Materials"))

        # Middle 90: include Building Materials and Steel (Infrastructure keywords)
        for i in range(6, 96):
            if i == 50:
                ranked.append(
                    _make_ranked_industry(
                        "Building Materials", 0.0, 50.0, "bullish", i, "Industrials"
                    )
                )
            elif i == 51:
                ranked.append(
                    _make_ranked_industry("Steel", -0.1, 49.0, "bearish", i, "Basic Materials")
                )
            elif i == 52:
                ranked.append(
                    _make_ranked_industry(
                        "Engineering & Construction", 0.1, 50.5, "bullish", i, "Industrials"
                    )
                )
            else:
                ranked.append(
                    _make_ranked_industry(
                        f"Industry_{i}", 10.0 - i * 0.2, 80.0 - i * 0.3, "bullish", i, "Other"
                    )
                )

        # Bottom 5 (lowest momentum)
        ranked.append(_make_ranked_industry("Solar", -18.0, 10.0, "bearish", 96, "Energy"))
        ranked.append(
            _make_ranked_industry(
                "Auto Manufacturers", -19.0, 8.0, "bearish", 97, "Consumer Cyclical"
            )
        )
        ranked.append(
            _make_ranked_industry("Utilities - Renewable", -20.0, 5.0, "bearish", 98, "Utilities")
        )
        ranked.append(
            _make_ranked_industry("Retail_A", -21.0, 4.0, "bearish", 99, "Consumer Cyclical")
        )
        ranked.append(
            _make_ranked_industry("Retail_B", -22.0, 3.0, "bearish", 100, "Consumer Cyclical")
        )

        themes = classify_themes(ranked, SAMPLE_THEMES_CONFIG, top_n=5)
        theme_names = [t["theme_name"] for t in themes]

        # AI keywords (Semiconductor, Software - App) are in top 5 -> AI detected
        assert "AI & Automation" in theme_names

        # Green keywords (Solar, Auto Manufacturers) are in bottom 5 -> Green detected
        assert "Green Energy Transition" in theme_names

        # Infrastructure keywords (Building Materials, Steel, E&C) are in middle -> NOT detected
        assert "Infrastructure Boom" not in theme_names

    def test_top_n_default_30(self):
        """Default top_n=30 works for small datasets (< 30 items)."""
        ranked = [
            _make_ranked_industry("Semiconductor", 15.0, 82.0, "bullish", 1, "Technology"),
            _make_ranked_industry("Software - Application", 12.0, 75.0, "bullish", 2, "Technology"),
        ]
        # Default top_n=30, with only 2 items, all should be considered
        themes = classify_themes(ranked, SAMPLE_THEMES_CONFIG)
        theme_names = [t["theme_name"] for t in themes]
        assert "AI & Automation" in theme_names

    def test_vertical_theme_only_in_top_bottom(self):
        """Vertical themes only detected from top/bottom N, not middle."""
        ranked = []
        # Top 3: mixed sectors
        ranked.append(_make_ranked_industry("Gold", 20.0, 95.0, "bullish", 1, "Basic Materials"))
        ranked.append(
            _make_ranked_industry("Banks - Regional", 18.0, 90.0, "bullish", 2, "Financial")
        )
        ranked.append(_make_ranked_industry("Solar", 16.0, 88.0, "bullish", 3, "Energy"))

        # Middle: 3 Technology industries (would trigger vertical if not filtered)
        for i, name in enumerate(["Semiconductor", "Software - Application", "IT Services"], 4):
            ranked.append(_make_ranked_industry(name, 5.0, 50.0, "bullish", i, "Technology"))

        # Bottom 3: mixed sectors
        ranked.append(_make_ranked_industry("Oil & Gas E&P", -15.0, 10.0, "bearish", 7, "Energy"))
        ranked.append(
            _make_ranked_industry("Retail_A", -18.0, 8.0, "bearish", 8, "Consumer Cyclical")
        )
        ranked.append(
            _make_ranked_industry("Retail_B", -20.0, 5.0, "bearish", 9, "Consumer Cyclical")
        )

        themes = classify_themes(ranked, SAMPLE_THEMES_CONFIG, top_n=3)
        vertical_names = [t["theme_name"] for t in themes if "Sector" in t["theme_name"]]
        # Technology industries are in middle -> no Technology vertical theme
        assert not any("Technology" in n for n in vertical_names)


# ---------------------------------------------------------------------------
# get_matched_industry_names
# ---------------------------------------------------------------------------


class TestGetMatchedIndustryNames:
    """Test extraction of matched industry names from classified themes."""

    def test_returns_matched_names(self):
        themes = [
            {
                "theme_name": "AI",
                "matching_industries": [
                    {"name": "Semiconductors"},
                    {"name": "Software - Application"},
                ],
            },
        ]
        names = get_matched_industry_names(themes)
        assert names == {"Semiconductors", "Software - Application"}

    def test_multiple_themes_combined(self):
        themes = [
            {"theme_name": "A", "matching_industries": [{"name": "X"}]},
            {"theme_name": "B", "matching_industries": [{"name": "Y"}, {"name": "Z"}]},
        ]
        names = get_matched_industry_names(themes)
        assert names == {"X", "Y", "Z"}

    def test_empty_themes_returns_empty_set(self):
        assert get_matched_industry_names([]) == set()

    def test_no_matching_industries_key(self):
        themes = [{"theme_name": "A"}]
        assert get_matched_industry_names(themes) == set()

    def test_deduplicates_across_themes(self):
        themes = [
            {"theme_name": "A", "matching_industries": [{"name": "X"}, {"name": "Y"}]},
            {"theme_name": "B", "matching_industries": [{"name": "Y"}, {"name": "Z"}]},
        ]
        names = get_matched_industry_names(themes)
        assert names == {"X", "Y", "Z"}


# ---------------------------------------------------------------------------
# theme_origin and name_confidence
# ---------------------------------------------------------------------------


class TestThemeOrigin:
    """Test that theme_origin and name_confidence are set correctly."""

    def test_cross_sector_has_seed_origin(self):
        ranked = [
            _make_ranked_industry("Semiconductor", 15.0, 82.0, "bullish", 1, "Technology"),
            _make_ranked_industry("Software - Application", 12.0, 75.0, "bullish", 2, "Technology"),
        ]
        themes = classify_themes(ranked, SAMPLE_THEMES_CONFIG)
        ai_theme = [t for t in themes if t["theme_name"] == "AI & Automation"][0]
        assert ai_theme["theme_origin"] == "seed"
        assert ai_theme["name_confidence"] == "high"

    def test_vertical_has_vertical_origin(self):
        ranked = [
            _make_ranked_industry("Semiconductor", 20.0, 90.0, "bullish", 1, "Technology"),
            _make_ranked_industry("Software - Application", 18.0, 85.0, "bullish", 2, "Technology"),
            _make_ranked_industry("IT Services", 15.0, 82.0, "bullish", 3, "Technology"),
        ]
        themes = classify_themes(ranked, SAMPLE_THEMES_CONFIG)
        vertical = [t for t in themes if "Sector" in t["theme_name"]]
        assert len(vertical) >= 1
        assert vertical[0]["theme_origin"] == "vertical"
        assert vertical[0]["name_confidence"] == "high"

"""Minimal E2E test for the Theme Detector pipeline.

Tests the full flow: raw industries -> rank -> classify -> score -> report
without network calls (all I/O is mocked).
"""

import json
import sys
from datetime import date

from calculators.heat_calculator import (
    breadth_signal_score,
    calculate_theme_heat,
    momentum_strength_score,
)

# Import the pieces of the pipeline
from calculators.industry_ranker import get_top_bottom_industries, rank_industries
from calculators.lifecycle_calculator import (
    calculate_lifecycle_maturity,
    classify_stage,
    estimate_duration_score,
)
from calculators.theme_classifier import classify_themes, get_matched_industry_names
from calculators.theme_discoverer import discover_themes
from config_loader import load_themes_config
from report_generator import generate_json_report, generate_markdown_report
from scorer import calculate_confidence, determine_data_mode, score_theme

# Minimal themes config for E2E
E2E_THEMES_CONFIG = {
    "cross_sector_min_matches": 2,
    "vertical_min_industries": 3,
    "cross_sector": [
        {
            "theme_name": "AI & Semiconductors",
            "matching_keywords": [
                "Semiconductors",
                "Software - Application",
                "Software - Infrastructure",
            ],
            "proxy_etfs": ["SMH", "SOXX"],
            "static_stocks": ["NVDA", "AVGO", "AMD"],
        },
    ],
}


def _make_raw_industry(name, sector, perf_1w, perf_1m, perf_3m, perf_6m=None):
    """Build a raw industry dict (simulates FINVIZ output, already in pct)."""
    return {
        "name": name,
        "sector": sector,
        "perf_1w": perf_1w,
        "perf_1m": perf_1m,
        "perf_3m": perf_3m,
        "perf_6m": perf_6m or perf_3m * 1.2,
    }


class TestThemeDetectorE2E:
    """Full pipeline integration test (no network)."""

    def test_full_pipeline_produces_valid_report(self):
        """Raw industries -> ranked -> classified -> scored -> JSON+Markdown."""
        # Step 1: Raw industry data (already in percent)
        raw = [
            _make_raw_industry("Semiconductors", "Technology", 5.0, 12.0, 25.0),
            _make_raw_industry("Software - Application", "Technology", 4.0, 10.0, 20.0),
            _make_raw_industry("Software - Infrastructure", "Technology", 3.5, 9.0, 18.0),
            _make_raw_industry("Banks - Diversified", "Financial", 2.0, 5.0, 8.0),
            _make_raw_industry("Oil & Gas E&P", "Energy", -1.0, -3.0, -5.0),
            _make_raw_industry("Department Stores", "Consumer Cyclical", -4.0, -10.0, -15.0),
        ]

        # Step 2: Rank
        ranked = rank_industries(raw)
        assert len(ranked) == 6
        assert ranked[0]["name"] == "Semiconductors"  # highest momentum
        assert ranked[0]["direction"] == "bullish"

        industry_rankings = get_top_bottom_industries(ranked, n=3)
        assert len(industry_rankings["top"]) == 3
        assert len(industry_rankings["bottom"]) == 3

        # Step 3: Classify
        themes = classify_themes(ranked, E2E_THEMES_CONFIG, top_n=30)
        assert len(themes) >= 1
        ai_theme = [t for t in themes if t["theme_name"] == "AI & Semiconductors"]
        assert len(ai_theme) == 1
        assert ai_theme[0]["direction"] == "bullish"
        assert len(ai_theme[0]["matching_industries"]) == 3

        # Step 4: Score (simplified - just for AI theme)
        theme = ai_theme[0]
        theme_wr = sum(ind.get("weighted_return", 0) for ind in theme["matching_industries"]) / len(
            theme["matching_industries"]
        )

        momentum = momentum_strength_score(theme_wr)
        volume = None  # no ETF data in E2E
        uptrend = None  # no uptrend data in E2E
        breadth = breadth_signal_score(1.0)  # all bullish -> ratio 1.0
        heat = calculate_theme_heat(momentum, volume, uptrend, breadth)
        assert 0 <= heat <= 100

        duration = estimate_duration_score(12.0, 25.0, 30.0, None, False)
        maturity = calculate_lifecycle_maturity(duration, 50, 50, 50, 30)
        stage = classify_stage(maturity)
        confidence = calculate_confidence(True, False, False, False)
        data_mode = determine_data_mode(False, False)

        score = score_theme(
            round(heat, 2), round(maturity, 2), stage, "bullish", confidence, data_mode
        )

        scored_theme = {
            "name": "AI & Semiconductors",
            "direction": "bullish",
            "heat": round(heat, 2),
            "maturity": round(maturity, 2),
            "stage": stage,
            "confidence": confidence,
            "heat_label": score["heat_label"],
            "heat_breakdown": {"momentum_strength": round(momentum, 2)},
            "maturity_breakdown": {"duration_estimate": round(duration, 2)},
            "representative_stocks": ["NVDA", "AVGO", "AMD"],
            "proxy_etfs": ["SMH", "SOXX"],
            "industries": ["Semiconductors", "Software - Application", "Software - Infrastructure"],
            "sector_weights": {"Technology": 1.0},
        }

        # Step 5: Generate reports
        metadata = {
            "generated_at": "2026-02-16 09:00:00",
            "data_mode": data_mode,
            "data_sources": {
                "scan_hits": {"path": "scan_hits.json", "rows": 1, "hits": 1, "skipped_rows": 0}
            },
        }
        json_report = generate_json_report([scored_theme], industry_rankings, {}, metadata)

        # Verify JSON structure
        assert json_report["report_type"] == "theme_detector"
        assert json_report["summary"]["total_themes"] == 1
        assert json_report["summary"]["bullish_count"] == 1
        assert json_report["summary"]["bearish_count"] == 0
        assert json_report["summary"]["top_bullish"] == "AI & Semiconductors"

        # Verify JSON is serializable
        serialized = json.dumps(json_report, default=str)
        assert len(serialized) > 100

        # Step 6: Generate Markdown
        md_report = generate_markdown_report(json_report, top_n_detail=3)
        assert "# Theme Detector Report" in md_report
        assert "AI & Semiconductors" in md_report
        assert "LEAD" in md_report
        assert "NVDA" in md_report
        assert "## 1. Theme Dashboard" in md_report
        assert "## 2. What Changed Today" in md_report
        assert "## 3. Leadership Evidence" in md_report
        assert "## 9. Methodology & Data Quality" in md_report

        # Verify perf values are reasonable (not multiplied by 100 again)
        assert "500.0%" not in md_report
        assert "1200.0%" not in md_report

    def test_dynamic_stocks_with_mock_selector(self):
        """Dynamic stock selection produces stock_details in scored_theme."""
        from unittest.mock import MagicMock

        from theme_detector import _get_representative_stocks

        # Mock selector
        mock_selector = MagicMock()
        mock_selector.select_stocks.return_value = [
            {
                "symbol": "NEM",
                "source": "finviz_public",
                "market_cap": 50_000_000_000,
                "matched_industries": ["Gold"],
                "reasons": ["Public screener: Gold"],
                "composite_score": 0.95,
            },
            {
                "symbol": "GOLD",
                "source": "finviz_public",
                "market_cap": 30_000_000_000,
                "matched_industries": ["Gold"],
                "reasons": ["Public screener: Gold"],
                "composite_score": 0.85,
            },
        ]

        theme = {
            "theme_name": "Gold & Precious Metals",
            "direction": "bullish",
            "matching_industries": [{"name": "Gold"}],
            "proxy_etfs": ["GDX"],
            "static_stocks": ["NEM", "GOLD", "AEM"],
        }

        tickers, details = _get_representative_stocks(theme, mock_selector, max_stocks=10)
        assert tickers == ["NEM", "GOLD"]
        assert len(details) == 2
        assert details[0]["source"] == "finviz_public"
        assert details[0]["composite_score"] == 0.95

    def test_static_fallback_without_selector(self):
        """Without selector, static_stocks are returned."""
        from theme_detector import _get_representative_stocks

        theme = {
            "theme_name": "Test",
            "direction": "bullish",
            "matching_industries": [],
            "proxy_etfs": [],
            "static_stocks": ["A", "B", "C"],
        }

        tickers, details = _get_representative_stocks(theme, None, max_stocks=10)
        assert tickers == ["A", "B", "C"]
        assert all(d["source"] == "static" for d in details)

    def test_config_loader_produces_same_themes(self):
        """Config loaded from YAML produces same themes as before."""
        raw = [
            _make_raw_industry("Semiconductors", "Technology", 5.0, 12.0, 25.0),
            _make_raw_industry("Software - Application", "Technology", 4.0, 10.0, 20.0),
            _make_raw_industry("Software - Infrastructure", "Technology", 3.5, 9.0, 18.0),
            _make_raw_industry("Banks - Diversified", "Financial", 2.0, 5.0, 8.0),
        ]
        ranked = rank_industries(raw)

        # Load config from YAML
        config, catalog = load_themes_config()
        themes = classify_themes(ranked, config, top_n=30)

        theme_names = [t["theme_name"] for t in themes]
        assert "AI & Semiconductors" in theme_names
        # Should have theme_origin set
        ai = [t for t in themes if t["theme_name"] == "AI & Semiconductors"][0]
        assert ai["theme_origin"] == "seed"
        assert ai["name_confidence"] == "high"

    def test_cli_scan_hits_history_no_update(self, monkeypatch, tmp_path, capsys):
        """CLI run emits leadership/history sections without mutating history."""
        import etf_scanner
        import finviz_performance_client
        import theme_detector
        import uptrend_client

        raw = [
            _make_raw_industry("Semiconductors", "Technology", 0.05, 0.12, 0.25),
            _make_raw_industry("Software - Application", "Technology", 0.04, 0.10, 0.20),
            _make_raw_industry("Software - Infrastructure", "Technology", 0.035, 0.09, 0.18),
            _make_raw_industry("Banks - Diversified", "Financial", 0.01, 0.02, 0.03),
        ]
        monkeypatch.setattr(finviz_performance_client, "get_industry_performance", lambda: raw)
        monkeypatch.setattr(
            finviz_performance_client, "cap_outlier_performances", lambda rows: rows
        )
        monkeypatch.setattr(
            uptrend_client,
            "fetch_sector_uptrend_data",
            lambda **_kwargs: {
                "Technology": {
                    "ratio": 0.55,
                    "ma_10": 0.45,
                    "slope": 0.01,
                    "trend": "up",
                    "latest_date": "2026-07-04",
                }
            },
        )
        monkeypatch.setattr(uptrend_client, "is_data_stale", lambda *_args, **_kwargs: False)
        monkeypatch.setattr(theme_detector, "_live_market_date", lambda: date(2026, 7, 4))

        class FakeScanner:
            def __init__(self, *args, **kwargs):
                pass

            def batch_stock_metrics(self, symbols):
                return [
                    {
                        "symbol": sym,
                        "rsi_14": 72,
                        "dist_from_52w_high": 0.02,
                        "dist_from_52w_low": 0.40,
                        "pe_ratio": 40,
                    }
                    for sym in symbols
                ]

            def batch_etf_volume_ratios(self, symbols):
                return {sym: {"symbol": sym, "vol_20d": 200.0, "vol_60d": 100.0} for sym in symbols}

            def backend_stats(self):
                return {
                    "fmp_calls": 0,
                    "fmp_failures": 0,
                    "yf_calls": 0,
                    "yf_fallbacks": 0,
                    "stock": {},
                    "etf": {},
                }

        monkeypatch.setattr(etf_scanner, "ETFScanner", FakeScanner)

        scan_hits = tmp_path / "scan_hits.json"
        scan_hits.write_text(
            json.dumps(
                [
                    {
                        "symbol": "NVDA",
                        "return_5d": 25,
                        "change_pct": 6,
                        "volume": 12_000_000,
                        "relative_volume": 2.4,
                        "atr_expansion": 1.8,
                        "close_location": 0.85,
                        "industry": "Semiconductors",
                    }
                ]
            ),
            encoding="utf-8",
        )
        history_file = tmp_path / "history.json"
        history_payload = {
            "version": 1,
            "themes": {
                "AI & Semiconductors": [
                    {
                        "date": "2026-07-03",
                        "theme": "AI & Semiconductors",
                        "heat": 45,
                        "leadership_counts": {"ep9m": 0},
                    }
                ]
            },
        }
        history_file.write_text(json.dumps(history_payload), encoding="utf-8")

        monkeypatch.setattr(
            sys,
            "argv",
            [
                "theme_detector.py",
                "--output-dir",
                str(tmp_path),
                "--scan-hits",
                str(scan_hits),
                "--history-file",
                str(history_file),
                "--no-history-update",
                "--as-of-date",
                "2026-07-04",
                "--max-themes",
                "1",
            ],
        )

        theme_detector.main()
        captured = capsys.readouterr()
        report = json.loads(captured.out)
        top_theme = report["themes"]["all"][0]
        assert top_theme["leadership_score"] is not None
        assert top_theme["leadership_counts"]["ep9m"] == 1
        assert top_theme["history_metrics"]["heat_delta_1d"] is not None
        assert report["what_changed_today"]
        assert report["leadership_evidence"][0]["ep9m_count"] == 1

        markdown = next(tmp_path.glob("theme_detector_*.md")).read_text()
        assert "## 2. What Changed Today" in markdown
        assert "## 3. Leadership Evidence" in markdown
        assert json.loads(history_file.read_text()) == history_payload

    def test_historical_live_as_of_fails_before_provider_access(self, capsys):
        import theme_detector

        rc = theme_detector.main(["--as-of-date", "2000-01-03"])

        captured = capsys.readouterr()
        assert rc == 2
        assert "not point-in-time" in captured.err
        assert captured.out == ""

    def test_discover_path_finds_unmatched_clusters(self):
        """Discover path finds themes from unmatched industries."""
        raw = [
            _make_raw_industry("Semiconductors", "Technology", 5.0, 12.0, 25.0),
            _make_raw_industry("Software - Application", "Technology", 4.0, 10.0, 20.0),
            # Unmatched industries that should cluster
            _make_raw_industry("Gold", "Basic Materials", 8.0, 15.0, 30.0),
            _make_raw_industry("Silver", "Basic Materials", 7.5, 14.0, 28.0),
            _make_raw_industry(
                "Other Precious Metals & Mining", "Basic Materials", 7.0, 13.0, 27.0
            ),
            # Filler for bottom
            _make_raw_industry("Department Stores", "Consumer Cyclical", -4.0, -10.0, -15.0),
            _make_raw_industry("Specialty Retail", "Consumer Cyclical", -3.5, -9.0, -14.0),
        ]
        ranked = rank_industries(raw)

        # Use a minimal config that only matches Semiconductors + Software
        mini_config = {
            "cross_sector_min_matches": 2,
            "vertical_min_industries": 3,
            "cross_sector": [
                {
                    "theme_name": "AI & Semiconductors",
                    "matching_keywords": [
                        "Semiconductors",
                        "Software - Application",
                    ],
                    "proxy_etfs": ["SMH"],
                    "static_stocks": ["NVDA"],
                },
            ],
        }
        themes = classify_themes(ranked, mini_config, top_n=30)
        matched = get_matched_industry_names(themes)
        discovered = discover_themes(ranked, matched, themes, top_n=30)

        # Gold/Silver/Precious Metals should form a discovered cluster
        if discovered:
            assert discovered[0]["theme_origin"] == "discovered"
            assert discovered[0]["proxy_etfs"] == []
            assert discovered[0]["name_confidence"] == "medium"

    def test_max_themes_applied_after_discover(self):
        """max_themes limits seed+discovered combined total."""
        from theme_detector import _theme_selection_priority

        raw = [
            _make_raw_industry("Semiconductors", "Technology", 5.0, 12.0, 25.0),
            _make_raw_industry("Software - Application", "Technology", 4.0, 10.0, 20.0),
            _make_raw_industry("Software - Infrastructure", "Technology", 3.5, 9.0, 18.0),
            _make_raw_industry("Gold", "Basic Materials", 8.0, 15.0, 30.0),
            _make_raw_industry("Silver", "Basic Materials", 7.5, 14.0, 28.0),
            _make_raw_industry("Copper", "Basic Materials", 7.0, 13.0, 27.0),
            _make_raw_industry("Department Stores", "Consumer Cyclical", -4.0, -10.0, -15.0),
        ]
        ranked = rank_industries(raw)
        mini_config = {
            "cross_sector_min_matches": 2,
            "vertical_min_industries": 3,
            "cross_sector": [
                {
                    "theme_name": "AI",
                    "matching_keywords": ["Semiconductors", "Software - Application"],
                    "proxy_etfs": [],
                    "static_stocks": [],
                },
            ],
        }
        themes = classify_themes(ranked, mini_config, top_n=30)
        matched = get_matched_industry_names(themes)
        discovered = discover_themes(ranked, matched, themes, top_n=30)
        themes.extend(discovered)

        # Apply max_themes=2 with the production selection priority.
        themes.sort(key=lambda theme: _theme_selection_priority(theme, None), reverse=True)
        themes = themes[:2]
        assert len(themes) <= 2

    def test_theme_selection_priority_keeps_strong_match_evidence(self):
        """A strong stock/ETF/narrative match can survive max_themes truncation."""
        from theme_detector import _theme_selection_priority

        broad_theme = {
            "theme_name": "Broad Software",
            "matching_industries": [
                {"name": "Software - Application", "weighted_return": 18.0},
                {"name": "Software - Infrastructure", "weighted_return": 17.0},
                {"name": "Information Technology Services", "weighted_return": 16.0},
            ],
        }
        narrow_confirmed = {
            "theme_name": "Cybersecurity",
            "matching_industries": [{"name": "Software - Infrastructure", "weighted_return": 8.0}],
        }

        ranked = sorted(
            [
                (broad_theme, {"theme_match_score": 45.0, "theme_match_coverage": 0.6}),
                (narrow_confirmed, {"theme_match_score": 92.0, "theme_match_coverage": 0.9}),
            ],
            key=lambda item: _theme_selection_priority(item[0], item[1]),
            reverse=True,
        )

        assert ranked[0][0]["theme_name"] == "Cybersecurity"

    def test_stock_details_in_scored_theme_and_report(self):
        """stock_details flows through to scored_theme and into markdown."""
        raw = [
            _make_raw_industry("Semiconductors", "Technology", 5.0, 12.0, 25.0),
            _make_raw_industry("Software - Application", "Technology", 4.0, 10.0, 20.0),
            _make_raw_industry("Software - Infrastructure", "Technology", 3.5, 9.0, 18.0),
        ]

        ranked = rank_industries(raw)
        themes = classify_themes(ranked, E2E_THEMES_CONFIG, top_n=30)
        theme = themes[0]
        theme_wr = sum(ind.get("weighted_return", 0) for ind in theme["matching_industries"]) / len(
            theme["matching_industries"]
        )

        momentum = momentum_strength_score(theme_wr)
        breadth = breadth_signal_score(1.0)
        heat = calculate_theme_heat(momentum, None, None, breadth)
        duration = estimate_duration_score(12.0, 25.0, 30.0, None, False)
        maturity = calculate_lifecycle_maturity(duration, 50, 50, 50, 30)
        stage = classify_stage(maturity)
        confidence = calculate_confidence(True, False, False, False)
        data_mode = determine_data_mode(False, False)
        score = score_theme(
            round(heat, 2), round(maturity, 2), stage, "bullish", confidence, data_mode
        )

        stock_details = [
            {
                "symbol": "NVDA",
                "source": "finviz_public",
                "market_cap": 2_800_000_000_000,
                "matched_industries": ["Semiconductors"],
                "reasons": ["Public screener"],
                "composite_score": 0.95,
            },
            {
                "symbol": "AVGO",
                "source": "finviz_public",
                "market_cap": 500_000_000_000,
                "matched_industries": ["Semiconductors"],
                "reasons": ["Public screener"],
                "composite_score": 0.80,
            },
        ]

        scored_theme = {
            "name": "AI & Semiconductors",
            "direction": "bullish",
            "heat": round(heat, 2),
            "maturity": round(maturity, 2),
            "stage": stage,
            "confidence": confidence,
            "heat_label": score["heat_label"],
            "heat_breakdown": {"momentum_strength": round(momentum, 2)},
            "maturity_breakdown": {"duration_estimate": round(duration, 2)},
            "representative_stocks": ["NVDA", "AVGO"],
            "stock_details": stock_details,
            "proxy_etfs": ["SMH", "SOXX"],
            "industries": ["Semiconductors", "Software - Application", "Software - Infrastructure"],
            "sector_weights": {"Technology": 1.0},
        }

        metadata = {
            "generated_at": "2026-02-16 09:00:00",
            "data_mode": data_mode,
            "data_sources": {
                "scan_hits": {"path": "scan_hits.json", "rows": 1, "hits": 1, "skipped_rows": 0}
            },
        }
        json_report = generate_json_report([scored_theme], {"top": [], "bottom": []}, {}, metadata)

        # Verify stock_details in JSON
        theme_in_report = json_report["themes"]["bullish"][0]
        assert "stock_details" in theme_in_report
        assert len(theme_in_report["stock_details"]) == 2

        # Verify markdown has source labels
        md_report = generate_markdown_report(json_report, top_n_detail=3)
        assert "NVDA[Fp]" in md_report
        assert "AVGO[Fp]" in md_report


class TestE2EFMPPath:
    """E2E tests for FMP backend integration in theme_detector."""

    def test_scanner_receives_fmp_key(self):
        """ETFScanner constructed with fmp_api_key stores it."""
        from etf_scanner import ETFScanner

        scanner = ETFScanner(fmp_api_key="test_key_123")
        assert scanner._fmp_api_key == "test_key_123"
        scanner_no_key = ETFScanner()
        assert scanner_no_key._fmp_api_key is None

    def test_metadata_contains_scanner_stats(self):
        """After batch_stock_metrics, backend_stats is populated."""
        from unittest.mock import MagicMock, patch

        from etf_scanner import ETFScanner

        scanner = ETFScanner(fmp_api_key="test_key", rate_limit_sec=0)

        with patch("etf_scanner._requests_lib") as mock_req:
            quote_resp = MagicMock()
            quote_resp.status_code = 200
            quote_resp.json.return_value = [
                {"symbol": "NVDA", "pe": 60, "price": 800, "yearHigh": 950, "yearLow": 400},
            ]
            hist_resp = MagicMock()
            hist_resp.status_code = 200
            hist_resp.json.return_value = {
                "historicalStockList": [
                    {
                        "symbol": "NVDA",
                        "historical": [{"close": float(800 - i)} for i in range(20)],
                    },
                ]
            }
            mock_req.get.side_effect = [quote_resp, hist_resp]
            scanner.batch_stock_metrics(["NVDA"])

        stats = scanner.backend_stats()
        assert "fmp_calls" in stats
        assert "fmp_failures" in stats
        assert "yf_calls" in stats
        assert "yf_fallbacks" in stats
        assert stats["fmp_calls"] > 0

    def test_metadata_still_has_yfinance_stocks_key(self):
        """yfinance_stocks key is preserved for backward compatibility.

        Simulates the metadata construction in theme_detector.main().
        """
        from etf_scanner import ETFScanner

        scanner = ETFScanner(fmp_api_key="test_key")
        metadata = {"data_sources": {}}

        # Simulate what theme_detector does
        all_metrics = [{"symbol": "NVDA"}, {"symbol": "AVGO"}]
        metadata["data_sources"]["yfinance_stocks"] = len(all_metrics)
        scanner_stats = scanner.backend_stats()
        metadata["data_sources"]["scanner_backend"] = scanner_stats

        # Both keys exist
        assert "yfinance_stocks" in metadata["data_sources"]
        assert "scanner_backend" in metadata["data_sources"]
        assert metadata["data_sources"]["yfinance_stocks"] == 2
        assert isinstance(metadata["data_sources"]["scanner_backend"], dict)

"""Tests for theme_discoverer module.

Covers unmatched industry extraction, proximity clustering with perf vectors,
auto-naming, duplicate detection, and full discover_themes flow.
"""

from calculators.theme_discoverer import (
    _auto_name_cluster,
    _build_theme_dict,
    _cluster_by_proximity,
    _get_unmatched_industries,
    _is_duplicate_of_existing,
    _perf_vector_distance,
    discover_themes,
)

# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------


def _ind(
    name, weighted_return, direction="bullish", sector="Tech", perf_1w=0.0, perf_1m=0.0, perf_3m=0.0
):
    return {
        "name": name,
        "weighted_return": weighted_return,
        "direction": direction,
        "sector": sector,
        "perf_1w": perf_1w,
        "perf_1m": perf_1m,
        "perf_3m": perf_3m,
    }


# ---------------------------------------------------------------------------
# TestGetUnmatchedIndustries
# ---------------------------------------------------------------------------


class TestGetUnmatchedIndustries:
    def test_excludes_matched_names(self):
        ranked = [
            _ind("A", 10.0, "bullish"),
            _ind("B", 8.0, "bullish"),
            _ind("C", 6.0, "bullish"),
            _ind("D", -5.0, "bearish"),
            _ind("E", -8.0, "bearish"),
            _ind("F", -10.0, "bearish"),
        ]
        matched = {"A", "E"}
        bull, bear = _get_unmatched_industries(ranked, matched, top_n=3)
        bull_names = {i["name"] for i in bull}
        bear_names = {i["name"] for i in bear}
        assert "A" not in bull_names
        assert "E" not in bear_names
        assert "B" in bull_names
        assert "F" in bear_names

    def test_separates_bullish_and_bearish(self):
        ranked = [
            _ind("A", 10.0, "bullish"),
            _ind("B", 8.0, "bullish"),
            _ind("C", -5.0, "bearish"),
            _ind("D", -10.0, "bearish"),
        ]
        bull, bear = _get_unmatched_industries(ranked, set(), top_n=2)
        assert all(i["direction"] == "bullish" for i in bull)
        assert all(i["direction"] == "bearish" for i in bear)

    def test_respects_top_n(self):
        ranked = [_ind(f"I{i}", 20.0 - i) for i in range(20)]
        bull, bear = _get_unmatched_industries(ranked, set(), top_n=5)
        # Top 5 + bottom 5 from 20 items
        assert len(bull) + len(bear) <= 10

    def test_empty_matched_returns_all(self):
        ranked = [_ind("A", 10.0), _ind("B", -10.0)]
        bull, bear = _get_unmatched_industries(ranked, set(), top_n=30)
        assert len(bull) + len(bear) == 2


# ---------------------------------------------------------------------------
# TestClusterByProximity
# ---------------------------------------------------------------------------


class TestClusterByProximity:
    def test_adjacent_within_both_thresholds_grouped(self):
        # Use 3+ items so range is wider than pairwise diff
        industries = [
            _ind("A", 10.0, perf_1w=5.0, perf_1m=10.0, perf_3m=15.0),
            _ind("B", 11.5, perf_1w=6.0, perf_1m=11.0, perf_3m=16.0),
            _ind("C", 13.0, perf_1w=15.0, perf_1m=20.0, perf_3m=25.0),
        ]
        # A-B gap=1.5 within 3.0, C far in perf space; A&B should cluster
        clusters = _cluster_by_proximity(industries, gap_threshold=3.0, vector_threshold=0.5)
        assert len(clusters) >= 1
        cluster_names = [set(i["name"] for i in c) for c in clusters]
        assert {"A", "B"} in cluster_names or any("A" in c and "B" in c for c in cluster_names)

    def test_gap_exceeds_splits(self):
        industries = [
            _ind("A", 10.0, perf_1w=5.0, perf_1m=10.0, perf_3m=15.0),
            _ind("B", 20.0, perf_1w=6.0, perf_1m=11.0, perf_3m=16.0),
        ]
        clusters = _cluster_by_proximity(industries, gap_threshold=3.0, vector_threshold=0.5)
        # Gap is 10.0 > 3.0, so separate clusters; each < min_size so empty
        assert len(clusters) == 0

    def test_vector_distance_exceeds_splits(self):
        """Different perf patterns should prevent clustering."""
        industries = [
            _ind("A", 10.0, perf_1w=20.0, perf_1m=5.0, perf_3m=1.0),
            _ind("B", 11.0, perf_1w=1.0, perf_1m=5.0, perf_3m=20.0),
        ]
        clusters = _cluster_by_proximity(industries, gap_threshold=3.0, vector_threshold=0.3)
        # Vector distance should be high despite close weighted_return
        assert len(clusters) == 0

    def test_single_industry_not_a_cluster(self):
        industries = [_ind("A", 10.0)]
        clusters = _cluster_by_proximity(industries, gap_threshold=3.0, vector_threshold=0.5)
        assert len(clusters) == 0

    def test_all_within_thresholds_single_cluster(self):
        # Add a distant outlier to widen the range, so close items normalize well
        industries = [
            _ind("A", 10.0, perf_1w=5.0, perf_1m=10.0, perf_3m=15.0),
            _ind("B", 11.0, perf_1w=5.5, perf_1m=10.5, perf_3m=15.5),
            _ind("C", 12.0, perf_1w=6.0, perf_1m=11.0, perf_3m=16.0),
            _ind("Far", 30.0, perf_1w=50.0, perf_1m=60.0, perf_3m=70.0),
        ]
        clusters = _cluster_by_proximity(industries, gap_threshold=3.0, vector_threshold=0.5)
        # A, B, C should form a single cluster (close in both gap and perf)
        abc_cluster = [c for c in clusters if len(c) >= 3]
        assert len(abc_cluster) >= 1
        names = {i["name"] for i in abc_cluster[0]}
        assert {"A", "B", "C"}.issubset(names)

    def test_empty_input_returns_empty(self):
        clusters = _cluster_by_proximity([], gap_threshold=3.0, vector_threshold=0.5)
        assert clusters == []


# ---------------------------------------------------------------------------
# TestPerfVectorDistance
# ---------------------------------------------------------------------------


class TestPerfVectorDistance:
    def test_identical_returns_zero(self):
        a = _ind("A", 10.0, perf_1w=5.0, perf_1m=10.0, perf_3m=15.0)
        b = _ind("B", 10.0, perf_1w=5.0, perf_1m=10.0, perf_3m=15.0)
        ranges = {"perf_1w": 20.0, "perf_1m": 20.0, "perf_3m": 20.0}
        assert _perf_vector_distance(a, b, ranges) == 0.0

    def test_different_patterns_high_distance(self):
        a = _ind("A", 10.0, perf_1w=20.0, perf_1m=0.0, perf_3m=0.0)
        b = _ind("B", 10.0, perf_1w=0.0, perf_1m=0.0, perf_3m=20.0)
        ranges = {"perf_1w": 20.0, "perf_1m": 20.0, "perf_3m": 20.0}
        dist = _perf_vector_distance(a, b, ranges)
        assert dist > 1.0  # sqrt(1^2 + 0 + 1^2) = sqrt(2) ~= 1.414


# ---------------------------------------------------------------------------
# TestAutoNameCluster
# ---------------------------------------------------------------------------


class TestAutoNameCluster:
    def test_top_two_tokens(self):
        industries = [
            _ind("Oil & Gas E&P", 10.0),
            _ind("Oil & Gas Integrated", 9.0),
            _ind("Oil & Gas Midstream", 8.0),
        ]
        name = _auto_name_cluster(industries)
        assert "Oil" in name
        assert "Gas" in name

    def test_stop_words_excluded(self):
        industries = [
            _ind("General Services", 10.0),
            _ind("Specialty Services", 9.0),
        ]
        name = _auto_name_cluster(industries)
        # "General", "Specialty", "Services" are stop words
        # Should still produce a name
        assert len(name) > 0

    def test_single_token_adds_related(self):
        industries = [
            _ind("Gold", 10.0),
            _ind("Silver", 9.0),
        ]
        name = _auto_name_cluster(industries)
        assert len(name) > 0

    def test_oil_gas_example(self):
        industries = [
            _ind("Oil & Gas E&P", 10.0),
            _ind("Oil & Gas Drilling", 9.0),
        ]
        name = _auto_name_cluster(industries)
        assert "Oil" in name or "Gas" in name

    def test_metals_mining_example(self):
        industries = [
            _ind("Other Industrial Metals & Mining", 10.0),
            _ind("Steel", 9.0),
            _ind("Copper", 8.0),
        ]
        name = _auto_name_cluster(industries)
        assert len(name) > 0


# ---------------------------------------------------------------------------
# TestDiscoverThemes
# ---------------------------------------------------------------------------


class TestDiscoverThemes:
    def test_discovers_cross_sector_cluster(self):
        ranked = [
            _ind("UnmatchedA", 15.0, "bullish", "Tech", perf_1w=5.0, perf_1m=10.0, perf_3m=15.0),
            _ind(
                "UnmatchedB",
                14.0,
                "bullish",
                "Industrials",
                perf_1w=5.0,
                perf_1m=10.0,
                perf_3m=15.0,
            ),
            _ind("MatchedC", 13.0, "bullish"),
            _ind("Bottom1", -10.0, "bearish"),
            _ind("Bottom2", -11.0, "bearish"),
        ]
        matched = {"MatchedC"}
        existing = []
        discovered = discover_themes(ranked, matched, existing, top_n=5)
        assert len(discovered) >= 1

    def test_single_sector_with_no_vertical_overlap_kept(self):
        """Single-sector cluster not overlapping vertical should be kept."""
        ranked = [
            _ind(
                "Drug Manufacturers - General",
                15.0,
                "bullish",
                "Healthcare",
                perf_1w=5.0,
                perf_1m=10.0,
                perf_3m=15.0,
            ),
            _ind(
                "Drug Manufacturers - Specialty & Generic",
                14.0,
                "bullish",
                "Healthcare",
                perf_1w=5.0,
                perf_1m=10.0,
                perf_3m=15.0,
            ),
        ] + [_ind(f"Filler{i}", -10.0 - i, "bearish") for i in range(8)]
        matched = set()
        existing_vertical = [
            {
                "theme_name": "Healthcare Sector Concentration",
                "direction": "bullish",
                "matching_industries": [
                    {"name": "Medical Devices"},
                    {"name": "Healthcare Plans"},
                    {"name": "Medical Care Facilities"},
                ],
            }
        ]
        discovered = discover_themes(ranked, matched, existing_vertical, top_n=5)
        # GLP-1 like cluster (Drug Manufacturers) should not overlap Healthcare Vertical
        if discovered:
            names = {i["name"] for d in discovered for i in d.get("matching_industries", [])}
            assert "Drug Manufacturers - General" in names or len(discovered) == 0

    def test_single_sector_with_vertical_overlap_excluded(self):
        """Single-sector cluster overlapping existing vertical should be excluded."""
        ranked = [
            _ind(
                "Medical Devices",
                15.0,
                "bullish",
                "Healthcare",
                perf_1w=5.0,
                perf_1m=10.0,
                perf_3m=15.0,
            ),
            _ind(
                "Healthcare Plans",
                14.0,
                "bullish",
                "Healthcare",
                perf_1w=5.0,
                perf_1m=10.0,
                perf_3m=15.0,
            ),
        ] + [_ind(f"Filler{i}", -10.0 - i, "bearish") for i in range(8)]
        matched = set()
        existing_vertical = [
            {
                "theme_name": "Healthcare Sector Concentration",
                "direction": "bullish",
                "matching_industries": [
                    {"name": "Medical Devices"},
                    {"name": "Healthcare Plans"},
                    {"name": "Medical Care Facilities"},
                ],
            }
        ]
        discovered = discover_themes(ranked, matched, existing_vertical, top_n=5)
        # Should be excluded due to high overlap with Healthcare Vertical
        for d in discovered:
            d_names = {i["name"] for i in d.get("matching_industries", [])}
            overlap = d_names & {"Medical Devices", "Healthcare Plans", "Medical Care Facilities"}
            # If cluster is detected, it shouldn't overlap significantly
            if d.get("direction") == "bullish":
                assert len(overlap) / max(len(d_names), 1) < 0.5

    def test_small_cluster_rejected(self):
        """Cluster with only 1 industry should be rejected."""
        ranked = [
            _ind("LoneIndustry", 15.0, "bullish", "Tech", perf_1w=5.0, perf_1m=10.0, perf_3m=15.0),
            _ind("FarAway", 50.0, "bullish", "Other", perf_1w=30.0, perf_1m=40.0, perf_3m=50.0),
        ] + [_ind(f"Bottom{i}", -10.0 - i, "bearish") for i in range(8)]
        discovered = discover_themes(ranked, set(), [], top_n=5)
        # Single-industry clusters don't form (min_cluster_size=2)
        for d in discovered:
            assert len(d.get("matching_industries", [])) >= 2

    def test_sets_theme_origin_discovered(self):
        ranked = [
            _ind("A", 15.0, "bullish", "Tech", perf_1w=5.0, perf_1m=10.0, perf_3m=15.0),
            _ind("B", 14.0, "bullish", "Tech", perf_1w=5.0, perf_1m=10.0, perf_3m=15.0),
        ] + [_ind(f"Bottom{i}", -10.0 - i, "bearish") for i in range(8)]
        discovered = discover_themes(ranked, set(), [], top_n=5)
        for d in discovered:
            assert d["theme_origin"] == "discovered"

    def test_sets_proxy_etfs_empty(self):
        ranked = [
            _ind("A", 15.0, "bullish", "Tech", perf_1w=5.0, perf_1m=10.0, perf_3m=15.0),
            _ind("B", 14.0, "bullish", "Tech", perf_1w=5.0, perf_1m=10.0, perf_3m=15.0),
        ] + [_ind(f"Bottom{i}", -10.0 - i, "bearish") for i in range(8)]
        discovered = discover_themes(ranked, set(), [], top_n=5)
        for d in discovered:
            assert d["proxy_etfs"] == []

    def test_name_confidence_is_medium(self):
        ranked = [
            _ind("A", 15.0, "bullish", "Tech", perf_1w=5.0, perf_1m=10.0, perf_3m=15.0),
            _ind("B", 14.0, "bullish", "Tech", perf_1w=5.0, perf_1m=10.0, perf_3m=15.0),
        ] + [_ind(f"Bottom{i}", -10.0 - i, "bearish") for i in range(8)]
        discovered = discover_themes(ranked, set(), [], top_n=5)
        for d in discovered:
            assert d["name_confidence"] == "medium"

    def test_no_unmatched_returns_empty(self):
        ranked = [_ind("A", 10.0), _ind("B", -10.0)]
        matched = {"A", "B"}
        discovered = discover_themes(ranked, matched, [], top_n=30)
        assert discovered == []

    def test_bearish_clusters_detected(self):
        ranked = [
            _ind("TopFiller", 20.0, "bullish"),
        ] + [
            _ind(
                f"BearInd{i}",
                -10.0 - i * 0.5,
                "bearish",
                "Consumer",
                perf_1w=-3.0 - i,
                perf_1m=-5.0 - i,
                perf_3m=-8.0 - i,
            )
            for i in range(5)
        ]
        discovered = discover_themes(ranked, set(), [], top_n=5)
        bearish = [d for d in discovered if d["direction"] == "bearish"]
        if bearish:
            assert bearish[0]["direction"] == "bearish"


# ---------------------------------------------------------------------------
# TestIsDuplicateOfExisting
# ---------------------------------------------------------------------------


class TestIsDuplicateOfExisting:
    def test_high_overlap_same_direction_is_duplicate(self):
        cluster = [{"name": "A"}, {"name": "B"}]
        existing = [
            {
                "direction": "bullish",
                "matching_industries": [{"name": "A"}, {"name": "B"}, {"name": "C"}],
            }
        ]
        assert _is_duplicate_of_existing(cluster, "bullish", existing, 0.5) is True

    def test_high_overlap_different_direction_not_duplicate(self):
        cluster = [{"name": "A"}, {"name": "B"}]
        existing = [
            {
                "direction": "bearish",
                "matching_industries": [{"name": "A"}, {"name": "B"}, {"name": "C"}],
            }
        ]
        assert _is_duplicate_of_existing(cluster, "bullish", existing, 0.5) is False

    def test_low_overlap_same_direction_not_duplicate(self):
        cluster = [{"name": "X"}, {"name": "Y"}]
        existing = [
            {
                "direction": "bullish",
                "matching_industries": [{"name": "A"}, {"name": "B"}, {"name": "C"}],
            }
        ]
        assert _is_duplicate_of_existing(cluster, "bullish", existing, 0.5) is False

    def test_glp1_not_duplicate_of_healthcare_vertical(self):
        """GLP-1 (Drug Manufacturers) should not be duplicate of Healthcare Vertical."""
        cluster = [
            {"name": "Drug Manufacturers - General"},
            {"name": "Drug Manufacturers - Specialty & Generic"},
        ]
        existing = [
            {
                "direction": "bullish",
                "matching_industries": [
                    {"name": "Medical Devices"},
                    {"name": "Healthcare Plans"},
                    {"name": "Medical Care Facilities"},
                ],
            }
        ]
        assert _is_duplicate_of_existing(cluster, "bullish", existing, 0.5) is False

    def test_subset_detected_by_overlap_coefficient(self):
        """Small cluster that is a subset of a large theme should be detected.

        Jaccard alone misses this: intersection=2, union=12 → Jaccard=0.17.
        Overlap coefficient catches it: intersection=2, min(2,12)=2 → coeff=1.0.
        """
        cluster = [{"name": "A"}, {"name": "B"}]
        existing = [
            {
                "direction": "bullish",
                "matching_industries": [{"name": n} for n in "ABCDEFGHIJKL"],
            }
        ]
        # Jaccard = 2/12 = 0.17 < 0.5 → would PASS Jaccard-only check
        # Overlap coeff = 2/2 = 1.0 >= 0.5 → caught by overlap coefficient
        assert _is_duplicate_of_existing(cluster, "bullish", existing, 0.5) is True

    def test_partial_overlap_below_both_thresholds_passes(self):
        """Cluster with low overlap on both metrics should not be duplicate."""
        cluster = [{"name": "A"}, {"name": "X"}, {"name": "Y"}, {"name": "Z"}]
        existing = [
            {
                "direction": "bullish",
                "matching_industries": [{"name": n} for n in "ABCDEFGHIJ"],
            }
        ]
        # Jaccard = 1/13 = 0.077 < 0.5
        # Overlap coeff = 1/4 = 0.25 < 0.5
        assert _is_duplicate_of_existing(cluster, "bullish", existing, 0.5) is False


# ---------------------------------------------------------------------------
# TestBuildThemeDict
# ---------------------------------------------------------------------------


class TestBuildThemeDict:
    def test_has_required_fields(self):
        industries = [_ind("A", 10.0, "bullish"), _ind("B", 9.0, "bullish")]
        result = _build_theme_dict("Test Cluster", industries)
        assert result["theme_name"] == "Test Cluster"
        assert result["direction"] == "bullish"
        assert result["proxy_etfs"] == []
        assert result["static_stocks"] == []
        assert result["theme_origin"] == "discovered"
        assert result["name_confidence"] == "medium"
        assert len(result["matching_industries"]) == 2
        assert "sector_weights" in result

"""Tests for Theme Detector history metrics."""

import json

from theme_history import (
    append_observations,
    build_observation,
    compute_history_metrics,
    load_history,
    save_history,
)


def test_history_metrics_use_prior_observations_only():
    history = {
        "AI": [
            {"date": "2026-07-01", "heat": 40},
            {"date": "2026-07-02", "heat": 50},
            {"date": "2026-07-03", "heat": 60},
        ]
    }
    metrics = compute_history_metrics(history, "AI", "2026-07-04", 70)
    assert metrics["prior_observations"] == 3
    assert metrics["duration_count"] == 4
    assert metrics["duration_score"] == 40.0
    assert metrics["heat_delta_1d"] == 10


def test_current_date_record_is_not_self_referenced():
    history = {
        "AI": [
            {"date": "2026-07-03", "heat": 60},
            {"date": "2026-07-04", "heat": 99},
        ]
    }
    metrics = compute_history_metrics(history, "AI", "2026-07-04", 70)
    assert metrics["prior_observations"] == 1
    assert metrics["heat_delta_1d"] == 10


def test_save_and_load_history_roundtrip(tmp_path):
    path = tmp_path / "history.json"
    history = append_observations(
        {},
        [
            {
                "date": "2026-07-04",
                "theme": "AI",
                "heat": 70,
                "leadership_counts": {"ep9m": 2},
            }
        ],
    )
    save_history(str(path), history)
    loaded = load_history(str(path))
    assert loaded["AI"][0]["leadership_counts"]["ep9m"] == 2


def test_build_observation_contains_leadership_counts():
    observation = build_observation(
        {
            "name": "AI",
            "direction": "bullish",
            "heat": 70,
            "base_heat": 65,
            "leadership_score": 80,
            "leadership_counts": {"ep9m": 2},
        },
        "2026-07-04",
    )
    assert observation["date"] == "2026-07-04"
    assert observation["theme"] == "AI"
    assert observation["leadership_counts"] == {"ep9m": 2}


def test_load_history_accepts_flat_list(tmp_path):
    path = tmp_path / "history.json"
    path.write_text(
        json.dumps([{"date": "2026-07-04", "theme": "AI", "heat": 70}]), encoding="utf-8"
    )
    assert load_history(str(path))["AI"][0]["heat"] == 70

"""Tests for reusable theme match scoring."""

from leadership import ScanHit
from theme_matcher import calculate_theme_match, load_narrative_scores


def _theme(name, static_stocks, proxy_etfs):
    industries = [
        {"name": "Software - Application", "weighted_return": 15.0},
        {"name": "Software - Infrastructure", "weighted_return": 14.0},
    ]
    return {
        "theme_name": name,
        "matching_industries": industries,
        "matching_keyword_count": 2,
        "static_stocks": static_stocks,
        "proxy_etfs": proxy_etfs,
    }


def test_overlapping_themes_rank_by_basket_and_etf_confirmation():
    themes = [
        _theme("AI & Semiconductors", ["NVDA", "AVGO"], ["SMH"]),
        _theme("Cybersecurity", ["CRWD", "PANW"], ["CIBR"]),
        _theme("Cloud Computing & SaaS", ["CRM", "NOW"], ["SKYY"]),
    ]
    scan_hits = [
        ScanHit(date="2026-07-04", symbol="CRWD", scan_type="ep9m"),
        ScanHit(date="2026-07-04", symbol="PANW", scan_type="range_expansion"),
    ]
    etf_volume = {
        "SMH": {"vol_ratio": 1.0},
        "CIBR": {"vol_ratio": 1.8},
        "SKYY": {"vol_ratio": 0.9},
    }

    scored = [
        calculate_theme_match(theme, scan_hits, etf_volume, scan_hits_available=True)
        for theme in themes
    ]
    ranked_names = [
        name
        for name, detail in sorted(
            zip([theme["theme_name"] for theme in themes], scored),
            key=lambda item: item[1]["theme_match_score"],
            reverse=True,
        )
    ]

    assert ranked_names[0] == "Cybersecurity"
    assert scored[1]["theme_match_components"]["static_stock_hit_score"] == 80.0
    assert scored[1]["proxy_etf_confirmation"]["confirmed"] is True


def test_stock_basket_hits_exclude_industry_only_evidence():
    theme = _theme("Cybersecurity", ["CRWD", "PANW"], ["CIBR"])
    scan_hits = [
        ScanHit(
            date="2026-07-04",
            symbol="DDOG",
            scan_type="ep9m",
            industry="Software - Infrastructure",
        )
    ]

    detail = calculate_theme_match(theme, scan_hits, {}, scan_hits_available=True)

    assert detail["theme_match_components"]["static_stock_hit_score"] == 0.0
    assert detail["static_stock_confirmation"]["hit_symbols"] == []


def test_missing_scan_hits_and_narrative_are_optional():
    theme = _theme("AI & Semiconductors", ["NVDA"], ["SMH"])

    detail = calculate_theme_match(theme, [], {}, scan_hits_available=False)

    assert detail["theme_match_score"] is not None
    assert detail["theme_match_components"]["static_stock_hit_score"] is None
    assert detail["theme_match_components"]["narrative_keyword_score"] is None
    assert "static_stock_hit_score" in detail["theme_match_missing_components"]
    assert "narrative_keyword_score" in detail["theme_match_missing_components"]


def test_narrative_scores_accept_wrapped_shape(tmp_path):
    path = tmp_path / "narrative.json"
    path.write_text(
        '{"themes": {"Cybersecurity": {"narrative_keyword_score": 82}}}', encoding="utf-8"
    )

    scores = load_narrative_scores(str(path))
    theme = _theme("Cybersecurity", ["CRWD"], ["CIBR"])
    detail = calculate_theme_match(theme, [], {}, False, narrative_scores=scores)

    assert scores == {"Cybersecurity": 82.0}
    assert detail["theme_match_components"]["narrative_keyword_score"] == 82.0

"""Tests for uptrend_client.is_data_stale (business day logic)."""

from datetime import date
from unittest.mock import MagicMock

import uptrend_client
from uptrend_client import fetch_sector_uptrend_data, is_data_stale


class TestIsDataStale:
    """is_data_stale should count business days, not calendar days."""

    def test_friday_to_sunday_not_stale(self):
        """Friday data checked on Sunday: 0 business days -> not stale."""
        assert is_data_stale("2026-02-13", as_of_date=date(2026, 2, 15)) is False

    def test_friday_to_monday_not_stale(self):
        """Friday data checked on Monday: 1 business day -> not stale (threshold=2)."""
        assert is_data_stale("2026-02-13", as_of_date=date(2026, 2, 16)) is False

    def test_friday_to_tuesday_not_stale(self):
        """Friday data checked on Tuesday: 2 business days -> not stale (threshold=2)."""
        assert is_data_stale("2026-02-13", as_of_date=date(2026, 2, 17)) is False

    def test_friday_to_wednesday_stale(self):
        """Friday data checked on Wednesday: 3 business days -> stale (threshold=2)."""
        assert is_data_stale("2026-02-20", as_of_date=date(2026, 2, 25)) is True

    def test_monday_to_wednesday_not_stale(self):
        """Monday data checked on Wednesday: 2 business days -> not stale."""
        assert is_data_stale("2026-02-16", as_of_date=date(2026, 2, 18)) is False

    def test_monday_to_thursday_stale(self):
        """Monday data checked on Thursday: 3 business days -> stale."""
        assert is_data_stale("2026-02-16", as_of_date=date(2026, 2, 19)) is True

    def test_same_day_not_stale(self):
        """Same day data -> 0 business days -> not stale."""
        assert is_data_stale("2026-02-16", as_of_date=date(2026, 2, 16)) is False

    def test_invalid_date_returns_true(self):
        """Invalid date string -> stale (safe default)."""
        assert is_data_stale("not-a-date") is True

    def test_custom_threshold(self):
        """Custom threshold_bdays works correctly."""
        # Friday to Tuesday = 2 sessions, threshold=1 -> stale.
        assert (
            is_data_stale(
                "2026-02-20",
                threshold_bdays=1,
                as_of_date=date(2026, 2, 24),
            )
            is True
        )

    def test_exchange_holiday_is_excluded(self):
        assert is_data_stale("2026-04-02", threshold_bdays=1, as_of_date=date(2026, 4, 6)) is False

    def test_future_source_date_is_stale(self):
        assert is_data_stale("2026-04-07", as_of_date=date(2026, 4, 6)) is True


def test_fetch_uses_latest_row_on_or_before_as_of(monkeypatch):
    response = MagicMock()
    response.text = (
        "date,worksheet,ratio,ma_10,slope,trend\n"
        "2026-04-02,sec_technology,0.20,0.18,0.01,up\n"
        "2026-04-07,sec_technology,0.99,0.90,0.50,up\n"
    )
    response.raise_for_status.return_value = None
    monkeypatch.setattr(uptrend_client.requests, "get", lambda *_args, **_kwargs: response)

    result = fetch_sector_uptrend_data(as_of_date=date(2026, 4, 6))

    assert result["Technology"]["latest_date"] == "2026-04-02"
    assert result["Technology"]["ratio"] == 0.20

#!/usr/bin/env python3
"""
Theme Detector - Main Orchestrator

Detects trending market themes (bullish and bearish) by combining:
- FINVIZ industry/sector performance data
- yfinance ETF volume and stock metrics
- Monty's Uptrend Ratio Dashboard data

Outputs: JSON + Markdown report saved to reports/ directory.

Usage:
    python3 theme_detector.py --output-dir reports/
    python3 theme_detector.py --fmp-api-key $FMP_API_KEY --finviz-api-key $FINVIZ_API_KEY
"""

from __future__ import annotations

import argparse
import json
import os
import sys
import time
from datetime import date, datetime
from zoneinfo import ZoneInfo

# Ensure scripts directory is on the path
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))

from calculators.heat_calculator import (
    breadth_signal_score,
    calculate_theme_heat_detailed,
    momentum_strength_score,
    uptrend_signal_score,
    volume_intensity_score,
)
from calculators.industry_ranker import get_top_bottom_industries, rank_industries
from calculators.lifecycle_calculator import (
    calculate_lifecycle_maturity_detailed,
    classify_stage,
    estimate_duration_score,
    etf_proliferation_score,
    extremity_clustering_score,
    price_extreme_saturation_score,
    valuation_premium_score,
)
from calculators.theme_classifier import classify_themes
from leadership import aggregate_leadership, blend_theme_heat, load_scan_hits
from report_generator import generate_json_report, generate_markdown_report, save_reports
from scorer import (
    calculate_confidence,
    determine_data_mode,
    score_theme,
)
from theme_history import (
    append_observations,
    build_observation,
    compute_history_metrics,
    load_history,
    resolve_run_date,
    save_history,
)
from theme_matcher import calculate_theme_match, load_narrative_scores

# Heavy-dependency modules (pandas/numpy/yfinance/finvizfinance) are imported
# lazily inside main() to allow lightweight helpers like _get_representative_stocks
# to be imported without triggering sys.exit(1) when pandas is absent.


# ---------------------------------------------------------------------------
# Industry-to-sector mapping (FINVIZ industry names -> sector)
# ---------------------------------------------------------------------------
INDUSTRY_TO_SECTOR: dict[str, str] = {
    # Technology
    "Semiconductors": "Technology",
    "Semiconductor Equipment & Materials": "Technology",
    "Software - Application": "Technology",
    "Software - Infrastructure": "Technology",
    "Information Technology Services": "Technology",
    "Computer Hardware": "Technology",
    "Electronic Components": "Technology",
    "Electronics & Computer Distribution": "Technology",
    "Scientific & Technical Instruments": "Technology",
    "Communication Equipment": "Technology",
    "Consumer Electronics": "Technology",
    # Healthcare
    "Biotechnology": "Healthcare",
    "Drug Manufacturers - General": "Healthcare",
    "Drug Manufacturers - Specialty & Generic": "Healthcare",
    "Medical Devices": "Healthcare",
    "Medical Instruments & Supplies": "Healthcare",
    "Diagnostics & Research": "Healthcare",
    "Healthcare Plans": "Healthcare",
    "Health Information Services": "Healthcare",
    "Medical Care Facilities": "Healthcare",
    "Pharmaceutical Retailers": "Healthcare",
    "Medical Distribution": "Healthcare",
    # Financial
    "Banks - Diversified": "Financial",
    "Banks - Regional": "Financial",
    "Insurance - Life": "Financial",
    "Insurance - Property & Casualty": "Financial",
    "Insurance - Diversified": "Financial",
    "Insurance - Specialty": "Financial",
    "Insurance Brokers": "Financial",
    "Asset Management": "Financial",
    "Capital Markets": "Financial",
    "Financial Data & Stock Exchanges": "Financial",
    "Credit Services": "Financial",
    "Mortgage Finance": "Financial",
    "Financial Conglomerates": "Financial",
    "Shell Companies": "Financial",
    # Consumer Cyclical
    "Auto Manufacturers": "Consumer Cyclical",
    "Auto Parts": "Consumer Cyclical",
    "Auto & Truck Dealerships": "Consumer Cyclical",
    "Recreational Vehicles": "Consumer Cyclical",
    "Furnishings, Fixtures & Appliances": "Consumer Cyclical",
    "Residential Construction": "Consumer Cyclical",
    "Textile Manufacturing": "Consumer Cyclical",
    "Apparel Manufacturing": "Consumer Cyclical",
    "Footwear & Accessories": "Consumer Cyclical",
    "Packaging & Containers": "Consumer Cyclical",
    "Personal Services": "Consumer Cyclical",
    "Restaurants": "Consumer Cyclical",
    "Apparel Retail": "Consumer Cyclical",
    "Department Stores": "Consumer Cyclical",
    "Home Improvement Retail": "Consumer Cyclical",
    "Luxury Goods": "Consumer Cyclical",
    "Internet Retail": "Consumer Cyclical",
    "Specialty Retail": "Consumer Cyclical",
    "Gambling": "Consumer Cyclical",
    "Leisure": "Consumer Cyclical",
    "Lodging": "Consumer Cyclical",
    "Resorts & Casinos": "Consumer Cyclical",
    "Travel Services": "Consumer Cyclical",
    # Consumer Defensive
    "Beverages - Non-Alcoholic": "Consumer Defensive",
    "Beverages - Brewers": "Consumer Defensive",
    "Beverages - Wineries & Distilleries": "Consumer Defensive",
    "Confectioners": "Consumer Defensive",
    "Farm Products": "Consumer Defensive",
    "Household & Personal Products": "Consumer Defensive",
    "Packaged Foods": "Consumer Defensive",
    "Education & Training Services": "Consumer Defensive",
    "Discount Stores": "Consumer Defensive",
    "Food Distribution": "Consumer Defensive",
    "Grocery Stores": "Consumer Defensive",
    "Tobacco": "Consumer Defensive",
    # Industrials
    "Aerospace & Defense": "Industrials",
    "Airlines": "Industrials",
    "Building Products & Equipment": "Industrials",
    "Business Equipment & Supplies": "Industrials",
    "Conglomerates": "Industrials",
    "Consulting Services": "Industrials",
    "Electrical Equipment & Parts": "Industrials",
    "Engineering & Construction": "Industrials",
    "Farm & Heavy Construction Machinery": "Industrials",
    "Industrial Distribution": "Industrials",
    "Infrastructure Operations": "Industrials",
    "Integrated Freight & Logistics": "Industrials",
    "Marine Shipping": "Industrials",
    "Metal Fabrication": "Industrials",
    "Pollution & Treatment Controls": "Industrials",
    "Railroads": "Industrials",
    "Rental & Leasing Services": "Industrials",
    "Security & Protection Services": "Industrials",
    "Specialty Business Services": "Industrials",
    "Specialty Industrial Machinery": "Industrials",
    "Staffing & Employment Services": "Industrials",
    "Tools & Accessories": "Industrials",
    "Trucking": "Industrials",
    "Waste Management": "Industrials",
    # Energy
    "Oil & Gas E&P": "Energy",
    "Oil & Gas Equipment & Services": "Energy",
    "Oil & Gas Integrated": "Energy",
    "Oil & Gas Midstream": "Energy",
    "Oil & Gas Refining & Marketing": "Energy",
    "Oil & Gas Drilling": "Energy",
    "Thermal Coal": "Energy",
    "Uranium": "Energy",
    "Solar": "Energy",
    # Basic Materials
    "Gold": "Basic Materials",
    "Silver": "Basic Materials",
    "Aluminum": "Basic Materials",
    "Copper": "Basic Materials",
    "Steel": "Basic Materials",
    "Other Industrial Metals & Mining": "Basic Materials",
    "Other Precious Metals & Mining": "Basic Materials",
    "Coking Coal": "Basic Materials",
    "Lumber & Wood Production": "Basic Materials",
    "Paper & Paper Products": "Basic Materials",
    "Chemicals": "Basic Materials",
    "Specialty Chemicals": "Basic Materials",
    "Agricultural Inputs": "Basic Materials",
    "Building Materials": "Basic Materials",
    # Communication Services
    "Telecom Services": "Communication Services",
    "Advertising Agencies": "Communication Services",
    "Publishing": "Communication Services",
    "Broadcasting": "Communication Services",
    "Entertainment": "Communication Services",
    "Internet Content & Information": "Communication Services",
    "Electronic Gaming & Multimedia": "Communication Services",
    # Real Estate
    "REIT - Diversified": "Real Estate",
    "REIT - Healthcare Facilities": "Real Estate",
    "REIT - Hotel & Motel": "Real Estate",
    "REIT - Industrial": "Real Estate",
    "REIT - Mortgage": "Real Estate",
    "REIT - Office": "Real Estate",
    "REIT - Residential": "Real Estate",
    "REIT - Retail": "Real Estate",
    "REIT - Specialty": "Real Estate",
    "Real Estate - Development": "Real Estate",
    "Real Estate - Diversified": "Real Estate",
    "Real Estate Services": "Real Estate",
    # Utilities
    "Utilities - Diversified": "Utilities",
    "Utilities - Independent Power Producers": "Utilities",
    "Utilities - Regulated Electric": "Utilities",
    "Utilities - Regulated Gas": "Utilities",
    "Utilities - Regulated Water": "Utilities",
    "Utilities - Renewable": "Utilities",
}


# ---------------------------------------------------------------------------
# Theme configuration is loaded from YAML / inline fallback via config_loader.
# DEFAULT_THEMES_CONFIG and ETF_CATALOG live in default_theme_config.py
# to avoid circular imports.
# ---------------------------------------------------------------------------


# ---------------------------------------------------------------------------
# CLI argument parsing
# ---------------------------------------------------------------------------
def _as_of_date_arg(value: str) -> str:
    try:
        return resolve_run_date(value)
    except (TypeError, ValueError) as exc:
        raise argparse.ArgumentTypeError("must be zero-padded YYYY-MM-DD") from exc


def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
    parser = argparse.ArgumentParser(
        description="Detect trending market themes from FINVIZ industry data"
    )
    parser.add_argument(
        "--fmp-api-key",
        default=os.environ.get("FMP_API_KEY"),
        help="Financial Modeling Prep API key (env: FMP_API_KEY)",
    )
    parser.add_argument(
        "--finviz-api-key",
        default=os.environ.get("FINVIZ_API_KEY"),
        help="FINVIZ Elite API key (env: FINVIZ_API_KEY)",
    )
    parser.add_argument(
        "--finviz-mode",
        choices=["public", "elite"],
        default=None,
        help="FINVIZ mode (auto-detected if not specified)",
    )
    parser.add_argument(
        "--output-dir",
        default="reports/",
        help="Output directory for reports (default: reports/)",
    )
    parser.add_argument(
        "--top",
        type=int,
        default=3,
        help="Number of top themes to show in detail (default: 3)",
    )
    parser.add_argument(
        "--max-themes",
        type=int,
        default=10,
        help="Maximum themes to analyze (default: 10)",
    )
    parser.add_argument(
        "--max-stocks-per-theme",
        type=int,
        default=10,
        help="Maximum stocks per theme (default: 10)",
    )
    parser.add_argument(
        "--themes-config",
        default=None,
        help="Path to custom themes.yaml (default: bundled)",
    )
    parser.add_argument(
        "--discover-themes",
        action="store_true",
        default=False,
        help="Enable automatic theme discovery for unmatched industries",
    )
    parser.add_argument(
        "--dynamic-stocks",
        action="store_true",
        default=False,
        help="Enable dynamic stock selection via FINVIZ screener (default: off)",
    )
    parser.add_argument(
        "--dynamic-min-cap",
        choices=["micro", "small", "mid"],
        default="small",
        help="Minimum market cap for dynamic stock selection (default: small=$300mln+)",
    )
    parser.add_argument(
        "--scan-hits",
        default=None,
        help="Optional JSON/JSONL/CSV stock leadership scan rows or hits",
    )
    parser.add_argument(
        "--narrative-scores",
        default=None,
        help="Optional JSON map of theme names to offline narrative scores",
    )
    parser.add_argument(
        "--history-file",
        default=None,
        help="Theme history JSON path (default: <output-dir>/theme_detector_history.json)",
    )
    parser.add_argument(
        "--no-history-update",
        action="store_true",
        default=False,
        help="Read history metrics without writing the current run",
    )
    parser.add_argument(
        "--as-of-date",
        type=_as_of_date_arg,
        default=None,
        help="Run date for deterministic history/scan processing (YYYY-MM-DD)",
    )
    return parser.parse_args(argv)


# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _add_sector_info(industries: list[dict]) -> list[dict]:
    """Add sector field to each industry dict from INDUSTRY_TO_SECTOR mapping."""
    for ind in industries:
        name = ind.get("name", "")
        ind["sector"] = INDUSTRY_TO_SECTOR.get(name, "Unknown")
    return industries


def _convert_perf_to_pct(industries: list[dict]) -> list[dict]:
    """Convert FINVIZ decimal performance values to percentage.

    finvizfinance returns 0.05 for 5%. Our calculators expect 5.0.
    """
    perf_keys = ["perf_1w", "perf_1m", "perf_3m", "perf_6m", "perf_1y", "perf_ytd"]
    for ind in industries:
        for key in perf_keys:
            val = ind.get(key)
            if val is not None:
                ind[key] = val * 100.0
    return industries


def _get_representative_stocks(
    theme: dict,
    selector,  # Optional[RepresentativeStockSelector]
    max_stocks: int,
) -> tuple[list[str], list[dict]]:
    """Get representative stocks and metadata for a theme.

    When selector is provided (--dynamic-stocks), uses FINVIZ/FMP fallback chain.
    Otherwise falls back to static_stocks from config.

    Returns:
        (tickers, stock_details)
        tickers: ["NVDA", "AVGO", ...] (backward compatible)
        stock_details: [{symbol, source, market_cap, matched_industries,
                         reasons, composite_score}, ...]
    """
    if selector is not None:
        details = selector.select_stocks(theme, max_stocks)
        tickers = [d["symbol"] for d in details]
        return tickers, details

    # Static fallback (selector is None = --dynamic-stocks not set)
    static = theme.get("static_stocks", [])[:max_stocks]
    details = [
        {
            "symbol": s,
            "source": "static",
            "market_cap": 0,
            "matched_industries": [],
            "reasons": ["Static config"],
            "composite_score": 0,
        }
        for s in static
    ]
    return static, details


def detect_divergence(heat_breakdown: dict, direction: str) -> dict | None:
    """Detect divergence between price momentum and breadth signals."""
    momentum = heat_breakdown.get("momentum_strength")
    uptrend = heat_breakdown.get("uptrend_signal")
    if momentum is None or uptrend is None:
        return None
    gap = momentum - uptrend
    if abs(gap) < 25:
        return None
    if gap > 0:
        return {
            "type": "narrow_rally",
            "gap": round(gap, 1),
            "description": "Price momentum strong but breadth weak — concentrated/fragile move",
        }
    else:
        desc = (
            "Breadth improving ahead of price — potential reversal candidate"
            if direction == "bearish"
            else "Breadth improving ahead of price — potential acceleration candidate"
        )
        return {
            "type": "internal_recovery",
            "gap": round(abs(gap), 1),
            "description": desc,
        }


def _calculate_breadth_ratio(theme: dict) -> float | None:
    """Estimate breadth ratio from theme's matching industries.

    For bullish: ratio of industries with positive weighted_return.
    For bearish: ratio of industries with negative weighted_return.
    """
    industries = theme.get("matching_industries", [])
    if not industries:
        return None

    is_bearish = theme.get("direction") == "bearish"
    if is_bearish:
        count = sum(1 for ind in industries if ind.get("weighted_return", 0) < 0)
    else:
        count = sum(1 for ind in industries if ind.get("weighted_return", 0) > 0)

    return count / len(industries)


def _get_theme_uptrend_data(theme: dict, sector_uptrend: dict) -> list[dict]:
    """Build sector_data for uptrend_signal_score from theme's sector weights.

    Maps theme sectors to uptrend data with weights.
    """
    sector_weights = theme.get("sector_weights", {})
    if not sector_weights or not sector_uptrend:
        return []

    sector_data = []
    for sector_name, weight in sector_weights.items():
        uptrend_entry = sector_uptrend.get(sector_name)
        if uptrend_entry and uptrend_entry.get("ratio") is not None:
            sector_data.append(
                {
                    "sector": sector_name,
                    "ratio": uptrend_entry["ratio"],
                    "ma_10": uptrend_entry.get("ma_10") or 0,
                    "slope": uptrend_entry.get("slope") or 0,
                    "weight": weight,
                }
            )

    return sector_data


def _get_theme_weighted_return(theme: dict) -> float:
    """Calculate aggregate weighted return for a theme from its industries."""
    industries = theme.get("matching_industries", [])
    if not industries:
        return 0.0

    returns = [ind.get("weighted_return", 0.0) for ind in industries]
    return sum(returns) / len(returns)


def _resolve_output_dir(output_dir: str) -> str:
    """Resolve output directory relative to repo root if needed."""
    if os.path.isabs(output_dir):
        return output_dir
    repo_root = os.path.dirname(
        os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
    )
    return os.path.join(repo_root, output_dir)


def _round_optional(value: float | None) -> float | None:
    return None if value is None else round(value, 2)


def _coverage_penalties(
    heat_detail: dict, maturity_detail: dict, scan_hits_available: bool
) -> list[str]:
    penalties = []
    if heat_detail.get("coverage", 0.0) < 0.75:
        penalties.append("heat_low_coverage")
    if maturity_detail.get("coverage", 0.0) < 0.60:
        penalties.append("lifecycle_low_coverage")
    if not scan_hits_available:
        penalties.append("leadership_scan_hits_missing")
    return penalties


def _legacy_theme_priority(theme: dict) -> float:
    """Legacy max-themes priority from theme size and industry strength."""
    industries = theme.get("matching_industries", [])
    n_industries = len(industries)
    avg_strength = sum(abs(ind.get("weighted_return", 0)) for ind in industries) / max(
        len(industries), 1
    )
    size_norm = min(n_industries / 10.0, 1.0)
    strength_norm = min(avg_strength / 30.0, 1.0)
    return size_norm * 0.5 + strength_norm * 0.5


def _theme_selection_priority(theme: dict, match_detail: dict | None = None) -> float:
    """Select max-themes candidates without dropping high-quality matches."""
    legacy_priority = _legacy_theme_priority(theme)
    if not match_detail:
        return legacy_priority

    match_score = match_detail.get("theme_match_score")
    match_coverage = match_detail.get("theme_match_coverage", 0.0) or 0.0
    if match_score is None or match_coverage < 0.50:
        return legacy_priority

    match_priority = max(0.0, min(float(match_score) / 100.0, 1.0))
    return max(legacy_priority, match_priority)


# ---------------------------------------------------------------------------
# Main orchestrator
# ---------------------------------------------------------------------------
def _live_market_date() -> date:
    return datetime.now(ZoneInfo("America/New_York")).date()


def main(argv: list[str] | None = None) -> int:
    args = parse_args(argv)
    run_date = args.as_of_date or _live_market_date().isoformat()
    run_date_value = date.fromisoformat(run_date)
    if args.as_of_date is not None and run_date_value != _live_market_date():
        print(
            "ERROR: historical --as-of-date is unavailable because FINVIZ, quote, "
            "profile, and uptrend live sources are not point-in-time",
            file=sys.stderr,
        )
        return 2

    # Lazy imports: these modules depend on pandas/numpy/yfinance/finvizfinance
    # and are only needed at runtime, not when importing helpers for testing.
    from calculators.theme_classifier import deduplicate_themes, enrich_vertical_themes
    from config_loader import load_themes_config
    from etf_scanner import ETFScanner
    from finviz_performance_client import cap_outlier_performances, get_industry_performance
    from uptrend_client import fetch_sector_uptrend_data, is_data_stale

    output_dir = _resolve_output_dir(args.output_dir)
    history_file = args.history_file or os.path.join(output_dir, "theme_detector_history.json")
    history = load_history(history_file)

    # -----------------------------------------------------------------------
    # Step 0: Load theme configuration (YAML or inline fallback)
    # -----------------------------------------------------------------------
    themes_config, etf_catalog = load_themes_config(args.themes_config)
    start_time = time.time()

    # Determine data mode
    finviz_mode = args.finviz_mode
    if finviz_mode is None:
        finviz_mode = "elite" if args.finviz_api_key else "public"
    fmp_available = args.fmp_api_key is not None
    data_mode = determine_data_mode(fmp_available, finviz_mode == "elite")

    print("Theme Detector starting...", file=sys.stderr)
    print(f"  Data mode: {data_mode}", file=sys.stderr)
    print(f"  FINVIZ mode: {finviz_mode}", file=sys.stderr)
    print(f"  FMP API: {'available' if fmp_available else 'not available'}", file=sys.stderr)
    print(f"  Max themes: {args.max_themes}", file=sys.stderr)
    print(f"  Max stocks/theme: {args.max_stocks_per_theme}", file=sys.stderr)

    metadata = {
        "generated_at": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
        "run_date": run_date,
        "data_mode": data_mode,
        "finviz_mode": finviz_mode,
        "fmp_available": fmp_available,
        "max_themes": args.max_themes,
        "max_stocks_per_theme": args.max_stocks_per_theme,
        "history_file": history_file,
        "history_update": not args.no_history_update,
        "data_sources": {},
    }

    # -----------------------------------------------------------------------
    # Step 1: Fetch FINVIZ industry performance
    # -----------------------------------------------------------------------
    print("Fetching FINVIZ industry performance...", file=sys.stderr)
    raw_industries = get_industry_performance()
    if not raw_industries:
        print("ERROR: No industry data from FINVIZ. Exiting.", file=sys.stderr)
        sys.exit(1)

    metadata["data_sources"]["finviz_industries"] = len(raw_industries)
    print(f"  Got {len(raw_industries)} industries", file=sys.stderr)

    # Convert decimal to percentage, filter outliers, and add sector info
    industries = _convert_perf_to_pct(raw_industries)
    industries = cap_outlier_performances(industries)
    industries = _add_sector_info(industries)

    # -----------------------------------------------------------------------
    # Step 2: Rank industries by momentum
    # -----------------------------------------------------------------------
    print("Ranking industries by momentum...", file=sys.stderr)
    ranked = rank_industries(industries)
    industry_rankings = get_top_bottom_industries(ranked, n=15)
    print(
        f"  Top: {ranked[0]['name']} ({ranked[0]['momentum_score']})" if ranked else "",
        file=sys.stderr,
    )

    # -----------------------------------------------------------------------
    # Step 3: Classify themes
    # -----------------------------------------------------------------------
    print("Classifying themes...", file=sys.stderr)
    themes = classify_themes(ranked, themes_config)
    print(f"  Detected {len(themes)} themes (seed + vertical)", file=sys.stderr)

    if not themes:
        print("WARNING: No themes detected. Generating empty report.", file=sys.stderr)

    # Step 3.3: Enrich vertical themes with ETFs + deduplicate
    enrich_vertical_themes(themes)
    themes = deduplicate_themes(themes)
    print(f"  After enrich/dedup: {len(themes)} themes", file=sys.stderr)

    # Step 3.5: Discover new themes from unmatched industries
    if args.discover_themes:
        from calculators.theme_classifier import get_matched_industry_names
        from calculators.theme_discoverer import discover_themes

        matched_names = get_matched_industry_names(themes)
        discovered = discover_themes(ranked, matched_names, themes, top_n=30)
        themes.extend(discovered)
        metadata["data_sources"]["discovered_themes"] = len(discovered)
        print(f"  Discovered {len(discovered)} new themes", file=sys.stderr)

    # -----------------------------------------------------------------------
    # Step 3.8: Load stock leadership and narrative evidence before truncation
    # -----------------------------------------------------------------------
    scan_hits = []
    scan_hits_available = args.scan_hits is not None
    if args.scan_hits:
        print(f"Loading leadership scan hits from {args.scan_hits}...", file=sys.stderr)
        scan_hits, scan_summary = load_scan_hits(args.scan_hits, run_date)
        metadata["data_sources"]["scan_hits"] = scan_summary
        print(
            f"  Scan rows: {scan_summary['rows']}, hits: {scan_summary['hits']}",
            file=sys.stderr,
        )
    else:
        metadata["data_sources"]["scan_hits"] = {
            "path": None,
            "rows": 0,
            "hits": 0,
            "skipped_rows": 0,
            "skipped_date_rows": 0,
        }

    narrative_scores = None
    if args.narrative_scores:
        print(f"Loading narrative scores from {args.narrative_scores}...", file=sys.stderr)
        narrative_scores = load_narrative_scores(args.narrative_scores)
        metadata["data_sources"]["narrative_scores"] = {
            "path": args.narrative_scores,
            "themes": len(narrative_scores),
        }
    else:
        metadata["data_sources"]["narrative_scores"] = {"path": None, "themes": 0}

    scanner = ETFScanner(fmp_api_key=args.fmp_api_key, as_of_date=run_date_value)

    preselection_etfs = sorted(
        {etf for theme in themes for etf in theme.get("proxy_etfs", []) if etf}
    )
    print("Fetching preliminary ETF volume data for theme selection...", file=sys.stderr)
    etf_volume_map: dict[str, dict] = scanner.batch_etf_volume_ratios(preselection_etfs)
    metadata["data_sources"]["preselection_etf_volume"] = len(etf_volume_map)

    preselection_matches = {
        id(theme): calculate_theme_match(
            theme,
            scan_hits,
            etf_volume_map,
            scan_hits_available,
            narrative_scores=narrative_scores,
        )
        for theme in themes
    }

    # Step 3.9: Limit to max_themes using legacy priority plus match evidence.
    candidate_theme_count = len(themes)
    themes.sort(
        key=lambda theme: _theme_selection_priority(theme, preselection_matches.get(id(theme))),
        reverse=True,
    )
    themes = themes[: args.max_themes]
    metadata["data_sources"]["theme_selection"] = {
        "candidate_themes": candidate_theme_count,
        "selected_themes": len(themes),
        "uses_match_evidence": True,
    }

    leadership_by_theme = aggregate_leadership(themes, scan_hits, history)

    # -----------------------------------------------------------------------
    # Step 4: Collect all stock symbols for batch download
    # -----------------------------------------------------------------------
    print("Selecting representative stocks...", file=sys.stderr)

    # Create dynamic selector if requested
    selector = None
    if args.dynamic_stocks:
        from representative_stock_selector import RepresentativeStockSelector

        selector = RepresentativeStockSelector(
            finviz_elite_key=args.finviz_api_key,
            fmp_api_key=args.fmp_api_key,
            finviz_mode=finviz_mode,
            rate_limit_sec=1.0,
            min_cap=args.dynamic_min_cap,
        )
        print("  Dynamic stock selection: ON", file=sys.stderr)

    # Use index-based keys to avoid collisions when multiple themes share
    # the same name (e.g. two "{Sector} Sector Concentration" themes for
    # top and bottom, or duplicate auto-names from the discoverer).
    theme_stocks: dict[int, list[str]] = {}
    theme_stock_details: dict[int, list[dict]] = {}
    all_symbols = set()

    for idx, theme in enumerate(themes):
        tickers, stock_details = _get_representative_stocks(
            theme, selector, args.max_stocks_per_theme
        )
        theme_stocks[idx] = tickers
        theme_stock_details[idx] = stock_details
        all_symbols.update(tickers)

    all_symbols_list = sorted(all_symbols)
    print(f"  Total unique stocks: {len(all_symbols_list)}", file=sys.stderr)

    if selector:
        print(f"  Dynamic stock queries: {selector.query_count}", file=sys.stderr)
        print(f"  Dynamic stock failures: {selector.failure_count}", file=sys.stderr)
        print(f"  Dynamic stock status: {selector.status}", file=sys.stderr)
        metadata["data_sources"]["dynamic_stocks_status"] = selector.status
        metadata["data_sources"]["dynamic_stocks_queries"] = selector.query_count
        metadata["data_sources"]["dynamic_stocks_failures"] = selector.failure_count
        metadata["data_sources"]["dynamic_stocks_source_states"] = {
            name: {"disabled": s.disabled, "failures": s.total_failures}
            for name, s in selector.source_states.items()
        }

    # -----------------------------------------------------------------------
    # Step 5: Batch fetch stock metrics (yfinance)
    # -----------------------------------------------------------------------
    stock_metrics_map: dict[str, dict] = {}

    if all_symbols_list:
        print(f"Batch downloading {len(all_symbols_list)} stocks...", file=sys.stderr)
        all_metrics = scanner.batch_stock_metrics(all_symbols_list)
        for m in all_metrics:
            stock_metrics_map[m["symbol"]] = m
        # Backward compatible key (1 release coexistence)
        metadata["data_sources"]["yfinance_stocks"] = len(all_metrics)
        print(f"  Got metrics for {len(all_metrics)} stocks", file=sys.stderr)

    # -----------------------------------------------------------------------
    # Step 6: Fetch ETF volume ratios for each theme's proxy ETFs
    # -----------------------------------------------------------------------
    all_etfs = set()
    for theme in themes:
        for etf in theme.get("proxy_etfs", []):
            all_etfs.add(etf)

    missing_etfs = sorted(etf for etf in all_etfs if etf not in etf_volume_map)
    if missing_etfs:
        print("Fetching ETF volume data...", file=sys.stderr)
        etf_volume_map.update(scanner.batch_etf_volume_ratios(missing_etfs))

    metadata["data_sources"]["etf_volume"] = len(etf_volume_map)

    # Capture backend stats after all scanner calls (stock + ETF)
    scanner_stats = scanner.backend_stats()
    metadata["data_sources"]["scanner_backend"] = scanner_stats
    stock_s = scanner_stats.get("stock", {})
    etf_s = scanner_stats.get("etf", {})
    print(
        f"  Scanner (stock): FMP {stock_s.get('fmp_calls', 0)} calls "
        f"({stock_s.get('fmp_failures', 0)} failures), "
        f"yfinance: {stock_s.get('yf_calls', 0)} calls "
        f"({stock_s.get('yf_fallbacks', 0)} fallbacks)",
        file=sys.stderr,
    )
    print(
        f"  Scanner (ETF):   FMP {etf_s.get('fmp_calls', 0)} calls "
        f"({etf_s.get('fmp_failures', 0)} failures), "
        f"yfinance: {etf_s.get('yf_calls', 0)} calls "
        f"({etf_s.get('yf_fallbacks', 0)} fallbacks)",
        file=sys.stderr,
    )

    # -----------------------------------------------------------------------
    # Step 7: Fetch uptrend-dashboard data
    # -----------------------------------------------------------------------
    print("Fetching uptrend ratio data...", file=sys.stderr)
    sector_uptrend = fetch_sector_uptrend_data(as_of_date=run_date_value)
    stale_data = False

    if sector_uptrend:
        stale_sectors = [
            sector
            for sector, values in sector_uptrend.items()
            if is_data_stale(
                values.get("latest_date", ""),
                threshold_bdays=2,
                as_of_date=run_date_value,
            )
        ]
        stale_data = bool(stale_sectors)
        if stale_data:
            print(
                "  WARNING: Uptrend data is stale for: " + ", ".join(stale_sectors),
                file=sys.stderr,
            )
        metadata["data_sources"]["uptrend_sectors"] = len(sector_uptrend)
        metadata["data_sources"]["uptrend_stale"] = stale_data
        metadata["data_sources"]["uptrend_stale_sectors"] = stale_sectors
    else:
        print("  WARNING: Uptrend data unavailable", file=sys.stderr)
        metadata["data_sources"]["uptrend_error"] = "fetch failed"

    # -----------------------------------------------------------------------
    # Step 8: Score each theme
    # -----------------------------------------------------------------------
    print("Scoring themes...", file=sys.stderr)
    scored_themes = []

    for idx, theme in enumerate(themes):
        theme_name = theme["theme_name"]
        direction = theme["direction"]
        is_bearish = direction == "bearish"
        stocks = theme_stocks.get(idx, [])

        # --- Theme Heat ---
        # Momentum: average weighted_return of matching industries
        theme_wr = _get_theme_weighted_return(theme)
        momentum = momentum_strength_score(theme_wr)

        # Volume: average ETF volume ratio
        etf_vol_ratios = []
        for etf_sym in theme.get("proxy_etfs", []):
            vol = etf_volume_map.get(etf_sym, {})
            if vol.get("vol_20d") is not None and vol.get("vol_60d") is not None:
                etf_vol_ratios.append((vol["vol_20d"], vol["vol_60d"]))

        if etf_vol_ratios:
            avg_20d = sum(r[0] for r in etf_vol_ratios) / len(etf_vol_ratios)
            avg_60d = sum(r[1] for r in etf_vol_ratios) / len(etf_vol_ratios)
            volume = volume_intensity_score(avg_20d, avg_60d)
        else:
            volume = None

        # Uptrend signal
        sector_data = _get_theme_uptrend_data(theme, sector_uptrend)
        if sector_data:
            uptrend = uptrend_signal_score(sector_data, is_bearish)
        else:
            uptrend = None

        # Breadth signal
        breadth_ratio = _calculate_breadth_ratio(theme)
        n_industries = len(theme.get("matching_industries", []))
        breadth = breadth_signal_score(breadth_ratio, industry_count=n_industries)

        heat_detail = calculate_theme_heat_detailed(momentum, volume, uptrend, breadth)
        base_heat = round(heat_detail["score"] or 0.0, 2)

        leadership = leadership_by_theme.get(theme_name, {})
        leadership_score = leadership.get("leadership_score")
        heat = blend_theme_heat(base_heat, leadership_score)
        history_metrics = compute_history_metrics(history, theme_name, run_date, heat)
        theme_match = calculate_theme_match(
            theme,
            scan_hits,
            etf_volume_map,
            scan_hits_available,
            narrative_scores=narrative_scores,
        )
        theme_match_score = theme_match.get("theme_match_score")
        theme_rank_score = (
            round(0.75 * heat + 0.25 * theme_match_score, 2)
            if theme_match_score is not None
            else round(heat, 2)
        )

        heat_breakdown = {
            "momentum_strength": _round_optional(momentum),
            "volume_intensity": _round_optional(volume),
            "uptrend_signal": _round_optional(uptrend),
            "breadth_signal": _round_optional(breadth),
            "leadership_score": _round_optional(leadership_score),
        }

        # --- Divergence detection ---
        divergence = detect_divergence(heat_breakdown, direction)

        # --- Lifecycle Maturity ---
        # Get stock-level metrics for this theme
        theme_stock_metrics = [stock_metrics_map[s] for s in stocks if s in stock_metrics_map]

        # Remap keys: rsi_14 -> rsi (lifecycle_calculator expects "rsi")
        for sm in theme_stock_metrics:
            if "rsi_14" in sm:
                sm["rsi"] = sm["rsi_14"]

        # Duration: prefer actual theme history, fall back to performance horizons
        avg_perfs = _average_industry_perfs(theme.get("matching_industries", []))
        duration = history_metrics["duration_score"]
        if not history_metrics["prior_observations"] and duration == 0:
            duration = estimate_duration_score(
                avg_perfs.get("perf_1m"),
                avg_perfs.get("perf_3m"),
                avg_perfs.get("perf_6m"),
                avg_perfs.get("perf_1y"),
                is_bearish,
            )

        # Extremity clustering
        extremity = extremity_clustering_score(theme_stock_metrics, is_bearish)

        # Price extreme saturation
        price_extreme = price_extreme_saturation_score(theme_stock_metrics, is_bearish)

        # Valuation premium
        valuation = valuation_premium_score(theme_stock_metrics)

        # ETF proliferation
        etf_count = etf_catalog.get(theme_name, 0)
        etf_prolif = etf_proliferation_score(etf_count)

        maturity_detail = calculate_lifecycle_maturity_detailed(
            duration, extremity, price_extreme, valuation, etf_prolif
        )
        maturity = round(maturity_detail["score"] or 0.0, 2)
        stage = classify_stage(maturity)

        maturity_breakdown = {
            "duration_score": _round_optional(duration),
            "extremity_clustering": _round_optional(extremity),
            "price_extreme_saturation": _round_optional(price_extreme),
            "valuation_premium": _round_optional(valuation),
            "etf_proliferation": _round_optional(etf_prolif),
        }

        # --- Confidence ---
        quant_confirmed = momentum > 50
        breadth_confirmed = (uptrend is not None and uptrend > 55) if uptrend else False
        narrative_confirmed = False  # Pending Claude WebSearch
        confidence_penalties = _coverage_penalties(
            heat_detail, maturity_detail, scan_hits_available
        )
        confidence = calculate_confidence(
            quant_confirmed,
            breadth_confirmed,
            narrative_confirmed,
            stale_data,
            coverage_penalty_count=len(confidence_penalties),
        )

        # --- Score theme ---
        score = score_theme(
            round(heat, 2),
            round(maturity, 2),
            stage,
            direction,
            confidence,
            data_mode,
        )

        # Lifecycle data quality flag
        lifecycle_quality = "sufficient" if theme_stock_metrics else "insufficient"

        # Build full theme result
        scored_theme = {
            "name": theme_name,
            "direction": direction,
            "heat": round(heat, 2),
            "base_heat": base_heat,
            "maturity": round(maturity, 2),
            "stage": stage,
            "confidence": confidence,
            "heat_label": score["heat_label"],
            "heat_breakdown": heat_breakdown,
            "heat_coverage": round(heat_detail["coverage"], 4),
            "heat_missing_components": heat_detail["missing_components"],
            "maturity_breakdown": maturity_breakdown,
            "maturity_coverage": round(maturity_detail["coverage"], 4),
            "maturity_missing_components": maturity_detail["missing_components"],
            "lifecycle_data_quality": lifecycle_quality,
            "representative_stocks": stocks,
            "stock_details": theme_stock_details.get(idx, []),
            "proxy_etfs": theme.get("proxy_etfs", []),
            "industries": [ind.get("name", "") for ind in theme.get("matching_industries", [])],
            "sector_weights": theme.get("sector_weights", {}),
            "stock_data": "available" if theme_stock_metrics else "unavailable",
            "data_mode": data_mode,
            "stale_data_penalty": stale_data,
            "confidence_penalties": confidence_penalties,
            "theme_origin": theme.get("theme_origin", "seed"),
            "name_confidence": theme.get("name_confidence", "high"),
            "divergence": divergence,
            "leadership_score": leadership_score,
            "leadership_coverage": leadership.get("leadership_coverage", 0.0),
            "leadership_counts": leadership.get("leadership_counts", {}),
            "leader_symbols": leadership.get("leader_symbols", []),
            "leader_candidates": leadership.get("leader_candidates", []),
            "fresh_leadership_symbols": leadership.get("fresh_leadership_symbols", []),
            "extended_symbols": leadership.get("extended_symbols", []),
            "theme_match_score": theme_match_score,
            "theme_match_components": theme_match.get("theme_match_components", {}),
            "theme_match_coverage": theme_match.get("theme_match_coverage", 0.0),
            "theme_match_missing_components": theme_match.get("theme_match_missing_components", []),
            "static_stock_confirmation": theme_match.get("static_stock_confirmation", {}),
            "proxy_etf_confirmation": theme_match.get("proxy_etf_confirmation", {}),
            "theme_rank_score": theme_rank_score,
            "history_metrics": history_metrics,
        }
        scored_themes.append(scored_theme)

    # Preserve legacy heat ordering while exposing explicit match-quality ranks.
    heat_sorted = sorted(scored_themes, key=lambda t: t["heat"], reverse=True)
    for rank, theme in enumerate(heat_sorted, start=1):
        theme["heat_rank"] = rank

    match_sorted = sorted(
        scored_themes,
        key=lambda t: (
            t.get("theme_match_score") is not None,
            t.get("theme_match_score") or -1.0,
            t.get("theme_rank_score", t.get("heat", 0)),
        ),
        reverse=True,
    )
    for rank, theme in enumerate(match_sorted, start=1):
        theme["theme_match_rank"] = rank

    scored_themes = heat_sorted

    # -----------------------------------------------------------------------
    # Step 9: Generate reports
    # -----------------------------------------------------------------------
    print("Generating reports...", file=sys.stderr)

    json_report = generate_json_report(
        scored_themes,
        industry_rankings,
        sector_uptrend,
        metadata,
        as_of_date=run_date_value,
    )
    md_report = generate_markdown_report(json_report, top_n_detail=args.top)

    paths = save_reports(json_report, md_report, output_dir)

    if not args.no_history_update:
        observations = [build_observation(theme, run_date) for theme in scored_themes]
        save_history(history_file, append_observations(history, observations))

    elapsed = time.time() - start_time
    print(f"\nDone in {elapsed:.1f}s", file=sys.stderr)
    print(f"  JSON:     {paths['json']}", file=sys.stderr)
    print(f"  Markdown: {paths['markdown']}", file=sys.stderr)
    print(f"  Themes:   {len(scored_themes)}", file=sys.stderr)

    # Print JSON to stdout for programmatic consumption
    print(json.dumps(json_report, indent=2, default=str))
    return 0


def _average_industry_perfs(industries: list[dict]) -> dict:
    """Average performance across theme's matching industries."""
    if not industries:
        return {}

    perf_keys = ["perf_1m", "perf_3m", "perf_6m", "perf_1y"]
    result = {}
    for key in perf_keys:
        vals = [ind.get(key) for ind in industries if ind.get(key) is not None]
        result[key] = sum(vals) / len(vals) if vals else None
    return result


if __name__ == "__main__":
    raise SystemExit(main())

#!/usr/bin/env python3
"""Theme Detector history and acceleration metrics."""

import json
import math
import os
from datetime import datetime
from pathlib import Path
from typing import Optional
from zoneinfo import ZoneInfo


def load_history(path: Optional[str]) -> dict[str, list[dict]]:
    """Load history from JSON. Missing files return an empty history."""
    if not path or not os.path.exists(path):
        return {}
    with open(path) as f:
        data = json.load(f)
    if isinstance(data, dict) and isinstance(data.get("themes"), dict):
        return {
            str(name): records if isinstance(records, list) else []
            for name, records in data["themes"].items()
        }
    if isinstance(data, list):
        history: dict[str, list[dict]] = {}
        for record in data:
            name = record.get("theme")
            if name:
                history.setdefault(name, []).append(record)
        return history
    return {}


def save_history(path: str, history: dict[str, list[dict]]) -> None:
    """Write history JSON atomically."""
    target = Path(path)
    target.parent.mkdir(parents=True, exist_ok=True)
    payload = {"version": 1, "themes": history}
    tmp = target.with_suffix(target.suffix + ".tmp")
    with tmp.open("w", encoding="utf-8") as f:
        json.dump(payload, f, indent=2, sort_keys=True)
    tmp.replace(target)


def compute_history_metrics(
    history: dict[str, list[dict]], theme_name: str, current_date: str, current_heat: float
) -> dict:
    """Compute current acceleration metrics against prior observations."""
    prior = _prior_records(history.get(theme_name, []), current_date)
    prior_heats = [_float(record.get("heat")) for record in prior]
    prior_heats = [value for value in prior_heats if value is not None]

    heat_delta_1d = current_heat - prior_heats[-1] if prior_heats else None
    heat_delta_5d = current_heat - prior_heats[-5] if len(prior_heats) >= 5 else None

    heat_z_20d = None
    acceleration_score = None
    window = prior_heats[-20:]
    if len(window) >= 2:
        mean = sum(window) / len(window)
        variance = sum((value - mean) ** 2 for value in window) / len(window)
        std = math.sqrt(variance)
        if std > 0:
            heat_z_20d = (current_heat - mean) / std
            acceleration_score = max(0.0, min(100.0, 50.0 + heat_z_20d * 15.0))

    duration_count = _duration_count(prior, current_heat)
    duration_score = min(100.0, duration_count * 10.0)

    return {
        "duration_count": duration_count,
        "duration_score": round(duration_score, 2),
        "heat_delta_1d": _round_or_none(heat_delta_1d),
        "heat_delta_5d": _round_or_none(heat_delta_5d),
        "heat_z_20d": _round_or_none(heat_z_20d),
        "acceleration_score": _round_or_none(acceleration_score),
        "prior_observations": len(prior),
    }


def append_observations(history: dict[str, list[dict]], observations: list[dict]) -> dict:
    """Return history with current observations appended."""
    updated = {name: list(records) for name, records in history.items()}
    for observation in observations:
        name = observation.get("theme")
        if not name:
            continue
        records = updated.setdefault(name, [])
        records = [r for r in records if r.get("date") != observation.get("date")]
        records.append(observation)
        records.sort(key=lambda r: r.get("date", ""))
        updated[name] = records[-260:]
    return updated


def build_observation(theme: dict, run_date: str) -> dict:
    """Build one persistent history observation for a scored theme."""
    return {
        "date": run_date,
        "theme": theme.get("name"),
        "direction": theme.get("direction"),
        "heat": theme.get("heat"),
        "base_heat": theme.get("base_heat"),
        "leadership_score": theme.get("leadership_score"),
        "leadership_counts": theme.get("leadership_counts", {}),
    }


def resolve_run_date(value: Optional[str] = None) -> str:
    """Return YYYY-MM-DD run date."""
    if value:
        parsed = datetime.strptime(value, "%Y-%m-%d").date()
        if parsed.isoformat() != value:
            raise ValueError("run date must be zero-padded YYYY-MM-DD")
        return parsed.isoformat()
    return datetime.now(ZoneInfo("America/New_York")).date().isoformat()


def _prior_records(records: list[dict], current_date: str) -> list[dict]:
    prior = [record for record in records if str(record.get("date", "")) < current_date]
    return sorted(prior, key=lambda record: record.get("date", ""))


def _duration_count(prior: list[dict], current_heat: float) -> int:
    count = 1 if current_heat >= 40.0 else 0
    if count == 0:
        return 0
    for record in reversed(prior):
        heat = _float(record.get("heat"))
        if heat is None or heat < 40.0:
            break
        count += 1
    return count


def _float(value) -> Optional[float]:
    if value in (None, ""):
        return None
    try:
        return float(value)
    except (TypeError, ValueError):
        return None


def _round_or_none(value: Optional[float]) -> Optional[float]:
    return None if value is None else round(value, 2)

#!/usr/bin/env python3
"""Theme match quality scoring for Theme Detector.

This module scores how specifically a detected theme is supported by its
industry matches, explicit stock basket, proxy ETFs, and optional narrative
input. It is intentionally deterministic and offline.
"""

import json
from pathlib import Path
from typing import Optional

THEME_MATCH_WEIGHTS = {
    "industry_match_score": 0.40,
    "static_stock_hit_score": 0.30,
    "proxy_etf_momentum_score": 0.20,
    "narrative_keyword_score": 0.10,
}


def load_narrative_scores(path: str) -> dict[str, float]:
    """Load optional caller-supplied narrative scores keyed by theme name.

    Supported shapes:
    - {"AI & Semiconductors": 82}
    - {"AI & Semiconductors": {"narrative_keyword_score": 82}}
    - {"themes": {"AI & Semiconductors": 82}}
    """
    data = json.loads(Path(path).read_text())
    if isinstance(data, dict) and isinstance(data.get("themes"), dict):
        data = data["themes"]
    if not isinstance(data, dict):
        return {}

    scores: dict[str, float] = {}
    for name, value in data.items():
        if isinstance(value, dict):
            value = value.get("narrative_keyword_score", value.get("narrative_score"))
        score = _float(value)
        if score is not None:
            scores[str(name)] = _clamp(score)
    return scores


def calculate_theme_match(
    theme: dict,
    scan_hits: list,
    etf_volume_map: dict[str, dict],
    scan_hits_available: bool,
    narrative_scores: Optional[dict[str, float]] = None,
) -> dict:
    """Calculate a 0-100 theme match quality score with components."""
    industry_score = _industry_match_score(theme)
    static_stock_score, static_stock_detail = _static_stock_hit_score(
        theme, scan_hits, scan_hits_available
    )
    etf_score, etf_confirmation = _proxy_etf_momentum_score(theme, etf_volume_map)
    narrative_score = _narrative_score(theme, narrative_scores)

    components = {
        "industry_match_score": industry_score,
        "static_stock_hit_score": static_stock_score,
        "proxy_etf_momentum_score": etf_score,
        "narrative_keyword_score": narrative_score,
    }
    score, coverage, missing = _weighted_available_score(components, THEME_MATCH_WEIGHTS)

    return {
        "theme_match_score": round(score, 2) if score is not None else None,
        "theme_match_components": {
            key: round(value, 2) if value is not None else None for key, value in components.items()
        },
        "theme_match_coverage": round(coverage, 4),
        "theme_match_missing_components": missing,
        "static_stock_confirmation": static_stock_detail,
        "proxy_etf_confirmation": etf_confirmation,
    }


def _industry_match_score(theme: dict) -> float:
    industries = theme.get("matching_industries", [])
    if not industries:
        return 0.0

    target_count = theme.get("matching_keyword_count") or max(len(industries), 5)
    count_score = min(len(industries) / max(float(target_count), 1.0), 1.0) * 100.0

    strengths = []
    for industry in industries:
        weighted_return = _float(industry.get("weighted_return"))
        if weighted_return is not None:
            strengths.append(min(abs(weighted_return) / 30.0, 1.0) * 100.0)
    strength_score = sum(strengths) / len(strengths) if strengths else count_score

    return _clamp(0.55 * count_score + 0.45 * strength_score)


def _static_stock_hit_score(
    theme: dict, scan_hits: list, scan_hits_available: bool
) -> tuple[Optional[float], dict]:
    basket = _theme_stock_basket(theme)
    if not scan_hits_available:
        return None, {
            "available": False,
            "basket_size": len(basket),
            "hit_symbols": [],
            "hit_count": 0,
        }

    if not basket:
        return 0.0, {
            "available": True,
            "basket_size": 0,
            "hit_symbols": [],
            "hit_count": 0,
        }

    hits_by_symbol: dict[str, set[str]] = {}
    for hit in scan_hits:
        symbol = getattr(hit, "symbol", None)
        if symbol in basket:
            hits_by_symbol.setdefault(symbol, set()).add(getattr(hit, "scan_type", "unknown"))

    hit_symbols = sorted(hits_by_symbol)
    hit_count = len(hit_symbols)
    score = min(100.0, hit_count * 40.0)
    return score, {
        "available": True,
        "basket_size": len(basket),
        "hit_symbols": hit_symbols,
        "hit_count": hit_count,
        "hit_scan_types": {
            symbol: sorted(scan_types) for symbol, scan_types in hits_by_symbol.items()
        },
    }


def _proxy_etf_momentum_score(
    theme: dict, etf_volume_map: dict[str, dict]
) -> tuple[Optional[float], dict]:
    proxy_etfs = [str(etf).upper() for etf in theme.get("proxy_etfs", [])]
    ratios = []
    matched = []
    missing = []

    for etf in proxy_etfs:
        data = etf_volume_map.get(etf, {})
        ratio = _float(data.get("vol_ratio"))
        if ratio is None:
            missing.append(etf)
            continue
        ratios.append(ratio)
        matched.append({"symbol": etf, "vol_ratio": ratio})

    if not ratios:
        return None, {
            "confirmed": False,
            "average_vol_ratio": None,
            "matched_etfs": matched,
            "missing_etfs": missing or proxy_etfs,
        }

    avg_ratio = sum(ratios) / len(ratios)
    score = _clamp(50.0 + (avg_ratio - 1.0) * 50.0)
    return score, {
        "confirmed": avg_ratio >= 1.2,
        "average_vol_ratio": round(avg_ratio, 4),
        "matched_etfs": matched,
        "missing_etfs": missing,
    }


def _narrative_score(theme: dict, narrative_scores: Optional[dict[str, float]]) -> Optional[float]:
    if not narrative_scores:
        return None
    name = theme.get("theme_name") or theme.get("name")
    value = narrative_scores.get(name)
    return _clamp(value) if value is not None else None


def _theme_stock_basket(theme: dict) -> set[str]:
    stocks = theme.get("static_stocks") or theme.get("representative_stocks") or []
    return {str(stock).upper() for stock in stocks if stock}


def _weighted_available_score(
    components: dict[str, Optional[float]], weights: dict[str, float]
) -> tuple[Optional[float], float, list[str]]:
    available = {key: value for key, value in components.items() if value is not None}
    total_possible = sum(weights.values())
    total_available = sum(weights[key] for key in available)
    if total_available <= 0:
        return None, 0.0, list(components)
    score = sum(available[key] * weights[key] for key in available) / total_available
    missing = [key for key, value in components.items() if value is None]
    return score, total_available / total_possible if total_possible else 0.0, missing


def _float(value) -> Optional[float]:
    if value in (None, ""):
        return None
    try:
        return float(value)
    except (TypeError, ValueError):
        return None


def _clamp(value: float, low: float = 0.0, high: float = 100.0) -> float:
    return min(high, max(low, float(value)))

# Theme Detector configuration
# Cross-sector theme definitions with proxy ETFs and representative stocks.
# Migrated from DEFAULT_THEMES_CONFIG + ETF_CATALOG inline constants.

cross_sector_min_matches: 2
vertical_min_industries: 3

cross_sector:
  - theme_name: "AI & Semiconductors"
    matching_keywords:
      - Semiconductors
      - Semiconductor Equipment & Materials
      - Software - Application
      - Software - Infrastructure
      - Information Technology Services
      - Computer Hardware
      - Electronic Components
    proxy_etfs: [SMH, SOXX, AIQ, BOTZ]
    static_stocks: [NVDA, AVGO, AMD, INTC, QCOM, MRVL, AMAT, LRCX, KLAC, TSM]
    etf_count: 8

  - theme_name: "Clean Energy & EV"
    matching_keywords:
      - Solar
      - Utilities - Renewable
      - Auto Manufacturers
      - Electrical Equipment & Parts
      - Specialty Chemicals
    proxy_etfs: [ICLN, QCLN, TAN, LIT]
    static_stocks: [ENPH, SEDG, FSLR, RUN, TSLA, RIVN, ALB, PLUG, BE, NEE]
    etf_count: 7

  - theme_name: "Cybersecurity"
    matching_keywords:
      - Software - Infrastructure
      - Software - Application
      - Information Technology Services
    proxy_etfs: [CIBR, HACK, BUG]
    static_stocks: [CRWD, PANW, FTNT, ZS, S, OKTA, NET, CYBR, QLYS, RPD]
    etf_count: 4

  - theme_name: "Cloud Computing & SaaS"
    matching_keywords:
      - Software - Application
      - Software - Infrastructure
      - Information Technology Services
    proxy_etfs: [SKYY, WCLD, CLOU]
    static_stocks: [CRM, NOW, SNOW, DDOG, MDB, NET, ZS, HUBS, WDAY, TEAM]
    etf_count: 4

  - theme_name: "Biotech & Genomics"
    matching_keywords:
      - Biotechnology
      - Drug Manufacturers - General
      - Drug Manufacturers - Specialty & Generic
      - Diagnostics & Research
    proxy_etfs: [XBI, IBB, ARKG]
    static_stocks: [AMGN, GILD, VRTX, REGN, MRNA, BIIB, ILMN, SRPT, ALNY, BMRN]
    etf_count: 5

  - theme_name: "Infrastructure & Construction"
    matching_keywords:
      - Engineering & Construction
      - Building Products & Equipment
      - Building Materials
      - Specialty Industrial Machinery
      - Farm & Heavy Construction Machinery
      - Infrastructure Operations
    proxy_etfs: [PAVE, IFRA]
    static_stocks: [CAT, DE, VMC, MLM, URI, PWR, EME, J, ACM, FAST]
    etf_count: 3

  - theme_name: "Gold & Precious Metals"
    matching_keywords:
      - Gold
      - Silver
      - Other Precious Metals & Mining
    proxy_etfs: [GDX, GDXJ, SLV, GLD]
    static_stocks: [NEM, GOLD, AEM, FNV, WPM, RGLD, KGC, AGI, HL, PAAS]
    etf_count: 4

  - theme_name: "Oil & Gas (Energy)"
    matching_keywords:
      - Oil & Gas E&P
      - Oil & Gas Equipment & Services
      - Oil & Gas Integrated
      - Oil & Gas Midstream
      - Oil & Gas Refining & Marketing
      - Oil & Gas Drilling
    proxy_etfs: [XLE, XOP, OIH]
    static_stocks: [XOM, CVX, COP, EOG, SLB, FANG, MPC, PSX, OXY, DVN]
    etf_count: 5

  - theme_name: "Financial Services & Banks"
    matching_keywords:
      - Banks - Diversified
      - Banks - Regional
      - Capital Markets
      - Insurance - Diversified
      - Financial Data & Stock Exchanges
      - Asset Management
    proxy_etfs: [XLF, KBE, KRE]
    static_stocks: [JPM, BAC, GS, MS, WFC, C, BLK, SCHW, ICE, CME]
    etf_count: 5

  - theme_name: "Healthcare & Pharma"
    matching_keywords:
      - Drug Manufacturers - General
      - Medical Devices
      - Medical Instruments & Supplies
      - Healthcare Plans
      - Medical Care Facilities
    proxy_etfs: [XLV, IHI, XBI]
    static_stocks: [UNH, JNJ, LLY, PFE, ABT, TMO, DHR, SYK, ISRG, MDT]
    etf_count: 5

  - theme_name: "Defense & Aerospace"
    matching_keywords:
      - Aerospace & Defense
      - Communication Equipment
      - Scientific & Technical Instruments
    proxy_etfs: [ITA, PPA, XAR]
    static_stocks: [LMT, RTX, NOC, GD, BA, LHX, HII, TDG, HWM, AXON]
    etf_count: 5

  - theme_name: "Real Estate & REITs"
    matching_keywords:
      - REIT - Diversified
      - REIT - Industrial
      - REIT - Residential
      - REIT - Retail
      - REIT - Specialty
      - REIT - Office
      - Real Estate - Development
      - Real Estate Services
    proxy_etfs: [VNQ, IYR, XLRE]
    static_stocks: [PLD, AMT, EQIX, CCI, SPG, O, WELL, DLR, PSA, AVB]
    etf_count: 4

  - theme_name: "Retail & Consumer"
    matching_keywords:
      - Internet Retail
      - Specialty Retail
      - Apparel Retail
      - Discount Stores
      - Home Improvement Retail
      - Department Stores
    proxy_etfs: [XRT, XLY]
    static_stocks: [AMZN, HD, LOW, TJX, COST, WMT, TGT, ROST, BURL, LULU]
    etf_count: 3

  - theme_name: "Crypto & Blockchain"
    matching_keywords:
      - Capital Markets
      - Software - Infrastructure
      - Financial Data & Stock Exchanges
    proxy_etfs: [BITO, BLOK, BITQ]
    static_stocks: [COIN, MSTR, MARA, RIOT, CLSK, HUT, BITF, HIVE, SQ, PYPL]
    etf_count: 4

  - theme_name: "Nuclear & Uranium"
    min_matching_industries: 1
    matching_keywords:
      - Uranium
      - Utilities - Independent Power Producers
      - Utilities - Regulated Electric
    proxy_etfs: [URA, URNM, NLR]
    static_stocks: [CCJ, UEC, NXE, DNN, LEU, SMR, UUUU, URG, OKLO, VST]
    etf_count: 3

#!/usr/bin/env python3
"""
Theme Detector - Uptrend Ratio Client

Fetches sector uptrend ratio data from Monty's Uptrend Ratio Dashboard
(GitHub CSV). No API key required.

Data Source: https://github.com/tradermonty/uptrend-dashboard
"""

import csv
import io
import sys
from datetime import date
from typing import Optional
from zoneinfo import ZoneInfo

from _market_calendar import count_sessions

try:
    import requests

    HAS_REQUESTS = True
except ImportError:
    HAS_REQUESTS = False


TIMESERIES_URL = (
    "https://raw.githubusercontent.com/tradermonty/uptrend-dashboard/"
    "main/data/uptrend_ratio_timeseries.csv"
)
SECTOR_SUMMARY_URL = (
    "https://raw.githubusercontent.com/tradermonty/uptrend-dashboard/main/data/sector_summary.csv"
)

# Map FINVIZ sector names to uptrend worksheet names
FINVIZ_TO_WORKSHEET = {
    "Technology": "sec_technology",
    "Healthcare": "sec_healthcare",
    "Financial": "sec_financial",
    "Consumer Cyclical": "sec_consumercyclical",
    "Consumer Defensive": "sec_consumerdefensive",
    "Industrials": "sec_industrials",
    "Energy": "sec_energy",
    "Basic Materials": "sec_basicmaterials",
    "Communication Services": "sec_communicationservices",
    "Real Estate": "sec_realestate",
    "Utilities": "sec_utilities",
}

# Reverse mapping for display
WORKSHEET_TO_DISPLAY = {v: k for k, v in FINVIZ_TO_WORKSHEET.items()}


def _safe_float(value) -> Optional[float]:
    """Convert to float, return None if empty or invalid."""
    if value is None or str(value).strip() == "":
        return None
    try:
        return float(value)
    except (ValueError, TypeError):
        return None


def _calculate_slope(values: list[float]) -> Optional[float]:
    """Calculate slope from a list of values using simple linear regression.

    Uses least-squares fit: slope = sum((x-xbar)(y-ybar)) / sum((x-xbar)^2)

    Args:
        values: List of float values (at least 2 required)

    Returns:
        Slope value or None if insufficient data
    """
    if not values or len(values) < 2:
        return None

    n = len(values)
    x_mean = (n - 1) / 2.0
    y_mean = sum(values) / n

    numerator = sum((i - x_mean) * (v - y_mean) for i, v in enumerate(values))
    denominator = sum((i - x_mean) ** 2 for i in range(n))

    if denominator == 0:
        return 0.0

    return round(numerator / denominator, 6)


def is_data_stale(
    latest_date_str: str,
    threshold_bdays: int = 2,
    *,
    as_of_date: Optional[date] = None,
) -> bool:
    """Check if the latest data is older than threshold business days.

    Counts XNYS sessions, so exchange holidays and weekends do not age data.

    Args:
        latest_date_str: Date string in YYYY-MM-DD format
        threshold_bdays: Maximum acceptable age in business days (default 2)

    Returns:
        True if data is stale (older than threshold)
    """
    try:
        latest = date.fromisoformat(latest_date_str)
        current_date = as_of_date or _live_et_date()
        if latest > current_date:
            return True
        bdays = count_sessions(
            "XNYS",
            latest,
            current_date,
            include_start=False,
            include_end=True,
        )
        return bdays > threshold_bdays
    except (ValueError, TypeError):
        return True


def _live_et_date() -> date:
    """Return the live XNYS calendar date without machine-local time."""
    from datetime import datetime

    return datetime.now(ZoneInfo("America/New_York")).date()


def fetch_sector_uptrend_data(*, as_of_date: Optional[date] = None) -> dict[str, dict]:
    """Fetch sector uptrend ratio data from timeseries CSV.

    Downloads the full timeseries, extracts the latest row per sector,
    and calculates slope from the last 5 data points.

    Returns:
        Dict mapping display sector name to:
        {
            "ratio": float,       # Current uptrend ratio (0.0-1.0)
            "ma_10": float,       # 10-period moving average
            "slope": float,       # Slope from last 5 data points
            "trend": str,         # "up" or "down"
            "latest_date": str,   # YYYY-MM-DD
        }
        Empty dict on failure.
    """
    if not HAS_REQUESTS:
        print("WARNING: requests library not installed.", file=sys.stderr)
        return {}

    try:
        response = requests.get(TIMESERIES_URL, timeout=30)
        response.raise_for_status()
    except Exception as e:
        print(f"WARNING: Failed to fetch uptrend timeseries: {e}", file=sys.stderr)
        return {}

    # Parse all rows, grouped by worksheet
    sector_rows: dict[str, list[dict]] = {}
    reader = csv.DictReader(io.StringIO(response.text))
    for row in reader:
        ws = row.get("worksheet", "").strip()
        if ws == "all" or not ws.startswith("sec_"):
            continue
        date_str = row.get("date", "").strip()
        ratio = _safe_float(row.get("ratio"))
        ma_10 = _safe_float(row.get("ma_10"))
        slope_csv = _safe_float(row.get("slope"))
        trend = row.get("trend", "").strip()

        if not date_str or ratio is None:
            continue
        try:
            row_date = date.fromisoformat(date_str)
        except ValueError:
            continue
        if as_of_date is not None and row_date > as_of_date:
            continue

        if ws not in sector_rows:
            sector_rows[ws] = []
        sector_rows[ws].append(
            {
                "date": date_str,
                "ratio": ratio,
                "ma_10": ma_10,
                "slope_csv": slope_csv,
                "trend": trend,
            }
        )

    # For each sector: sort by date, get latest, calculate slope from last 5
    result = {}
    for ws_name, rows in sector_rows.items():
        rows.sort(key=lambda r: r["date"])
        latest = rows[-1]
        display_name = WORKSHEET_TO_DISPLAY.get(ws_name, ws_name)

        # Calculate slope from last 5 ratio values
        last_n = rows[-5:] if len(rows) >= 5 else rows
        ratio_values = [r["ratio"] for r in last_n if r["ratio"] is not None]
        calculated_slope = _calculate_slope(ratio_values)

        # Prefer CSV slope if available, otherwise use calculated
        slope = latest["slope_csv"] if latest["slope_csv"] is not None else calculated_slope

        result[display_name] = {
            "ratio": latest["ratio"],
            "ma_10": latest["ma_10"],
            "slope": slope,
            "trend": latest["trend"] or ("up" if slope and slope > 0 else "down"),
            "latest_date": latest["date"],
        }

    return result


def build_summary_from_timeseries(sector_timeseries: dict[str, dict]) -> list[dict]:
    """Build a sector summary list from timeseries data.

    Fallback when sector_summary.csv is unavailable.

    Args:
        sector_timeseries: Dict from fetch_sector_uptrend_data()

    Returns:
        List of dicts with: Sector, Ratio, 10MA, Trend, Slope, Status
    """
    OVERBOUGHT = 0.37
    OVERSOLD = 0.097

    rows = []
    for sector_name, data in sector_timeseries.items():
        ratio = data.get("ratio")
        status = (
            "Overbought"
            if ratio is not None and ratio > OVERBOUGHT
            else "Oversold"
            if ratio is not None and ratio < OVERSOLD
            else "Normal"
        )
        rows.append(
            {
                "Sector": sector_name,
                "Ratio": ratio,
                "10MA": data.get("ma_10"),
                "Trend": (data.get("trend") or "").capitalize(),
                "Slope": data.get("slope"),
                "Status": status,
            }
        )
    return rows


def get_sector_uptrend_3point(sector_name: str, all_data: dict[str, dict]) -> Optional[dict]:
    """Get ratio + ma_10 + slope for a specific sector.

    Args:
        sector_name: Display name (e.g., "Technology")
        all_data: Output from fetch_sector_uptrend_data()

    Returns:
        Dict with ratio, ma_10, slope, trend, latest_date. None if not found.
    """
    return all_data.get(sector_name)

---
name: theme-detector
description: Detect and analyze trending market themes across sectors. Use when user asks about current market themes, trending sectors, sector rotation, thematic investing, what themes are hot or cold, or wants to identify bullish and bearish market narratives with lifecycle analysis.
---

# Theme Detector

## Overview

This skill detects and ranks trending market themes by analyzing cross-sector momentum, volume, and breadth signals. It identifies both bullish (upward momentum) and bearish (downward pressure) themes, assesses lifecycle maturity (Emerging/Accelerating/Trending/Mature/Exhausting), and provides a confidence score combining quantitative data with narrative analysis.

**3-Dimensional Scoring Model:**
1. **Theme Heat** (0-100): Direction-neutral strength of the theme (momentum, volume, uptrend ratio, breadth)
2. **Lifecycle Maturity**: Stage classification (Emerging / Accelerating / Trending / Mature / Exhausting) based on duration, extremity clustering, valuation, and ETF proliferation
3. **Confidence** (Low / Medium / High): Reliability of the detection, combining quantitative breadth with narrative confirmation. Script output is capped at Medium; Claude's WebSearch narrative confirmation step can elevate to High.
4. **Stock Leadership**: Optional daily scan-hit evidence from 5D+20%, EP9M, range expansion, new highs, and high-RS stocks. When supplied, this is blended into Theme Heat v2; when absent, it lowers confidence coverage but does not force leadership to zero.
5. **Theme Match Quality**: Specificity of the theme match from industry participation, explicit stock-basket hits, proxy ETF confirmation, and optional offline narrative scores. This is evidence quality, not a trade recommendation.

**Key Features:**
- Cross-sector theme detection using FINVIZ industry data
- Direction-aware scoring (bullish and bearish themes)
- Lifecycle maturity assessment to identify crowded vs. emerging trades
- ETF proliferation scoring (more ETFs = more mature/crowded theme)
- Integration with uptrend-dashboard for 3-point evaluation
- Stock-level leadership evidence via `--scan-hits`
- Theme match quality via explicit stock baskets and proxy ETF confirmation
- Leader candidate evidence ranked by abnormal move/volume/range/RS metrics, with market cap shown only as a risk bucket
- Theme history and acceleration metrics via `--history-file`
- Dual-mode operation: FINVIZ Elite (fast) or public scraping (slower, limited)
- WebSearch-based narrative confirmation for top themes

---

## When to Use This Skill

**Explicit Triggers:**
- "What market themes are trending right now?"
- "Which sectors are hot/cold?"
- "Detect current market themes"
- "What are the strongest bullish/bearish narratives?"
- "Is AI/clean energy/defense still a strong theme?"
- "Where is sector rotation heading?"
- "Show me thematic investing opportunities"

**Implicit Triggers:**
- User wants to understand broad market narrative shifts
- User is looking for thematic ETF or sector allocation ideas
- User asks about crowded trades or late-cycle themes
- User wants to know which themes are emerging vs. exhausted

**When NOT to Use:**
- Individual stock analysis (use us-stock-analysis instead)
- Specific sector deep-dive with chart reading (use sector-analyst instead)
- Portfolio rebalancing (use portfolio-manager instead)
- Dividend/income investing (use value-dividend-screener instead)

---

## Prerequisites

**Required:**
- Python 3.9+ with core dependencies.
  ```bash
  pip install -r skills/theme-detector/requirements.txt
  ```

**Cron / mixed-Python fallback:** If the active `python3` is older than 3.10, or a newer Hermes venv lacks the data-science dependencies, run the detector through `uv` with an explicit modern interpreter and temporary dependencies instead of editing the environment mid-cron:
```bash
uv run --python 3.12 \
  --with requests --with beautifulsoup4 --with lxml \
  --with pandas --with numpy --with yfinance \
  --with finvizfinance --with PyYAML \
  python skills/theme-detector/scripts/theme_detector.py \
  --finviz-api-key "$FINVIZ_API_KEY" \
  --fmp-api-key "$FMP_API_KEY" \
  --output-dir reports/
```
Use this as a setup workaround, not as evidence that the detector is broken; still report FINVIZ/FMP/API-data caveats separately.

**Optional API Keys:**

FINVIZ Elite (recommended for full industry coverage and speed):
```bash
export FINVIZ_API_KEY=your_finviz_elite_api_key_here
```

FMP API (optional, for P/E ratio valuation data):
```bash
export FMP_API_KEY=your_fmp_api_key_here
```

The requirements include `finvizfinance`, PyYAML, pandas/numpy, requests, and
yfinance because normal public-mode execution imports or uses each of them.

Without FINVIZ Elite, the skill uses public FINVIZ scraping (limited to ~20 stocks per industry, slower rate limits).

---

## Workflow

### Step 1: Verify Environment

Check that API keys are configured (see Prerequisites):

```bash
# Verify FINVIZ Elite API key (optional but recommended)
echo $FINVIZ_API_KEY

# Verify FMP API key (optional)
echo $FMP_API_KEY
```

### Step 2: Execute Theme Detection Script

Run the main detection script:

```bash
python3 skills/theme-detector/scripts/theme_detector.py \
  --output-dir reports/
```

**Script Options:**
```bash
# Full run (public FINVIZ mode, no API key required)
python3 skills/theme-detector/scripts/theme_detector.py \
  --output-dir reports/

# With FINVIZ Elite API key
python3 skills/theme-detector/scripts/theme_detector.py \
  --finviz-api-key $FINVIZ_API_KEY \
  --output-dir reports/

# With FMP API key for enhanced stock data
python3 skills/theme-detector/scripts/theme_detector.py \
  --fmp-api-key $FMP_API_KEY \
  --output-dir reports/

# Custom limits
python3 skills/theme-detector/scripts/theme_detector.py \
  --max-themes 5 \
  --max-stocks-per-theme 10 \
  --output-dir reports/

# Explicit FINVIZ mode
python3 skills/theme-detector/scripts/theme_detector.py \
  --finviz-mode public \
  --output-dir reports/

# Add Stockbee/Pradeep-style leadership evidence
python3 skills/theme-detector/scripts/theme_detector.py \
  --scan-hits data/theme_scan_hits_YYYY-MM-DD.json \
  --narrative-scores data/theme_narrative_scores_YYYY-MM-DD.json \
  --history-file reports/theme_detector_history.json \
  --as-of-date YYYY-MM-DD \
  --output-dir reports/
```

**Scan-hit input contract:** `--scan-hits` accepts JSON, JSONL, or CSV. Rows may be pre-labeled with `scan_type` / `scan_types`, or raw rows with fields such as `symbol`, `return_5d`, `change_pct`, `volume`, `avg_volume_50d`, `relative_volume`, `true_range`, `atr_20`, `atr_expansion`, `close_location`, `industry`, `sector`, and `theme_guess`. A raw row can expand into multiple hits when it satisfies multiple rules.

Initial scan rules:
- `five_day_20pct`: `return_5d >= 20`
- `ep9m`: `volume >= 9,000,000`, `relative_volume >= 2.0`, and `change_pct >= 4`
- `range_expansion`: `change_pct >= 4`, `true_range / atr_20 >= 1.5` or `atr_expansion >= 1.5`, and `close_location >= 0.75`
- `new_high`: explicit `new_high` / `is_new_high`, or 52-week high evidence
- `high_rs`: `rs_rating >= 90` or normalized `relative_strength >= 0.90`

**Narrative-score input contract:** `--narrative-scores` is an offline JSON input, not a live WebSearch call. It accepts either `{"Theme Name": 82}` or `{"themes": {"Theme Name": {"narrative_keyword_score": 82}}}`. Missing narrative input leaves `narrative_keyword_score` as `null` and reduces `theme_match_coverage`; it does not fail the run.

**Expected Execution Time:**
- FINVIZ Elite mode: ~2-3 minutes (14+ themes)
- Public FINVIZ mode: ~5-8 minutes (rate-limited scraping)

### Step 3: Read and Parse Detection Results

The script generates two output files:
- `theme_detector_YYYY-MM-DD_HHMMSS.json` - Structured data for programmatic use
- `theme_detector_YYYY-MM-DD_HHMMSS.md` - Human-readable report

Read the JSON output to understand quantitative results:

```bash
# Find the latest report
ls -lt reports/theme_detector_*.json | head -1

# Read the JSON output
cat reports/theme_detector_YYYY-MM-DD_HHMMSS.json
```

### Step 4: Perform Narrative Confirmation via WebSearch

For the top 5 themes (by Theme Heat score), execute WebSearch queries to confirm narrative strength:

**Search Pattern:**
```
"[theme name] stocks market [current month] [current year]"
"[theme name] sector momentum [current month] [current year]"
```

**Evaluate narrative signals:**
- **Strong narrative**: Multiple major outlets covering the theme, analyst upgrades, policy catalysts
- **Moderate narrative**: Some coverage, mixed sentiment, no clear catalyst
- **Weak narrative**: Little coverage, or predominantly contrarian/skeptical tone

Update Confidence levels based on findings:
- Quantitative High + Narrative Strong = **High** confidence
- Quantitative High + Narrative Weak = **Medium** confidence (possible momentum divergence)
- Quantitative Low + Narrative Strong = **Medium** confidence (narrative may lead price)
- Quantitative Low + Narrative Weak = **Low** confidence

### Step 5: Analyze Results and Provide Recommendations

Cross-reference detection results with knowledge bases:

**Reference Documents to Consult:**
1. `references/cross_sector_themes.md` - Theme definitions and constituent industries
2. `references/thematic_etf_catalog.md` - ETF exposure options by theme
3. `references/theme_detection_methodology.md` - Scoring model details
4. `references/finviz_industry_codes.md` - Industry classification reference

**Analysis Framework:**

For **Hot Bullish Themes** (Heat >= 70, Direction = Bullish):
- Identify lifecycle stage (Emerging = opportunity, Mature/Exhausting = caution)
- List top-performing industries within the theme
- Recommend proxy ETFs for exposure
- Flag if ETF proliferation is high (crowded trade warning)

For **Hot Bearish Themes** (Heat >= 70, Direction = Bearish):
- Identify industries under pressure
- Assess if bearish momentum is accelerating or decelerating
- Recommend hedging strategies or sectors to avoid
- Note potential mean-reversion opportunities if lifecycle is Mature/Exhausting

For **Emerging Themes** (Heat 40-69, Lifecycle = Emerging):
- These may represent early rotation signals
- Recommend monitoring with watchlist
- Identify catalyst events that could accelerate the theme

For **Exhausted Themes** (Heat >= 60, Lifecycle = Exhausting):
- Warn about crowded trade risk
- High ETF count confirms excessive retail participation
- Consider contrarian positioning or reducing exposure

### Step 6: Generate Final Report

Present the final report to the user using the report template structure:

```markdown
# Theme Detection Report
**Date:** YYYY-MM-DD
**Mode:** FINVIZ Elite / Public
**Themes Analyzed:** N
**Data Quality:** [note any limitations]

## Theme Dashboard
[Top themes table with Heat, Direction, Lifecycle, Confidence]

## What Changed Today
[Newly emerging themes, largest heat acceleration, new EP9M clusters, fading themes]

## Leadership Evidence
[5D+20%, EP9M, range expansion, new highs, high-RS counts and leader symbols]

## Bullish Themes Detail
[Detailed analysis of bullish themes sorted by Heat]

## Bearish Themes Detail
[Detailed analysis of bearish themes sorted by Heat]

## All Themes Summary
[Complete theme ranking table]

## Industry Rankings
[Top performing and worst performing industries]

## Sector Uptrend Ratios
[Sector-level aggregation if uptrend data available]

## Methodology Notes
[Brief explanation of scoring model]
```

Save the report to `reports/` directory.

---

## Output

The skill generates two output files in the `reports/` directory:

**JSON Output** (`theme_detector_YYYY-MM-DD_HHMMSS.json`):
```json
{
  "report_type": "theme_detector",
  "generated_at": "2026-04-18 10:30:00",
  "metadata": {
    "generated_at": "2026-04-18 10:30:00",
    "data_mode": "full",
    "finviz_mode": "elite",
    "fmp_available": true,
    "max_themes": 14,
    "max_stocks_per_theme": 5,
    "data_sources": {
      "finviz_industries": 152,
      "yfinance_stocks": 68,
      "etf_volume": 24
    }
  },
  "summary": {
    "total_themes": 14,
    "bullish_count": 8,
    "bearish_count": 6,
    "top_bullish": "AI & Machine Learning",
    "top_bearish": "Regional Banks"
  },
  "themes": {
    "all": [
      {
        "name": "AI & Machine Learning",
        "direction": "bullish",
        "heat": 85.3,
        "maturity": 42.1,
        "stage": "Accelerating",
        "confidence": "Medium",
        "heat_label": "Hot",
        "industries": ["Software - Infrastructure", "Semiconductors"],
        "representative_stocks": ["NVDA", "MSFT"],
        "stock_details": [{"symbol": "NVDA"}, {"symbol": "MSFT"}],
        "proxy_etfs": ["BOTZ", "ROBO"],
        "theme_match_score": 78.4,
        "theme_match_components": {
          "industry_match_score": 84.2,
          "static_stock_hit_score": 80.0,
          "proxy_etf_momentum_score": 70.0,
          "narrative_keyword_score": null
        },
        "leader_candidates": [
          {
            "symbol": "NVDA",
            "leader_score": 91.2,
            "scan_types": ["ep9m", "range_expansion"],
            "risk_bucket": "mega"
          }
        ],
        "theme_origin": "seed"
      }
    ],
    "bullish": [...],
    "bearish": [...],
    "match_ranked": [...]
  },
  "industry_rankings": {
    "top": [...],
    "bottom": [...]
  },
  "sector_uptrend": {...},
  "data_quality": {...}
}
```

**Markdown Report** (`theme_detector_YYYY-MM-DD_HHMMSS.md`):
- Theme Dashboard with sortable rankings
- Bullish/Bearish theme detail sections
- Industry performance rankings
- Sector uptrend ratio summary
- Methodology notes

**Key Output Fields (per theme):**
| Field | Description |
|-------|-------------|
| `heat` | 0-100 direction-neutral theme strength |
| `direction` | `"bullish"` (LEAD) or `"bearish"` (LAG) |
| `stage` | Emerging / Accelerating / Trending / Mature / Exhausting |
| `confidence` | Low / Medium / High (script caps at Medium; WebSearch can elevate) |
| `representative_stocks` | Top ticker symbols for the theme |
| `stock_details` | Optional stock metric objects for the selected representatives |
| `proxy_etfs` | Thematic ETF tickers (length = ETF count; higher = more crowded) |
| `theme_match_score` | 0-100 evidence-quality score from industries, stock basket hits, proxy ETF confirmation, and optional narrative input |
| `theme_match_components` | Inspectable sub-scores explaining the theme match |
| `leader_candidates` | Evidence-ranked symbols for the theme; not entry/stop/invalidation guidance |
| `fresh_leadership_symbols` | Current-run symbols with EP9M, range expansion, or new-high evidence |
| `extended_symbols` | Current-run symbols with 5D+20% evidence; used as overextension evidence only |
| `theme_origin` | `"seed"` (from YAML config) or `"discovered"` (auto-clustered) |

---

## Resources

### Scripts Directory (`scripts/`)

**Main Scripts:**
- `theme_detector.py` - Main orchestrator script
  - Coordinates industry data collection, theme classification, and scoring
  - Generates JSON + Markdown output
  - Usage: `python3 theme_detector.py [options]`

- `theme_classifier.py` - Maps industries to cross-sector themes
  - Reads theme definitions from `cross_sector_themes.md`
  - Calculates theme-level aggregated scores
  - Determines direction (bullish/bearish) from constituent industries
  - Display mapping: "bullish" → "LEAD", "bearish" → "LAG" (see report_generator.py::_direction_label())

- `finviz_industry_scanner.py` - FINVIZ industry data collection
  - Elite mode: CSV export with full stock data per industry
  - Public mode: Web scraping with rate limiting
  - Extracts: performance, volume, change%, avg volume, market cap

- `calculators/lifecycle_calculator.py` - Lifecycle maturity assessment
  - Duration scoring, extremity clustering, valuation analysis
  - ETF proliferation scoring from thematic_etf_catalog.md
  - Stage classification: Emerging / Accelerating / Trending / Mature / Exhausting

- `report_generator.py` - Report output generation
  - Markdown report from template
  - JSON structured output
  - Theme dashboard formatting

### References Directory (`references/`)

**Knowledge Bases:**
- `cross_sector_themes.md` - Theme definitions with industries, ETFs, stocks, and matching criteria
- `thematic_etf_catalog.md` - Comprehensive thematic ETF catalog with counts per theme
- `finviz_industry_codes.md` - Complete FINVIZ industry-to-filter-code mapping
- `theme_detection_methodology.md` - Technical documentation of the 3D scoring model

### Assets Directory (`assets/`)

- `report_template.md` - Markdown template for report generation with placeholder format

---

## Exchange Calendar and Replay

Install `requirements.txt` before running the detector. `--as-of-date` is a
strict `YYYY-MM-DD` ceiling used by both uptrend freshness checks and provider
history windows. Because FINVIZ, quote, profile, and uptrend inputs are live
rather than PIT snapshots, a non-current `--as-of-date` fails closed. Freshness
counts XNYS sessions and excludes future-dated source rows.

## Important Notes

### FINVIZ Mode Differences

| Feature | Elite Mode | Public Mode |
|---------|-----------|-------------|
| Industry coverage | All ~145 industries | All ~145 industries |
| Stocks per industry | Full universe | ~20 stocks (page 1) |
| Rate limiting | 0.5s between requests | 2.0s between requests |
| Data freshness | Real-time | 15-min delayed |
| API key required | Yes ($39.50/mo) | No |
| Execution time | ~2-3 minutes | ~5-8 minutes |

### Direction Detection Logic

Theme direction is determined by majority vote of constituent industries' relative rank:

1. **Industry ranking**: All ~145 industries are ranked by multi-timeframe momentum score
2. **Rank-based direction**: Industries in the top half of the ranked list are classified as "bullish"; bottom half as "bearish"
3. **Theme majority vote**: `_majority_direction()` counts bullish vs. bearish industries within each theme; the majority wins

Display mapping: "bullish" → **LEAD**, "bearish" → **LAG** (see `report_generator.py::_direction_label()`)

A LEAD theme indicates relative outperformance of its constituent industries. A LAG theme may still have positive absolute returns — it indicates relative underperformance, not a short signal.

### Known Limitations

1. **Survivorship bias**: Only analyzes currently listed stocks and ETFs
2. **Lag**: FINVIZ data may lag intraday moves by 15 minutes (public mode)
3. **Theme boundaries**: Some stocks fit multiple themes; classification uses primary industry
4. **ETF proliferation**: Catalog is static and may not capture very new ETFs
5. **Narrative scoring**: WebSearch-based and inherently subjective
6. **Public mode limitation**: ~20 stocks per industry may miss small-cap signals

### Disclaimer

**This analysis is for educational and informational purposes only.**
- Not investment advice
- Past thematic trends do not guarantee future performance
- Theme detection identifies momentum, not fundamental value
- Conduct your own research before making investment decisions

---

**Version:** 1.0
**Last Updated:** 2026-02-16
**API Requirements:** FINVIZ Elite (recommended) or public mode (free); FMP API optional
**Execution Time:** ~2-8 minutes depending on mode
**Output Formats:** JSON + Markdown
**Themes Covered:** 14+ cross-sector themes