Skip to contents

Changelog

Source: NEWS.md

Refinitiv 0.2.4

Security & Credential Handling

  • Credential vault isolation (#15, #2, #5) — refinitiv_vault_set() no longer writes api_key through to options(.EikonApiKey) (while refinitiv_vault_get("api_key") retains a read-only fallback for legacy callers, and .onLoad() clears .EikonApiKey after migrating it into the private vault). DataStreamConnect() now stores credentials in the private vault and passes username and password directly to DatastreamDSWS2R::dsws$new() without writing to or clearing options(Datastream.Username) / options(Datastream.Password). TestDataStreamCredentials() now uses https://, URL-encodes credentials with reserved = TRUE, repeated = TRUE, and scrubs passwords and query strings from diagnostic messages and warnings.
  • Token lifecycle & handshake defaults (#9, #16b, #16c) — rd_handshake() now defaults to force = FALSE, reusing valid cached bearer tokens instead of re-probing ports and fetching a new token on every call. Routine token expiry checks via rd_VerifyToken(..., warn = FALSE) no longer emit a warning during normal refresh, StreamManager invalidates the cached token when the server closes the Login stream (Status: Closed), rd_handshake() uses get_pkg_version() as its fallback version, and get_rdp_streaming_url() no longer sends duplicate HTTP requests.

Streaming, Caching & Request Pipeline Fixes

  • Batched WebSocket frames & dynamic port resolution (#1, #13, #1) — StreamManager$.process_message() now iterates over all elements in batched OMM JSON array frames instead of dropping multi-element frames, with per-message tryCatch isolation so a malformed message cannot drop subsequent messages in the same frame. When REFINITIV_PORT is unset at load time, streaming_port is left NULL and resolved dynamically from getOption("eikon_port") (with 9000L fallback). Imported %||% from rlang in NAMESPACE for full compatibility with R 4.1–4.3.
  • HTTP status code validation & fail-fast on permanent 4xx errors (#6, #14) — send_json_request() now checks HTTP status codes (>= 400), guards resp_body_json() against non-JSON error responses (such as HTML 502/503 proxy pages), fails immediately on initial synchronous ErrorCode == 400 responses (while still allowing retry on async ticket-poll 400 responses), uses 5 * (counter + 1L) linear backoff (5s, 10s, 15s, ...), and signals refinitiv_http_error on permanent HTTP 4xx errors (400, 401, 403, 404) so retry() and chunked_download() fail fast without sleeping through chunk retries.
  • Input order preservation & Eikon_Error_Data row synchronization (#4, #3) — When caching is active, rd_GetData(), EikonGetData(), and EikonGetSymbology() restore the caller’s input instrument order (including repeated instruments) via .restore_input_order(), and EikonGetData() remaps 0-based Eikon_Error_Data$row indices to match the reordered PostProcessedEikonGetData rows (as well as offsetting Eikon_Error_Data$row across multiple chunks in EikonPostProcessor()). Fixed if (!isFALSE(verbose)) on cache hits so options(refinitiv_progress = "verbose") no longer errors.
  • Chunked download, pagination, date parsing & URL encoding (#5, #7, #8, #10, #11, #12) — chunked_download() no longer sleeps refinitiv_chunk_sleep after the final chunk, preserves NULL chunk returns via results[j] <- list(...), and tracks pre-fetch cache hits separately for accurate (N from cache) reporting. rd_GetOwnership() guards unlimited pagination with duplicate-page detection and getOption("refinitiv_ownership_max_pages", 100L). rd_OutputProcesser() and rd_GetHistory() safely guard as.Date() coercion on date-named columns containing non-ISO strings (e.g. "Q1 2023"). Historical pricing endpoints now URL-encode universe and query parameters via build_get_query_string(), and manage_custom_instrument respects its debug parameter instead of hardcoding debug = TRUE.

Refinitiv 0.2.3

Robustness & AQL Collapse Recovery

  • Shared AQL collapse detection & adaptive ≤25\le 25-RIC sub-chunk fallback — EikonGetData() and rd_GetData() now validate chunks with eikon_data_is_success() and .is_eikon_aql_collapse() to detect collapsed 1-column HTTP 200 payloads (totalColumnsCount <= 1 or single unnamed displayName = NULL header with cell errors code %in% c(100, 400, 500, 2504) or "Asynchronous Query library internal error"). When an oversized chunk hits an LSEG AQL collapse (code = 100), .fetch_with_aql_subchunks() automatically splits the chunk into ≤25\le 25-RIC sub-chunks (getOption("refinitiv_aql_fallback_rics", 25L)), merges the raw payloads via .merge_eikon_raw_subchunks(), and steps down remaining oversized chunks in the same call.
  • Multi-period chunk sizing & 10,000-row DataGrid truncation guard — EikonChunker() now estimates rows per instrument when Duration or multi-period Parameters (SDate/EDate/FRQ) are supplied so chunks stay below LSEG DataGrid’s 10,000-row limit. EikonGetData() and rd_GetData() emit an explicit warning if any DataGrid chunk returns ≥10,000\ge 10,000 rows.
  • Session cache reference isolation & partial-failure protection — cache_set() and cache_get() now deep-copy cached objects (data.table::copy(), including EikonResult wrappers) so caller-side in-place data.table mutations (:=, setnames(), setcolorder()) never corrupt cached results. Multi-chunk downloads in EikonGetData(), EikonGetTimeseries(), and rd_GetHistoricalPricing() no longer poison the top-level session cache when on_failure = "warning" produces partial NA chunks.
  • Async ticket poll payload restoration — send_json_request() restores json <- original_json before retrying after a retriable ErrorCode (2504, 500, 400) so subsequent retries do not send a stale async ticket poll payload.

Bug Fixes

  • rd_GetHistory() data integrity & edge-case fixes — Fixed 0-row merge crashes when either or both endpoints return empty tables, applied end (EDate) even when start is NULL, mapped interval to parameters[["FRQ"]] (case-insensitively preserving user-supplied FRQ/Frq), forwarded cache = cache to underlying rd_GetData() and rd_GetHistoricalPricing() calls, and replaced .. column selection with .SD/.SDcols.
  • Intraday POSIXct timestamp preservation & "6M" interval handling — PostProcessTimeSeriesRequest(), rd_OutputProcesser(), and rd_GetHistory() now preserve full UTC POSIXct timestamps for intraday data ("tick", "minute", "hour", and ISO 8601 intraday intervals) instead of truncating to Date. Removed silent coercion of unsupported "6m"/"6M" to quarterly "P3M" in translate_to_iso8601_duration().
  • WebSocket streaming resilience & custom API key propagation — Fixed StreamManager$.attempt_reconnect() to pass the vaulted access token to rd_VerifyToken(), preserved the reconnect attempt counter for on_reconnected callbacks before resetting, unregistered reconnect callbacks on Stream$close(), inspected OMM State$Data == "Suspect" and State$Stream == "Closed" in Stream, wired StreamManager’s base_url to URL construction, and propagated custom API keys from the credential vault across rd_handshake() and WebSocket OMM login headers.
  • Parser hardening, ADC diagnostics & pagination — Sanitized collapsed all-NULL headers in JsonHeaderAnalyzer() and EikonPostProcessor() (Instrument / Column_<idx>) with safe fallback to headers[["name"]], guarded DateValueCol renaming in rd_OutputProcesser(), tightened date column regex matching ([._]date$|\bdate\b|^date_time$) so columns like ConsolidatedUpdate or Mandate are not misclassified as dates, hardened ProcessSymbology() when leading chunk symbols fail to resolve, added unresolved-identifier warnings in process_adc_response(), fixed non-zero initial offset pagination in rd_GetOwnership(), and vectorized CheckifCustomInstrument().
  • Interval alias resolution in rd_GetHistory() — rd_GetHistory() normalizes friendly interval aliases (e.g., "daily", "1d", "1D", "1min", "weekly", etc.) via translate_to_iso8601_duration() before forwarding to rd_GetHistoricalPricing().
  • Preserve ISO 8601 durations in translate_to_iso8601_duration() — Valid ISO 8601 duration inputs (such as "PT1M", "P1D", "PT5M") are preserved instead of defaulting to daily intervals.
  • Parse integer YYYYMMDD dates in rd_OutputProcesser() — Date fields returning numeric YYYYMMDD integer values (such as EXPIR_DATE) are converted to Date objects correctly instead of treating values as days since epoch.

Testing & Quality

  • 100.00% unit test coverage — Added comprehensive regression tests in test-v023-audit-fixes.R covering every v0.2.3 audit fix and edge case, bringing overall package line coverage across all 37 files in R/ to 100.00% (2,305 passing test assertions, 0 failures).

Refinitiv 0.2.2

Robustness

  • Survive LSEG proxy connection resets — Under sustained bulk load the LSEG Workspace proxy can forcibly close the TCP connection (WinError 10054), which surfaces as an httr2 "Recv failure" / "Connection was reset" error thrown before any HTTP response exists. httr2::req_retry() only inspects responses, so it never fired on these transport-level failures and a single reset could fail an entire chunked download. send_json_request() now retries req_perform() on genuine connection-reset signatures (WinError 10054, forcibly closed, Connection was reset, Recv/Send failure, reset by peer) with exponential backoff. It deliberately does not match httr2’s generic "Failed to perform HTTP request" wrapper, so ordinary errors (connection refused, DNS, timeout) still fail fast.

  • Inter-chunk pacing in chunked_download() — Adds a configurable sleep between sequential chunk requests and a longer jittered backoff between retry rounds, giving the proxy room to recover after a reset.

New Features

  • Tunable resilience options — All of the new resilience delays are driven by options so they can be tuned in production (or zeroed in tests/CI) without code changes: refinitiv_reset_retries (default 3), refinitiv_reset_base_wait (default 5s), refinitiv_reset_max_wait (default 30s), refinitiv_chunk_sleep (default 1s), and refinitiv_chunk_backoff (default 3s). EikonGetData(), EikonGetTimeseries() and rd_GetData() inherit the single chunked_download() default rather than carrying their own sleep values.

CI/CD & Infrastructure

  • .gitignore / .Rbuildignore hardened — Ignore the local mcps/ tool-definition cache and README.html so they are no longer flagged as non-standard top-level files by R CMD check.

Refinitiv 0.2.1

New Features

  • rd_ConvertSymbol() — New function for converting between symbology types (RIC, ISIN, CUSIP, SEDOL, etc.) with multi-tiered fallback: SymbologySearch → EikonGetSymbology → GetData. Supports DelistingDate natively and handles ISIN→RIC, RIC→ISIN, RIC→MIC, and more.

  • fields parameter alias in rd_GetData() — fields can now be used as an alias for Eikonformulas, aligning the R API with the lseg-data Python library. Eikonformulas continues to work but is soft-deprecated.

  • EikonResult S3 class — EikonGetData() now returns an EikonResult object that wraps both data and error components. as.data.table() and as.data.frame() methods extract only the data portion, eliminating recycling warnings. Fully backward-compatible: $PostProcessedEikonGetData and $Eikon_Error_Data still work.

Bug Fixes

  • Cache key canonicalization (STOXX-2) — cache_key() now sorts character vectors before hashing, ensuring the same set of RICs produces the same cache key regardless of input order. Diagnostic logging available via options(refinitiv_verbose_cache = TRUE).

  • EikonGetSymbology() multi-chunk fix (STOXX-3) — EikonGetSymbology() now correctly combines results across multiple chunks. Previously only the first chunk was processed, silently discarding all subsequent chunks for requests with >300 symbols.

CI/CD & Infrastructure

  • Windows MAX_PATH fix — Shortened httptest2 API fixture paths in the redactor (api/rdp/data/ → d/, news/v1/top-news → tn, etc.) and removed an accidentally committed Refinitiv.Rcheck/ directory (456 files) that contained old long-path fixtures. Package download size dropped from 6.23 MB to 2.83 MB.

  • Installation test workflow — New install-test.yaml validates that the package installs and loads successfully on Ubuntu, macOS, and Windows.

  • Node.js 24 migration — Bumped actions/checkout to v5 and codecov/codecov-action to v6 (both native Node 24). Removed the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24 workaround.

  • Modern installation docs — README now recommends pak::pak() instead of the deprecated devtools::install_github().

  • .gitignore / .Rbuildignore hardened — Added *.Rcheck/ patterns to prevent future accidental commits of R CMD check output.

Refinitiv 0.2.0

Breaking Changes

  • Python/reticulate support removed. The package now uses direct JSON API calls exclusively. No Python installation is required.

New Features

  • Real-time streaming data — New WebSocket-based streaming API via R6 classes (rd_get_streaming_data(), StreamManager, Stream, StreamDefinition). Supports multi-instrument subscriptions, callbacks (on_refresh, on_update, on_error), data history buffering, summary statistics, and built-in live plotting with Shiny.

    • Auto-reconnect with jittered exponential backoff and automatic re-subscription on disconnect.
    • Sequential stream IDs for multi-instrument support.
    • Debug logging via options(refinitiv_streaming_debug = TRUE).

  • ESG data — rd_GetESG() retrieves ESG scores, measures, basic overviews, and universe data with optional fiscal year filtering.

  • Consensus estimates — rd_GetEstimates() retrieves I/B/E/S actuals, summary, and KPI data across 15 views with package-level validation.

  • Ownership data — rd_GetOwnership() retrieves consolidated, fund, insider, investor, and organizational ownership data across 19 views with automatic pagination.

  • Session-scoped request cache — Opt-in caching layer (R/cache.R) with configurable TTLs per function. Enable globally with options(refinitiv_cache = TRUE) or per-call with cache = TRUE. Helpers: rd_ClearCache(), rd_CacheInfo(). Instrumented across 13 exported functions.

  • Custom instruments — rd_ManageCustomInstruments() for create, read, update, and delete operations, including basket instruments with custom holidays.

  • Zero-config connection — rd_connection() provides a lazy, cached singleton connection. All 20+ API functions default to it, eliminating the need to pass connection objects explicitly.

  • Environment variable configuration — Set REFINITIV_PORT, REFINITIV_BASE_URL, and/or REFINITIV_APP_KEY in .Renviron for headless/CI environments. Skips interactive terminal detection.

  • EikonRepairMic() updated — Operating MIC lookup data refreshed for current exchange codes.

Bug Fixes & Improvements

  • retry() rewritten (C2) — Now accepts a zero-argument closure with iterative loop and exponential backoff. Max attempts configurable via options(refinitiv_max_retries). Previously the retry mechanism did not actually re-execute failed calls.

  • retry() throws on exhaustion by default (#17) — retry() now defaults to on_failure = "stop", throwing an error with the last failure message when all attempts are exhausted. Previously it returned NA with a warning, causing confusing downstream errors. Use on_failure = "NA" for the chunked_download() use case where per-chunk success is tracked externally.

  • HTTP 429/503 rate-limit handling (#18) — send_json_request() now uses httr2::req_retry() to automatically retry on HTTP 429 (Too Many Requests) and 503 (Service Unavailable) responses with jittered exponential backoff. Respects Retry-After headers in both seconds and HTTP-date formats. Configurable via options(refinitiv_rate_limit_max_wait) (default 60s).

  • Retry architecture documented (#19) — The three-layer retry system (retry(), httr2::req_retry(), server polling loop) is now documented in ARCHITECTURE.md and retry() roxygen.

  • DELETE dead assignment fixed (#21) — Removed unreachable results <- NA after break in send_json_request() DELETE code path.

  • Package version in cache key (#22) — cache_key() now includes the package version, so a mid-session upgrade automatically invalidates stale cached entries.

  • Credential vault (C3) — Bearer tokens are now stored in a package-private environment invisible to getOption(), save.image(), and str(options()), replacing the previous plaintext options() storage.

  • Token expiry buffer (H1) — rd_VerifyToken() subtracts a 60-second buffer (configurable via options(refinitiv_token_buffer_seconds)) before checking token expiration, preventing clock-skew failures.

  • Unified HTTP request handling (H2) — Extracted build_request() helper to DRY four HTTP method branches in send_json_request(). All methods now use consistent error handling. Debug output changed from print() to message(). Timeouts normalized to 30s across all methods.

  • Honest handshake identity (H5) — Handshake now identifies as RefinitivR with the actual package version, replacing the previously spoofed Python library identity.

  • Hostname leakage fixed (M4) — Streaming login uses 127.0.0.1/localhost instead of the real hostname.

  • NA_cleaning default flipped (M3) — rd_OutputProcesser() now defaults to NA_cleaning = FALSE, matching the behavior of all internal callers.

  • chunked_download() extracted (M2) — Generic chunk-retry coordinator replaces 7 duplicated download-coordinator patterns. Supports parameterized sleep, max retries, success predicates, and configurable failure behavior.

  • Search port bug fixed — Construct_url("rdp") and bearer token handshake URL now correctly use eikon_port instead of rdp_port, fixing silent fallback to port 80 when rdp_port was NULL.

  • HistoricalPricing.r setnames crash fixed — No longer fails when multiple V-prefixed columns are present in the response.

  • EikonTimeSeriesPreprocessor NA handling — Defensive checks prevent crashes when Duration or Limit are NA.

  • Int64 overflow fixed (Q1) — Large integer values (exceeding 2^31-1) from LSEG responses are now converted with as.numeric() instead of as.integer(), preventing silent truncation.

  • retry() handles NULL returns (A1) — retry() now correctly treats NULL as a valid return value rather than a failure signal.

  • rd_GetHistory() merge improvements (Q2) — Duplicate column deduplication, granularity mismatch warning, NA-row filtering, and merge_info attribute on merged results.

  • Type coercion guard tightened (Q4) — Column type coercion now requires all non-NA values to be parseable before converting.

  • Optimal chunking (Q5) — Even-distribution chunking eliminates runt chunks and unnecessary extra API calls.

  • Linear backoff in polling loop (Q6) — Server polling now uses linear backoff between iterations instead of fixed sleep.

  • UTC timezone standardization (Q7) — All date conversions now explicitly use tz = "UTC".

  • EikonGetSymbology() multi-chunk fix (STOXX-3) — EikonGetSymbology() now correctly combines results across multiple chunks. Previously, ProcessSymbology() only processed the first chunk, silently discarding all subsequent chunks. Requests with >300 symbols returned only ~250 rows instead of the full result set.

Test Infrastructure

  • httptest2 integration — 24 fixture sets recorded against live Workspace terminal with a custom redactor (strips auth headers, normalizes port to 9000). All fixtures verified clean of API keys and real ports.

  • Golden sample cross-validation — 11 scenarios with both R (golden_samples_r.rds) and Python (truth_lseg_data.json) truth files, 23 cross-validation tests.

  • Test suite results — FAIL 0 | WARN 0 | SKIP 42 | PASS 1382 (1424 expectations, measured via covr::package_coverage()). 86.8% code coverage across 35 source files. 12 files at 100% coverage.

  • Option pollution fixed — All options() mutations in tests wrapped with withr::local_options() to prevent cross-test contamination.

CI

  • Removed Python/Conda/reticulate from R-CMD-check.yaml and test-coverage.yaml workflows. Both now use standard r-lib/actions patterns with automatic R package caching.

Refinitiv 0.1.3

  • Previous release with Python/reticulate support.