Skip to main content

Use External Data

Bring your own signals, indicators, or other numeric observations into MesoSim with a CSV. Its columns become variables you can use in strategy expressions.

The workflow is prepare → publish → load → use. This guide covers all four steps, whether you work in the AI Job Editor or with an AI Agent.

Prepare the CSV​

Use a header row followed by dated numeric observations. For example, this short excerpt contains an entry signal and an asset price:

date,my_entry_signal,my_asset_price
2024-01-02,0,13.5
2024-01-03,1,13.8
2024-01-04,0,14.2
  • Name the first column date or datetime (case-insensitive). Use YYYY-MM-DD or YYYY-MM-DD hh:mm:ss values in strictly increasing order.
  • Give each remaining column a unique variable name, such as my_entry_signal. Use letters, digits, and underscores, starting with a letter or underscore, and avoid names already used by the strategy or simulator.
  • Fill data columns with numeric values. Use 0 and 1 for an off/on signal.
  • Provide observations for the backtest period, including at least two days before its start and two days after its end. The excerpt above illustrates the format, not a complete dataset.
  • Keep the CSV at or below 2 MiB.

See the External CSV Data reference for the complete requirements.

Match the time when the information was available

A date-only observation, or a timestamp at midnight, is treated as an end-of-day sample and becomes available the next day. Other timestamps make the value available at that time. For example, the January 3 signal above can be used on January 4.

Use the time when the signal was actually known to avoid using future information in a backtest.

Publish the CSV​

MesoSim must be able to download the file without signing in. Use GitHub Gist or Google Sheets, and publish only data you intend to make accessible through the link. A file on your computer is not available to the backtesting service until you publish it.

GitHub Gist​

  1. Sign in to GitHub Gist and create a new gist.
  2. Give the file a .csv filename and paste the CSV contents.
  3. Create a public or secret gist. A secret gist is unlisted, but anyone with its URL can read it; see GitHub's gist guide.
  4. Open the file's Raw view and copy its gist.githubusercontent.com URL for use in MesoSim.
  5. Check the URL in a signed-out browser window: it should display or download CSV text without a login page.

Google Sheets​

  1. Open Google Sheets and import or paste your CSV. Each field must occupy its own column. If pasted rows remain in one column, select it and use Data → Split text to columns, with comma as the separator.
  2. Choose File → Share → Publish to web.
  3. Select the sheet containing your data and Comma-separated values (.csv) as the format, then publish.
  4. Copy the published URL and check that it downloads the CSV without requiring a sign-in. Use this CSV URL in MesoSim.

See Google's publishing guide for publishing settings and updates.

Load the data into your strategy​

Set ExternalData.CsvUrl to your published URL. This fragment uses the dataset from the built-in [FEAT-ExternalData-Csv] template:

{
  "ExternalData": {
    "CsvUrl": "https://gist.githubusercontent.com/tibkiss/54f9c01c6d569d6e78a1a9b23e59de68/raw/fcda115c21c9d0abaf9aea9c8513cb0bb307a2a5/mesosim-SPX-ExternalData-Template.csv"
  }
}

Merge the fragment into your complete Strategy Definition. In the AI Job Editor, select Validate to download and validate the file. Its loaded variables appear under External Data.

With an AI Agent, supply the published URL and describe the intended rule. For example:

Add this CSV to my strategy. Check its column names and date coverage, then require my_entry_signal == 1 in addition to the existing entry logic. Combine it using Lua and, preserving the grouping of any existing alternatives. Keep the other settings unchanged, validate the definition, and run a comparison with the baseline.

You can also ask the AI Assistant to help prepare the definition.

Use the imported variables​

The built-in example enters when my_entry_signal == 1 and exits when my_asset_price > 14. These fragments show the relevant conditions:

{
  "Entry": {
    "Conditions": [
      "my_entry_signal == 1"
    ]
  },
  "Exit": {
    "Conditions": [
      "my_asset_price > 14"
    ]
  }
}

Use your CSV's actual column names and preserve the other fields and conditions in your strategy. Imported variables are available in the supported expression scopes. For the complete runnable example, open [FEAT-ExternalData-Csv] from the Strategy Library.

Separate Entry.Conditions items are OR alternatives. To require the signal alongside an existing rule, combine them with and in the same expression; see Entry conditions.

Validate and compare​

MesoSim uses the most recent available observation until a newer one arrives. A selected sample more than five days old causes a data error. Check for gaps as well as coverage at the beginning and end of your backtest.

After editing a published file, verify that its CSV URL shows the updated data. For MesoSim to refresh a cached file, allow at least one minute without requesting the same URL, or use a new URL. Google Sheets publishing updates can also take a few minutes.

Keep a baseline and compare a run with only the intended signal change. Use Clone in the Portal, or have the agent create the revised run. Inspect trade counts, performance, and drawdowns in Backtest Results and Tearsheets.

To explore how an entry signal relates to final position PnL, see Analytics with DataVoyager. Find further strategy and research examples on the Deltaray blog.