# Tulipa as the market model ## Overview Instead of using the simple, Python-based market model, annular can use `Tulipa` for computing the equilibrium in the market. [Tulipa](https://tulipaenergy.github.io/TulipaEnergyModel.jl/stable/) is a Julia library for modeling energy systems. Using `Tulipa` as the market in annular allows representing the market more realistically, for instance with power imports and exports. ```{admonition} This is a proof of concept. This feature has many rough edges. ``` ## Installation First, the machine running annular needs to have Julia installed and accessible with the `julia` executable. Next, we need to install the Julia packages that support `Tulipa`. The easiest is to use the installer provided by annular, by calling: ```bash annular install-tulipa -w my-workspace ``` This creates a new Julia environment in `my-workspace`. Alternatively, to manage your own Julia environment, you can install `Tulipa` with the following `Project.toml`: ```{literalinclude} ../../src/annular/market/tulipa/Project.toml :language: toml :caption: Project.toml ``` ## Defining data for Tulipa When using annular with `Tulipa`, the main use case is to simulate the market with an existing `Tulipa` model, but you can go both ways. ### Adapting an existing `base` market to `Tulipa` The function [`create_tulipa_input_data`] can help convert the market model definition from a `base` model to `Tulipa`. ### Using an existing `Tulipa` model `Tulipa` models are defined through data. Consult the [Tulipa documentation][tulipa-docs] for details. To integrate a `Tulipa` model with annular, the data for the `Tulipa` model need to be placed in a particular folder. Specifically, for each table `table_name` in the `Tulipa` database, there should be a csv file with the same schema. The csv file can contain the exact same data as in the `Tulipa` database. However, because the simulation in annular creates a model by combining assets in `Tulipa` and satellite bids, it may be necessary to adjust the `Tulipa` input data and remove assets that are now represented by annular satellites. ## Example project with Tulipa market model Building on the example introduced in [Defining a basic simulation][basic-simulation], you can download all the necessary data as a [zip file][tulipa-zip]. After installing `Tulipa` and unzipping the simulation data, your working directory should look like this: ``` simulation-with-tulipa/ ├── my-workspace/ (1) ├── tulipa_input/ (2) ├── simulation_configuration_with_tulipa.ymmsl (3) ├── tulipa_config.yml (4) ├── generators/ │ └── G01.yaml ├── satellite-configurations/ │ ├── battery-flexible.yaml │ └── cronian-configurations/ │ └── P01.yaml └── csv_files/ ├── availability_factors.csv ├── battery-flexible-demand.csv └── ... ``` Compared to the earlier example, there are additional files and configurations, which we now describe. ### Configuration #### (1) Julia environment This is the `my-workspace` folder into which we installed `TulipaEnergyModel` earlier. #### (2) Input data The folder `tulipa_input` has the `csv` files previously described. They come from the `Tulipa` case study named "Rolling Horizon". It is a basic example of a `Tulipa` model. #### (3) Simulation configuration file In the `ymmsl` configuration file, two keys are added, as follows: ```{literalinclude} ../static/example-project/simulation_configuration_with_tulipa.ymmsl :language: yaml :caption: simulation-configuration-with-tulipa.ymmsl ``` To use `Tulipa` as the market, set `market` to `tulipa`. To switch back to the base market, set it to `base`. In addition, when using `Tulipa`, it is necessary to specify a `tulipa_config_file`. It is a path relative to the `simulation-configuration-with-tulipa.ymmsl` file and indicates additional configuration for running `Tulipa`: ```{literalinclude} ../static/example-project/tulipa_config.yml :language: yaml :caption: tulipa_config.yml ``` #### (4) `Tulipa` configuration file This file contains settings for `Tulipa`. **Mandatory arguments** `input_data_folder` indicates the path to a folder with csv input files from which `Tulipa` can build a model. In our example, this is `tulipa_input`. `julia_environment` is the path to the Julia environment in which `Tulipa` will run.[^julia-project-arg] In our example, this is `my-workspace` where we installed `Tulipa`. Both `input_data_folder` and `julia_environment` need to be relative to the `tulipa_config.yml` file they are defined in. **Optional arguments** Both of these arguments are fully managed by annular and do not need to be set, in which case default values are used. `communication_folder` is the path to a folder through which annular and `Tulipa` can exchange data. If it already exists at the start of an invocation of the market model, annular will give an error. By default, for each iteration, a separate hidden folder is created in the location of python process running the simulation, and removed after the simulation. If a value is given for `communication_folder`, it is interpreted relative to the location of the `tulipa_config.yml` file. When using the `-vv` option in the annular command-line interface, the communication folders are not deleted after the simulation, which can be useful for debugging. `db_file` is the name of the database created inside `communication_folder`, and defaults to `tulipa.duckdb`. ### Running the simulation Now, you can run the simulation with the `Tulipa` market model as follows: ```bash annular run -v simulation_configuration_with_tulipa.ymmsl ``` ### Analyzing the results The results are stored similarly to the base market model, with one difference: instead of a `csv` file with the market solution, `annular` stores the database file from the `TulipaEnergyModel` with the name `tulipa_iteration-ID.duckdb`, where `ID` is the iteration of the model. Iteration 0 refers to the solution for the first optimization window, iteration 1 for the second, etc. To process the results in this database file, it is necessary to understand the data schema used by `Tulipa`. A starting point is in the [Tulipa output documentation][tulipa-output-docs]. ## Limitations of this feature In the most basic setup, an energy system defined by `annular` can be converted to a `Tulipa` model. In simple cases, both market models yield the same solution if the underlying problem has a unique solution.[^multiple-solutions] The more common use case is `annular` is integrated with an existing `Tulipa` model, in which case there are limitations. ### 1. Solutions for bids depend on the network structure in the `Tulipa` model The optimization problem in `Tulipa` is a graph where nodes are assets and edges are flows. There are production, consumption, storage, and conversion assets. In `Tulipa`, all bids from the `annular` satellites are connected to an arbitrary consumption asset in the `Tulipa` model.[^tulipa-bid-docs] This is a reasonable first approximation, but has limitations: 1. If there is excess supply from the `annular` satellites, it is not guaranteed that this supply can flow back to the system at large. This can impact the optimal solution; exactly how depends on how the model is set up. 2. The market price for the bids depends on which consumption asset the bids are connected to. There is no good intuition for when this matters for market prices, except for the fact that the larger the `Tulipa` model, the more it matters. As a rule of thumb, use the `Tulipa` market model only for nation-wide simulations where geographical price differences are small and carefully investigate the results. ### 2. `Tulipa` (storage) assets do not look forward and have no memory This is a serious shortcoming,[^annular-tulipa-limitations] but was chosen deliberately because of time constraints, and because the `base` market model in `annular` only works with flexible generators, where this is not a problem. #### Not forward-looking This is because the coupling module in `annular` uses the bidding window (24h) as the market's optimization window. For `Tulipa`, it means that the capabilities of storage assets in the future are not considered when solving the market problem. #### No memory While plain `Tulipa` is solved for an entire simulation period---for instance 1 year---, `annular` solves the market problem separately for each day in the simulation period---365 days---for both types of market models. When using `Tulipa`, this means that a new model is defined from scratch for each iteration, and previous solutions are not stored. This matters particularly for assets that have storage---their storage level in the next iteration will start from the same level as in the current iteration. ### 3. Bids cannot be curtailed in either market model While `Tulipa` can curtail demand or supply of an asset, this curtailment works differently from how for instance EPEX curtails block bids. Specifically, in `Tulipa`, curtailment truncates the peak of a profile, while in EPEX, curtailment shifts the entire profile. For instance, for a block bid with a quantity profile such as `{19h: 4, 20h: 8, 21h: 2}`, a 50% curtailment means that a profile with the following minimum amounts can be allocated: `{19h: 2, 20h: 4, 21h: 1}`. In contrast, `Tulipa` curtails with a `min_operating_point` constraint that is between 0 and 1. With a `min_operating_point` of `0.5`, `Tulipa` can curtail the same profile down to `{19h: 4, 20h: 4, 21h: 2}`. The practical implication is that currently, bids cannot be curtailed for neither of the markets. [`create_tulipa_input_data`]: #annular.market.tulipa.utils.create_tulipa_input_data [tulipa-zip]: ../static/example-project/simulation-with-tulipa.zip [basic-simulation]: ../basic-project.md [tulipa-docs]: https://tulipaenergy.github.io/TulipaEnergyModel.jl/stable/ [tulipa-output-docs]: https://tulipaenergy.github.io/TulipaEnergyModel.jl/stable/20-user-guide/55-outputs/ [^multiple-solutions]: If there are multiple solutions, different optimizers may choose different solutions. [^tulipa-bid-docs]: See the [Tulipa documentation](https://tulipaenergy.github.io/TulipaEnergyModel.jl/stable/10-tutorials/40-bids-workaround/) for how it incorporates bids. [^annular-tulipa-limitations]: See [this issue](https://gitlab.tudelft.nl/demoses/annular/-/issues/152) in annular. [^julia-project-arg]: If you are familiar with Julia: this is the `--project` argument when calling Julia.