Skip to content
Merged
Show file tree
Hide file tree
Changes from 137 commits
Commits
Show all changes
141 commits
Select commit Hold shift + click to select a range
f94d570
Wip CSV/Parquet
JasonMarechal25 Sep 29, 2025
008ad11
Add parquet
JasonMarechal25 Sep 29, 2025
431265b
Implement parquet table writer
JasonMarechal25 Sep 29, 2025
114c932
Use writer
JasonMarechal25 Sep 30, 2025
e55efde
Sort
JasonMarechal25 Oct 6, 2025
56f980b
Merge remote-tracking branch 'remotes/origin/feature/parquet' into fe…
Mar 9, 2026
0f56acf
Fix parquet build
flomnes Mar 17, 2026
779fe04
Fix absl/re2 version conflict between vcpkg and OR-Tools
flomnes Mar 17, 2026
808e264
Merge remote-tracking branch 'origin/develop' into fix/parquet-deps
flomnes Mar 17, 2026
bb519fe
Fix absl/re2 version conflict in installed AntaresConfig.cmake
flomnes Mar 17, 2026
25190d1
Fix Arrow/Parquet linkage for shared library builds (Windows)
flomnes Mar 17, 2026
6081c84
Static linkage for absl & re2 to avoid windows conflicts
flomnes Mar 17, 2026
11cf1cc
Fix missing DLL on windows
flomnes Mar 17, 2026
e678bf4
[skip ci] ANT-4633 - parquet : shrink license headers
Mar 23, 2026
ef4625b
ANT-4633 - parquet : add comments in code + small simplification
Mar 24, 2026
c748a24
ANT-4633 - parquet : code formatting
Mar 24, 2026
b940a91
ANT-4633 - parquet : change a lambda to free function
Mar 24, 2026
46d3dee
[skip ci] ANT-4633 - parquet : clarifications
Mar 24, 2026
1abfc98
Merge remote-tracking branch 'remotes/origin/develop' into fix/parque…
Mar 24, 2026
ed7ddd7
ANT-4633 - parquet : renaming
Mar 25, 2026
da94ac6
ANT-4633 - parquet : handle --parquet cmd line option
Mar 25, 2026
28ca2c2
ANT-4633 - parquet (modeler side) : bring down the output format to w…
Mar 25, 2026
98b2c7f
ANT-4633 - parquet (modeler side) : remove code not required.
Mar 25, 2026
d366790
ANT-4633 - parquet : fix cmake config
Mar 27, 2026
75373ad
ANT-4633 - parquet : fix a crash
Mar 27, 2026
86f7a2b
ANT-4633 - parquet : code formatting
Mar 27, 2026
ad7fd91
ANT-4633 - parquet > solver : define --parquet option + remove useles…
Mar 27, 2026
b539331
ANT-4633 - parquet : remove useless lock guard (we're not in a thread…
Mar 30, 2026
c22240a
ANT-4633 - parquet : make comment more clear
Mar 30, 2026
3fceb46
ANT-4633 - parquet : fix in modeler + simplification + comments in code
Mar 30, 2026
39dcec7
[skip ci] ANT-4633 - parquet : adding suggestion / questions on forma…
Mar 31, 2026
3467e32
ANT-4633 - parquet : compilation fix
Mar 31, 2026
e66c17d
[skip ci] ANT-4633 - parquet :
Mar 31, 2026
76bf7e1
ANT-4633 - parquet : fix an coding error + renaming
Mar 31, 2026
4310b6b
[skip ci] ANT-4633 - parquet : adding design suggestions in modeler
Mar 31, 2026
b4a71c6
[skip ci] ANT-4633 - parquet : more suggestions on modeler design
Mar 31, 2026
b5ec827
ANT-4633 - parquet > modeler : work to prepare separation between mak…
Mar 31, 2026
c2003e7
ANT-4633 - parquet : remove class ISimulationTable, useless
Mar 31, 2026
108f3d3
ANT-4633 - parquet : compilation fix after interface class removal
Mar 31, 2026
da53c69
ANT-4633 - parquet : separate simulation table and writing simulation…
Mar 31, 2026
58c3b74
ANT-4633 - parquet : fix to make tests pass (after many refactoring)
Apr 1, 2026
abf296e
[skip ci] ANT-4633 - parquet : update refactoring report
Apr 1, 2026
62b2196
ANT-4633 - parquet : towards class SimulationTableCsvFile removal
Apr 1, 2026
7520128
Relax expressions in port definition : fix unit test due to previous …
Apr 2, 2026
20e4000
ANT-4633 - parquet : remove class SimulationTableCsvFile
Apr 2, 2026
0a35d56
ANT-4633 - parquet : renaming class SimulationTableCsv into Simulatio…
Apr 2, 2026
5449229
ANT-4633 - parquet : update refactoring state progress file
Apr 2, 2026
0e7aab9
ANT-4633 - parquet : make a target for simulation table writers only
Apr 3, 2026
96d05b3
ANT-4633 - parquet : simplification in cmake
Apr 3, 2026
99b1a80
ANT-4633 - parquet : split simulation-table into 2 targets
Apr 3, 2026
9dc58c1
ANT-4633 - parquet : make simulation table writer take a table and a …
Apr 3, 2026
a25f274
[skip ci] ANT-4633 - parquet : update things that should be done
Apr 3, 2026
70f4748
ANT-4633 - parquet : move definition of output file extension into sp…
Apr 7, 2026
623c42d
Fix linux build
flomnes Apr 8, 2026
f33e5dd
Format
flomnes Apr 8, 2026
acc6d1d
ANT-4633 - parquet : move output file check to parent class
Apr 8, 2026
6cb9898
ANT-4633 - parquet : add unit tests on writing tables in parquet form…
Apr 8, 2026
701d27a
ANT-4633 - parquet : in unit tests, read back parquet file and check …
Apr 9, 2026
08eb4f6
ANT-4633 - parquet : remove use of a table that combines chunks, not …
Apr 9, 2026
437dfe3
ANT-4633 - parquet : remove not needed static casts
Apr 9, 2026
0ce202e
ANT-4633 - parquet : make unit tests more clear
Apr 9, 2026
e9593cc
ANT-4633 - parquet : make tests more clear
Apr 9, 2026
7a9b9d6
ANT-4633 - parquet : mainly add name to (as data member of) column
Apr 9, 2026
b6ff37b
ANT-4633 - parquet : simplifying simulation table storage after previ…
Apr 9, 2026
7a2992e
ANT-4633 - parquet : fix compilation error on Ubuntu
Apr 9, 2026
2eef562
ANT-4633 - parquet writer : adapting columns storage type to column t…
Apr 9, 2026
34cbf26
ANT-4633 - parquet : converting simulation columns into arrow columns…
Apr 10, 2026
80c3dbc
ANT-4633 - parquet : finish the job started in previous commit
Apr 10, 2026
59460f1
ANT-4633 - parquet : move macros specific to arrow into a header
Apr 10, 2026
4525ed6
ANT-4633 - parquet : extract functions in parquet writer to make code…
Apr 10, 2026
155603a
ANT-4633 - parquet writer : make a first (failing) unit tests
Apr 13, 2026
34424b7
ANT-4633 - parquet : finalize first unit test
Apr 13, 2026
85b318d
ANT-4633 - parquet : adding a new unit test that writes a table with …
Apr 13, 2026
b8f13ec
ANT-4633 - parquet : clang formatting
Apr 13, 2026
c8ec2cf
ANT-4633 - parquet : adding a more thorough test + improvements in un…
Apr 15, 2026
dba815e
ANT-4633 - parquet : simplify arrow adapters for std::optional<T> case
Apr 17, 2026
a14222c
ANT-4633 - parquet : make empty value (=invalid value) coherent betwe…
Apr 17, 2026
99deb55
ANT-4633 - parquet : make unit tests more readable
Apr 17, 2026
39a61f9
Merge remote-tracking branch 'remotes/origin/develop' into fix/parque…
Apr 17, 2026
78346a2
Merge remote-tracking branch 'remotes/origin/fix/parquet-deps' into f…
Apr 17, 2026
e1da3ee
ANT-4633 - parquet : adapt the reader after merge from develop
Apr 20, 2026
c4fdfac
ANT-4633 - parquet : fix Sonar code smells
Apr 20, 2026
a74aef9
ANT-4633 - parquet : remove unit tests used to test arrow / parquet
Apr 21, 2026
9858a6d
ANT-4633 - parquet (legacy solver) : simplification
Apr 21, 2026
570f817
ANT-4633 - parquet (legacy solver) : renaming for more clarity
Apr 21, 2026
ee128c4
ANT-4633 - parquet (legacy solver) : write simulation table within a …
Apr 21, 2026
a6f76c3
ANT-4633 - parquet (legacy solver) : make unit tests compile
Apr 22, 2026
94ffce0
ANT-4633 - parquet (legacy solver) : adding a new writer for optim si…
Apr 22, 2026
32cc90a
ANT-4633 - parquet (legacy solver) : many simplifications
Apr 22, 2026
729cb06
Merge remote-tracking branch 'origin/develop' into feature/parquet-fo…
flomnes May 4, 2026
bf4d50e
Apply formatting
flomnes May 4, 2026
850b4cf
Fix std::path, headers, message
flomnes May 4, 2026
a8e990f
fix api tests by skipping legacy output writer without output folder
flomnes May 4, 2026
62d7089
Use unique_ptr around OptimisationsSimulationTable
flomnes May 4, 2026
4139291
Improve error handling for unknown columns
flomnes May 4, 2026
555ecec
Clean up
flomnes May 4, 2026
72fe139
sonar
flomnes May 4, 2026
d8dea4e
ANT-4633 - parquet (legacy solver) : remove useless method Simulation…
Apr 22, 2026
3ae1c6a
ANT-4633 - parquet (legacy solver) : removal
Apr 22, 2026
0aaaaed
ANT-4633 - parquet (legacy solver) : mainly reverting code added when…
May 4, 2026
295a5d9
ANT-4633 - parquet (legacy solver) : removing a strange and useless c…
May 4, 2026
74732e9
ANT-4633 - parquet (legacy solver) : logging the correction absolute …
May 5, 2026
bfe2f26
ANT-4633 - parquet (legacy solver) : fixing compilation error
May 5, 2026
b3777b5
ANT-4633 - parquet (legacy solver) : fix due to review + fix human error
May 5, 2026
4082e33
ANT-4633 - parquet (legacy solver) : trial to fix failures on tests i…
May 5, 2026
9095209
Merge branch 'develop' into feature/parquet-for-legacy-solver
JasonMarechal25 May 6, 2026
4aadc1d
ANT-4633 - parquet (legacy solver) : fix error in collecting the simu…
May 6, 2026
f211b86
ANT-4633 - parquet (legacy solver) : avoid that hybrid studies are ru…
May 7, 2026
7995e9f
Merge remote-tracking branch 'remotes/origin/develop' into feature/pa…
May 7, 2026
9666b70
ANT-4633 - parquet (legacy solver) : fix after merge
May 7, 2026
03e96af
ANT-4633 - parquet (legacy solver) : clang format
May 7, 2026
2600a06
ANT-4633 - parquet (legacy solver) : adapt cucumber to the case where…
May 7, 2026
279f7bd
ANT-4633 - parquet (legacy solver) : update the submodule of NR tests
May 7, 2026
fce0a15
Fix - don't expose arrow's symbols in parquet_writer's API (#3654)
flomnes May 20, 2026
f8d8043
Merge remote-tracking branch 'remotes/origin/develop' into feature/pa…
May 20, 2026
4aaee88
ANT-4633 - parquet (legacy solver) : fix due to review
May 20, 2026
c116648
ANT-4633 - parquet (legacy solver) : trial for fixing a cucumber test
May 20, 2026
a9a0b47
ANT-4633 - parquet (legacy solver) : fix for CI
May 20, 2026
b6f9ba3
ANT-4633 - parquet (legacy solver) : we don't care about end of lines…
May 20, 2026
57dfdd2
ANT-4633 - parquet (legacy solver) : update test NR repository
May 20, 2026
d35a11c
Parquet - fix review of PR 3586 : revert changes on submodule vcpkg
May 26, 2026
c5ff2cc
test: update sha1
JasonMarechal25 May 26, 2026
be39c27
test: update sha1
JasonMarechal25 May 26, 2026
0882fc6
fix: remove test NR checkout
JasonMarechal25 May 26, 2026
5bfcd63
Merge branch 'develop' into feature/parquet-for-legacy-solver
JasonMarechal25 May 26, 2026
131216f
fix: e2e tags
May 27, 2026
8530bd7
fix: throw dedicated errors
May 27, 2026
b265d5b
fix: move OptimisationsSimulationTable in IO::Outputs namespace
May 27, 2026
f0825a6
format
May 27, 2026
f24a27b
Revert "format"
May 27, 2026
8a0ead6
Revert "fix: move OptimisationsSimulationTable in IO::Outputs namespace"
May 27, 2026
89da1ad
fix: for review of PR 3586 [ANT-4633] (#3657)
guilpier-code May 27, 2026
947137d
parquet part 3 - parquet for legacy solver #3586 : update submodule t…
May 27, 2026
05f32f4
parquet part 3 - parquet for legacy solver #3586 : restore table format
May 27, 2026
f835350
ANT-4633 - parquet (legacy solver) : fix coding error
May 28, 2026
a352a27
Merge remote-tracking branch 'official/develop' into feature/parquet-…
May 29, 2026
ceed9b0
fix: build
May 29, 2026
bfc4c08
Merge branch 'develop' into feature/parquet-for-legacy-solver
JasonMarechal25 May 29, 2026
bf4efd5
Remove table_writer sources & headers from result_writer (#3677)
flomnes May 29, 2026
d8e8377
[skip ci] Modeler - trial for removing FileWriter : add some doc
May 29, 2026
443ef77
refacto: removing useless class FileWriter (#3675)
guilpier-code Jun 1, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 0 additions & 95 deletions docs/developer-guide/comments-on-parquet-subject.txt

This file was deleted.

2 changes: 1 addition & 1 deletion src/api/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ target_link_libraries(single_problem_api
add_executable(antares-problem-generator main.cpp)
target_link_libraries(antares-problem-generator
single_problem_api)
copy_project_shared_lib(parquet_writer antares-problem-generator)
copy_project_shared_lib(simulation-table-writers antares-problem-generator)

install(DIRECTORY include/antares
DESTINATION "include")
3 changes: 3 additions & 0 deletions src/io/outputs/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,12 @@
# ===================================
set(SRC_SIMU_TABLE
SimulationTable.cpp
OptimisationsSimulationTable.cpp

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

class OptimisationsSimulationTable was moved here

include/antares/io/outputs/OptimisationsSimulationTable.h
include/antares/io/outputs/SimulationTableEntry.h
include/antares/io/outputs/SimulationTable.h
include/antares/io/outputs/columns.h
include/antares/io/outputs/IColumnAdapterVisitor.h
include/antares/io/outputs/storage.h)

add_library(simulation-table STATIC ${SRC_SIMU_TABLE})
Expand Down
23 changes: 23 additions & 0 deletions src/io/outputs/OptimisationsSimulationTable.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
// Copyright 2007-2026, RTE (https://www.rte-france.com)
// SPDX-License-Identifier: MPL-2.0

#include "include/antares/io/outputs/OptimisationsSimulationTable.h"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this class is now so short that it could now be a struct


namespace Antares::IO::Outputs
{
SimulationTable* OptimisationsSimulationTable::firstOptimSimulationTable()
{
return &firstOptimSimulationTable_;
}

SimulationTable* OptimisationsSimulationTable::secondOptimSimulationTable()
{
return &secondOptimSimulationTable_;
}

void OptimisationsSimulationTable::clear()
{
firstOptimSimulationTable_.clear();
secondOptimSimulationTable_.clear();
}
} // namespace Antares::IO::Outputs
52 changes: 4 additions & 48 deletions src/io/outputs/SimulationTable.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -12,19 +12,18 @@ namespace Antares::IO::Outputs
SimulationTable::SimulationTable()

{
storage_.addIntegralColumn<unsigned int>("block");
storage_.addIntegralColumn("block");
storage_.addOptionalColumn<std::string>("component");
storage_.addStringColumn("output");
storage_.addOptionalColumn<unsigned int>("absolute_time_index");
storage_.addOptionalColumn<unsigned int>("block_time_index");
storage_.addIntegralColumn<unsigned int>("scenario_index");
storage_.addIntegralColumn("scenario_index");
storage_.addOptionalColumn<double>("value");
storage_.addOptionalColumn<Antares::Optimisation::LinearProblemApi::MipBasisStatus>(
"basis_status");
}

SimulationTable::SimulationTable(SimulationTable&& other) noexcept:
buffer_(std::move(other.buffer_)),
storage_(std::move(other.storage_))
{
}
Expand All @@ -41,51 +40,14 @@ void SimulationTable::addEntry(const SimulationTableEntry& entry)
storage_.addValue("basis_status", entry.status);
}

std::string SimulationTable::headerCsvFormat() const
{
std::ostringstream os;
bool first = true;
for (const auto& column: storage_.columns())
{
if (!first)
{
os << ',';
}
first = false;
os << column->name();
}
return os.str();
}

void SimulationTable::writeHeaderToBuffer()
{
buffer_ << headerCsvFormat() << '\n';
}

const std::vector<std::unique_ptr<IColumn>>& SimulationTable::columns() const
{
return storage_.columns();
}

void SimulationTable::writeToBuffer()
size_t SimulationTable::rowCount() const
{
const size_t row_count = storage_.rowCount();
const auto& columns = storage_.columns();

for (size_t row = 0; row < row_count; ++row)
{
bool first = true;
for (const auto& col: columns)
{
if (!first)
{
buffer_ << ',';
}
first = false;
buffer_ << col->toString(row);
}
buffer_ << '\n';
}
return storage_.rowCount();
}

std::vector<std::vector<std::string>> SimulationTable::storageIntoRows() const
Expand All @@ -111,11 +73,5 @@ std::vector<std::vector<std::string>> SimulationTable::storageIntoRows() const
void SimulationTable::clear()
{
storage_.clear();
buffer_.str("");
}

std::string SimulationTable::buffer() const
{
return buffer_.str();
}
} // namespace Antares::IO::Outputs
47 changes: 47 additions & 0 deletions src/io/outputs/include/antares/io/outputs/IColumnAdapterVisitor.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
// Copyright 2007-2026, RTE (https://www.rte-france.com)
// SPDX-License-Identifier: MPL-2.0

#pragma once

#include "antares/optimisation/linear-problem-api/hasStatus.h"

namespace Antares::Writer
{
class IColumnAdapter;
} // namespace Antares::Writer

using namespace Antares::Writer;

Check failure on line 13 in src/io/outputs/include/antares/io/outputs/IColumnAdapterVisitor.h

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Remove this using-directive.

See more on https://sonarcloud.io/project/issues?id=AntaresSimulatorTeam_Antares_Simulator&issues=AZ5pwehL_7VPFyX-dACw&open=AZ5pwehL_7VPFyX-dACw&pullRequest=3586

namespace Antares::IO::Outputs
{

// Forward declare template
template<typename T>
class TypedColumn;

// Define type aliases
using StringColumn = TypedColumn<std::string>;
using IntegralColumn = TypedColumn<unsigned>;
using DoubleColumn = TypedColumn<double>;

template<typename T>
using OptionalColumn = TypedColumn<std::optional<T>>;

// ================================
// Class IColumnAdapterVisitor
// ================================
class IColumnAdapterVisitor
{
public:
virtual ~IColumnAdapterVisitor() = default;
virtual std::shared_ptr<IColumnAdapter> visit(const StringColumn&) = 0;
virtual std::shared_ptr<IColumnAdapter> visit(const DoubleColumn&) = 0;
virtual std::shared_ptr<IColumnAdapter> visit(const IntegralColumn&) = 0;
virtual std::shared_ptr<IColumnAdapter> visit(const OptionalColumn<std::string>&) = 0;
virtual std::shared_ptr<IColumnAdapter> visit(const OptionalColumn<double>&) = 0;
virtual std::shared_ptr<IColumnAdapter> visit(const OptionalColumn<unsigned>&) = 0;

virtual std::shared_ptr<IColumnAdapter>
visit(const OptionalColumn<Optimisation::LinearProblemApi::MipBasisStatus>&) = 0;
};
} // namespace Antares::IO::Outputs
Original file line number Diff line number Diff line change
Expand Up @@ -5,28 +5,17 @@

#include "antares/io/outputs/SimulationTable.h"

namespace Antares::Solver
namespace Antares::IO::Outputs
{
class IResultWriter;
}

class OptimisationsSimulationTable

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As you can see, this class was :

  • moved to another location (next to class SimulationTable)
  • simplified : indeed another class (LegacySimulationTablesWriter) takes care of the writing part

{
public:
void clear();

std::pair<std::string, std::string> moveBuffers();

void write();

void writeTo(const std::string& filePrefix, Antares::Solver::IResultWriter& writer);
Antares::IO::Outputs::SimulationTable* firstOptimSimulationTable();
Antares::IO::Outputs::SimulationTable* secondOptimSimulationTable();
[[nodiscard]] std::string headerCsvFormat() const;
void clear();

private:
Antares::IO::Outputs::SimulationTable firstOptimSimulationTable_;
Antares::IO::Outputs::SimulationTable secondOptimSimulationTable_;
std::string firstOptimBuffer_;
std::string secondOptimBuffer_;
};
} // namespace Antares::IO::Outputs
12 changes: 1 addition & 11 deletions src/io/outputs/include/antares/io/outputs/SimulationTable.h
Original file line number Diff line number Diff line change
Expand Up @@ -17,22 +17,12 @@ class SimulationTable
SimulationTable();
SimulationTable(SimulationTable&& other) noexcept;
void addEntry(const SimulationTableEntry& entry);
[[nodiscard]] std::string headerCsvFormat() const;
const std::vector<std::unique_ptr<IColumn>>& columns() const;
void writeToBuffer();
size_t rowCount() const;
void clear();

// const std::string& buffer() const;
[[nodiscard]] std::string buffer() const;

// gp : never called in production code, only from tests
// gp : should be moved as a free function in tests
void writeHeaderToBuffer();

std::vector<std::vector<std::string>> storageIntoRows() const;

private:
std::ostringstream buffer_;
ColumnBasedStorage storage_;
};
} // namespace Antares::IO::Outputs
Loading
Loading