feat(integration-tests): Add core binary tests for the log-converter and clp-s pipeline.
#1591
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| { | ||
| "timestamp": "01 May 2025 01:02:03,456", | ||
| "message": " INFO f81d4fae-7dec-11d0-a765-00a0c91e6bf6 [9] (2) a.b.c: Hello world\n" | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| 01 May 2025 01:02:03,456 INFO f81d4fae-7dec-11d0-a765-00a0c91e6bf6 [9] (2) a.b.c: Hello world |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| {"timestamp": "08 Oct 2025 20:00:31,125", "message": " ERROR cc994739-866c-4789-9596-cbd08b4a5ebd [pool-5-thread-515] c.a.r.c.e.PreferenceBasedRulesExecutor Vasanth PreferenceBasedRulesExecutor processInvalidRuleException: {FAILURE:\"MVEL failure: could not execute facts for rule.\",EXCEPTION_TYPE:\"org.mvel.UnresolveablePropertyException \",RETRY_COUNT:2,REASON:\"unable to resolve token: \\u201c$ABCString \",VARIABLE_VALUES:{abc:\"XXX \",$randomVariable:\"000 \"}\\n\n"} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| 08 Oct 2025 20:00:31,125 ERROR cc994739-866c-4789-9596-cbd08b4a5ebd [pool-5-thread-515] c.a.r.c.e.PreferenceBasedRulesExecutor Vasanth PreferenceBasedRulesExecutor processInvalidRuleException: {FAILURE:"MVEL failure: could not execute facts for rule.",EXCEPTION_TYPE:"org.mvel.UnresolveablePropertyException ",RETRY_COUNT:2,REASON:"unable to resolve token: \u201c$ABCString ",VARIABLE_VALUES:{abc:"XXX ",$randomVariable:"000 "}\n |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| {"message":" INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties\n","timestamp":"2015-03-23 05:38:01,440"} | ||
| {"message":" INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).\n","timestamp":"2015-03-23 05:38:02,096"} | ||
| {"message":" INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started\n","timestamp":"2015-03-23 05:38:02,096"} | ||
| {"message":" INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens:\n","timestamp":"2015-03-23 05:38:02,164"} | ||
| {"message":" INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1427088391284_0001, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@482dbdc2)\n","timestamp":"2015-03-23 05:38:02,164"} | ||
| {"message":" INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.\n","timestamp":"2015-03-23 05:38:02,703"} | ||
| {"message":" INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1427088391284_0001\n","timestamp":"2015-03-23 05:38:04,418"} | ||
| {"message":" INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id\n","timestamp":"2015-03-23 05:38:09,630"} | ||
| {"message":" INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ]\n","timestamp":"2015-03-23 05:38:11,829"} | ||
| {"message":" INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: hdfs://172.31.17.135:8120/HiBench/Hive/temp/dummy:113+3\n","timestamp":"2015-03-23 05:38:12,508"} | ||
| {"message":" INFO [main] org.apache.hadoop.mapred.MapTask: numReduceTasks: 96\n","timestamp":"2015-03-23 05:38:12,620"} | ||
| {"message":" INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)\n","timestamp":"2015-03-23 05:38:13,101"} | ||
| {"message":" INFO [main] org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100\n","timestamp":"2015-03-23 05:38:13,101"} | ||
| {"message":" INFO [main] org.apache.hadoop.mapred.MapTask: soft limit at 83886080\n","timestamp":"2015-03-23 05:38:13,101"} | ||
| {"message":" INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600\n","timestamp":"2015-03-23 05:38:13,101"} | ||
| {"message":" INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600\n","timestamp":"2015-03-23 05:38:13,101"} | ||
| {"message":" INFO [main] org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer\n","timestamp":"2015-03-23 05:38:13,112"} | ||
| {"message":" INFO [main] HiBench.HtmlCore: WARNING: dict empty!!!\n","timestamp":"2015-03-23 05:38:13,120"} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| 2015-03-23 05:38:01,440 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties | ||
| 2015-03-23 05:38:02,096 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). | ||
| 2015-03-23 05:38:02,096 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started | ||
| 2015-03-23 05:38:02,164 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens: | ||
| 2015-03-23 05:38:02,164 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1427088391284_0001, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@482dbdc2) | ||
| 2015-03-23 05:38:02,703 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now. | ||
| 2015-03-23 05:38:04,418 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1427088391284_0001 | ||
| 2015-03-23 05:38:09,630 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id | ||
| 2015-03-23 05:38:11,829 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] | ||
| 2015-03-23 05:38:12,508 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: hdfs://172.31.17.135:8120/HiBench/Hive/temp/dummy:113+3 | ||
| 2015-03-23 05:38:12,620 INFO [main] org.apache.hadoop.mapred.MapTask: numReduceTasks: 96 | ||
| 2015-03-23 05:38:13,101 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) | ||
| 2015-03-23 05:38:13,101 INFO [main] org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 | ||
| 2015-03-23 05:38:13,101 INFO [main] org.apache.hadoop.mapred.MapTask: soft limit at 83886080 | ||
| 2015-03-23 05:38:13,101 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 | ||
| 2015-03-23 05:38:13,101 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 | ||
| 2015-03-23 05:38:13,112 INFO [main] org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer | ||
| 2015-03-23 05:38:13,120 INFO [main] HiBench.HtmlCore: WARNING: dict empty!!! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| {"message":" DEBUG oslo_messaging._drivers.amqpdriver [-] received message msg_id: None reply to None from (pid=25803) __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:201\n","timestamp":"2016-05-09 09:48:24.502"} | ||
| {"message":" DEBUG cinder.scheduler.host_manager [req-d1f3baf7-1d6e-4336-a28e-7f5d10fb16ca None None] Received volume service update from openstack-0@lvmdriver-1: {u'filter_function': None, u'goodness_function': None, u'volume_backend_name': u'lvmdriver-1', u'driver_version': u'3.0.0', u'sparse_copy_volume': False, u'pools': [{u'pool_name': u'lvmdriver-1', u'filter_function': None, u'goodness_function': None, u'total_volumes': 0, u'multiattach': True, u'provisioned_capacity_gb': 0.0, u'allocated_capacity_gb': 0, u'thin_provisioning_support': False, u'free_capacity_gb': 10.01, u'location_info': u'LVMVolumeDriver:openstack-0:stack-volumes-lvmdriver-1:default:0', u'total_capacity_gb': 10.01, u'thick_provisioning_support': True, u'reserved_percentage': 0, u'QoS_support': False, u'max_over_subscription_ratio': 1.0}], u'vendor_name': u'Open Source', u'storage_protocol': u'iSCSI'} from (pid=25803) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:444\n","timestamp":"2016-05-09 09:48:24.504"} | ||
| {"message":" DEBUG oslo_concurrency.lockutils [req-fd08fa95-2092-4b6c-9f2e-522c892d468b None None] Acquired semaphore \"singleton_lock\" from (pid=25803) lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:212\n","timestamp":"2016-05-09 09:49:17.967"} | ||
| {"message":" DEBUG oslo_concurrency.lockutils [req-fd08fa95-2092-4b6c-9f2e-522c892d468b None None] Releasing semaphore \"singleton_lock\" from (pid=25803) lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:225\nTerminated\nc-sch failed to start\nopenstack@openstack-0:~/devstack$\n","timestamp":"2016-05-09 09:49:17.968"} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| 2016-05-09 09:48:24.502 DEBUG oslo_messaging._drivers.amqpdriver [-] received message msg_id: None reply to None from (pid=25803) __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:201 | ||
| 2016-05-09 09:48:24.504 DEBUG cinder.scheduler.host_manager [req-d1f3baf7-1d6e-4336-a28e-7f5d10fb16ca None None] Received volume service update from openstack-0@lvmdriver-1: {u'filter_function': None, u'goodness_function': None, u'volume_backend_name': u'lvmdriver-1', u'driver_version': u'3.0.0', u'sparse_copy_volume': False, u'pools': [{u'pool_name': u'lvmdriver-1', u'filter_function': None, u'goodness_function': None, u'total_volumes': 0, u'multiattach': True, u'provisioned_capacity_gb': 0.0, u'allocated_capacity_gb': 0, u'thin_provisioning_support': False, u'free_capacity_gb': 10.01, u'location_info': u'LVMVolumeDriver:openstack-0:stack-volumes-lvmdriver-1:default:0', u'total_capacity_gb': 10.01, u'thick_provisioning_support': True, u'reserved_percentage': 0, u'QoS_support': False, u'max_over_subscription_ratio': 1.0}], u'vendor_name': u'Open Source', u'storage_protocol': u'iSCSI'} from (pid=25803) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:444 | ||
| 2016-05-09 09:49:17.967 DEBUG oslo_concurrency.lockutils [req-fd08fa95-2092-4b6c-9f2e-522c892d468b None None] Acquired semaphore "singleton_lock" from (pid=25803) lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:212 | ||
| 2016-05-09 09:49:17.968 DEBUG oslo_concurrency.lockutils [req-fd08fa95-2092-4b6c-9f2e-522c892d468b None None] Releasing semaphore "singleton_lock" from (pid=25803) lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:225 | ||
| Terminated | ||
| c-sch failed to start | ||
| openstack@openstack-0:~/devstack$ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| {"message":" DEBUG org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to choose remote rack (location = ~/default-rack), fallback to local rack\norg.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy$NotEnoughReplicasException:\n at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:829)\n at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRemoteRack(BlockPlacementPolicyDefault.java:691)\n at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTargetInOrder(BlockPlacementPolicyDefault.java:496)\n at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:418)\n at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:294)\n at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:147)\n at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:163)\n at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1810)\n at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:265)\n at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2563)\n at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:846)\n at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:510)\n at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\n at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:503)\n at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)\n at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:871)\n at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:817)\n at java.security.AccessController.doPrivileged(Native Method)\n at javax.security.auth.Subject.doAs(Subject.java:422)\n at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1889)\n at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2606)\n","timestamp":"2018-06-12 00:24:09,357"} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| 2018-06-12 00:24:09,357 DEBUG org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to choose remote rack (location = ~/default-rack), fallback to local rack | ||
| org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy$NotEnoughReplicasException: | ||
| at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:829) | ||
| at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRemoteRack(BlockPlacementPolicyDefault.java:691) | ||
| at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTargetInOrder(BlockPlacementPolicyDefault.java:496) | ||
| at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:418) | ||
| at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:294) | ||
| at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:147) | ||
| at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:163) | ||
| at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1810) | ||
| at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:265) | ||
| at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2563) | ||
| at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:846) | ||
| at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:510) | ||
| at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) | ||
| at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:503) | ||
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989) | ||
| at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:871) | ||
| at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:817) | ||
| at java.security.AccessController.doPrivileged(Native Method) | ||
| at javax.security.auth.Subject.doAs(Subject.java:422) | ||
| at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1889) | ||
| at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2606) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -54,6 +54,11 @@ def clp_s_binary_path(self) -> Path: | |
| """:return: The absolute path to the core binary `clp-s`.""" | ||
| return self.clp_core_bins_dir / "clp-s" | ||
|
|
||
| @property | ||
| def log_converter_binary_path(self) -> Path: | ||
| """:return: The absolute path to the core binary `log-converter`.""" | ||
| return self.clp_core_bins_dir / "log-converter" | ||
|
|
||
|
|
||
| @dataclass(frozen=True) | ||
| class PackageConfig: | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See high-level comment.