Introduce published segment cache in broker #6901

surekhasaharan · 2019-01-22T23:37:39Z

This PR introduces a published segments cache on broker, as proposed in #6834 . This implements the "phase 1" of the proposal. Added a new class MetadataSegmentView which polls coordinator periodically to get published segments and maintains a synced copy of same in cache. Another change is to use DataSegmentInterner#INTERNER while storing DataSegment object in ServerSelector and MetadataSegmentView in order reuse the reference for overlapping segments.

Testing on our test cluster shows sub-second execution time for sys.segments queries which were taking multiple seconds without this patch.

This feature is optional, it is turned off by default. It can be turned on by adding this config to broker druid.sql.planner.metadataSegmentCacheEnable=true. To get better sys tables query performance, it should be turned on.

Adding some memory comparison charts with and without the patch to compare broker heap, gc activity etc. These are one day's compares with more than 67K segments in our test cluster. Grey is without patch and blue line is with this patch. We can see that this patch causes slightly more used memory and higher GC activity, which is expected as it's collecting published segments from coordinator and as DataSegment objects are deserialized in broker and destroyed, so more GC activity is caused, but the overall used memory in broker heap is not much bigger than before.

drcrallen · 2019-01-23T18:44:27Z

Heads up, I very briefly experimented with interning data segments back in #3286 but had to back it out because real-time nodes publish with a size of 0, which makes the segment size calculations all break.

I have not dug into the code here deep enough to know if it falls to the same issue.

drcrallen · 2019-01-23T18:45:29Z

Can you also please record the impact on heap size in the broker? We already have pretty massive broker heaps and putting more pressure on them would be problematic.

surekhasaharan · 2019-01-24T01:11:28Z

Heads up, I very briefly experimented with interning data segments back in #3286 but had to back it out because real-time nodes publish with a size of 0, which makes the segment size calculations all break.

Thanks @drcrallen This issue can indeed happen, I looked at the trail of issues surrounding #3286, I am thinking of addressing #3287 before this PR to avoid incorrect segment sizes.

…uals only and size is preserved if set * Added a trueEquals to DataSegment class

…t-caching

surekhasaharan · 2019-01-28T18:25:45Z

Heads up, I very briefly experimented with interning data segments back in #3286 but had to back it out because real-time nodes publish with a size of 0, which makes the segment size calculations all break.

Thanks @drcrallen This issue can indeed happen, I looked at the trail of issues surrounding #3286, I am thinking of addressing #3287 before this PR to avoid incorrect segment sizes.

The size issue is resolved with my latest commit. Changed the DataSegmentInterner to not rely on the DataSegment#equals, it instead uses separate interners for historical and realtime segments.

…t-caching

jihoonson · 2019-01-28T21:37:10Z

server/src/main/java/org/apache/druid/client/MetadataSegmentView.java

+{
+
+  private static final int DEFAULT_POLL_PERIOD_IN_MS = 60000;
+  private static final EmittingLogger log = new EmittingLogger(MetadataSegmentView.class);


Can be Logger since it emits nothing.

ok, changed to Logger

changed back to EmittingLogger, since it emits something now :)

jihoonson · 2019-01-28T21:46:13Z

server/src/main/java/org/apache/druid/client/MetadataSegmentView.java

+  private final BytesAccumulatingResponseHandler responseHandler;
+  private final BrokerSegmentWatcherConfig segmentWatcherConfig;
+
+  private final Map<DataSegment, DateTime> publishedSegments = new ConcurrentHashMap<>();


Please change its type to ConcurrentMap (#6898).

jihoonson · 2019-01-28T21:47:17Z

server/src/main/java/org/apache/druid/client/MetadataSegmentView.java

+          segmentWatcherConfig.getWatchedDataSources()
+      );
+      publishedSegments.keySet()
+                       .removeIf(key -> !segmentWatcherConfig.getWatchedDataSources().contains(key.getDataSource()));


This should be getting only the segments of watched dataSources in the first place rather than filtering later.

Good point, thanks. Changed the /segments api in MetadataResource to take datasources, so we can pass watchedDataSources to it and filtering can happen before broker gets the published segments.

I think this is not needed now because getMetadataSegments() returns only filtered segments.

yeah, not needed anymore, missed to remove it.

jihoonson · 2019-01-28T21:49:16Z

server/src/main/java/org/apache/druid/client/MetadataSegmentView.java

+      publishedSegments.put(interned, ts);
+    }
+    // filter the segments from cache which may not be present in subsequent polling
+    publishedSegments.values().removeIf(v -> v != ts);


Would you elaborate more on what this is doing? And what happens if someone reads a segment by calling getPublishedSegments which is supposed to be removed but haven't yet?

Added more details in the comment.

And what happens if someone reads a segment by calling getPublishedSegments which is supposed to be removed but haven't yet?

If that happens, the segment will be removed in the next poll, so publishedSegments will be eventually consistent.

jihoonson · 2019-01-28T21:50:50Z

sql/src/main/java/org/apache/druid/sql/calcite/schema/SystemSchema.java

 import java.util.Objects;
 import java.util.Set;

+


Please remove unnecessary change.

oops, removed.

jihoonson · 2019-01-28T22:02:20Z

sql/src/main/java/org/apache/druid/sql/calcite/schema/SystemSchema.java

      return authorizedSegments.iterator();
    }

-    private CloseableIterator<DataSegment> getAuthorizedPublishedSegments(


Is this authorization not needed anymore?

I don't think it's needed because, MetadataResource#getDatabaseSegments does the authorization check as well.

Sorry, this authorization check is still required, because broker->coordinator call uses escalated client, so MetadataResource is using the escalated client. Will add this back.

jihoonson · 2019-01-28T22:03:08Z

server/src/main/java/org/apache/druid/client/MetadataSegmentView.java

+    while (metadataSegments.hasNext()) {
+      final DataSegment currentSegment = metadataSegments.next();
+      final DataSegment interned;
+      if (currentSegment.getSize() > 0) {


Please add a method to DataSegmentInterner which has this check.

added common method getInterner(segment)

👍 for pushing this down, and also using in ServerSelector constructor

…t-caching

jihoonson · 2019-01-30T01:25:12Z

server/src/test/java/org/apache/druid/client/CachingClusteredClientTest.java

        );
        serverExpectations.get(lastServer).addExpectation(expectation);
-
+        EasyMock.expect(mockSegment.getSize()).andReturn(-1L).anyTimes();


I guess you're simulating the segment from realtime tasks. Then, it would be more realistic if it returns 0.

okay, changed to 0

jihoonson · 2019-01-30T01:32:59Z

server/src/main/java/org/apache/druid/server/http/MetadataResource.java

-  public Response getDatabaseSegments(@Context final HttpServletRequest req)
+  public Response getDatabaseSegments(
+      @Context final HttpServletRequest req,
+      @QueryParam("datasources") String datasources


This can be simply a list or set.

Didn't find any existing API's taking in a list as query param in druid code, so went with json string. Also the URL with list will look like /druid/coordinator/v1/metadata/segments?datasources="foo"&datasources="bar"&datasources="baz" instead of /druid/coordinator/v1/metadata/segments?datasources=["foo","bar","baz"]. But I can change if there's any benefit.

jihoonson · 2019-01-30T01:50:27Z

server/src/main/java/org/apache/druid/client/MetadataSegmentView.java

+    // since the presence of a segment with an earlier timestamp indicates that
+    // "that" segment is not returned by coordinator in latest poll, so it's
+    // likely deleted and therefore we remove it from publishedSegments
+    publishedSegments.values().removeIf(v -> v != timestamp);


Thanks for adding a comment. How about replacing all segments like in CoordinatorRuleManager rather than checking and removing individual segment?

If I did the replacing, then I'd be allocating memory for a map containing all published segments, (possibly) filling the young gen on every poll period, I thought it would be more memory efficient by removing the segments from existing map.

jihoonson · 2019-01-30T01:52:17Z

server/src/main/java/org/apache/druid/client/selector/ServerSelector.java

  )
  {
-    this.segment = new AtomicReference<>(segment);
+    this.segment = new AtomicReference<>(DataSegmentInterner.intern(segment));


I'm not sure segment still needs to be an AtomicReference.. Let's figure it out in #6952.

jihoonson · 2019-01-30T01:53:37Z

server/src/main/java/org/apache/druid/client/MetadataSegmentView.java

+  public MetadataSegmentView(
+      final @Coordinator DruidLeaderClient druidLeaderClient,
+      ObjectMapper jsonMapper,
+      BytesAccumulatingResponseHandler responseHandler,


Please add final to these two variables.

jihoonson · 2019-01-30T01:54:38Z

server/src/main/java/org/apache/druid/client/MetadataSegmentView.java

+import java.util.concurrent.TimeUnit;
+
+@ManageLifecycle
+public class MetadataSegmentView


It would be great if you can add a simple description about what this class does and how it's being used.

okay, added description.

* change the coordinator api * removeall segments at once from MetadataSegmentView in order to serve a more consistent view of published segments * Change the poll behaviour to avoid multiple poll execution at same time

…t-caching

surekhasaharan · 2019-01-30T23:04:45Z

Can you also please record the impact on heap size in the broker? We already have pretty massive broker heaps and putting more pressure on them would be problematic.

This is used memory comparison on our test broker over a day with about 67K segments, grey line is without this patch and blue line is with this patch. There is slightly higher memory usage with current published segments cache on broker.

jihoonson · 2019-01-30T22:55:34Z

server/src/main/java/org/apache/druid/client/MetadataSegmentView.java

+    @Override
+    public Void call()
+    {
+      poll();


I think it's important to let users know if poll() fails. Please revert logging back. Probably it's worth to catch all exceptions like

try { poll(); } catch (Exception e) { // emit exception }

And since it catches all exceptions, PollTask can be Runnable.

ok, catching all exceptions in the Runnable

jihoonson · 2019-01-30T22:57:30Z

server/src/main/java/org/apache/druid/client/MetadataSegmentView.java

+    public Void call()
+    {
+      poll();
+      scheduledExec.schedule(new PollTask(), DEFAULT_POLL_PERIOD_IN_MS, TimeUnit.MILLISECONDS);


This means the next poll would happen after DEFAULT_POLL_PERIOD_IN_MS which I don't think you intended here. You should probably check how long poll() took and compute the wait time for the next poll. It should be 0 if it already passed the next poll time.

Added computing the delay after every poll() call

jihoonson · 2019-01-30T22:58:13Z

server/src/main/java/org/apache/druid/client/MetadataSegmentView.java

+public class MetadataSegmentView
+{
+
+  private static final int DEFAULT_POLL_PERIOD_IN_MS = 60000;


Are you going to make this configurable? If so, please add a configuration. Otherwise, please rename to POLL_PERIOD_IN_MS.

Yes, I'd want to make it configurable, may be in a follow-up PR. For now, i renamed the variable.

jihoonson · 2019-01-30T23:00:48Z

server/src/main/java/org/apache/druid/client/MetadataSegmentView.java

+  private final BytesAccumulatingResponseHandler responseHandler;
+  private final BrokerSegmentWatcherConfig segmentWatcherConfig;
+
+  private final ConcurrentMap<DataSegment, DateTime> publishedSegments = new ConcurrentHashMap<>();


I think the size of this map would be usually very large. It would be better to initialize with a larger initial size like 1000.

yeah, right, will set initialCapacity

jihoonson · 2019-01-30T23:21:44Z

server/src/main/java/org/apache/druid/client/MetadataSegmentView.java

+          segmentWatcherConfig.getWatchedDataSources()
+      );
+      publishedSegments.keySet()
+                       .removeIf(key -> !segmentWatcherConfig.getWatchedDataSources().contains(key.getDataSource()));


I think this is not needed now because getMetadataSegments() returns only filtered segments.

jihoonson · 2019-01-30T23:23:21Z

server/src/main/java/org/apache/druid/client/MetadataSegmentView.java

+    if (watchedDataSources != null && !watchedDataSources.isEmpty()) {
+      final StringBuilder sb = new StringBuilder();
+      for (String ds : watchedDataSources) {
+        sb.append("datasources=" + ds + "&");


sb.append("datasources=").append(ds).append("&")

You also need to delete the last &.

yeah, doing that later by setting the sb length to sb.length()-1

Ok. Please check https://github.com/apache/incubator-druid/pull/6901/files#r252479790.

oh, missed that one

jihoonson · 2019-01-30T23:24:10Z

server/src/main/java/org/apache/druid/client/MetadataSegmentView.java

+      for (String ds : watchedDataSources) {
+        sb.append("datasources=" + ds + "&");
+      }
+      sb.setLength(Math.max(sb.length() - 1, 0));


I think sb.length() is always larger than 0?

in this case, yes

jihoonson · 2019-01-30T23:42:42Z

server/src/main/java/org/apache/druid/client/DataSegmentInterner.java

+import com.google.common.collect.Interners;
+import org.apache.druid.timeline.DataSegment;
+
+public class DataSegmentInterner


Would you please add a javadoc about what this is doing and why we need two interners?

Interners with weak references shouldn't be used: #7395

jihoonson · 2019-01-30T23:43:18Z

server/src/main/java/org/apache/druid/client/DataSegmentInterner.java

+
+  public static DataSegment intern(DataSegment segment)
+  {
+    return segment.getSize() > 0 ? HISTORICAL_INTERNER.intern(segment) : REALTIME_INTERNER.intern(segment);


It's worth to comment why size can be an indicator of a realtime or historical segment.

added a comment

jihoonson · 2019-01-31T01:42:33Z

server/src/main/java/org/apache/druid/client/MetadataSegmentView.java

+  private void poll()
+  {
+    log.info("polling published segments from coordinator");
+    //get authorized published segments from coordinator


I guess this comment is not correct? The authorization happens in SystemSchema.

ah yeah, since i copied over this code from SystemSchema . Will remove.

jihoonson · 2019-01-31T01:43:43Z

server/src/main/java/org/apache/druid/client/MetadataSegmentView.java

+    // since the presence of a segment with an earlier timestamp indicates that
+    // "that" segment is not returned by coordinator in latest poll, so it's
+    // likely deleted and therefore we remove it from publishedSegments
+    publishedSegments.entrySet().removeIf(e -> e.getValue() != timestamp);


Would you please also add a comment about publishedSegments will be eventually consistent with the segments in the coordinator?

jihoonson · 2019-01-31T01:48:15Z

server/src/main/java/org/apache/druid/client/MetadataSegmentView.java

+        if (pollTimeMs > POLL_PERIOD_IN_MS) {
+          delayMS = 0;
+        } else {
+          delayMS = pollTimeMs;


pollTimeMs is how long poll took, so probably delayMs = Math.max(POLL_PERIOD_IN_MS - pollTimeMs, 0)

jihoonson · 2019-01-31T01:50:35Z

server/src/main/java/org/apache/druid/client/MetadataSegmentView.java

+    if (watchedDataSources != null && !watchedDataSources.isEmpty()) {
+      final StringBuilder sb = new StringBuilder();
+      for (String ds : watchedDataSources) {
+        sb.append("datasources=" + ds + "&");


Ok. Please check https://github.com/apache/incubator-druid/pull/6901/files#r252479790.

jihoonson · 2019-01-31T02:03:43Z

@surekhasaharan thanks for checking the memory usage. Would you please add how many segments were in the cluster?

…t-caching

surekhasaharan · 2019-01-31T05:23:44Z

@surekhasaharan thanks for checking the memory usage. Would you please add how many segments were in the cluster?

At that time, there would be roughly around 67K segments.

…t-caching

…is still in discussion in PR

jihoonson · 2019-02-02T00:47:56Z

sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java

+        try {
+          poll();
+        }
+        finally {


Please add a catch block to emit exceptions.

added catch block

jihoonson · 2019-02-02T00:48:45Z

sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java

+          poll();
+        }
+        finally {
+          scheduledExec.schedule(new PollTask(), 0, TimeUnit.MILLISECONDS);


Please set the proper wait time instead of starting poll() immediately.

yeah, changed to use pollPeriodinMS

jon-wei · 2019-02-02T01:05:30Z

sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java

+  {
+    if (isCacheEnabled) {
+      Preconditions.checkState(
+          lifecycleLock.awaitStarted(1, TimeUnit.MILLISECONDS),


Suggest changing this precondition check message to "lifecycle start not completed", since the lifecycle starting doesn't necessarily indicate that the cache is ready (the initial poll() might have failed).

We could adjust this class so that the publishedSegments map is initially null, and initialize it upon the first successful getMetadataSegments() call in poll(). Then, in this method, add a check for "map != null" with the "hold on, still syncing published segments" message if the check fails.

made changes, so the message correctly implies that published segments are still syncing.

clintropolis

LGTM, is now 'opt in' behavior 👍

* Add published segment cache in broker * Change the DataSegment interner so it's not based on DataSEgment's equals only and size is preserved if set * Added a trueEquals to DataSegment class * Use separate interner for realtime and historical segments * Remove trueEquals as it's not used anymore, change log message * PR comments * PR comments * Fix tests * PR comments * Few more modification to * change the coordinator api * removeall segments at once from MetadataSegmentView in order to serve a more consistent view of published segments * Change the poll behaviour to avoid multiple poll execution at same time * minor changes * PR comments * PR comments * Make the segment cache in broker off by default * Added a config to PlannerConfig * Moved MetadataSegmentView to sql module * Add doc for new planner config * Update documentation * PR comments * some more changes * PR comments * fix test * remove unintentional change, whether to synchronize on lifecycleLock is still in discussion in PR * minor changes * some changes to initialization * use pollPeriodInMS * Add boolean cachePopulated to check if first poll succeeds * Remove poll from start() * take the log message out of condition in stop()

leventov · 2019-03-14T19:54:33Z

server/src/main/java/org/apache/druid/server/http/MetadataResource.java

+    if (datasources != null && !datasources.isEmpty()) {
+      druidDataSources = druidDataSources.stream()
+                                         .filter(src -> datasources.contains(src.getName()))
+                                         .collect(Collectors.toSet());


@surekhasaharan we don't want to collect ImmutableDruidDataSource objects, which are already known to be unique here, into a set. It induces expensive O(n) hash code computations (which is also totally unnecessary; ImmutableDruidDataSource.hashCode() should be cheaper. But that's another question.)

thanks @leventov for catching this, will change to list in a follow up PR.

leventov · 2019-04-01T13:13:01Z

sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java

+  {
+    if (isCacheEnabled) {
+      Preconditions.checkState(
+          lifecycleLock.awaitStarted(1, TimeUnit.MILLISECONDS) && cachePopulated.get(),


How is the client of this method expected to wait until cachePopulated contains true without running into IllegalStateException?

yeah, client would get IllegalStateException with the error message hold on, still syncing published segments, and they'll have to retry.

leventov · 2019-04-01T13:21:57Z

sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java

+      Set<String> watchedDataSources
+  )
+  {
+    String query = "/druid/coordinator/v1/metadata/segments";


Would be nice to do something like this: #7390

ok, will check that, thanks.

leventov · 2019-04-01T13:34:36Z

sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java

+    }
+  }
+
+  // Note that coordinator must be up to get segments


will resolve with #7391 , thanks.

leventov · 2019-04-01T13:36:16Z

sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java

+      }
+      finally {
+        if (!Thread.currentThread().isInterrupted()) {
+          scheduledExec.schedule(new PollTask(), delayMS, TimeUnit.MILLISECONDS);


Why doesn't this class use ScheduledExecutorService.sheduleAtFixedRate() or sheduleWithFixedDelay()?

i changed to use schedule because if poll is executed with fixed delay, there might be a possibility of two or more polls to be executed at the same time, if there's huge GC or getMetadataSegments takes longer than poll period.

Both scheduleAtFixedRate and sheduleWithFixedDelay explicitly prohibit concurrent execution in their Javadocs. Also, this is not imaginably possible, because your sheduledExec has only a single thread.

leventov · 2019-04-01T13:42:30Z

sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java

+      throw new ISE("can't start.");
+    }
+    try {
+      if (isCacheEnabled) {


Looks like this class violates the Single Responsibility Principle, see #7392.

hmm I need to read up more on Single Responsibility Principle, another way to look at it, is the responsibility of this class is to get metadata segments, the public method is getPublishedSegments, then it kind of does one thing i.e. get metadata segments.

Probably SRP is a wrong link in this case, cohesion may be a more correct reference. Anyway, it seems to me that isCacheEnabled replaces a perfectly healthy polymorphism "CachingMetadataSegmentView"/"DirectMetadataSegmentView" here. Somebody may want to add "CachingWithDirectMetadataStoreAccess" (vs. using Coordinator as a proxy), etc.

From the discussion [here](#6901 (comment)) Remove the collection and filter datasources from the stream. Also remove StreamingOutput and JsonFactory constructs.

Add published segment cache in broker

e8b0a23

Surekha Saharan added 2 commits January 26, 2019 21:46

Change the DataSegment interner so it's not based on DataSEgment's eq…

6ec4911

…uals only and size is preserved if set * Added a trueEquals to DataSegment class

Merge branch 'master' of github.com:druid-io/druid into broker-segmen…

8b6d453

…t-caching

Surekha Saharan added 3 commits January 28, 2019 13:18

Use separate interner for realtime and historical segments

281600d

Merge branch 'master' of github.com:druid-io/druid into broker-segmen…

cdc751e

…t-caching

Remove trueEquals as it's not used anymore, change log message

8fed80a

jihoonson reviewed Jan 28, 2019

View reviewed changes

Surekha Saharan added 5 commits January 28, 2019 16:22

PR comments

0d0f89f

PR comments

8fc84b3

Merge branch 'master' of github.com:druid-io/druid into broker-segmen…

adf133f

…t-caching

Fix tests

d46edc5

Merge branch 'master' of github.com:druid-io/druid into broker-segmen…

92601d0

…t-caching

jihoonson mentioned this pull request Jan 29, 2019

ServerSelector should always return the best segment #6952

Closed

jihoonson reviewed Jan 30, 2019

View reviewed changes

Surekha Saharan added 4 commits January 29, 2019 23:12

PR comments

a993482

Few more modification to

a4cbcfc

* change the coordinator api * removeall segments at once from MetadataSegmentView in order to serve a more consistent view of published segments * Change the poll behaviour to avoid multiple poll execution at same time

Merge branch 'master' of github.com:druid-io/druid into broker-segmen…

e376df3

…t-caching

minor changes

0fbb48c

jihoonson reviewed Jan 30, 2019

View reviewed changes

PR comments

8df2d96

jihoonson reviewed Jan 31, 2019

View reviewed changes

Surekha Saharan added 2 commits January 30, 2019 19:59

PR comments

d5b7d79

Merge branch 'master' of github.com:druid-io/druid into broker-segmen…

2bfa396

…t-caching

Surekha Saharan added 4 commits February 1, 2019 14:47

Merge branch 'master' of github.com:druid-io/druid into broker-segmen…

33751a4

…t-caching

remove unintentional change, whether to synchronize on lifecycleLock …

2ddb7a1

…is still in discussion in PR

minor changes

1b37493

some changes to initialization

60dbf41

jihoonson approved these changes Feb 2, 2019

View reviewed changes

use pollPeriodInMS

8860053

jon-wei reviewed Feb 2, 2019

View reviewed changes

Surekha Saharan added 2 commits February 1, 2019 17:58

Add boolean cachePopulated to check if first poll succeeds

ca21779

Remove poll from start()

3a70f39

clintropolis approved these changes Feb 2, 2019

View reviewed changes

jon-wei approved these changes Feb 2, 2019

View reviewed changes

take the log message out of condition in stop()

e2a9af7

clintropolis merged commit 7baa330 into apache:master Feb 3, 2019

surekhasaharan deleted the broker-segment-caching branch February 4, 2019 18:45

gianm mentioned this pull request Feb 19, 2019

[Proposal] Add published segment cache in broker #6834

Open

jon-wei added this to the 0.14.0 milestone Feb 20, 2019

jon-wei mentioned this pull request Feb 22, 2019

0.14.0-incubating release notes #7126

Closed

leventov reviewed Mar 14, 2019

View reviewed changes

leventov mentioned this pull request Mar 20, 2019

Reconcile terminology and method naming to 'used/unused segments'; Rename MetadataSegmentManager to MetadataSegmentsManager #7306

Merged

surekhasaharan mentioned this pull request Mar 27, 2019

Remove unnecessary collection #7350

Merged

leventov reviewed Apr 1, 2019

View reviewed changes

leventov mentioned this pull request Apr 1, 2019

Published segments cache in Brokers shouldn't use Interners #7395

Open

leventov pushed a commit that referenced this pull request Apr 15, 2019

Remove unnecessary collection (#7350)

4654e1e

From the discussion [here](#6901 (comment)) Remove the collection and filter datasources from the stream. Also remove StreamingOutput and JsonFactory constructs.

leventov mentioned this pull request May 13, 2019

Refactor SQLMetadataSegmentManager; Change contract of REST methods in DataSourcesResource #7653

Merged

kaijianding mentioned this pull request Dec 29, 2020

reuse DataSegment object when a segment found on another server #10715

Merged

9 tasks

Introduce published segment cache in broker #6901

Introduce published segment cache in broker #6901

Uh oh!

Conversation

surekhasaharan commented Jan 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

drcrallen commented Jan 23, 2019

Uh oh!

drcrallen commented Jan 23, 2019

Uh oh!

surekhasaharan commented Jan 24, 2019

Uh oh!

surekhasaharan commented Jan 28, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

surekhasaharan commented Jan 30, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

surekhasaharan commented Jan 22, 2019 •

edited

Loading

surekhasaharan commented Jan 28, 2019 •

edited

Loading

surekhasaharan commented Jan 30, 2019 •

edited

Loading