Skip to content

Commit d8bce91

Browse files
aozarovaozarov
committed
Merge pull request #582 from ajkannan/test-paging-in-datastore-example
Use pagination in DatastoreExample
2 parents c5bae84 + 360e048 commit d8bce91

File tree

2 files changed

+71
-61
lines changed

2 files changed

+71
-61
lines changed

‎gcloud-java-examples/README.md‎

Lines changed: 47 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -39,54 +39,54 @@ To run examples from your command line:
3939

4040
4. Run an example using Maven from command line.
4141

42-
Here's an example run of `BigQueryExample`.
43-
44-
Before running the example, go to the [Google Developers Console][developers-console] to ensure
45-
that BigQuery API is enabled. You can upload a CSV file `my_csv_file` to the `my_bucket` bucket
46-
(replace `my_csv_file` and `my_bucket` with actual file and bucket names) using the GCS
47-
[web browser](https://console.developers.google.com/storage/browser). The CSV file will be used to
48-
load data into a BigQuery table and should look something like:
49-
```csv
50-
value1
51-
value2
52-
value3
53-
```
54-
Then you are ready to run the following example:
55-
```
56-
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="create dataset new_dataset_id"
57-
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="create table new_dataset_id new_table_id field_name:string"
58-
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="list tables new_dataset_id"
59-
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="load new_dataset_id new_table_id CSV gs://my_bucket/my_csv_file"
60-
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="query 'select * from new_dataset_id.new_table_id'"
61-
```
62-
63-
Here's an example run of `DatastoreExample`.
42+
* Here's an example run of `BigQueryExample`.
43+
44+
Before running the example, go to the [Google Developers Console][developers-console] to ensure
45+
that BigQuery API is enabled. You can upload a CSV file `my_csv_file` to the `my_bucket` bucket
46+
(replace `my_csv_file` and `my_bucket` with actual file and bucket names) using the GCS
47+
[web browser](https://console.developers.google.com/storage/browser). The CSV file will be used to
48+
load data into a BigQuery table and should look something like:
49+
```csv
50+
value1
51+
value2
52+
value3
53+
```
54+
Then you are ready to run the following example:
55+
```
56+
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="create dataset new_dataset_id"
57+
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="create table new_dataset_id new_table_id field_name:string"
58+
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="list tables new_dataset_id"
59+
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="load new_dataset_id new_table_id CSV gs://my_bucket/my_csv_file"
60+
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="query 'select * from new_dataset_id.new_table_id'"
61+
```
62+
63+
* Here's an example run of `DatastoreExample`.
6464
65-
Note that you have to enable the Google Cloud Datastore API on the [Google Developers Console][developers-console] before running the following commands.
66-
```
67-
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" -Dexec.args="my_name add my\ comment"
68-
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" -Dexec.args="my_name display"
69-
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" -Dexec.args="my_name delete"
70-
```
71-
72-
Here's an example run of `ResourceManagerExample`.
73-
74-
Be sure to change the placeholder project ID "my-project-id" with your own globally unique project ID.
75-
```
76-
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" -Dexec.args="create my-project-id"
77-
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" -Dexec.args="list"
78-
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" -Dexec.args="get my-project-id"
79-
```
80-
81-
Here's an example run of `StorageExample`.
82-
83-
Before running the example, go to the [Google Developers Console][developers-console] to ensure that Google Cloud Storage API is enabled and that you have a bucket. Also ensure that you have a test file (`test.txt` is chosen here) to upload to Cloud Storage stored locally on your machine.
84-
```
85-
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="upload /path/to/test.txt <bucket_name>"
86-
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="list <bucket_name>"
87-
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="download <bucket_name> test.txt"
88-
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="delete <bucket_name> test.txt"
89-
```
65+
Be sure to change the placeholder project ID "your-project-id" with your own project ID. Also note that you have to enable the Google Cloud Datastore API on the [Google Developers Console][developers-console] before running the following commands.
66+
```
67+
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" -Dexec.args="your-project-id my_name add my\ comment"
68+
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" -Dexec.args="your-project-id my_name display"
69+
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" -Dexec.args="your-project-id my_name delete"
70+
```
71+
72+
* Here's an example run of `ResourceManagerExample`.
73+
74+
Be sure to change the placeholder project ID "your-project-id" with your own globally unique project ID.
75+
```
76+
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" -Dexec.args="create your-project-id"
77+
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" -Dexec.args="list"
78+
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" -Dexec.args="get your-project-id"
79+
```
80+
81+
* Here's an example run of `StorageExample`.
82+
83+
Before running the example, go to the [Google Developers Console][developers-console] to ensure that Google Cloud Storage API is enabled and that you have a bucket. Also ensure that you have a test file (`test.txt` is chosen here) to upload to Cloud Storage stored locally on your machine.
84+
```
85+
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="upload /path/to/test.txt <bucket_name>"
86+
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="list <bucket_name>"
87+
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="download <bucket_name> test.txt"
88+
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="delete <bucket_name> test.txt"
89+
```
9090
9191
Troubleshooting
9292
---------------

‎gcloud-java-examples/src/main/java/com/google/gcloud/examples/DatastoreExample.java‎

Lines changed: 24 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,8 @@
2525
import com.google.gcloud.datastore.Key;
2626
import com.google.gcloud.datastore.KeyFactory;
2727
import com.google.gcloud.datastore.Query;
28-
import com.google.gcloud.datastore.Query.ResultType;
2928
import com.google.gcloud.datastore.QueryResults;
29+
import com.google.gcloud.datastore.StructuredQuery;
3030
import com.google.gcloud.datastore.StructuredQuery.PropertyFilter;
3131
import com.google.gcloud.datastore.Transaction;
3232

@@ -100,21 +100,31 @@ public void run(Transaction tx, Key userKey, String... args) {
100100
return;
101101
}
102102
System.out.printf("User '%s' has %d comment[s].%n", userKey.name(), user.getLong("count"));
103-
// ORDER BY timestamp";
104-
String gql = "SELECT * FROM " + COMMENT_KIND + " WHERE __key__ HAS ANCESTOR @1";
105-
Query<Entity> query = Query.gqlQueryBuilder(ResultType.ENTITY, gql)
106-
.namespace(NAMESPACE)
107-
.addBinding(userKey)
108-
.build();
109-
QueryResults<Entity> results = tx.run(query);
110-
// We could have added "ORDER BY timestamp" to the query to avoid the sorting bellow
111-
// but that would require adding an ancestor index for timestamp
112-
// see: https://cloud.google.com/datastore/docs/tools/indexconfig
103+
int limit = 200;
113104
Map<DateTime, String> sortedComments = new TreeMap<>();
114-
while (results.hasNext()) {
115-
Entity result = results.next();
116-
sortedComments.put(result.getDateTime("timestamp"), result.getString("content"));
105+
StructuredQuery<Entity> query =
106+
Query.entityQueryBuilder()
107+
.namespace(NAMESPACE)
108+
.kind(COMMENT_KIND)
109+
.filter(PropertyFilter.hasAncestor(userKey))
110+
.limit(limit)
111+
.build();
112+
while (true) {
113+
QueryResults<Entity> results = tx.run(query);
114+
int resultCount = 0;
115+
while (results.hasNext()) {
116+
Entity result = results.next();
117+
sortedComments.put(result.getDateTime("timestamp"), result.getString("content"));
118+
resultCount++;
119+
}
120+
if (resultCount < limit) {
121+
break;
122+
}
123+
query = query.toBuilder().startCursor(results.cursorAfter()).build();
117124
}
125+
// We could have added "ORDER BY timestamp" to the query to avoid sorting, but that would
126+
// require adding an ancestor index for timestamp.
127+
// See: https://cloud.google.com/datastore/docs/tools/indexconfig
118128
for (Map.Entry<DateTime, String> entry : sortedComments.entrySet()) {
119129
System.out.printf("\t%s: %s%n", entry.getKey(), entry.getValue());
120130
}

0 commit comments

Comments
 (0)